Video quality assessment model production method and its device, equipment, and medium

Abstract

The present application relates to a method for producing a video quality assessment model in the field of image recognition technology and its device, equipment, and medium. The method includes: obtaining an initialized instance of a video quality assessment model that has been trained to a converged state; obtaining a training configuration list of the video quality assessment model; using the converged instance as a teacher network, sequentially calling a single instance in the training configuration list as a student network, reusing the weight of the teacher network to train it to a converged state, replacing the teacher network to continue iterative knowledge distillation training on other instances until the last instance is trained to convergence; repeatedly training the last instance to a converged state, taking the average of the weight parameters obtained by the multiple trainings of the instance, configuring it as the actual weight of the last instance, and using the instance as the output of the video quality assessment model. The present application can realize a lightweight model for accurately assessing video quality.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method for producing a video quality assessment model and its corresponding device, computer equipment, and computer-readable storage medium. Background Art

[0002] Video quality assessment is a method that has been studied in academia and industry since the emergence of video. It is mainly divided into reference assessment and non-reference assessment. Reference assessment compares the effect difference frame by frame based on the given reference video, so that a relatively accurate assessment score can be obtained more easily. Video non-reference assessment is a method that gives the user's subjective experience score of the current video quality. According to the score of the non-reference assessment, the quality indicators such as the bit rate, frame rate, and resolution of network transmission can be dynamically adjusted to achieve the purpose of cost control. Through adjustment, the user's subjective video viewing experience is basically unaffected. This implementation not only meets user needs, but also saves network transmission resources in disguise, which can bring considerable economic benefits.

[0003] The difficulty of no-reference evaluation methods is very great. Traditional algorithms can only play a very limited role in no-reference video evaluation. Until the rapid development of artificial intelligence in recent years, no-reference video evaluation methods based on CNN (Convolutional Neural Network) have received attention and developed rapidly, becoming the main research and application direction. With the rapid development of AI, various CNN-based video no-reference evaluation technologies and model compression technologies continue to emerge. The industry has achieved good accuracy and robustness in no-reference evaluation of video quality by using different AI algorithms, but there are still some shortcomings, mainly reflected in the model output efficiency and cost, and failed to achieve good overall benefits. Summary of the invention

[0004] The purpose of the present application is to solve at least one of the above problems and to provide a video quality assessment model production method and its corresponding device, computer equipment, and computer-readable storage medium.

[0005] To meet the various purposes of this application, the following technical solutions are adopted:

[0006] Obtaining an instance of initialization of a video quality assessment model trained to a converged state, the model comprising a multi-level backbone module for downsampling sample data of multiple image frames of the same video step by step to extract features, and a prediction module for predicting a quality score of image feature information output by a terminal backbone module, wherein each level of the backbone module comprises one or more convolutional layers;

[0007] Obtaining a training configuration list of the video quality assessment model, the training configuration list sequentially comprising configuration information of the convolutional layer scales of the backbone modules at various levels corresponding to multiple other instances of the model, wherein the overall scale of the convolutional layer of the subsequent instance is smaller than that of the previous instance and does not exceed that of the initialized instance;

[0008] Using the converged instance as the teacher network, calling the single instance in the training configuration list in sequence as the student network, reusing the weight of the teacher network to train it to a converged state, and then replacing the teacher network to continue iterative knowledge distillation training on other instances until the last instance is trained to convergence;

[0009] Repeat the training of the last instance multiple times until convergence, take the average of the weight parameters obtained from the multiple trainings of the instance, configure it as the actual weight of the last instance, and use the instance as the output of the video quality assessment model.

[0010] In a further embodiment, the training process of the initialized instance of the video quality assessment model comprises the following steps:

[0011] Acquire multiple image frames of a video in a single training sample, perform data enhancement processing on the multiple image frames to obtain sample data, and input the sample data into the instance;

[0012] After the backbone modules at each level of the instance extract image feature information from the sample data level by level, the prediction module outputs the quality score of the video;

[0013] The model loss value of the quality score is calculated according to the pre-labeled supervision score of the training sample, and the supervision score is a subjective quality score. When the model loss value does not reach a preset threshold, the weight of the instance is updated, and other training samples are continuously called to implement iterative training.

[0014] In a further embodiment, the training process of the instance of the video quality assessment model configured according to the training configuration list comprises the following steps:

[0015] Acquire multiple image frames of a video in a single training sample, perform data enhancement processing on the multiple image frames to obtain sample data, and synchronously input the sample data into the teacher network and the student network;

[0016] After extracting image feature information from the sample data level by level through the backbone modules at each level in the teacher network and the student network, the quality score of the video is output through their respective prediction modules;

[0017] Calculating a model loss value of the student network, the loss value comprising a loss value between a supervised score pre-labeled by the training sample and a quality score predicted by the student network, the supervised score being a subjective quality score, and a feature loss value between image feature information obtained by backbone modules at various levels between the teacher network and the student network;

[0018] Determine whether the model loss value reaches the preset threshold. If it does not reach the preset threshold, update the weight of the instance and continue to call other training samples to implement iterative training.

[0019] In a further embodiment, in the training configuration list, the scale of convolutional layers corresponding to the backbone modules at each level is expressed in quantity, wherein the number of convolutional layers of some backbone modules of the later-ordered instances is greater than the number of convolutional layers of the corresponding backbone modules of the earlier instances, while the number of convolutional layers of the remaining backbone modules remains consistent.

[0020] In a further embodiment, in the video quality assessment model, the output of the front backbone module is processed by the maximum pooling layer and then input into the subsequent backbone module, and the output of the last-level backbone module is mean pooled by the average pooling layer and then input into the prediction module.

[0021] In a further embodiment, in the video quality assessment model, the convolutional layer scale of a subsequent backbone module is not smaller than the convolutional layer scale of a previous backbone module, and there is at least one subsequent backbone module whose convolutional layer scale is larger than the convolutional layer scale of a previous backbone module.

[0022] In a further embodiment, after configuring the mean weight of the weights of all trained instances as the corresponding weight of the last trained instance so that the instance becomes the output video quality assessment model, the following steps are included:

[0023] Input multiple continuous image frames in the live video stream of the network into the generated video quality assessment model to predict their corresponding quality scores;

[0024] The encoding frame rate of the encoder of the video stream is adjusted according to the quality score.

[0025] On the other hand, a video quality assessment model production device provided to meet one of the purposes of the present application includes an initialization module, an instance configuration module, an iterative distillation module, and a model output module, wherein: the initialization module is used to obtain an initialized instance of a video quality assessment model that has been trained to a converged state, the model includes a multi-level backbone module for downsampling sample data of multiple image frames of the same video step by step to extract features, and a prediction module for predicting the quality score of image feature information output by the terminal backbone module, and each level of the backbone module includes one or more convolutional layers; the instance configuration module obtains a training configuration list of the video quality assessment model, the training configuration list sequentially includes configuration information of the convolutional layer scale of each level of the backbone module corresponding to other multiple instances of the model, wherein the overall scale of the convolutional layer of the subsequent instance is smaller than that of the previous instance and does not exceed the initialized instance; the iterative distillation module is used to use the converged instance as a teacher network, sequentially call a single instance in the training configuration list as a student network, reuse the weight of the teacher network to train it to a converged state, and then replace the teacher network to continue to implement iterative knowledge distillation training on other instances until the last instance is trained to converge;

[0026] In a further embodiment, the training process of the initialized instance of the video quality assessment model of the initialization module includes: a first sample input submodule, used to obtain multiple image frames of a video in a single training sample, perform data enhancement processing on the multiple image frames to obtain sample data, and input the sample data into the instance; a first feature extraction submodule, used to extract image feature information from the sample data step by step by the backbone modules of each level of the instance, and output the quality score of the video through a prediction module; a first iterative training submodule, used to calculate the model loss value of the quality score based on the supervised score pre-labeled by the training sample, the supervised score is a subjective quality score, and when the model loss value does not reach a preset threshold, the instance is weighted updated, and other training samples are continued to be called to implement iterative training.

[0027] In a further embodiment, the training process of the instance of the video quality assessment model configured according to the training configuration list of the instance configuration module includes: a second sample input submodule, which is used to obtain multiple image frames of the video in a single training sample, obtain sample data after data enhancement processing of the multiple image frames, and synchronously input them into the teacher network and the student network; a second feature extraction submodule, which is used to extract image feature information from the sample data step by step through the backbone modules at various levels in the teacher network and the student network, and output the quality score of the video through their respective prediction modules; a loss calculation submodule, which is used to calculate the model loss value of the student network, the loss value includes the loss value between the pre-labeled supervision score of the training sample and the quality score predicted by the student network, the supervision score is a subjective quality score, and the feature loss value between the image feature information obtained by the backbone modules at various levels between the teacher network and the student network; a second iterative training submodule, which is used to determine whether the model loss value reaches a preset threshold, and when the preset threshold is not reached, the weight of the instance is updated, and other training samples are continued to be called to perform iterative training.

[0028] In a further embodiment, in the training configuration list of the instance configuration module, the scale of convolutional layers corresponding to the backbone modules at each level is expressed in quantity, wherein the number of convolutional layers of some backbone modules of the later-ordered instances is greater than the number of convolutional layers of the corresponding backbone modules of the earlier instances, while the number of convolutional layers of the remaining backbone modules remains consistent.

[0029] In a further embodiment, in the video quality assessment model of the instance configuration module, the output of the front backbone module is processed by the maximum pooling layer and then input into the subsequent backbone module, and the output of the last-level backbone module is mean pooled by the average pooling layer and then input into the prediction module.

[0030] In a further embodiment, in the video quality assessment model of the instance configuration module, the convolutional layer scale of the subsequent backbone module is not smaller than the convolutional layer scale of the previous backbone module, and there is at least one subsequent backbone module whose convolutional layer scale is larger than the convolutional layer scale of the previous backbone module.

[0031] In a further embodiment, the model generation module includes: a model prediction submodule, used to input multiple continuous image frames in the live network video stream into the generated video quality assessment model to predict their corresponding quality scores; a frame rate adjustment submodule, used to adjust the encoding frame rate of the encoder of the video stream according to the quality score.

[0032] On the other hand, a computer device provided to meet one of the purposes of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the video quality assessment model production method described in the present application.

[0033] On the other hand, a computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the video quality assessment model production method in the form of computer-readable instructions, and when the computer program is called and executed by a computer, it executes the steps included in the method.

[0034] Compared with the prior art, the present application obtains an initialized instance of a video quality assessment model that has been trained to a converged state, and compresses part or all of the backbone modules in the model, that is, reduces the scale of the convolutional layer corresponding to the backbone module, thereby deriving multiple instances accordingly, and constructing a training configuration list with the scale of the convolutional layer corresponding to the backbone modules at each level of the multiple instances. Further, the converged instance is used as the teacher network, and a single instance in the training configuration list is called in sequence as the student network. After the weight of the teacher network is reused and trained to a converged state, the teacher network is replaced to continue iterative knowledge distillation training for other instances until the last instance is trained to convergence, and the last instance is repeatedly trained to a converged state multiple times, and the weight parameters obtained from the multiple trainings of the instance are averaged and configured as the actual weight of the last instance, and the instance is used as the output of the video quality assessment model. It can be understood that the present application can obtain many advantages, including but not limited to:

[0035] First, the student network is trained through iterative distillation, and the final video quality assessment model has the same capabilities as the teacher network and can also accurately predict the corresponding video quality of the video. In addition, the produced video quality assessment model is lightweight and can be deployed online to run real-time assessment of video quality and predict the corresponding quality score.

[0036] Secondly, during the iterative distillation training process, the student network in each iteration reuses part of the weights of the teacher network, which can not only maintain the accuracy of the student network's predictions, but also accelerate the training of the student network and improve its convergence speed.

[0037] In addition, the present application takes the average of the weight parameters obtained by repeatedly training the last instance until the convergence state, and configures it as the actual weight of the last instance, which can increase the generalization ability and robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0039] Figure 1 A flowchart of a typical embodiment of the method for producing a video quality assessment model of the present application;

[0040] Figure 2 This is a schematic diagram of the structure of a video quality assessment model exemplified in an embodiment of the present application;

[0041] Figure 3 This is a schematic diagram of the structure of the backbone module in the exemplary video quality assessment model in the embodiment of the present application;

[0042] Figure 4 A flowchart of an example training process of initializing the video quality assessment model described in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of a flow chart of a training process of an instance of the video quality assessment model configured according to the training configuration list in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of knowledge distillation for backbone modules at various levels corresponding to a teacher network and a student network in an embodiment of the present application;

[0045] Figure 7 A schematic diagram of a process for adjusting the encoding frame rate of a video stream by predicting a quality score corresponding to the video stream according to the output video quality assessment model in an embodiment of the present application;

[0046] Figure 8 A block diagram of the video quality assessment model production device of this application;

[0047] Fig. 9 A schematic diagram of the structure of a computer device used in this application. DETAILED DESCRIPTION

[0048] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0049] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0050] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0051] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0052] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0053] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0054] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for access.

[0055] The neural network models referenced or may be referenced in this application, unless expressly specified, can be deployed on a remote server and remotely called on the client, or can be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0056] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as it is suitable for being called by the technical solution of this application.

[0057] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0058] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0059] A video quality assessment model production method of the present application can be programmed as a computer program product and deployed in a client or server for execution. For example, in the exemplary application scenario of the present application, it can be deployed and implemented in a server of an e-commerce platform, thereby accessing an interface opened after the computer program product is running and performing human-computer interaction with the process of the computer program product through a graphical user interface to execute the method.

[0060] See also Figure 1 The video quality assessment model production method of the present application, in its typical embodiment, comprises the following steps:

[0061] Step S1100, obtaining an initialized instance of a video quality assessment model that has been trained to a converged state, wherein the model includes a multi-level backbone module for gradually downsampling sample data of multiple image frames of the same video to extract features, and a prediction module for predicting a quality score of image feature information output by a terminal backbone module, wherein each level of the backbone module includes one or more convolutional layers;

[0062] The video quality assessment model of the present application will be used to predict the no-reference score for the video stream. Accordingly, in one embodiment, the video quality assessment model of the present application can be deployed on the media server corresponding to the live broadcast on the e-commerce platform to provide corresponding services, and the service can be used to predict the no-reference score for the video stream generated by the live broadcast on the e-commerce platform. The video stream is composed of image frames that are continuous in time, and the data of the corresponding multiple image frames can be obtained by decoding the video stream as the sample data, which is input into the model to extract features for training the model.

[0063] An example of initialization of the video quality assessment model is as follows: Figure 2 As shown, it includes a multi-level backbone module 200, an average pooling layer 202, and a prediction module 203. The multi-level backbone module 200 is connected to the average pooling layer 202, and the average pooling layer 202 is connected to the prediction module 203. The two adjacent backbone modules are connected through a maximum pooling layer 201 to achieve feature downsampling extracted by the previous backbone module, and each backbone module includes one or more convolutional layers. The output of the previous backbone module is sampled by the maximum pooling layer and then input into the subsequent backbone module for processing. The output of the last backbone module, i.e., the terminal backbone module, i.e., the image feature information, is average-pooled by the average pooling layer 202 and then input into the prediction module, and then the prediction module predicts the quality score corresponding to the image feature information. The prediction module 203 includes a fully connected layer 204.

[0064] In one embodiment, in a multi-level backbone module, the convolutional layer scale of the subsequent backbone module is not less than the convolutional layer scale of the previous backbone module, and there is at least one subsequent backbone module whose convolutional layer scale is larger than the convolutional layer scale of the previous backbone module. For example, if the convolutional layer scale of the subsequent backbone module is 16 convolutional layers, then the number of convolutional layers corresponding to the convolutional layer scale of the previous backbone module can be any number smaller than 16, such as 8, 6, 4, etc. By gradually increasing the number of convolutional layers in each level of backbone module, the deep semantic information of the image frame is extracted from shallow to deep, so that the terminal backbone module can effectively represent the deep semantic information of the image frame at all scales.

[0065] In one embodiment, the structure of the backbone module is exemplified as follows: Figure 3 As shown, it includes a combination of one or more groups of convolutional layers 300 and BN (Batch Normalization) layers 301, each of which is connected. The BN layer uses a certain normalization method to forcibly pull the distribution of the input values ​​corresponding to the neurons in the convolutional layer connected to it back to a standard normal distribution with a mean of 0 and a variance of 1, that is, to force the increasingly biased distribution back to a relatively standard distribution, so that the activation input value falls in the area where the nonlinear function is more sensitive to the input, the output of the network will not be very large, and a relatively large gradient can be obtained to avoid the problem of gradient disappearance. Moreover, a larger gradient means that the learning convergence speed is fast, which can greatly speed up the training speed. According to the principles disclosed herein, in another embodiment, those skilled in the art can flexibly choose the BN layer in the combination of the convolutional layer and the BN layer in the backbone module according to business needs, and can abandon the use of the BN layer and use the convolutional layer alone.

[0066] The multi-level backbone module that downsamples level by level to extract features means that the corresponding shallow-level backbone modules extract basic features such as details and edges in the sample data, and the corresponding deep-level backbone modules extract deep semantic features and high-level logical features of the sample data.

[0067] Step S1200, obtaining a training configuration list of the video quality assessment model, the training configuration list sequentially containing configuration information of the convolutional layer scales of the backbone modules at various levels corresponding to other multiple instances of the model, wherein the overall scale of the convolutional layer of the subsequent instance is smaller than that of the previous instance and does not exceed that of the initialized instance;

[0068] Furthermore, in order to achieve model compression of the video quality assessment model, the scale of the convolutional layers in the full or partial level backbone modules in the model is reduced, that is, the number of convolutional layers corresponding to the scale of the convolutional layers is reduced. For example, if the number of convolutional layers corresponding to the scale of the convolutional layers of the terminal backbone module in the model is 16, then it can be reduced to 8 or 4, etc., which is less than 16. Due to the particularity of CNN, that is, in the multi-level backbone modules of the model, the corresponding shallow-level backbone modules extract basic features such as details and edges in the sample data, and the corresponding deep-level backbone modules extract deep semantic features and high-level logical features of the sample data. Among them, there is more redundancy in the basic features extracted by the shallow-level backbone modules, which has a better effect on model compression. Based on this, in one embodiment, the scale of the convolutional layers of the shallow-level backbone modules can be reduced first, while the scale of the convolutional layers of the deep-level backbone modules is relatively retained unchanged, so that the configuration information of the scale of the convolutional layers of the backbone modules at all levels corresponding to the multiple instances of the model is configured, and then the deep-level backbone modules are further compressed. The convolutional layer scale of the backbone module is reduced so that the configuration information of the convolutional layer scale of each level of the backbone module corresponding to other multiple instances of the model is reconfigured, wherein the overall scale of the convolutional layer of the subsequent instance is smaller than that of the previous instance and does not exceed the initialized instance. For example, the number of convolutional layers of the convolutional layer scale corresponding to the backbone modules at all levels of the instance of the current video quality assessment model is [2, 4, 6, 8, 16]. When the video quality assessment model is compressed, the corresponding configuration information of the other multiple instances is configured, wherein the convolutional layer scale of each level of the backbone module adopted for each instance can be configured in the following manner in a list format for each instance in turn:

[0069] [1, 2, 6, 8, 16]

[0070] [1, 2, 4, 8, 16]

[0071] [1, 2, 4, 4, 16]

[0072] [1, 2, 4, 4, 8]

[0073] [1, 2, 4, 4, 4]

[0074] According to the disclosure herein, those skilled in the art can flexibly and adapt the training configuration list of the video quality assessment model according to business needs in a descending manner, so that the instance with the largest compression in the training configuration list, that is, the instance with the smallest convolutional layer size corresponding to the overall backbone modules, can meet business needs and be suitable for deployment after subsequent knowledge distillation. It can run the evaluation of the video quality in real time, predict the corresponding quality score, and then store the training configuration list so that it can be obtained from the storage in this step.

[0075] Step S1300: using the converged instance as the teacher network, calling the single instance in the training configuration list in sequence as the student network, reusing the weight of the teacher network to train it to a converged state, replacing the teacher network to continue iterative knowledge distillation training on other instances until the last instance is trained to convergence;

[0076] The initialized instance of the video quality assessment model trained to a convergent state is used as the teacher network, and the first single instance in the training configuration list is called as the student network. After the teacher network's weight is reused to train it to a convergent state, it is replaced with the teacher network, and the next single instance in the training configuration list is called in sequence as the student network. Accordingly, iterative knowledge distillation training is implemented until the last instance in the training configuration list is trained to convergence. It can be understood that after the iterative knowledge distillation training, the features corresponding to the extracted image frames learned by the teacher network can be migrated to the lightweight student network, so that after the student network is trained to convergence, it has at least the same capabilities as the teacher network, thereby achieving model compression and obtaining a model instance with a smaller network volume, which can be more lightweight and convenient for online deployment.

[0077] Since the characteristics of the quality difference of the image frames of the video stream are highly correlated with the content features existing in the image frames, they are also correlated with the coding features existing in the image frames. However, the coding features are not very significant relative to the content features. Accordingly, the weights of the convolutional layers corresponding to the backbone modules at all levels of the teacher network and the weights of the fully connected layers corresponding to the prediction modules are reused as the weights of the convolutional layers corresponding to the backbone modules at all levels of the student network and the weights of the fully connected layers corresponding to the prediction modules. It can be understood that the student network and the teacher network have the same number of backbone modules, but the scale of the convolutional layers of the former corresponding to part or all of them is smaller than that of the latter. Therefore, the student network reuses the teacher network relatively here. The weights of some convolutional layers corresponding to the multi-level backbone modules of the student network are given as follows: for example, the scale of convolutional layers corresponding to the multi-level backbone modules of the student network is [1, 2, 6, 8, 16], and the scale of convolutional layers corresponding to the multi-level backbone modules of the teacher network is [2, 4, 6, 8, 16]. The first backbone module of the student network reuses the weight of the first convolutional layer in the first backbone module of the teacher network, and the second backbone module of the student network reuses the weights of the first and second convolutional layers in the second backbone module of the teacher network. The third, fourth, and fifth backbone modules of the student network have the same scale of convolutional layers as the backbone modules of the teacher network, so the weights can be directly reused. In addition, the fully connected layers in the prediction modules corresponding to the student network and the teacher network are the same, so the weights can also be directly reused.

[0078] Step S1400, repeatedly training the last instance multiple times until convergence, taking the average of the weight parameters obtained from the multiple trainings of the instance, configuring the average as the actual weight of the last instance, and using the instance as the output of the video quality assessment model.

[0079] Generally speaking, the convolutional layer corresponding to the multi-level backbone module of the last instance is small in scale, prone to overfitting and poor in robustness. Therefore, in order to make up for the generalization ability of the last instance after knowledge distillation, in one embodiment, a fixed learning rate can be added to the last instance, and the training is repeated multiple times until convergence. During this period, the weights corresponding to multiple model instances are saved, and the weight parameters of the multiple models are averaged and configured as the actual weight of the last instance, and the instance is used as the output of the video quality assessment model. The learning rate and the number of saved multiple model instances can be flexibly set by those skilled in the art based on prior knowledge or experimental experience. It is recommended that the learning rate can be 1e-6 and the number of saved multiple models is 10.

[0080] Compared with the prior art, the present application obtains an initialized instance of a video quality assessment model that has been trained to a converged state, and compresses part or all of the backbone modules in the model, that is, reduces the scale of the convolutional layer corresponding to the backbone module, thereby deriving multiple instances accordingly, and constructing a training configuration list with the scale of the convolutional layer corresponding to the backbone modules at each level of the multiple instances. Further, the converged instance is used as the teacher network, and a single instance in the training configuration list is called in sequence as the student network. After the weight of the teacher network is reused and trained to a converged state, the teacher network is replaced to continue iterative knowledge distillation training for other instances until the last instance is trained to convergence, and the last instance is repeatedly trained to a converged state multiple times, and the weight parameters obtained from the multiple trainings of the instance are averaged and configured as the actual weight of the last instance, and the instance is used as the output of the video quality assessment model. It can be understood that the present application can obtain many advantages, including but not limited to:

[0081] First, the student network is trained through iterative distillation, and the final video quality assessment model has the same capabilities as the teacher network and can also accurately predict the corresponding video quality of the video. In addition, the produced video quality assessment model is lightweight and can be deployed online to run real-time assessment of video quality and predict the corresponding quality score.

[0082] Secondly, during the iterative distillation training process, the student network in each iteration reuses part of the weights of the teacher network, which can not only maintain the accuracy of the student network's predictions, but also accelerate the training of the student network and improve its convergence speed.

[0083] In addition, the present application takes the average of the weight parameters obtained by repeatedly training the last instance until the convergence state, and configures it as the actual weight of the last instance, which can increase the generalization ability and robustness of the model.

[0084] See also Figure 4 In a further embodiment, step S1100, the training process of the initialized instance of the video quality assessment model comprises the following steps:

[0085] Step S1000: obtaining multiple image frames of a video in a single training sample, performing data enhancement processing on the multiple image frames to obtain sample data, and inputting the sample data into the instance;

[0086] In one embodiment, based on hard indicators of video quality such as bit rate, frame rate, and resolution, a sufficient amount of videos can be manually collected and organized as a training set, wherein each video serves as a single training sample, and each single training sample has a corresponding subjective quality score annotated for it, which is used as a supervision score. The subjective quality score can be obtained by having multiple people perform subjective scoring on the video quality of the videos corresponding to the training samples and calculating the average of these subjective scores.

[0087] A video of a single training sample is obtained from the training set, and the corresponding multiple image frames are obtained by decoding the video. For example, the image data type corresponding to the image frame is unit8, which is converted to float data type and then divided by 255 to normalize to the corresponding value range of [0, 1]. The multiple image frames are subjected to data enhancement processing to obtain sample data. The data enhancement processing can be mirroring, rotation, scaling, adjusting brightness, contrast, Gaussian noise, Mosaic, Mixup, Cutout, CutMix, etc. Those skilled in the art can flexibly select and implement according to actual business needs, and then input the sample data into the initialized instance of the video quality assessment model.

[0088] Generally speaking, the initialization instance of the video quality assessment model is a model with a relatively large network scale, which has strong learning and generalization capabilities, and requires a large amount of computing resources when used, so as to accurately extract information from each image frame of a very large and highly redundant video. For example, a parameterized deep CNN neural network model is used, and a regularization strategy (such as dropout) is added to it, or an integrated model, that is, many weak CNN models are integrated together. Those skilled in the art can flexibly select and implement it according to the disclosure herein.

[0089] In one embodiment, the initialization of the video quality assessment model is exemplified by: Figure 2As shown, it includes a multi-level backbone module 200, an average pooling layer 202, and a prediction module 203. The multi-level backbone module 200 is connected to the average pooling layer 202, and the average pooling layer 202 is connected to the prediction module 203. Each level of backbone modules is connected through a maximum pooling layer 201, wherein the step size of the maximum pooling layer is 2, the kernel size is 2x2, the number of convolutional layers of the first backbone module is 2, the number of convolutional layers of the second backbone module is 4, and the number of convolutional layers of the third backbone module is The number of layers is 6, the number of convolutional layers of the fourth backbone module is 8, and the number of convolutional layers of the fifth backbone module is 16. The corresponding convolution kernels are all 3x3, with a step size of 1. The number of channels corresponding to the first and second backbone modules is 64, the number of channels corresponding to the third backbone module is 128, the number of channels corresponding to the fourth backbone module is 256, and the number of channels corresponding to the fifth backbone module is 512. The prediction module 203 contains two fully connected layers, and the corresponding parameters are 1280 and 512.

[0090] Step S1010: After extracting image feature information from the sample data at each level by the backbone modules of the instance, the prediction module outputs the quality score of the video;

[0091] The backbone modules at each level of the instance extract corresponding image features from the sample data, wherein the corresponding shallow-level backbone modules extract basic features such as details and edges in the sample data, and the corresponding deep-level backbone modules extract deep semantic features and high-level logical features of the sample data. In one embodiment, the backbone modules at each level of the instance are connected via a maximum pooling layer, whereby the features extracted by the lower-level backbone modules are input into the upper-level backbone modules after being processed by the maximum pooling layer. Then, after the backbone modules at each level extract image features from the sample data step by step, the terminal backbone module inputs the image features into the average pooling layer connected thereto. The average pooling layer performs spatial averaging on the image features and expands them into a one-dimensional vector in the channel direction, namely, the image feature information. Then, the vector is input into a prediction module connected to the average pooling layer, and the fully connected layer in the prediction module maps the vector to the corresponding classification space. The quality score of the video corresponding to the sample data is calculated and output as a prediction result.

[0092] Step S1020, calculating the model loss value of the quality score according to the pre-labeled supervisory score of the training sample, wherein the supervisory score is a subjective quality score, and when the model loss value does not reach a preset threshold, performing weight update on the instance, and continuing to call other training samples to perform iterative training.

[0093] Referring to step S1000, the supervised score is a subjective quality score, and a preset regression loss function is called to calculate the model loss value of the quality score according to the supervised score pre-labeled by the training sample. For example, the preset regression loss function is:

[0094]

[0095] Among them, Loss mos is the model loss value for calculating the quality score according to the pre-labeled supervision score of the training sample, n is the total number of image frames after video decoding corresponding to the sample data, y out The quality score of the video corresponding to the sample data is predicted by the quality score model, y gt A supervised score is pre-labeled for the training sample.

[0096] Whether to continue iterative training is determined based on the model loss value. If the model loss value does not reach the preset threshold, the weight of the instance is updated, the weights of the backbone module and / or the fully connected layer are updated accordingly, and other training samples in the training set are continued to be called to implement iterative training. The preset threshold can be flexibly set by technicians in this field based on prior knowledge or experimental experience. If the model loss value reaches the preset threshold, it indicates that the instance has been trained to convergence, the model has been initialized, and it can be put into subsequent use.

[0097] In this embodiment, the training process of the initialized instance of the video quality assessment model is disclosed. It can be seen that under the supervised scoring training, the instance has the ability to perform quality scoring on each image frame corresponding to the video, so that the model can accurately predict the quality score corresponding to the video.

[0098] See also Figure 5 In a further embodiment, step S1200, the training process of the instance of the video quality assessment model configured according to the training configuration list, comprises the following steps:

[0099] Step S1210: obtaining multiple image frames of a video in a single training sample, performing data enhancement processing on the multiple image frames to obtain sample data, and synchronously inputting the sample data into the teacher network and the student network;

[0100] In one embodiment, based on hard indicators of video quality such as bit rate, frame rate, and resolution, a sufficient amount of videos can be manually collected and organized as a training set, wherein each video serves as a single training sample, and each single training sample has a corresponding subjective quality score annotated for it, which is used as a supervision score. The subjective quality score can be obtained by having multiple people perform subjective scoring on the video quality of the videos corresponding to the training samples and calculating the average of these subjective scores.

[0101] A video of a single training sample is obtained from the training set, and the corresponding multiple image frames are obtained by decoding the video. The multiple image frames are subjected to data enhancement processing to obtain sample data. The data enhancement processing can be mirroring, rotation, scaling, adjusting brightness, contrast, Gaussian noise, Mosaic, Mixup, Cutout, CutMix, etc. Technical personnel in this field can flexibly select and implement it according to actual business needs, and then synchronously input the sample data into the teacher network and the student network.

[0102] Please refer to step S1200 for revealing the teacher network and the student network, and this step will not be described in detail.

[0103] Step S1220, after extracting image feature information from the sample data level by level through the backbone modules at each level in the teacher network and the student network, the quality score of the video is output through their respective prediction modules;

[0104] In one embodiment, the knowledge distillation diagram corresponding to the teacher network and the student network is as follows: Figure 6 It can be understood that the corresponding backbone module structures at all levels in the teacher network and the student network are the same, and the corresponding number of channels is the same, only the scale of the convolution layer in the partial or full backbone module is different. It can be seen that the model structures corresponding to the teacher network and the student network are the same, and the disclosure of the corresponding weights of the student network reused by the teacher network in step S1300 can be referred to, and it can be known that the convolution layers corresponding to the partial or full backbone modules in the student network reuse the weights of the convolution layers corresponding to the partial or full backbone modules of the corresponding teacher network. Further, the disclosure of the quality score corresponding to the predicted video by the example in step S1010 can be referred to, and the quality score corresponding to the predicted video by the teacher network and the student network in this step can be known. This step will not be repeated.

[0105] Step S1230, calculating the model loss value of the student network, the loss value including the loss value between the pre-labeled supervision score of the training sample and the quality score predicted by the student network, the supervision score being a subjective quality score, and the feature loss value between the image feature information obtained by the backbone modules at each level between the teacher network and the student network;

[0106] Referring to step S1020, the preset regression loss function is called to calculate the loss value corresponding to the loss value between the pre-labeled supervision score of the training sample and the quality score predicted by the student network. mos This step will not be described in detail. In addition, the preset loss function is called to calculate the feature loss value between the image feature information obtained by the backbone modules at all levels between the teacher network and the student network. For example, the preset loss function is:

[0107]

[0108] Among them, Loss distill is the feature loss value between the image feature information obtained by the backbone modules at all levels between the teacher network and the student network, n is the total number of backbone modules in the student network, st is the current corresponding backbone module (such as st = 1, indicating the first backbone module in the student network), c is the number of channels of the backbone module corresponding to st, w and h correspond to the width and height of the feature map extracted by the backbone module corresponding to st, is the eigenvalue of the teacher network in the kth row and jth column of the feature graph of channel i, is the feature value of the student network in the kth row and jth column of the feature map of channel i.

[0109] In summary, the Loss mos and Loss distill Added together as the model loss value of the student network.

[0110] Step S1240, determine whether the model loss value reaches a preset threshold value, and if it does not reach the preset threshold value, perform weight update on the instance, and continue to call other training samples to perform iterative training.

[0111] Furthermore, based on the model loss value of the student network, it is determined whether the model loss value reaches a preset threshold. When the preset threshold is not reached, the weight of the instance is updated, and the corresponding weights of the backbone module and / or the fully connected layer of the student network are updated, so that the student network can fit the probability of the teacher network mapping to each classification space, and continue to call other training samples in the training set to implement iterative training. When the preset threshold is reached, it is characterized that the student network has been trained to convergence.

[0112] In this embodiment, the training process of the instance of the video quality assessment model configured by the training configuration list is disclosed. It can be seen that under the supervised scoring training and knowledge distillation, the student network has the same ability as the teacher network, that is, the ability to perform quality scoring on each image frame corresponding to the video, so that the student network can accurately predict the quality score corresponding to the video. Furthermore, since the student network is obtained by compressing the teacher network, it becomes more lightweight and can be deployed online, and can predict the quality score corresponding to the video in real time. Secondly, the knowledge distillation between the teacher network and the student network is not only to distill the prediction modules of the two, but also to distill the backbone modules at all levels between the two, so that the student network can more comprehensively extract the features extracted by the teacher network, so as to have the same generalization ability and robustness as the teacher network.

[0113] See also Figure 7In a further embodiment, after step S1400, configuring the mean weight of the weights of all trained instances as the corresponding weight of the last trained instance so that the instance becomes the output video quality assessment model, the following steps are included:

[0114] Step S1500: input multiple continuous image frames in the live video stream into the generated video quality assessment model to predict their corresponding quality scores;

[0115] In one embodiment, in a live broadcast scenario on an e-commerce platform, the live broadcast user submits a video stream to the server of the e-commerce platform, which is decoded to obtain image frames in YUV or RGB format. At this time, the video quality assessment model produced by the present application deployed on the server can be run in real time to predict the scores corresponding to each continuous image frame in time, thereby obtaining the quality score corresponding to the video stream.

[0116] In another embodiment, in a live broadcast scenario of a terminal device, a camera unit collects data frames corresponding to image frames. One way is to draw the data frame texture to render and display an image user interface to the live broadcast user, and the other way is to convert the data frame into an image frame in YUV or RGB format through corresponding decoding operations. At this time, the video quality assessment model produced by the present application deployed on the terminal device can be run in real time to predict the quality scores corresponding to each temporally continuous image frame.

[0117] Step S1600: adjusting the encoding frame rate of the encoder of the video stream according to the quality score.

[0118] Further, the encoding frame rate of the encoder of the video stream is adjusted according to the quality score, so that the bit rate and / or resolution of the video stream is correspondingly reduced, and the adjusted video stream is output to the media server of the live broadcast network so as to be pushed to the devices of the audience users in the live broadcast room.

[0119] In this embodiment, it is disclosed that the generated video quality assessment model is deployed online and applied to the video stream of live broadcast on the Internet, so that the encoding frame rate of the encoder of the video stream can be adjusted according to the quality score of the video stream predicted by the model, so that the adjusted video stream can not only meet the video quality required by the business, but also save the bandwidth for transmitting the video stream.

[0120] See also Figure 8, a video quality assessment model production device provided to meet one of the purposes of the present application is a functional embodiment of the video quality assessment model production method of the present application, the device includes an initialization module 1100, an instance configuration module 1200, an iterative distillation module 1300, and a model output module 1400, wherein: the initialization module 1100 is used to obtain an initialized instance of a video quality assessment model that has been trained to a convergent state, the model includes a multi-level backbone module for gradually downsampling sample data of multiple image frames of the same video to extract features, and a prediction module for predicting the quality score of the image feature information output by the terminal backbone module, each level of the backbone module includes a one or more convolutional layers; an instance configuration module 1200, for obtaining a training configuration list of the video quality assessment model, the training configuration list sequentially containing configuration information of the convolutional layer scales of the backbone modules at all levels corresponding to the other multiple instances of the model, wherein the overall scale of the convolutional layer of the subsequent instance is smaller than that of the previous instance and does not exceed the initialized instance; an iterative distillation module 1300, for using the converged instance as the teacher network, sequentially calling a single instance in the training configuration list as the student network, reusing the weights of the teacher network to train it to a converged state, and then replacing the teacher network to continue iterative knowledge distillation training on other instances until the last instance is trained to converge;

[0121] In a further embodiment, the training process of the initialized instance of the video quality assessment model of the instance configuration module 1200 includes: a first sample input submodule, used to obtain multiple image frames of a video in a single training sample, perform data enhancement processing on the multiple image frames to obtain sample data, and input the sample data into the instance; a first feature extraction submodule, used to extract image feature information from the sample data step by step by the backbone modules of each level of the instance, and output the quality score of the video through a prediction module; a first iterative training submodule, used to calculate the model loss value of the quality score based on the supervised score pre-labeled by the training sample, the supervised score is a subjective quality score, and when the model loss value does not reach a preset threshold, the weight of the instance is updated, and other training samples are continued to be called to implement iterative training.

[0122] In a further embodiment, the training process of the instance of the video quality assessment model configured according to the training configuration list of the instance configuration module 1200 includes: a second sample input submodule, which is used to obtain multiple image frames of a video in a single training sample, obtain sample data after data enhancement processing of the multiple image frames, and synchronously input them into the teacher network and the student network; a second feature extraction submodule, which is used to extract image feature information from the sample data step by step through the backbone modules at various levels in the teacher network and the student network, and then output the quality score of the video through their respective prediction modules; a loss calculation submodule, which is used to calculate the model loss value of the student network, the loss value including the loss value between the pre-labeled supervision score of the training sample and the quality score predicted by the student network, the supervision score being a subjective quality score, and the feature loss value between the image feature information obtained by the backbone modules at various levels between the teacher network and the student network; a second iterative training submodule, which is used to determine whether the model loss value reaches a preset threshold, and when the preset threshold is not reached, weight update is performed on the instance, and other training samples are continued to be called to perform iterative training.

[0123] In a further embodiment, in the training configuration list of the initialization module 1100, the scales of convolutional layers corresponding to the backbone modules at each level are expressed in quantity, wherein the number of convolutional layers of some backbone modules of the later-ordered instances is greater than the number of convolutional layers of the corresponding backbone modules of the earlier instances, while the number of convolutional layers of the remaining backbone modules remains consistent.

[0124] In a further embodiment, in the video quality assessment model of the instance configuration module 1200, the output of the front backbone module is processed by the maximum pooling layer and then input into the subsequent backbone module, and the output of the last-level backbone module is mean pooled by the average pooling layer and then input into the prediction module.

[0125] In a further embodiment, in the video quality assessment model of the instance configuration module 1200, the convolutional layer scale of a subsequent backbone module is not smaller than the convolutional layer scale of a previous backbone module, and there is at least one subsequent backbone module whose convolutional layer scale is larger than the convolutional layer scale of a previous backbone module.

[0126] In a further embodiment, the model generation module 1400 includes: a model prediction submodule, used to input multiple continuous image frames in the live network video stream into the generated video quality assessment model to predict their corresponding quality scores; a frame rate adjustment submodule, used to adjust the encoding frame rate of the encoder of the video stream according to the quality score.

[0127] In order to solve the above technical problems, the present application also provides a computer device. Fig. 9As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a video quality assessment model production method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the video quality assessment model production method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art can understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0128] In this embodiment, the processor is used to execute Figure 8 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the video quality assessment model production device of the present application, and the server can call the program code and data of the server to execute the functions of all submodules.

[0129] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the video quality assessment model production method of any embodiment of the present application.

[0130] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0131] To summarize, this application first trains the student network through iterative distillation, and the final video quality assessment model has the same capabilities as the teacher network, and can also accurately predict the video quality corresponding to the video. In addition, the produced video quality assessment model is lightweight and can be deployed online to run the video quality assessment in real time and predict the corresponding quality score.

[0132] Secondly, during the iterative distillation training process, the student network in each iteration reuses part of the weights of the teacher network, which can not only maintain the accuracy of the student network's predictions, but also accelerate the training of the student network and improve its convergence speed.

[0133] In addition, the present application takes the average of the weight parameters obtained by repeatedly training the last instance until the convergence state, and configures it as the actual weight of the last instance, which can increase the generalization ability and robustness of the model.

[0134] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.

[0135] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for producing a video quality assessment model, characterized in that: The steps include: Obtaining an instance of initialization of a video quality assessment model trained to a converged state, the model comprising a multi-level backbone module for downsampling sample data of multiple image frames of the same video step by step to extract features, and a prediction module for predicting a quality score of image feature information output by a terminal backbone module, wherein each level of the backbone module comprises one or more convolutional layers; Obtaining a training configuration list of the video quality assessment model, the training configuration list sequentially containing configuration information of the convolutional layer scales of the backbone modules at various levels corresponding to other multiple instances of the model, wherein the overall scale of the convolutional layer of the subsequent instance is smaller than that of the previous instance and does not exceed that of the initialized instance; in the training configuration list, the scales of the convolutional layers corresponding to the backbone modules at various levels are expressed in quantity, wherein the number of convolutional layers of some backbone modules of the subsequent instance is larger than the number of convolutional layers of the corresponding backbone modules of the previous instance, while the number of convolutional layers of the remaining backbone modules remains consistent; Using the converged instance as the teacher network, calling the single instance in the training configuration list in sequence as the student network, reusing the weight of the teacher network to train it to a converged state, and then replacing the teacher network to continue iterative knowledge distillation training on other instances until the last instance is trained to convergence; Repeat the training of the last instance multiple times until convergence, take the average of the weight parameters obtained from the multiple trainings of the instance, configure it as the actual weight of the last instance, and use the instance as the output of the video quality assessment model.

2. The method for producing a video quality assessment model according to claim 1, characterized in that: The training process of the initialized instance of the video quality assessment model comprises the following steps: Acquire multiple image frames of a video in a single training sample, perform data enhancement processing on the multiple image frames to obtain sample data, and input the sample data into the instance; After the backbone modules at each level of the instance extract image feature information from the sample data level by level, the prediction module outputs the quality score of the video; The model loss value of the quality score is calculated according to the pre-labeled supervision score of the training sample, and the supervision score is a subjective quality score. When the model loss value does not reach a preset threshold, the weight of the instance is updated, and other training samples are continuously called to implement iterative training.

3. The method for producing a video quality assessment model according to claim 1, characterized in that: The training process of the instance of the video quality assessment model configured according to the training configuration list comprises the following steps: Acquire multiple image frames of a video in a single training sample, perform data enhancement processing on the multiple image frames to obtain sample data, and synchronously input the sample data into the teacher network and the student network; After extracting image feature information from the sample data level by level through the backbone modules at each level in the teacher network and the student network, the quality score of the video is output through their respective prediction modules; Calculating a model loss value of the student network, the loss value comprising a loss value between a supervised score pre-labeled by the training sample and a quality score predicted by the student network, the supervised score being a subjective quality score, and a feature loss value between image feature information obtained by backbone modules at various levels between the teacher network and the student network; Determine whether the model loss value reaches the preset threshold. If it does not reach the preset threshold, update the weight of the instance and continue to call other training samples to implement iterative training.

4. The method for producing a video quality assessment model according to any one of claims 1 to 3, characterized in that: In the video quality assessment model, the output of the front backbone module is input into the rear backbone module for processing via the maximum pooling layer, and the output of the last backbone module is input into the prediction module via the average pooling layer for mean pooling.

5. The method for producing a video quality assessment model according to any one of claims 1 to 3, characterized in that: In the video quality assessment model, the convolutional layer scale of the subsequent backbone module is not smaller than the convolutional layer scale of the previous backbone module, and there is at least one subsequent backbone module whose convolutional layer scale is larger than the convolutional layer scale of the previous backbone module.

6. The method for producing a video quality assessment model according to any one of claims 1 to 3, characterized in that: After configuring the mean weight of all trained instances as the corresponding weight of the last trained instance so that the instance becomes the output video quality assessment model, the following steps are included: Input multiple consecutive image frames in the live video stream of the network into the generated video quality assessment model to predict their corresponding quality scores; The encoding frame rate of the encoder of the video stream is adjusted according to the quality score.

7. A video quality assessment model production device, characterized in that: include: An initialization module, used to obtain an initialized instance of a video quality assessment model trained to a converged state, wherein the model includes a multi-level backbone module for downsampling sample data of multiple image frames of the same video step by step to extract features, and a prediction module for predicting a quality score of image feature information output by a terminal backbone module, wherein each level of the backbone module includes one or more convolutional layers; An instance configuration module obtains a training configuration list of the video quality assessment model, the training configuration list sequentially containing configuration information of the convolutional layer scales of the backbone modules at various levels corresponding to other multiple instances of the model, wherein the overall scale of the convolutional layer of the subsequent instance is smaller than that of the previous instance and does not exceed that of the initialized instance; in the training configuration list, the scales of the convolutional layers corresponding to the backbone modules at various levels are expressed in quantity, wherein the number of convolutional layers of some backbone modules of the subsequent instance is larger than the number of convolutional layers of the corresponding backbone modules of the previous instance, while the number of convolutional layers of the remaining backbone modules remains consistent; An iterative distillation module, used to use the converged instance as the teacher network, sequentially call a single instance in the training configuration list as the student network, reuse the weight of the teacher network to train it to a converged state, and then replace the teacher network to continue to perform iterative knowledge distillation training on other instances until the last instance is trained to convergence; The model output module is used to configure the mean weight of the weights of all trained instances as the corresponding weight of the last trained instance, so that the instance becomes the output video quality assessment model.

8. A computer device comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 6 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images