Video quality assessment model production method and its device, equipment, and medium

By sampling and jointly training sub-network structures from a pre-trained super-network model, the method addresses the inefficiencies of existing video quality assessment models, producing a lightweight model that efficiently assesses video quality with high accuracy and reduced resource consumption.

CN115115986BActive Publication Date: 2024-12-03GUANGZHOU HUANJU SHIDAI INFORMATION TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202210749997.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-28
Publication Date
2024-12-03
Estimated Expiration
2042-06-28

AI Technical Summary

Technical Problem

Existing video quality assessment models suffer from low accuracy and inefficient operation due to complex model structures that are difficult to train and compress, failing to meet user needs and network resource optimization.

Method used

A method involving sampling sub-network structures from a pre-trained super-network model, jointly training these sub-networks, and verifying their performance to produce a lightweight video quality assessment model that can efficiently converge and maintain accuracy.

Benefits of technology

The method enables efficient training of a lightweight video quality assessment model that can perform intelligent, reference-free video quality assessment without significant manpower costs, improving operating efficiency and maintaining accuracy and generalization ability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115115986B_ABST
    Figure CN115115986B_ABST
Patent Text Reader

Abstract

The present application relates to a method for producing a video quality assessment model in the field of image recognition technology and its device, equipment, and medium. The method includes: obtaining a sampling space corresponding to a network structure of a super network model pre-trained to convergence and a search space of a corresponding sub-network structure; sampling and determining multiple sub-network structures from the corresponding sampling space according to the network structure of the super network model, and obtaining a sub-network model corresponding to each sub-network structure; jointly training each sub-network model corresponding to the multiple sub-network structures with the super network model, and obtaining a sub-network model trained to convergence as a candidate model; verifying the performance of each candidate model, and selecting at least one candidate model as a video quality assessment model. The present application can produce an efficient video quality assessment model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method for producing a video quality assessment model and its corresponding device, computer equipment, and computer-readable storage medium. Background Art

[0002] Video quality assessment is a method that has been studied in academia and industry since the emergence of video. It is mainly divided into reference assessment and non-reference assessment. Reference assessment compares the effect difference frame by frame based on the given reference video, so that a relatively accurate assessment score can be obtained more easily. Video non-reference assessment is a method that gives the user's subjective experience score of the current video quality. According to the score of the non-reference assessment, the quality indicators such as the bit rate, frame rate, and resolution of network transmission can be dynamically adjusted to achieve the purpose of cost control. Through adjustment, the user's subjective video viewing experience is basically unaffected. This implementation not only meets user needs, but also saves network transmission resources in disguise, which can bring considerable economic benefits.

[0003] In order to improve the accuracy of the video quality assessment model implemented by traditional technology, the model structure is generally complex and the operation efficiency is low. In addition, in order to improve the operation efficiency, the traditional model compression technology solution cannot grasp the structure of the compressed model, which often leads to a model structure that is too simple, making it difficult to ensure accuracy and unable to meet actual business needs.

[0004] In view of the shortcomings of traditional technology, this application makes corresponding explorations. Summary of the invention

[0005] The primary purpose of the present application is to solve at least one of the above problems and provide a video quality assessment model production method and its corresponding device, computer equipment, and computer-readable storage medium.

[0006] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0007] A method for producing a video quality assessment model adapted to one of the purposes of this application comprises the following steps:

[0008] Obtaining a sampling space corresponding to a network structure of a super network model pre-trained to convergence and a search space of a corresponding sub-network structure;

[0009] According to the network structure of the super network model, sampling from the corresponding sampling space determines a plurality of sub-network structures, and obtains a sub-network model corresponding to each sub-network structure;

[0010] Jointly train the sub-network models corresponding to the multiple sub-network structures with the super-network model, and obtain the sub-network models trained to convergence as candidate models;

[0011] Verify the performance of each candidate model and select at least one candidate model as a video quality assessment model.

[0012] In a further embodiment, the training process of the hypernetwork model comprises the following steps:

[0013] Acquire multiple image frames of a video in a single training sample, perform data enhancement processing on the multiple image frames to obtain sample data and input the sample data into the hypernetwork model;

[0014] After the hypernetwork model extracts image feature information from the sample data, the prediction module outputs a prediction of the quality score corresponding to the sample data;

[0015] The loss value of the hypernetwork model is calculated, and when the loss value of the model does not reach a preset threshold, the weight of the model is updated, and other training samples are continuously called to perform iterative training until the model converges.

[0016] In a preferred embodiment, the loss value calculation includes the following steps:

[0017] Calculate the regression loss value of the predicted quality score obtained by predicting the sample data using the hypernetwork model according to the supervised score corresponding to the sample data, wherein the supervised score is a subjective quality score;

[0018] Calculate the corresponding cross entropy loss value according to the regression loss value;

[0019] The prediction quality score obtained by the hypernetwork model predicting the sample data and the supervision score corresponding to the sample data are subjected to the same shuffle processing respectively, and the difference between the prediction quality score and the shuffled prediction quality score after the shuffle processing is used as the input of the preset function, and the absolute value of the difference between the supervision score and the shuffled supervision score after the shuffle processing and the sum of the function output result are calculated, and the maximum value between the sum and 0 is taken as the shuffle loss value;

[0020] The sum of the regression loss value, the cross entropy loss value, and the out-of-order loss value is calculated as the loss value.

[0021] In a further embodiment, the joint training process comprises the following steps:

[0022] Acquire multiple image frames of a video in a single training sample, perform data enhancement on the multiple image frames to obtain sample data, and synchronously input the sample data into the super network model and each sub-network model;

[0023] After the super network model and each sub network model extract image feature information from the sample data respectively, a prediction module predicts a quality score corresponding to the sample data;

[0024] According to the corresponding loss values ​​of the super network model and each sub-network model, the corresponding joint loss value is calculated. When the joint loss value does not reach the preset threshold, the weights of each sub-network model are updated accordingly, and other training samples are continued to be called to implement iterative training.

[0025] In a further embodiment, the step of verifying the performance of each candidate model and selecting at least one candidate model as a video quality assessment model includes the following steps:

[0026] Using a proxy validation set to perform performance measurement on each candidate model to obtain comprehensive performance corresponding to each candidate model, wherein the comprehensive performance includes running time and / or accuracy index;

[0027] The candidate model corresponding to the comprehensive performance that meets the preset conditions is selected and output as the video quality assessment model.

[0028] In a further embodiment, the step of using a proxy validation set to perform performance measurement on each candidate model to obtain the comprehensive performance corresponding to each candidate model includes the following steps:

[0029] Perform channel search from the sub-network structures corresponding to the candidate models, and determine multiple compression candidate models corresponding to the sub-network structures in the target compression channel;

[0030] Using the input corresponding to the training samples in the proxy validation set to the multiple compression candidate models, and calculating the corresponding running time and accuracy indicators;

[0031] Determine the overall performance based on the run time and accuracy metrics.

[0032] In a further embodiment, after the step of determining the comprehensive performance according to the running time and the accuracy index, the following steps are also included:

[0033] For the network structure shared by each compression candidate model and the super network model, reuse the weight of the super network model corresponding to the shared network structure;

[0034] Each compression candidate model is jointly trained with the hypernetwork model, and the trained compression candidate model with comprehensive performance that meets preset conditions is selected and output as a video quality assessment model.

[0035] On the other hand, a video quality assessment model production device provided to meet one of the purposes of the present application includes a sampling acquisition module, a structure sampling module, a joint training module and a performance verification module, wherein: the sampling acquisition module is used to obtain a sampling space corresponding to the network structure of a super network model pre-trained to convergence and a search space of its corresponding sub-network structure; the structure sampling module is used to sample and determine multiple sub-network structures from the corresponding sampling space according to the network structure of the super network model, and obtain a sub-network model corresponding to each sub-network structure; the joint training module is used to jointly train each sub-network model corresponding to the multiple sub-network structures with the super network model, and obtain a sub-network model trained to convergence as a candidate model; the performance verification module is used to verify the performance of each candidate model and screen out at least one candidate model as a video quality assessment model.

[0036] In a further embodiment, the sampling acquisition module includes: a first sample input submodule, which is used to acquire multiple image frames of a video in a single training sample, and perform data enhancement processing on the multiple image frames to obtain sample data and input them into the hypernetwork model; a first quality prediction submodule, which is used to extract image feature information from the sample data by the hypernetwork model, and then predict the quality score corresponding to the sample data through the prediction module output; a first weight update submodule, which is used to calculate the loss value of the hypernetwork model, implement weight update on the model when the loss value of the model does not reach a preset threshold, and continue to call other training samples to implement iterative training until the model converges.

[0037] In a preferred embodiment, the weight update submodule includes: a regression loss submodule, which is used to calculate the regression loss value of the predicted quality score obtained by the hypernetwork model predicting the sample data according to the supervised score corresponding to the sample data, and the supervised score is a subjective quality score; a cross entropy loss submodule, which is used to calculate the corresponding cross entropy loss value according to the regression loss value; a disorder loss submodule, which is used to perform the same disorder processing on the predicted quality score obtained by the hypernetwork model predicting the sample data and the supervised score corresponding to the sample data, and use the difference between the predicted quality score and the disordered predicted quality score after the disorder processing as the input of the preset function, calculate the absolute value of the difference between the supervised score and the disordered supervised score after the disorder processing and the sum of the function output result, and take the maximum value between the sum and 0 as the disorder loss value; a loss value submodule, which is used to calculate the sum of the regression loss value, the cross entropy loss value and the disorder loss value as the loss value.

[0038] In a further embodiment, the joint training module includes: a second sample input submodule, used to obtain multiple image frames of a video in a single training sample, perform data enhancement on the multiple image frames to obtain sample data, and synchronously input them into the super network model and each sub-network model; a second quality prediction submodule, used to extract image feature information from the sample data respectively by the super network model and the each sub-network model, and then predict the quality score corresponding to the sample data through a prediction module; a second weight update submodule, used to calculate the corresponding joint loss value according to the corresponding loss values ​​of the super network model and the each sub-network model, and implement weight update for each sub-network model accordingly when the joint loss value does not reach a preset threshold, and continue to call other training samples to implement iterative training.

[0039] In a further embodiment, the performance verification module includes: a performance calculation submodule, which is used to use a proxy verification set to perform performance measurement on each candidate model to obtain the comprehensive performance corresponding to each candidate model, and the comprehensive performance includes running time and / or accuracy indicators; a model output submodule, which is used to select the candidate model corresponding to the comprehensive performance that meets preset conditions and output it as a video quality assessment model.

[0040] In a further embodiment, the performance calculation submodule includes: a channel search unit, which is used to perform channel search from the sub-network structures corresponding to each candidate model, and determine multiple compression candidate models corresponding to the sub-network structure in the target compression channel; a model performance estimation unit, which is used to use the input corresponding to the training samples in the proxy verification set to the multiple compression candidate models, and calculate the corresponding running time and accuracy indicators; a performance determination unit, which is used to determine the comprehensive performance based on the running time and accuracy indicators.

[0041] In a further embodiment, the performance determination unit further includes: a weight reuse unit for reusing the weights of the super network model corresponding to the shared network structure for the respective compression candidate models and the super network model; and a model output unit for jointly training the respective compression candidate models with the super network model, selecting the trained compression candidate models with comprehensive performance that meet preset conditions, and outputting them as video quality assessment models.

[0042] On the other hand, a computer device provided to meet one of the purposes of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the video quality assessment model production method described in the present application.

[0043] On the other hand, a computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the video quality assessment model production method in the form of computer-readable instructions, and when the computer program is called and executed by a computer, it executes the steps included in the method.

[0044] The technical solution of this application has many advantages, including but not limited to the following aspects:

[0045] On the one hand, the network structure of the super network model is sampled through the sampling space to determine the corresponding sub-network model, which is jointly trained with the super network model to obtain the sub-network model trained to convergence as the candidate model, and then the performance of the candidate model is verified, and the candidate model with better performance is screened out as the video quality assessment model. It can be understood that the network structure corresponding to the sub-network model determined by sampling in the sampling space is a simplified version of the network structure of the super network model, that is, a lightweight sub-network structure is realized, which can effectively improve the operating efficiency of the final video quality assessment model, and the lightweight model is easier to be trained to convergence, which can achieve efficient training. In addition, due to the use of joint training, the accuracy and generalization ability of the obtained candidate model can be guaranteed.

[0046] On the other hand, the video quality assessment model produced by the present application can perform reference-free video quality assessment on video streams in an intelligent and efficient manner without requiring a large amount of manpower costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0048] Figure 1 A flowchart of a typical embodiment of the method for producing a video quality assessment model of the present application;

[0049] Figure 2 A schematic diagram of an exemplary example of a network structure of a super network model in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of an exemplary example of a backbone module of a network structure of a super network model in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of an exemplary example of a sub-network model in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of a hypernetwork model training process in an embodiment of the present application;

[0053] Figure 6A schematic diagram of a process for calculating a loss value in an embodiment of the present application;

[0054] Figure 7 A schematic diagram of the joint training process in an embodiment of the present application;

[0055] Figure 8 A schematic diagram of a process for generating a video quality assessment model in an embodiment of the present application;

[0056] Fig. 9 A schematic diagram of a process for determining comprehensive performance in an embodiment of the present application;

[0057] Fig.10 A schematic diagram of a process for training a video quality assessment model in an embodiment of the present application;

[0058] Fig.11 A block diagram of the video quality assessment model production device of this application;

[0059] Fig.12 A schematic diagram of the structure of a computer device used in this application. DETAILED DESCRIPTION

[0060] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0061] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0062] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0063] It will be understood by those skilled in the art that the terms "client", "terminal" and "terminal device" as used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capability, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication over a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service), which may combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or adapted and / or configured to run locally, and / or in a distributed form, running at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MI D (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or may be a smart TV, a set-top box and other devices.

[0064] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0065] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0066] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for access.

[0067] The neural network models referenced or may be referenced in this application, unless expressly specified, can be deployed on a remote server and remotely called on the client, or can be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0068] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as it is suitable for being called by the technical solution of this application.

[0069] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0070] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0071] A video quality assessment model production method of the present application can be programmed as a computer program product and deployed in a client or server for execution. For example, in the exemplary application scenario of the present application, it can be deployed and implemented in a server of an e-commerce platform, thereby accessing an interface opened after the computer program product is running and performing human-computer interaction with the process of the computer program product through a graphical user interface to execute the method.

[0072] See also Figure 1 The video quality assessment model production method of the present application, in its typical embodiment, comprises the following steps:

[0073] Step S1100, obtaining a sampling space corresponding to a network structure of a super network model pre-trained to convergence and a search space of a corresponding sub-network structure;

[0074] The network structure of the hypernetwork model is exemplified as follows: Figure 2 As shown, it includes a backbone module 200, a global average pooling layer 201, and a prediction module 202. The backbone module is connected to the global average pooling layer, and the global average pooling layer is connected to the prediction module. The backbone module 200 includes multiple stages of convolution modules, and the convolution modules of the previous stage are connected to the convolution modules of the corresponding next stage. Figure 3 As shown, the convolution module includes a convolution kernel and a RELU layer, and the prediction module 202 includes an FC layer (fully connected layer). For example, the backbone module of the hypernetwork model includes 5 stages of convolution modules, and the number of channels corresponding to each convolution module is [64, 64, 128, 256, 512]. This structure is close to the VGG model, which helps to ensure the robustness and generalization ability of the hypernetwork model. Of course, those skilled in the art can flexibly set the convolution module and its corresponding number of channels according to the disclosure herein.

[0075] The corresponding sampling space can be preset according to the network structure of the super network model, so as to facilitate the subsequent sampling and determination of the corresponding sub-network structure. In one embodiment, the corresponding channel number of each convolution module of the backbone module of the model can be gradually simplified, and the gradual simplification can be an exponential decrease with a base of 2, and the channel number interval corresponding to the multiple channel numbers that are gradually simplified is confirmed, and the channel number interval corresponding to each convolution module is used as the sampling space. For example, the channel data corresponding to the convolution module of the first stage is 64, and the corresponding channel number interval can be [16, 32, 48, 64]. For example, the channel data corresponding to the convolution scale of the fourth stage is 256, and the corresponding channel number space can be [16, 32, 64, 128, 256]. It can be seen that the difference between the adjacent channel numbers in the channel number interval is an exponent with a base of 2. Those skilled in the art can flexibly set the channel number space corresponding to each convolution module according to the disclosure herein. Those skilled in the art should know that the sampling space realized by such reasonable planning, while ensuring a certain compression of the network structure, determines fewer options for subsequent sub-network structure sampling, greatly improving the execution efficiency of sampling.

[0076] A search space of a corresponding sub-network structure may be further preset according to the sampling space. In one embodiment, the sampling space is shown in the following table:

[0077] Stage 1, [16,32,48,64]

[0078] Stage 2, [16, 32, 48, 64]

[0079] Stage 3, [16,32,64,96,128]

[0080] Stage 4, [16,32,64,128,256]

[0081] Stage 5, [128, 256, 384, 512]

[0082] Among them, Stage1, Stage2, Stage3, Stage4, and Stage5 are convolution modules, and "[]" is the channel number space corresponding to the convolution module. Further, a subset of the middle part of each channel number space can be selected as the channel number space corresponding to each convolution module in the search space, thereby obtaining the search space. The exemplary examples are shown in the following table:

[0083] Stage 1, [32,48]

[0084] Stage 2, [32, 48]

[0085] Stage 3, [32, 64, 96]

[0086] Stage 4, [16,32,128]

[0087] Stage5, [256,384]

[0088] Those skilled in the art should be aware that the search space implemented above can ensure that the number of channels in the network structure of the model obtained by subsequent channel search in the search space is sufficient, thereby ensuring the accuracy and generalization ability of the model.

[0089] The hypernetwork model may be supervised and trained using a video stream carrying manually annotated scores until the model is trained to convergence. The specific implementation is further disclosed in the subsequent embodiments, and this step is not discussed for the time being.

[0090] Step S1200: Determine a plurality of sub-network structures by sampling from the corresponding sampling space according to the network structure of the super-network model, and obtain a sub-network model corresponding to each sub-network structure;

[0091] In one embodiment, according to each convolution module in the backbone module of the network structure of the hypernetwork model, the number of channels corresponding to each convolution module is determined by sampling from the sampling space, and then the corresponding partial structure of each convolution module is obtained as the backbone module of the sub-network structure. A sampling rule can be set here to constrain the sampling. The sampling rule is that when sampling each convolution module, it can only start from the first channel. For example, the number of channels of the convolution module at the final stage is 512, and the corresponding channel numbers are 1 to 512. If the number of sampled channels is 384, then the convolution module at the final stage of sampling is the partial structure of the channel numbers from 1 to 384, and cannot be the partial structure of the channel numbers from 11 to 394. It can be seen that the use of this sampling rule for sampling can greatly reduce the number of partial structures corresponding to different starting channel numbers when sampling the same number of channels, thereby improving the operating efficiency of sampling. Further, it can be understood that since the fully connected layer in the prediction module is connected to the convolution module of the final stage in the backbone module through the global average pooling, the fully connected layer needs to adapt to the changes of the convolution module of the final stage and change appropriately. For example, the number of channels of the convolution module of the final stage is 512, the parameter of the fully connected layer is 1024 and the input dimension is 512, and the corresponding parameter amount is 1024*512, which is equal to 524,288. At this time, when the number of channels of the convolution module of the final stage is 256, the parameter of the fully connected layer is 1024 and the input dimension is 256. At this time, the corresponding parameter amount is 1024*256, which is equal to 262,144. It can be seen that the parameters of the fully connected layer change with the change of the number of channels of the convolution module of the final stage, and the compression ratio is consistent. Based on this, dynamic adaptation of the fully connected layer can be achieved according to the consistency of the compression ratio. Based on the above, the backbone module and prediction module in the sub-network structure are obtained and the global average pooling layer is kept unchanged to achieve the sampling of the backbone module of the super network. That is, the network structure of the super network is complete, and the sub-network structure is obtained by using parts of each convolution module in the backbone module of the super network. Thus, the sub-network model corresponding to each sub-network structure is obtained. The schematic diagram of the sub-network model is shown as an example. Figure 4 As shown, the number of channels corresponding to sub-network 1 is [32, 32, 64, 256, 512], the number of channels corresponding to sub-network 2 is [32, 32, 128, 256, 384], and the number of channels corresponding to sub-network 3 is [32, 32, 64, 128, 256].

[0092] Step S1300: jointly train each sub-network model corresponding to the multiple sub-network structures with the super-network model to obtain a sub-network model trained to convergence as a candidate model;

[0093] The sub-network models corresponding to the multiple sub-network structures are jointly trained with the super-network model, during which the weights of the convolutional module structure shared by the sub-network model and the super-network model are shared, so that the weights of each sub-network model can be optimized during the training process. In one embodiment, the loss function used in the joint training is exemplified by the following formula:

[0094]

[0095] Among them: Loss ofa is the loss value of the joint training of the super network model and the sub-network model, n is the number of sub-networks sampled in a single weight update, i is the sub-network sampled in the sampling space, is the loss value of the sub-network model, Loss supernet is the loss value of the hypernetwork model.

[0096] The loss value of the sub-network model and the loss value of the super-network model may be any loss value known to those skilled in the art, such as a regression loss value, a cross entropy loss value, or a loss value obtained by mixing multiple loss values.

[0097] The preset training set is used to jointly train each sub-network model with the super-network model. It can be understood that the Loss ofa Determine the error of the prediction results of each sub-network model. ofa When the preset threshold is met, it is determined that the sub-network model is trained to convergence, thereby obtaining the sub-network model trained to convergence as a candidate model. The training set can be constructed by collecting multiple video streams of different video qualities, and in addition, the corresponding real labels are annotated for each video stream, and the real labels can be obtained by manually scoring the video stream.

[0098] Step S1400: verify the performance of each candidate model and select at least one candidate model as a video quality assessment model.

[0099] In one embodiment, a network structure search is performed on each candidate model in the search space of the sub-network structure, and the performance of each candidate model is verified using a preset verification set. For each backbone module in the network structure of each candidate model, the number of channels, loss value, and running time of each corresponding convolution module are recorded. Furthermore, the loss value and running time are used as performance evaluation indicators of the candidate model, and at least one candidate model whose loss value and running time reach a preset threshold is screened out as a video quality model. The threshold can be flexibly set by a person skilled in the art according to business needs.

[0100] The video quality assessment model of the present application will be used to predict the no-reference score for the video stream. Accordingly, in one embodiment, the video quality assessment model of the present application can be deployed on the media server corresponding to the live broadcast on the e-commerce platform to provide corresponding services, and the service can be used to predict the no-reference score for the video stream generated by the live broadcast on the e-commerce platform. The video stream is composed of image frames that are continuous in time, and the data of the corresponding multiple image frames can be obtained by decoding the video stream as the sample data, which is input into the model to extract features for training the model.

[0101] It can be known from the typical embodiments of the present application that the technical solution of the present application has many advantages, including but not limited to the following aspects:

[0102] On the one hand, the network structure of the super network model is sampled through the sampling space to determine the corresponding sub-network model, which is jointly trained with the super network model to obtain the sub-network model trained to convergence as the candidate model, and then the performance of the candidate model is verified, and the candidate model with better performance is screened out as the video quality assessment model. It can be understood that the network structure corresponding to the sub-network model determined by sampling in the sampling space is a simplified version of the network structure of the super network model, that is, a lightweight sub-network structure is realized, which can effectively improve the operating efficiency of the final video quality assessment model, and the lightweight model is easier to be trained to convergence, which can achieve efficient training. In addition, due to the use of joint training, the accuracy and generalization ability of the obtained candidate model can be guaranteed.

[0103] On the other hand, the video quality assessment model produced by the present application can perform reference-free video quality assessment on video streams in an intelligent and efficient manner without requiring a large amount of manpower costs.

[0104] See also Figure 5 In a further embodiment, step S1100, the training process of the hypernetwork model comprises the following steps:

[0105] Step S1110, obtaining multiple image frames of a video in a single training sample, performing data enhancement processing on the multiple image frames to obtain sample data and inputting the sample data into the hypernetwork model;

[0106] In one embodiment, a sufficient number of video streams can be manually collected and organized as training sets based on hard indicators of video quality such as bit rate, frame rate, and resolution, wherein each video stream serves as a single training sample, and each single training sample has a corresponding subjective quality score annotated therefor, which is used as a supervision score. The subjective quality score can be obtained by having multiple people perform subjective scoring on the video quality of each image frame in the video stream corresponding to the training sample, and calculating the average of these subjective scores.

[0107] A video stream of a single training sample is obtained from the training set, and the corresponding multiple image frames are obtained by decoding the video stream. The multiple image frames are subjected to data enhancement processing to obtain sample data. The data enhancement processing can be mirroring, rotation, scaling, adjusting brightness, contrast, Gaussian noise, Mosaic, Mixup, Cutout, CutMix, etc. Technical personnel in this field can flexibly select and implement it according to actual business needs, and then input the sample data into the hypernetwork model.

[0108] Step S1120, after the hypernetwork model extracts image feature information from the sample data, the prediction module outputs a prediction of the quality score corresponding to the sample data;

[0109] The backbone module in the hypernetwork model extracts the visual features of the sample data and outputs them as the image feature information. The global average pooling layer of the model then performs spatial averaging on the image feature information and inputs it into the prediction module of the model. The fully connected layer therein expands it into a one-dimensional vector according to the corresponding channel direction and maps it to the corresponding classification space to obtain the predicted quality score corresponding to the sample data.

[0110] Step S1130, calculating the loss value of the hypernetwork model, performing weight update on the model when the loss value of the model does not reach a preset threshold, and continuing to call other training samples to perform iterative training until the model converges.

[0111] The loss value of the hypernetwork model is calculated using a preset loss value calculation function. The loss value calculation function includes the calculation of the regression loss value, the disorder loss value, and the cross entropy loss value of the model. The specific implementation is further revealed by the subsequent embodiments, and this step is not discussed for the time being.

[0112] Determine whether the loss value of the hypernetwork model reaches a preset threshold. When the preset threshold is not reached, the model is weighted updated, the corresponding weights of the backbone modules of the hypernetwork model are updated, and other training samples in the training set are continued to be called to perform iterative training. When the preset threshold is reached, it is indicated that the hypernetwork model has been trained to convergence.

[0113] In this embodiment, the training process of the hypernetwork model is disclosed. It can be seen that after the hypernetwork model is trained to convergence, it can perform reference-free evaluation on the video stream and predict the corresponding quality score. Based on this, in actual application scenarios, it can replace human labor with artificially defined standards to intelligently and quickly evaluate the quality of the video stream, which is very efficient.

[0114] See also Figure 6 In a preferred embodiment, step S1130, the loss value calculation, comprises the following steps:

[0115] Step S1131, calculating the regression loss value of the predicted quality score obtained by predicting the sample data using the hypernetwork model according to the supervised score corresponding to the sample data, wherein the supervised score is a subjective quality score;

[0116] In one embodiment, the supervised score, ie, the subjective quality score, can be obtained by having multiple people perform subjective scoring on the video quality of each image frame in the video stream corresponding to the training sample and calculating the average of the subjective scores.

[0117] Calculate the supervision score of each image frame in the video stream corresponding to the sample data, and the predicted quality score of each image frame in the video stream corresponding to the sample data predicted by the hypernetwork model. The absolute value of the difference between the two is used as the regression loss value. The exemplary formula is as follows:

[0118]

[0119] Among them: Loss mos is the regression loss value, n is the number of image frames in the video stream, equal to batch_size, y out The predicted quality score corresponding to each image frame in the video stream corresponding to the sample data predicted by the hypernetwork model, y gt Provides supervision scores corresponding to each image frame in the video stream corresponding to the pre-labeled sample data.

[0120] Step S1132, calculating a corresponding cross entropy loss value according to the regression loss value;

[0121] Based on the regression loss value, 1-Loss mos As the positive sample output probability, the corresponding cross entropy loss value is calculated, and the exemplary formula is as follows:

[0122]

[0123] Among them: Loss ce is the cross entropy loss value, n is the number of image frames in the video stream, equal to batch_size, y outThe predicted quality score corresponding to each image frame in the video stream corresponding to the sample data predicted by the hypernetwork model, y gt Provides supervision scores corresponding to each image frame in the video stream corresponding to the pre-labeled sample data.

[0124] Step S1133: Perform the same shuffling process on the prediction quality score obtained by the hypernetwork model predicting the sample data and the supervision score corresponding to the sample data, respectively, and use the difference between the prediction quality score and the shuffled prediction quality score as the input of the preset function, calculate the sum of the absolute value of the difference between the supervision score and the shuffled supervision score after the shuffling process and the function output result, and take the maximum value between the sum and 0 as the shuffling loss value;

[0125] By improving the difference between the prediction quality scores obtained by the hypernetwork model predicting the sample data, the accuracy of the model prediction is improved. Accordingly, the prediction quality scores obtained by the hypernetwork model predicting each image frame in the video stream corresponding to the sample data, and the supervision scores corresponding to the image frames in the video stream corresponding to the sample data are subjected to the same shuffle processing, and the shuffled prediction quality scores and supervision scores are obtained accordingly, for example: the prediction quality score is [1,2,3,4,5], the supervision score is [2,3,3,4,3], the shuffled prediction quality score after shuffle processing is [5,4,1,3,2], and the shuffled supervision score after shuffle processing is [3,4,2,3,3]. Further, the difference between the prediction score and the shuffled prediction quality score is used as the input of a preset function, the preset function is sgn, and the sgn is a sign function that can indicate the positive or negative of the content input to it. Then, the absolute value of the difference between the supervision score and the supervision score after the shuffle processing and the sum of the function output result are calculated, and the maximum value between the sum and 0 is taken as the shuffle loss value. The exemplary formula is as follows:

[0126]

[0127] Among them: Loss rank is the out-of-order loss value, n is the number of image frames in the video stream, equal to batch_size, y gt is the supervision score corresponding to each image frame in the video stream corresponding to the pre-labeled sample data, r gt is the supervision score of the disordered data after disorder processing, y out The predicted quality score of each image frame in the video stream corresponding to the sample data predicted by the hypernetwork model, r out Score the prediction quality of the shuffled images after shuffling.

[0128] Those skilled in the art should know that in the formula of the example, |y gt -r gt| Provides the adaptive change in the prediction quality score difference after shuffling, sgn(y out -r out ) provides the direction of change in the difference in prediction quality scores after shuffling.

[0129] Step S1134: Calculate the sum of the regression loss value, the cross entropy loss value, and the out-of-order loss value as the loss value.

[0130] The regression loss value, the cross entropy loss value, and the disorder loss value are summed up and calculated as the loss value. The exemplary formula is:

[0131] Loss=Loss mos +Loss ce +Loss rank

[0132] Among them: Loss is the loss value corresponding to the sample data predicted by the hypernetwork model, Loss mos is the regression loss value, Loss ce is the cross entropy loss value, Loss rank is the out-of-order loss value.

[0133] In this embodiment, the calculation of the loss value corresponding to the sample data predicted by the hypernetwork model and the calculation of the three loss values ​​contained therein are disclosed. It can be understood that, on the one hand, the loss value calculated by this embodiment can accurately reflect the accuracy of the model prediction, and on the other hand, when training the hypernetwork model, applying the loss value calculated by this embodiment can accelerate the model convergence and improve the training efficiency.

[0134] See also Figure 7 In a further embodiment, step S1300, the process of the joint training, comprises the following steps:

[0135] Step S1310: Acquire multiple image frames of a video in a single training sample, perform data enhancement on the multiple image frames to obtain sample data, and synchronously input the sample data into the super network model and each sub-network model;

[0136] In one embodiment, a sufficient number of video streams can be manually collected and organized as training sets based on hard indicators of video quality such as bit rate, frame rate, and resolution, wherein each video stream serves as a single training sample, and each single training sample has a corresponding subjective quality score annotated therefor, which is used as a supervision score. The subjective quality score can be obtained by having multiple people perform subjective scoring on the video quality of each image frame in the video stream corresponding to the training sample, and calculating the average of these subjective scores.

[0137] A video stream of a single training sample is obtained from the training set, and the corresponding multiple image frames are obtained by decoding the video stream. The multiple image frames are subjected to data enhancement processing to obtain sample data. The data enhancement processing can be mirroring, rotation, scaling, adjusting brightness, contrast, Gaussian noise, Mosaic, Mixup, Cutout, CutMix, etc. Technical personnel in this field can flexibly select and implement it according to actual business needs, and then synchronously input the sample data into the supernetwork model and each sub-network model.

[0138] Step S1320: After the super network model and each sub network model extract image feature information from the sample data respectively, a prediction module predicts a quality score corresponding to the sample data;

[0139] The corresponding backbone modules in the hypernetwork model and each sub-network model extract visual features of the sample data and output them as the image feature information. The global average pooling layer corresponding to the two models then performs spatial averaging on the image feature information and inputs it into the prediction modules corresponding to the two models. The corresponding fully connected layers in the two models expand it into a one-dimensional vector according to the corresponding channel direction and map it to the corresponding classification space to obtain the predicted quality score corresponding to the sample data.

[0140] Step S1330: Calculate the corresponding joint loss value according to the corresponding loss values ​​of the super network model and each sub-network model, and when the joint loss value does not reach the preset threshold, update the weights of each sub-network model accordingly, and continue to call other training samples to implement iterative training.

[0141] The loss value of the super network model is calculated according to steps S1131-1134. Those skilled in the art should know that the corresponding loss values ​​of the sub network models can also be calculated according to the disclosure of steps S1131-1134. Further, according to the corresponding loss values ​​of the super network model and the sub network models, the corresponding joint loss value is calculated. The exemplary formula is as follows:

[0142]

[0143] Among them: Loss ofa is the loss value of the joint training of the super network model and the sub-network model, n is the number of sub-networks sampled in a single weight update, i is the sub-network sampled in the sampling space, is the loss value of the sub-network model, Loss supernet is the loss value of the hypernetwork model.

[0144] Determine whether the joint loss value of the super network model and each sub network model reaches a preset threshold value. When the preset threshold value is not reached, the super network model and each sub network model are weight updated. The weight of the convolution module structure shared by the sub network model and the super network model is shared. The corresponding weights of the backbone modules corresponding to the super network model and each sub network model are updated, and other training samples in the training set are continuously called to implement iterative training. When the preset threshold value is reached, it is characterized that the super network model and each sub network model have been trained to convergence.

[0145] In this embodiment, the joint training process of each sub-network model and the super-network model is disclosed, so that the weight of each sub-network model can be optimized during the training process. In addition, it can be understood that after each sub-network model is trained to convergence, the video stream can be evaluated without reference and the corresponding quality score can be predicted. Based on this, in actual application scenarios, it can replace human labor with artificially defined standards, and intelligently and quickly evaluate the quality of the video stream, which is very efficient.

[0146] See also Figure 8 In a further embodiment, step S1400, verifying the performance of each candidate model and selecting at least one candidate model as a video quality assessment model, includes the following steps:

[0147] Step S1410: Use the proxy verification set to perform performance measurement on each candidate model to obtain the comprehensive performance corresponding to each candidate model, where the comprehensive performance includes running time and / or accuracy index;

[0148] Part of the training samples during the training of the hypernetwork model may be used as the proxy verification set. Optionally, 10% of the training samples may be randomly sampled from the training set to obtain the part of the training samples.

[0149] The proxy validation set is input into each candidate model for performance measurement to obtain the running time and loss value required for prediction corresponding to each candidate model. Those skilled in the art should know that each candidate model can also calculate the corresponding loss value according to the disclosure of steps S1131-1134. Further, the result of 1 minus the loss value multiplied by 100% is calculated as the accuracy index, and then the accuracy index and / or running time are used as the comprehensive performance corresponding to each candidate model.

[0150] Step S1420: Select a candidate model corresponding to the comprehensive performance that meets preset conditions and output it as a video quality assessment model.

[0151] A corresponding comprehensive performance index can be preset according to the comprehensive performance, for example, an accuracy index of 85% and / or a running time of 0.3S, etc. Of course, those skilled in the art can flexibly set the comprehensive performance index according to actual business needs. Accordingly, at least one candidate model that meets the corresponding comprehensive performance index is selected and output as a video quality assessment model.

[0152] In this embodiment, the performance of the candidate model is measured by using a proxy validation set, so that the candidate model with the performance reaching the standard is selected as the video quality assessment model. It can be seen that the video quality assessment model is rationally selected so that when it is put into use in actual business scenarios, considerable results can be achieved.

[0153] See also Fig. 9 In a further embodiment, step S1410, using a proxy validation set to measure the performance of each candidate model to obtain the comprehensive performance corresponding to each candidate model, includes the following steps:

[0154] Step S1411, performing channel search from the sub-network structures corresponding to the candidate models, and determining a plurality of compression candidate models corresponding to the sub-network structures in the target compression channel;

[0155] A search space corresponding to the sub-network structure can be established based on the sampling space, and the specific implementation can be found in the corresponding part of step S1100. Accordingly, a channel search is performed from the sub-network structure corresponding to each candidate model according to the search space, and multiple compression candidate models corresponding to the sub-network structure in which each convolution module is in the corresponding target compression channel in the search space are determined. For example, for example, the number of channels corresponding to the convolution module of the final stage of the backbone module of the candidate model is 256, and the target compression channel corresponding to the convolution module of the final stage in the search space is [256,384]. At this time, it can be determined that the convolution module of the final stage is in the target compression channel, and then when the convolution modules of other stages of the backbone module of the candidate model are in the target compression channel, the candidate model can be determined to be a compression candidate model.

[0156] Step S1412: using the input corresponding to the training samples in the proxy validation set to the multiple compression candidate models, and calculating the corresponding running time and accuracy index;

[0157] Part of the training samples during the training of the hypernetwork model may be used as the proxy validation set. Optionally, 10% of the training samples may be randomly sampled from the training set to obtain the part of the training samples.

[0158] The proxy validation set is input into each compression candidate model for performance measurement to obtain the running time and loss value corresponding to the prediction of each compression candidate model. Those skilled in the art should know that each compression candidate model can also calculate the corresponding loss value according to the disclosure of steps S1131-1134. Further, the result of 1 minus the loss value multiplied by 100% is calculated as the accuracy index.

[0159] Step S1413: Determine the comprehensive performance according to the running time and accuracy indicators.

[0160] Furthermore, the accuracy index and the running time are used as the comprehensive performance corresponding to each compression candidate model.

[0161] In this embodiment, the compressed candidate model is searched out from the candidate models through the search space, and it can be seen that the compression of the candidate models is achieved.

[0162] See also Fig.10 In a further embodiment, after step S1413, the step of determining the comprehensive performance according to the running time and accuracy index, the following steps are also included:

[0163] Step S1414: reuse the weights of the super network model corresponding to the shared network structures of the respective compression candidate models and the super network model;

[0164] The structure shared by each convolution module in the backbone network corresponding to the compression candidate model and the hypernetwork model is reused, and the weight of the hypernetwork model corresponding to the shared structure is reused.

[0165] Step S1415: jointly train each compression candidate model with the hypernetwork model, select the trained compression candidate model with comprehensive performance that meets preset conditions, and output it as a video quality assessment model.

[0166] The compression candidate model is jointly trained with the hypernetwork model. For specific implementation, please refer to the disclosure of steps S1310-1330, and this step will not be described in detail. Furthermore, the corresponding comprehensive performance indicators can be preset according to the comprehensive performance corresponding to the compression candidate model, such as an accuracy index of 85% and a running time of 0.3S. Of course, those skilled in the art can flexibly set the comprehensive performance indicators according to actual business needs. Accordingly, at least one compression candidate model that meets the comprehensive performance indicators is selected as the video quality assessment model.

[0167] In this embodiment, by jointly training the compression candidate models, the accuracy and generalization ability of the compression candidate models are further improved, and considerable results can be achieved when they are put into use in actual business scenarios.

[0168] See also Fig.11 A video quality assessment model production device provided to meet one of the purposes of the present application is a functional embodiment of the video quality assessment model production method of the present application, and the device includes a sampling acquisition module 1100, a structure sampling module 1200, a joint training module 1300 and a performance verification module 1400, wherein: the sampling acquisition module 1100 is used to obtain the sampling space corresponding to the network structure of the super network model pre-trained to convergence and the search space of its corresponding sub-network structure; the structure sampling module 1200 is used to sample and determine multiple sub-network structures from the corresponding sampling space according to the network structure of the super network model, and obtain the sub-network model corresponding to each sub-network structure; the joint training module 1300 is used to jointly train each sub-network model corresponding to the multiple sub-network structures with the super network model, and obtain the sub-network model trained to convergence as a candidate model; the performance verification module 1400 is used to verify the performance of each candidate model and screen out at least one candidate model as a video quality assessment model.

[0169] In a further embodiment, the sampling acquisition module 1100 includes: a first sample input submodule, used to acquire multiple image frames of a video in a single training sample, and perform data enhancement processing on the multiple image frames to obtain sample data to input into the hypernetwork model; a first quality prediction submodule, used to extract image feature information from the sample data by the hypernetwork model, and output a prediction module to predict the quality score corresponding to the sample data; a first weight update submodule, used to calculate the loss value of the hypernetwork model, implement weight updates on the model when the loss value of the model does not reach a preset threshold, and continue to call other training samples to implement iterative training until the model converges.

[0170] In a preferred embodiment, the weight update submodule includes: a regression loss submodule, which is used to calculate the regression loss value of the predicted quality score obtained by the hypernetwork model predicting the sample data according to the supervised score corresponding to the sample data, and the supervised score is a subjective quality score; a cross entropy loss submodule, which is used to calculate the corresponding cross entropy loss value according to the regression loss value; a disorder loss submodule, which is used to perform the same disorder processing on the predicted quality score obtained by the hypernetwork model predicting the sample data and the supervised score corresponding to the sample data, and use the difference between the predicted quality score and the disordered predicted quality score after the disorder processing as the input of the preset function, calculate the absolute value of the difference between the supervised score and the disordered supervised score after the disorder processing and the sum of the function output result, and take the maximum value between the sum and 0 as the disorder loss value; a loss value submodule, which is used to calculate the sum of the regression loss value, the cross entropy loss value and the disorder loss value as the loss value.

[0171] In a further embodiment, the joint training module 1300 includes: a second sample input submodule, used to obtain multiple image frames of a video in a single training sample, perform data enhancement on the multiple image frames to obtain sample data, and synchronously input them into the super network model and each sub-network model; a second quality prediction submodule, used to extract image feature information from the sample data respectively by the super network model and the each sub-network model, and then predict the quality score corresponding to the sample data through a prediction module; a second weight update submodule, used to calculate the corresponding joint loss value according to the corresponding loss values ​​of the super network model and the each sub-network model, and implement weight update for each sub-network model accordingly when the joint loss value does not reach a preset threshold, and continue to call other training samples to implement iterative training.

[0172] In a further embodiment, the performance verification module 1400 includes: a performance calculation submodule, which is used to use a proxy verification set to perform performance measurement on each candidate model to obtain the comprehensive performance corresponding to each candidate model, and the comprehensive performance includes running time and / or accuracy indicators; a model output submodule, which is used to select the candidate model corresponding to the comprehensive performance that meets preset conditions and output it as a video quality assessment model.

[0173] In a further embodiment, the performance calculation submodule includes: a channel search unit, which is used to perform channel search from the sub-network structures corresponding to each candidate model, and determine multiple compression candidate models corresponding to the sub-network structure in the target compression channel; a model performance estimation unit, which is used to use the input corresponding to the training samples in the proxy verification set to the multiple compression candidate models, and calculate the corresponding running time and accuracy indicators; a performance determination unit, which is used to determine the comprehensive performance based on the running time and accuracy indicators.

[0174] In a further embodiment, the performance determination unit further includes: a weight reuse unit for reusing the weights of the super network model corresponding to the shared network structure for the respective compression candidate models and the super network model; and a model output unit for jointly training the respective compression candidate models with the super network model, selecting the trained compression candidate models with comprehensive performance that meet preset conditions, and outputting them as video quality assessment models.

[0175] In order to solve the above technical problems, the present application also provides a computer device. Fig.12As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a video quality assessment model production method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the video quality assessment model production method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art can understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0176] In this embodiment, the processor is used to execute Fig.11 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the video quality assessment model production device of the present application, and the server can call the program code and data of the server to execute the functions of all submodules.

[0177] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the video quality assessment model production method of any embodiment of the present application.

[0178] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0179] In summary, the present application produces a video quality assessment model through network structure search and compression of the hypernetwork model, thereby achieving lightweight model and being able to run efficiently in actual application scenarios.

[0180] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.

[0181] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for producing a video quality assessment model, characterized in that: The steps include: Obtaining a sampling space corresponding to a network structure of a super network model pre-trained to convergence and a search space of a corresponding sub-network structure; According to the network structure of the super network model, sampling from the corresponding sampling space determines a plurality of sub-network structures, and obtains a sub-network model corresponding to each sub-network structure; Jointly train the sub-network models corresponding to the multiple sub-network structures with the super-network model, and obtain the sub-network models trained to convergence as candidate models; Verify the performance of each candidate model and select at least one candidate model as a video quality assessment model; The training process of the hypernetwork model includes the following steps: obtaining multiple image frames of a video in a single training sample, performing data enhancement processing on the multiple image frames to obtain sample data and inputting the sample data into the hypernetwork model; extracting image feature information from the sample data by the hypernetwork model, and outputting a prediction module to predict the quality score corresponding to the sample data; calculating the loss value of the hypernetwork model, and updating the weight of the model when the loss value of the model does not reach a preset threshold, and continuing to call other training samples to perform iterative training until the model converges; The calculation of the loss value of the hypernetwork model includes the following steps: calculating the regression loss value of the predicted quality score obtained by predicting the sample data by the hypernetwork model according to the supervised score corresponding to the sample data, wherein the supervised score is a subjective quality score; calculating the corresponding cross entropy loss value according to the regression loss value; performing the same shuffling processing on the predicted quality score obtained by predicting the sample data by the hypernetwork model and the supervised score corresponding to the sample data, taking the difference between the predicted quality score and the shuffled predicted quality score after the shuffling processing as the input of a preset function, calculating the sum of the absolute value of the difference between the supervised score and the shuffled supervised score after the shuffling processing and the output result of the function, and taking the maximum value between the sum and 0 as the shuffling loss value; calculating the sum of the regression loss value, the cross entropy loss value and the shuffling loss value as the loss value.

2. The method for producing a video quality assessment model according to claim 1, characterized in that: The joint training process includes the following steps: Acquire multiple image frames of a video in a single training sample, perform data enhancement on the multiple image frames to obtain sample data, and synchronously input the sample data into the super network model and each sub-network model; After the super network model and each sub network model extract image feature information from the sample data respectively, a prediction module predicts a quality score corresponding to the sample data; According to the corresponding loss values ​​of the super network model and each sub-network model, the corresponding joint loss value is calculated. When the joint loss value does not reach the preset threshold, the weights of each sub-network model are updated accordingly, and other training samples are continued to be called to implement iterative training.

3. The method for producing a video quality assessment model according to claim 1, characterized in that: The step of verifying the performance of each candidate model and selecting at least one candidate model as a video quality assessment model includes the following steps: Using a proxy validation set to perform performance measurement on each candidate model to obtain comprehensive performance corresponding to each candidate model, wherein the comprehensive performance includes running time and / or accuracy index; The candidate model corresponding to the comprehensive performance that meets the preset conditions is selected and output as the video quality assessment model.

4. The method for producing a video quality assessment model according to claim 3, characterized in that: The step of using a proxy validation set to perform performance measurement on each candidate model to obtain the comprehensive performance corresponding to each candidate model includes the following steps: Perform channel search from the sub-network structures corresponding to the candidate models, and determine multiple compression candidate models corresponding to the sub-network structures in the target compression channel; Using the input corresponding to the training samples in the proxy validation set to the multiple compression candidate models, and calculating the corresponding running time and accuracy indicators; Determine the overall performance based on the run time and accuracy metrics.

5. The method for producing a video quality assessment model according to claim 4, characterized in that: After determining the comprehensive performance based on the running time and accuracy indicators, the following steps are also included: For the network structure shared by each compression candidate model and the super network model, reuse the weight of the super network model corresponding to the shared network structure; Each compression candidate model is jointly trained with the hypernetwork model, and the trained compression candidate model with comprehensive performance that meets preset conditions is selected and output as a video quality assessment model.

6. A video quality assessment model production device, characterized in that: include: A sampling acquisition module, used to obtain a sampling space corresponding to the network structure of a super network model pre-trained to convergence and a search space of its corresponding sub-network structure; A structure sampling module, used to sample and determine multiple sub-network structures from the corresponding sampling space according to the network structure of the super-network model, and obtain a sub-network model corresponding to each sub-network structure; A joint training module is used to jointly train each sub-network model corresponding to the multiple sub-network structures with the super-network model to obtain the sub-network model trained to convergence as a candidate model; A performance verification module, used to verify the performance of each candidate model and select at least one candidate model as a video quality assessment model; The sampling acquisition module includes: a first sample input submodule, which is used to obtain multiple image frames of a video in a single training sample, perform data enhancement processing on the multiple image frames to obtain sample data and input it into the hypernetwork model; a first quality prediction submodule, which is used to extract image feature information from the sample data by the hypernetwork model, and output a prediction module to predict the quality score corresponding to the sample data; a first weight update submodule, which is used to calculate the loss value of the hypernetwork model, implement weight update on the model when the loss value of the model does not reach a preset threshold, and continue to call other training samples to implement iterative training until the model converges; The weight updating submodule includes: a regression loss submodule, which is used to calculate the regression loss value of the predicted quality score obtained by the hypernetwork model predicting the sample data according to the supervised score corresponding to the sample data, wherein the supervised score is a subjective quality score; a cross entropy loss submodule, which is used to calculate the corresponding cross entropy loss value according to the regression loss value; a disorder loss submodule, which is used to perform the same disorder processing on the predicted quality score obtained by the hypernetwork model predicting the sample data and the supervised score corresponding to the sample data, respectively, and use the difference between the predicted quality score and the disordered predicted quality score after the disorder processing as the input of the preset function, calculate the absolute value of the difference between the supervised score and the disordered supervised score after the disorder processing and the sum of the function output result, and take the maximum value between the sum and 0 as the disorder loss value; a loss value submodule, which is used to calculate the sum of the regression loss value, the cross entropy loss value and the disorder loss value as the loss value.

7. A computer device comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 5 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Citation Information

Patent Citations

  • Neural network structure search method and device, computer equipment and storage medium

    CN112949842A

  • Neural network architecture channel number searching method and device, equipment and storage medium

    CN114118360A

  • Neural network model deployment method and device, equipment and storage medium

    CN114330692A

  • Method for acquiring neural network model, and image processing method and apparatus

    WO2021218517A1