Method for creating video quality analysis engine, video quality analysis method and device

Abstract

The present disclosure provides a method for creating a video quality analysis engine, a video quality analysis method, and a device. The method for creating the video quality analysis engine includes: obtaining a multi-task model, wherein the multi-task model includes: multiple machine learning models corresponding to multiple video quality analysis tasks; creating multiple models corresponding to the multiple machine learning models in the same network, wherein the multiple models conform to the format of the inference engine to be created; calling a builder to create the inference engine based on the network including the multiple models, wherein the inference engine is a video quality analysis engine for running the multiple models, wherein when the video quality analysis engine is running, the input of the video quality analysis engine serves as the input of the multiple models, and the output of the multiple models serves as the output of the video quality analysis engine.

Description

Technical Field

[0001] The present disclosure generally relates to the field of electronic technology, and more particularly, to a method for creating a video quality analysis engine, a video quality analysis method, and a device. Background Art

[0002] With the booming development of short videos, video quality analysis and assessment have broad practical applications in many fields, such as image compression, video encoding and decoding, and video surveillance. Common video quality analysis and assessment features include noise, focus, exposure, jitter, clarity, and color. Compared to subjective manual assessment, automatic prediction using objective mathematical models not only reduces assessment costs but also remains unaffected by subjective factors such as viewing distance, display device, lighting conditions, and mood, offering high practical value.

[0003] Analyzing different video quality characteristics often requires the use of different models, a process known as multi-task video quality analysis. Compared to single-task analysis, multi-task video quality analysis can provide a more detailed and effective description of video quality. However, in actual deployment, multi-task models consume more GPU memory and require longer inference times than single-task models. Therefore, effectively reducing the GPU memory requirements for deploying multi-task models and improving inference speed are pressing challenges. Summary of the Invention

[0004] The exemplary embodiments of the present disclosure provide a method for creating a video quality analysis engine, a video quality analysis method, and a device to at least solve the problems in the above-mentioned related technologies, and may not solve any of the above-mentioned problems.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for creating a video quality analysis engine is provided, comprising: obtaining a multi-task model, wherein the multi-task model comprises: a plurality of machine learning models corresponding to a plurality of video quality analysis tasks; creating a plurality of models corresponding to the plurality of machine learning models in the same network, wherein the plurality of models conform to the format of an inference engine to be created; calling a builder to create the inference engine based on the network including the plurality of models, wherein the inference engine is a video quality analysis engine for running the plurality of models, wherein when the video quality analysis engine runs, the input of the video quality analysis engine serves as the input of the plurality of models, and the prediction results of the plurality of models serve as the output of the video quality analysis engine.

[0006] Optionally, the multiple video quality analysis tasks include at least one of the following items: a noise feature analysis task, a jitter feature analysis task, an exposure feature analysis task, and a blur feature analysis task.

[0007] Optionally, the multiple machine learning models are deep learning models, wherein the step of obtaining a multi-task model includes: separately training the machine learning model corresponding to each of the multiple video quality analysis tasks; and performing layer parameter fusion on each trained machine learning model to obtain each updated machine learning model.

[0008] Optionally, for each trained machine learning model, the step of performing layer parameter fusing on each machine learning model includes: for each trained machine learning model, replacing the normalization layer in each machine learning model with a 1×1 convolutional layer, and merging the parameters of the convolutional layer and the convolutional layer adjacent to the normalization layer.

[0009] Optionally, the creation method further includes: serializing the created video quality analysis engine; and saving the serialized video quality analysis engine.

[0010] Optionally, the step of creating multiple models corresponding to the multiple machine learning models in the same network includes: redefining the model structure of the multiple machine learning models using the model definition method of the inference engine, and defining the method of assigning the parameters of the multiple machine learning models to the corresponding multiple models; in the same network, creating the multiple models corresponding to the multiple machine learning models according to the redefined model structure of the multiple machine learning models and the defined assignment method.

[0011] Optionally, the step of creating multiple models corresponding to the multiple machine learning models in the same network also includes: generating an optimization profile for each of the multiple models created; calling the builder to create a profile definition of the network; and placing the optimization profile of each model into the profile definition.

[0012] Optionally, the step of calling a builder to create the inference engine based on the network including the multiple models includes: calling the builder to create the inference engine based on the network and the analysis definition.

[0013] Optionally, the optimization profile of each model includes at least one of the following items: a minimum size of a model input vector, an optimal size of a model input vector, and a maximum size of a model input vector.

[0014] According to a second aspect of an embodiment of the present disclosure, a video quality analysis method is provided, comprising: reading a single inference engine that has been created for running multiple models; inputting a vector into the single inference engine to obtain prediction results of the multiple models output by the single inference engine for the vector, wherein the single inference engine is created by creating the multiple models corresponding to a multi-task model in the same network and based on the network, wherein the multi-task model includes multiple machine learning models corresponding to multiple video quality analysis tasks.

[0015] Optionally, the multiple video quality analysis tasks include at least one of the following items: a noise feature analysis task, a jitter feature analysis task, an exposure feature analysis task, and a blur feature analysis task.

[0016] Optionally, the multi-task model is obtained in the following manner: separately training the machine learning model corresponding to each video quality analysis task in the multiple video quality analysis tasks; and performing layer parameter fusion on each trained machine learning model to obtain each updated machine learning model.

[0017] Optionally, multiple models corresponding to the multiple machine learning models are created in the same network in the following manner: the model structure of the multiple machine learning models is redefined using the model definition method of the single inference engine, and the method of assigning the parameters of the multiple machine learning models to the corresponding multiple models is defined; in the same network, the multiple models corresponding to the multiple machine learning models are created according to the redefined model structure of the multiple machine learning models and the defined assignment method; an optimization profile is generated for each of the multiple models created; the builder is called to create a profiling definition of the network; and the optimization profile of each model is placed in the profiling definition.

[0018] Optionally, the single inference engine is created based on the network in the following manner: calling the builder to create the single inference engine based on the network and the analysis definition.

[0019] According to a third aspect of an embodiment of the present disclosure, a device for creating a video quality analysis engine is provided, comprising: a multi-task model acquisition unit, configured to acquire a multi-task model, wherein the multi-task model comprises: a plurality of machine learning models corresponding to a plurality of video quality analysis tasks; a model creation unit, configured to create a plurality of models corresponding to the plurality of machine learning models in the same network, wherein the plurality of models conform to the format of an inference engine to be created; an engine creation unit, configured to call a builder to create the inference engine based on a network including the plurality of models, wherein the inference engine is a video quality analysis engine for running the plurality of models, wherein when the video quality analysis engine runs, the input of the video quality analysis engine serves as the input of the plurality of models, and the prediction results of the plurality of models serve as the output of the video quality analysis engine.

[0020] Optionally, the multiple video quality analysis tasks include at least one of the following items: a noise feature analysis task, a jitter feature analysis task, an exposure feature analysis task, and a blur feature analysis task.

[0021] Optionally, the multiple machine learning models are deep learning models, wherein the multi-task model acquisition unit is configured to respectively train the machine learning model corresponding to each of the multiple video quality analysis tasks; and perform layer parameter fusion on each trained machine learning model to obtain each updated machine learning model.

[0022] Optionally, the multi-task model acquisition unit is configured to replace the normalization layer in each trained machine learning model with a 1×1 convolutional layer, and merge the parameters of the convolutional layer with the convolutional layer adjacent to the normalization layer.

[0023] Optionally, the creation device further includes: an engine storage unit configured to serialize the created video quality analysis engine; and save the serialized video quality analysis engine.

[0024] Optionally, the model creation unit is configured to redefine the model structure of the multiple machine learning models using the model definition method of the inference engine, and to define a method of assigning the parameters of the multiple machine learning models to the corresponding multiple models; in the same network, the multiple models corresponding to the multiple machine learning models are created according to the redefined model structure of the multiple machine learning models and the defined assignment method.

[0025] Optionally, the model creation unit is configured to generate an optimization profile for each of the multiple models created; call the builder to create a profile definition of the network; and place the optimization profile of each model into the profile definition.

[0026] Optionally, the engine creation unit is configured to call the builder to create the inference engine based on the network and the analysis definition.

[0027] Optionally, the optimization profile of each model includes at least one of the following items: a minimum size of a model input vector, an optimal size of a model input vector, and a maximum size of a model input vector.

[0028] According to a fourth aspect of an embodiment of the present disclosure, a video quality analysis device is provided, comprising: an engine reading unit configured to read a single inference engine that has been created for running multiple models; a prediction unit configured to input a vector into the single inference engine to obtain prediction results of the multiple models output by the single inference engine for the vector, wherein the single inference engine is created by creating the multiple models corresponding to a multi-task model in the same network and based on the network, wherein the multi-task model includes multiple machine learning models corresponding to multiple video quality analysis tasks.

[0029] Optionally, the multiple video quality analysis tasks include at least one of the following items: a noise feature analysis task, a jitter feature analysis task, an exposure feature analysis task, and a blur feature analysis task.

[0030] Optionally, the multi-task model is obtained in the following manner: separately training the machine learning model corresponding to each video quality analysis task in the multiple video quality analysis tasks; and performing layer parameter fusion on each trained machine learning model to obtain each updated machine learning model.

[0031] Optionally, multiple models corresponding to the multiple machine learning models are created in the same network in the following manner: the model structure of the multiple machine learning models is redefined using the model definition method of the single inference engine, and the method of assigning the parameters of the multiple machine learning models to the corresponding multiple models is defined; in the same network, the multiple models corresponding to the multiple machine learning models are created according to the redefined model structure of the multiple machine learning models and the defined assignment method; an optimization profile is generated for each of the multiple models created; the builder is called to create a profiling definition of the network; and the optimization profile of each model is placed in the profiling definition.

[0032] Optionally, the single inference engine is created based on the network in the following manner: calling the builder to create the single inference engine based on the network and the analysis definition.

[0033] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the method for creating a video quality analysis engine as described above and / or the video quality analysis method as described above.

[0034] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, which, when instructions in the computer-readable storage medium are executed by at least one processor, prompts the at least one processor to execute the method for creating a video quality analysis engine as described above and / or the video quality analysis method as described above.

[0035] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by at least one processor, implement the method for creating a video quality analysis engine and / or the method for video quality analysis as described above.

[0036] According to the video quality analysis engine creation method, video quality analysis method, and device of the exemplary embodiments of the present disclosure, a multi-task single-engine deployment framework is proposed for video quality analysis.

[0037] In addition, the technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0038] Optimize multi-task models by integrating multiple engines into a single engine, effectively reducing the graphics memory occupied during deployment and accelerating the model's execution speed during actual deployment to improve inference speed.

[0039] A single-engine solution is introduced during the multi-task model deployment process, merging the multiple engines corresponding to the multi-task model into a single engine. This reduces GPU memory usage while maintaining the prediction accuracy of each task and improving / maintaining the model's running speed.

[0040] It fully utilizes the GPU's computing resources, improves GPU utilization efficiency, and reserves sufficient video memory resources for the subsequent introduction of more tasks.

[0041] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0043] Figure 1 A flowchart illustrating a method for creating a video quality analysis engine according to an exemplary embodiment of the present disclosure;

[0044] Figure 2 An example of a video quality analysis engine created according to an exemplary embodiment of the present disclosure is shown;

[0045] Figure 3 A flowchart showing a video quality analysis method according to an exemplary embodiment of the present disclosure is shown;

[0046] Figure 4 A structural block diagram illustrating a device for creating a video quality analysis engine according to an exemplary embodiment of the present disclosure is shown;

[0047] Figure 5 A structural block diagram of a video quality analysis device according to an exemplary embodiment of the present disclosure is shown;

[0048] Figure 6 A structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0049] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0050] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0051] It should be noted that the phrase "at least one of the items" in this disclosure includes three types of parallel situations: "any one of the items", "a combination of any multiple items of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. For another example, "performing at least one of step 1 and step 2" includes the following three parallel situations: (1) performing step 1; (2) performing step 2; and (3) performing steps 1 and 2.

[0052] Figure 1 A flowchart illustrating a method for creating a video quality analysis engine according to an exemplary embodiment of the present disclosure is shown.

[0053] Reference Figure 1 , in step S101, a multi-task model is obtained.

[0054] Here, the multi-task model includes: a plurality of machine learning models corresponding to the multiple tasks, for example, a plurality of machine learning models corresponding one-to-one to the multiple tasks.

[0055] As an example, the multiple tasks may be multiple video quality analysis tasks, and the multi-task model may include: multiple machine learning models corresponding to the multiple video quality analysis tasks. It should be understood that the multiple tasks may also be multiple tasks of other types, and this disclosure is not limited to this.

[0056] As an example, the multiple video quality analysis tasks may include, but are not limited to, at least one of the following: a noise feature analysis task, a jitter feature analysis task, an exposure feature analysis task, and a blur feature analysis task. It should be understood that the multiple video quality analysis tasks may include various appropriate tasks for analyzing video quality, and this disclosure is not limited thereto. For example, the multiple video quality analysis tasks may include tasks for analyzing aspects of a video, such as noise, focus, exposure, jitter, clarity, and color.

[0057] As an example, when the multiple tasks are multiple video quality analysis tasks, step S101 may include: acquiring multiple video quality analysis models corresponding one-to-one to the multiple video quality analysis tasks.

[0058] Regarding the video quality analysis model corresponding to the video quality analysis task, it can be understood that the video image (or the vector of the video image) is input into the video quality analysis model, and the prediction result of the video quality analysis model is the analysis result of the corresponding video quality analysis task. As an example, the video quality analysis model corresponding to the noise feature analysis task (also referred to as the noise feature analysis model) is used to analyze the noise features of the video image, for example, the noise feature analysis model is used to predict the probability that the video image contains noise. As an example, the video quality analysis model corresponding to the jitter feature analysis task (also referred to as the jitter feature analysis model) is used to analyze the jitter features of the video image, for example, the jitter feature analysis model is used to predict the probability that the video image contains jitter. As an example, the video quality analysis model corresponding to the blur feature analysis task (also referred to as the blur feature analysis model) is used to analyze the blur features of the video image, for example, the blur feature analysis model is used to predict the probability that the video image contains blur.

[0059] As an example, multiple machine learning models corresponding to multiple tasks can be trained based on the training data sets respectively.

[0060] As an example, the multiple machine learning models corresponding to multiple tasks may be deep learning models (DL models). It should be understood that the multiple machine learning models corresponding to multiple tasks may also be other types of machine learning models, such as reinforcement learning models, etc., and this disclosure does not limit this.

[0061] As an example, multiple machine learning models corresponding to multiple tasks can be trained using an appropriate framework. For example, the multiple machine learning models corresponding to multiple tasks can be PyTorch models or TensorFlow models trained using the PyTorch or TensorFlow frameworks.

[0062] As an example, when the multiple machine learning models are deep learning models, step S101 may include: separately training the machine learning model corresponding to each task in the multiple tasks; performing layer parameter fusion on each trained machine learning model to obtain the updated machine learning model, thereby obtaining a multi-task model.

[0063] As an example, for each trained machine learning model, the convolution layer and the normalization layer of the machine learning model can be re-parameterized and fused. As an example, for each trained machine learning model, the normalization layer in each machine learning model can be equivalently replaced with a 1×1 convolution layer, and the convolution layer and the convolution layer that is adjacent to the normalization layer are merged with parameters. Specifically, layer fusion (Layer Merge) can be performed for each trained machine learning model, for example, the normalization layer (Batch Normalization Layer) is equivalent to a 1x1 convolution layer, and then the 1x1 convolution layer and the convolution layer (ConvolutionalLayer) that is adjacent to the normalization layer are merged with parameters, thereby reducing the number of layers.

[0064] In step S102, multiple models corresponding to the multiple machine learning models are created in the same network, wherein the multiple models conform to the format of the inference engine to be created.

[0065] The inference engine to be created may be an inference engine of an appropriate type, for example, a TensorRT inference engine. It should be understood that it may also be an inference engine of other types, and the present disclosure does not limit this. As an example, when the inference engine to be created is a TensorRT inference engine, multiple TensorRT models corresponding one to one to the multiple machine learning models may be created in the same TensorRT network. A TensorRT model is a model in the TensorRT format. For example, a machine learning model trained using a framework such as PyTorch or TensorFlow can be converted into a model in the TensorRT format for deployment, thereby improving the running speed when running these models using the TensorRT inference engine.

[0066] It should be understood that the multiple machine learning models are converted into multiple models that conform to the format of the inference engine to be created for deployment. Although the formats of these multiple models and these multiple machine learning models are different, their prediction functions and prediction capabilities for video quality are completely consistent.

[0067] As an example, step S102 may include: redefining the model structure of the multiple machine learning models using the model definition method of the inference engine, and defining the method of assigning the parameters of the multiple machine learning models to the corresponding multiple models; in the same network, creating the multiple models corresponding to the multiple machine learning models according to the redefined model structure of the multiple machine learning models and the defined assignment method.

[0068] As an example, the TensorRT model definition method can be used to redefine the model structure of the multiple machine learning models, and define a method for assigning the parameters of the multiple machine learning models to the corresponding multiple TensorRT models; and in the same TensorRT network, according to the redefined model structure of the multiple machine learning models and the defined assignment method, create multiple TensorRT models corresponding to the multiple machine learning models.

[0069] As an example, step S102 may further include: generating an optimization configuration file for each of the multiple models created; calling the builder to create a profile definition of the network; and placing the optimization configuration file of each model into the profile definition.

[0070] As an example, you can define an optimization configuration file for each TensorRT model created; call the TensorRT builder to create a profile definition of the TensorRT network; and put the optimization configuration file of each TensorRT model into the profile definition.

[0071] As an example, the optimization profile of each model may include, but is not limited to, at least one of the following items: the minimum size of the model input vector (i.e., the minimum size of the processed image), the optimal size of the model input vector (i.e., the optimization dimensions of the processed image), and the maximum size of the model input vector (i.e., the maximum size of the processed image).

[0072] As an example, a TensorRT initialization builder can be used to create a TensorRT network through the builder, and multiple TensorRT models corresponding to the multiple machine learning models can be created in the TensorRT network to place the multiple machine learning models into the TensorRT network; a profile definition can be created using the builder, and the optimization profile of each TensorRT model can be placed in the profile definition.

[0073] In step S103, a builder is called to create the inference engine based on the network including the multiple models, wherein the inference engine is a video quality analysis engine for running the multiple models.

[0074] Here, when the video quality analysis engine is running, the input of the video quality analysis engine serves as the input of the multiple models, and the prediction results of the multiple models serve as the output of the video quality analysis engine.

[0075] As an example, a TensorRT builder may be called to create an engine for running the multi-task model based on a TensorRT network including the multiple TensorRT models.

[0076] As an example, the created engine may be a runtime engine.

[0077] As an example, the builder may be called to create the inference engine based on the network and the parsing definition. For example, the TensorRT builder may be called to create an engine for running the multiple models based on the TensorRT network and the parsing definition.

[0078] As an example, the multi-task model running method according to an exemplary embodiment of the present disclosure may further include: serializing the created engine to convert it into a storable form; and saving the serialized engine so that it can be quickly deployed and rebuilt at runtime.

[0079] The present disclosure takes into account that video memory usage is often linearly related to the number of models. In actual deployment, multi-task models occupy more GPU video memory than single-task models and require longer inference time. Therefore, in order to effectively reduce the video memory requirements for the deployment of multi-task video quality analysis models and improve the inference speed, the present disclosure introduces a single-engine solution in the multi-task video analysis processing deployment process, and merges the multiple engines corresponding to the multi-task models into a single engine, thereby reducing the GPU video memory usage without changing the prediction accuracy of each task, while increasing / maintaining the running speed, improving the GPU utilization efficiency, and reserving sufficient video memory resources for the subsequent introduction of more video feature analysis. It solves the problem of high video memory resource usage and computing resource competition when multiple models are deployed on a single hardware.

[0080] Figure 2 An example of a video quality analysis engine created according to an exemplary embodiment of the present disclosure is shown. Figure 2As shown, the single inference engine created by the present disclosure can be used to run multiple computing models corresponding to multiple video quality analysis tasks. Compared with multi-tasking and multi-engines (i.e., each engine is used to run the computing model corresponding to a single task), on the one hand, it can reduce the competition for multi-model resources to improve operating efficiency. The multi-tasking single engine enables multiple models to operate in parallel through a single instruction, greatly reducing a series of problems such as uneven memory allocation, uneven computing resource allocation, and unreasonable memory layout during the call of multiple models, thereby maximizing the potential of high-performance computing hardware. On the other hand, it can improve the memory reuse rate of multiple model detection graphics cards, so that the same memory can be time-division multiplexed between different models, greatly reducing the overall memory usage and improving memory utilization. In addition, it can improve memory bandwidth utilization. For high-concurrency computing hardware, the bandwidth of global memory is one of the main reasons that limit its performance. In the process of multi-engine calling, the memory application order is not fixed, which will cause the overall arrangement to be chaotic, making the call become a disorderly call, reducing the available bandwidth. The multi-model single engine allocates the memory required for each model in a nearby memory location through a one-time memory request, making the memory call method mainly linear, greatly improving the utilization of memory bandwidth and alleviating the limitations of the memory wall to a certain extent.

[0081] like Figure 2 As shown, the multi-task single-engine deployment approach proposed in this disclosure can effectively reduce GPU memory usage compared to the multi-task multi-engine approach, while simultaneously improving / maintaining runtime speed and fully utilizing GPU computing resources. Table 1 shows the performance difference between the multi-task single-engine deployment approach and the multi-task multi-engine approach for video quality analysis.

[0082] Table 1 Performance differences between multi-task single engine and multi-task multi-engine

[0083] Multi-tasking and multi-engine Multi-tasking single engine Video memory usage (Memory) 9650MB 1750MB Inference Time 15.24ms 15.2ms

[0084] Figure 3 A flowchart of a video quality analysis method according to an exemplary embodiment of the present disclosure is shown.

[0085] like Figure 3 As shown, in step S201, a single inference engine that has been created for running multiple models is read.

[0086] As an example, a saved serialized inference engine can be read through deserialization for use.

[0087] As an example, the single inference engine is obtained by creating the multiple models corresponding to the multi-task model in the same network and based on the network, wherein the multi-task model includes multiple machine learning models corresponding to multiple video quality analysis tasks. The multiple models conform to the format of the single inference engine.

[0088] As an example, the plurality of video quality analysis tasks may include at least one of the following items: a noise feature analysis task, a jitter feature analysis task, an exposure feature analysis task, and a blur feature analysis task.

[0089] In step S202, the vector is input into the single inference engine to obtain prediction results of the multiple models output by the single inference engine for the vector.

[0090] As an example, vectors of images extracted from a video may be input into the single inference engine to obtain analysis results of multiple video quality analysis tasks.

[0091] Specifically, the extracted image vectors are input into the single inference engine, which then inputs the vectors into the various video quality analysis models (i.e., the multiple models) deployed in the single inference engine, and outputs the predicted results of each video quality analysis model (i.e., the analysis results of the multiple video quality analysis tasks). As an example, the prediction results output by each video quality analysis model can be the probability of the presence of each feature or the score of each quality feature. For example, the probability that the extracted video image contains noise, blur, or jitter, etc.

[0092] As an example, the multi-task model can be obtained in the following manner: separately training the machine learning model corresponding to each of the multiple video quality analysis tasks; and performing layer parameter fusion on each trained machine learning model to obtain each updated machine learning model.

[0093] As an example, multiple models corresponding to the multiple machine learning models can be created in the same network in the following manner: redefine the model structure of the multiple machine learning models using the model definition method of the single inference engine, and define the method of assigning the parameters of the multiple machine learning models to the corresponding multiple models; in the same network, create the multiple models corresponding to the multiple machine learning models according to the redefined model structure of the multiple machine learning models and the defined assignment method; generate an optimization profile for each of the multiple models created; call the builder to create a profiling definition of the network; and place the optimization profile of each model into the profiling definition.

[0094] As an example, the single inference engine may be created based on the network in the following manner: calling the builder to create the single inference engine based on the network and the analysis definition.

[0095] As an example, the single inference engine may be created by the method for creating a video quality analysis engine as described in the above exemplary embodiment, and the specific details are omitted here.

[0096] Figure 4 A structural block diagram of a device for creating a video quality analysis engine according to an exemplary embodiment of the present disclosure is shown.

[0097] like Figure 4 As shown, the video quality analysis engine creation device 10 according to an exemplary embodiment of the present disclosure includes: a multi-task model acquisition unit 101 , a model creation unit 102 , and an engine creation unit 103 .

[0098] Specifically, the multi-task model acquisition unit 101 is configured to acquire a multi-task model, wherein the multi-task model includes: multiple machine learning models corresponding to multiple video quality analysis tasks.

[0099] The model creation unit 102 is configured to create multiple models corresponding to the multiple machine learning models in the same network, wherein the multiple models conform to the format of the inference engine to be created.

[0100] The engine creation unit 103 is configured to call a builder to create the inference engine based on the network including the multiple models, wherein the inference engine is a video quality analysis engine for running the multiple models.

[0101] When the video quality analysis engine is running, the input of the video quality analysis engine serves as the input of the multiple models, and the prediction results of the multiple models serve as the output of the video quality analysis engine.

[0102] As an example, the plurality of video quality analysis tasks may include at least one of the following items: a noise feature analysis task, a jitter feature analysis task, an exposure feature analysis task, and a blur feature analysis task.

[0103] As an example, the multiple machine learning models are deep learning models, wherein the multi-task model acquisition unit 101 can be configured to respectively train the machine learning model corresponding to each of the multiple video quality analysis tasks; and perform layer parameter fusion on each trained machine learning model to obtain each updated machine learning model.

[0104] As an example, the multi-task model acquisition unit 101 can be configured to replace the normalization layer in each trained machine learning model with a 1×1 convolutional layer, and merge the parameters of the convolutional layer with the convolutional layer adjacent to the normalization layer.

[0105] As an example, the creation device 10 further includes: an engine storage unit (not shown), which is configured to serialize the created video quality analysis engine and save the serialized video quality analysis engine.

[0106] As an example, the model creation unit 102 can be configured to redefine the model structure of the multiple machine learning models using the model definition method of the inference engine, and define the method of assigning the parameters of the multiple machine learning models to the corresponding multiple models; in the same network, according to the redefined model structure of the multiple machine learning models and the defined assignment method, create the multiple models corresponding to the multiple machine learning models.

[0107] As an example, the model creation unit 102 may be configured to generate an optimization profile for each of the multiple models created; call the builder to create a profile definition of the network; and place the optimization profile of each model into the profile definition.

[0108] As an example, the engine creation unit 103 may be configured to call the builder to create the inference engine based on the network and the analysis definition.

[0109] As an example, the optimization profile of each model may include at least one of the following items: a minimum size of a model input vector, an optimal size of a model input vector, and a maximum size of a model input vector.

[0110] Figure 5 A structural block diagram of a video quality analysis device according to an exemplary embodiment of the present disclosure is shown.

[0111] like Figure 5 As shown, the video quality analysis device 20 according to an exemplary embodiment of the present disclosure includes: an engine reading unit 201 and a prediction unit 202 .

[0112] Specifically, the engine reading unit 201 is configured to read a single inference engine that has been created for running multiple models.

[0113] The prediction unit 202 is configured to input a vector into the single inference engine to obtain prediction results of the multiple models for the vector output by the single inference engine.

[0114] The single inference engine is obtained by creating the multiple models corresponding to the multi-task model in the same network and based on the network, wherein the multi-task model includes multiple machine learning models corresponding to multiple video quality analysis tasks.

[0115] As an example, the plurality of video quality analysis tasks may include at least one of the following items: a noise feature analysis task, a jitter feature analysis task, an exposure feature analysis task, and a blur feature analysis task.

[0116] As an example, the multi-task model can be obtained in the following manner: separately training the machine learning model corresponding to each of the multiple video quality analysis tasks; and performing layer parameter fusion on each trained machine learning model to obtain each updated machine learning model.

[0117] As an example, multiple models corresponding to the multiple machine learning models can be created in the same network in the following manner: redefine the model structure of the multiple machine learning models using the model definition method of the single inference engine, and define the method of assigning the parameters of the multiple machine learning models to the corresponding multiple models; in the same network, create the multiple models corresponding to the multiple machine learning models according to the redefined model structure of the multiple machine learning models and the defined assignment method; generate an optimization profile for each of the multiple models created; call the builder to create a profiling definition of the network; and place the optimization profile of each model into the profiling definition.

[0118] As an example, the single inference engine may be created based on the network in the following manner: calling the builder to create the single inference engine based on the network and the analysis definition.

[0119] Regarding the video quality analysis engine creation device 10 and the video quality analysis device 20 in the above embodiment, the specific manner in which each unit performs operations has been described in detail in the embodiment of the method and will not be elaborated here.

[0120] In addition, it should be understood that the various units in the video quality analysis engine creation device 10 and the video quality analysis device 20 according to the exemplary embodiment of the present disclosure can be implemented as hardware components and / or software components. Those skilled in the art can implement each unit using, for example, a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC), depending on the processing performed by each defined unit.

[0121] Figure 6 FIG. 1 shows a structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure. Figure 6 The electronic device 30 includes: at least one memory 301 and at least one processor 302, wherein the at least one memory 301 stores a set of computer-executable instructions. When the computer-executable instruction set is executed by the at least one processor 302, the method for creating a video quality analysis engine and / or the video quality analysis method as described in the above exemplary embodiments are executed.

[0122] As an example, the electronic device 30 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 30 is not necessarily a single electronic device, but may also be any collection of devices or circuits capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 30 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0123] In the electronic device 30, the processor 302 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor 302 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0124] The processor 302 can execute instructions or codes stored in the memory 301, wherein the memory 301 can also store data. Instructions and data can also be sent and received over the network via the network interface device, wherein the network interface device can use any known transmission protocol.

[0125] Memory 301 may be integrated with processor 302, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. Furthermore, memory 301 may comprise a separate device, such as an external disk drive, a storage array, or any other storage device usable by a database system. Memory 301 and processor 302 may be operatively coupled or may communicate with each other, for example, via an I / O port, a network connection, or the like, such that processor 302 can access files stored in memory.

[0126] In addition, the electronic device 30 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 30 may be connected to each other via a bus and / or a network.

[0127] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the method for creating a video quality analysis engine and / or the video quality analysis method as described in the above exemplary embodiments. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or ultra-fast digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device configured to store the computer program and any associated data, data files and data structures in a non-transitory manner and provide the computer program and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0128] According to exemplary embodiments of the present disclosure, a computer program product may also be provided. Instructions in the computer program product may be executed by at least one processor to implement the method for creating a video quality analysis engine and / or the method for analyzing video quality as described in the above exemplary embodiments.

[0129] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0130] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for creating a video quality analysis engine, characterized in that: include: Acquire a multi-task model, wherein the multi-task model includes: multiple machine learning models corresponding to multiple video quality analysis tasks; Creating a plurality of models corresponding to the plurality of machine learning models in the same network, wherein the plurality of models conform to a format of an inference engine to be created; Calling a builder to create the inference engine based on the network including the multiple models, wherein the inference engine is a video quality analysis engine for running the multiple models; Serialize the created inference engine to convert it into a storable form; Save the serialized inference engine; Wherein, when the video quality analysis engine is running, the input of the video quality analysis engine is used as the input of the multiple models, and the prediction results of the multiple models are used as the output of the video quality analysis engine; The step of creating multiple models corresponding to the multiple machine learning models in the same network includes: Redefine the model structures of the multiple machine learning models using the model definition method of the inference engine, and define the method for assigning the parameters of the multiple machine learning models to the corresponding multiple models; In the same network, creating the multiple models corresponding to the multiple machine learning models according to the redefined model structures of the multiple machine learning models and the defined assignment method; generating an optimization profile for each of the plurality of models created; Invoking the builder to create a profile definition of the network; Putting the optimization profile for each model into the profile definition; The step of calling a builder to create the inference engine based on the network including the multiple models includes: calling the builder to create the inference engine based on the network and the analysis definition.

2. The creation method according to claim 1, characterized in that: The multiple machine learning models are deep learning models, The steps of obtaining the multi-task model include: training a machine learning model corresponding to each of the plurality of video quality analysis tasks respectively; For each trained machine learning model, layer parameters of each machine learning model are fused to obtain each updated machine learning model.

3. The creation method according to claim 2, characterized in that: For each trained machine learning model, the step of fusing layer parameters of each machine learning model includes: For each trained machine learning model, the normalization layer in each machine learning model is equivalently replaced by a convolutional layer of size 11, and the parameters of the convolutional layer and the convolutional layer adjacent to the normalization layer are merged.

4. The creation method according to claim 1, characterized in that: The optimization profile for each model includes at least one of the following: Minimum size of model input vector, optimal size of model input vector, maximum size of model input vector.

5. The creation method according to claim 1, characterized in that: The plurality of video quality analysis tasks include at least one of the following: Noise feature analysis task, jitter feature analysis task, exposure feature analysis task, blur feature analysis task.

6. A video quality analysis method, characterized in that: include: Read a single inference engine that has been created to run multiple models; Inputting the vector into the single inference engine to obtain prediction results of the multiple models output by the single inference engine for the vector, wherein the single inference engine is obtained by creating the multiple models corresponding to the multi-task model in the same network and based on the network, wherein the multi-task model includes multiple machine learning models corresponding to the multiple video quality analysis tasks; Wherein, multiple models corresponding to the multiple machine learning models are created in the same network in the following manner: Redefine the model structures of the multiple machine learning models using the model definition method of the single inference engine, and define the method of assigning parameters of the multiple machine learning models to the corresponding multiple models; In the same network, creating the multiple models corresponding to the multiple machine learning models according to the redefined model structures of the multiple machine learning models and the defined assignment method; generating an optimization profile for each of the plurality of models created; Invoking a builder to create a profile definition of the network; Putting the optimization profile for each model into the profile definition; The single inference engine is created based on the network in the following manner: calling the builder to create the single inference engine based on the network and the analysis definition.

7. The video quality analysis method according to claim 6, characterized in that: The multi-task model is obtained in the following way: training a machine learning model corresponding to each of the plurality of video quality analysis tasks respectively; For each trained machine learning model, layer parameters of each machine learning model are fused to obtain each updated machine learning model.

8. The video quality analysis method according to claim 6, wherein: The plurality of video quality analysis tasks include at least one of the following: Noise feature analysis task, jitter feature analysis task, exposure feature analysis task, blur feature analysis task.

9. A device for creating a video quality analysis engine, characterized in that: include: A multi-task model acquisition unit is configured to acquire a multi-task model, wherein the multi-task model includes: a plurality of machine learning models corresponding to a plurality of video quality analysis tasks; a model creation unit configured to create a plurality of models corresponding to the plurality of machine learning models in the same network, wherein the plurality of models conform to a format of an inference engine to be created; An engine creation unit is configured to call a builder to create the inference engine based on a network including the multiple models, wherein the inference engine is a video quality analysis engine for running the multiple models; The engine storage unit is configured to serialize the created inference engine to convert it into a storable form and save the serialized inference engine; When the video quality analysis engine is running, the input of the video quality analysis engine serves as the input of the multiple models, and the prediction results of the multiple models serve as the output of the video quality analysis engine; The model creation unit is configured to redefine the model structures of the multiple machine learning models using the model definition method of the inference engine, and to define the method for assigning the parameters of the multiple machine learning models to the corresponding multiple models; in the same network, create the multiple models corresponding to the multiple machine learning models according to the redefined model structures of the multiple machine learning models and the defined method for assigning; generate an optimization profile for each of the multiple models created; call the builder to create a profile definition of the network; and place the optimization profile of each model into the profile definition; The engine creation unit is configured to call the builder to create the inference engine based on the network and the analysis definition.

10. The creation device according to claim 9, characterized in that The multiple machine learning models are deep learning models, Among them, the multi-task model acquisition unit is configured to respectively train the machine learning model corresponding to each video quality analysis task in the multiple video quality analysis tasks; and perform layer parameter fusion on each trained machine learning model to obtain each updated machine learning model.

11. The creation device according to claim 10, characterized in that The multi-task model acquisition unit is configured to replace the normalization layer in each trained machine learning model with a convolutional layer of size 11, and merge the parameters of the convolutional layer with the convolutional layer adjacent to the normalization layer.

12. The creation device according to claim 9, characterized in that The optimization profile for each model includes at least one of the following: Minimum size of model input vector, optimal size of model input vector, maximum size of model input vector.

13. The creation device according to claim 9, characterized in that The plurality of video quality analysis tasks include at least one of the following: Noise feature analysis task, jitter feature analysis task, exposure feature analysis task, blur feature analysis task.

14. A video quality analysis device, characterized in that: include: An engine reading unit, configured to read a single inference engine that has been created for running multiple models; A prediction unit is configured to input a vector into the single inference engine to obtain prediction results of the multiple models output by the single inference engine for the vector, wherein the single inference engine is obtained by creating the multiple models corresponding to the multi-task model in the same network and based on the network, wherein the multi-task model includes multiple machine learning models corresponding to the multiple video quality analysis tasks; Wherein, multiple models corresponding to the multiple machine learning models are created in the same network in the following manner: Redefine the model structures of the multiple machine learning models using the model definition method of the single inference engine, and define the method of assigning parameters of the multiple machine learning models to the corresponding multiple models; In the same network, creating the multiple models corresponding to the multiple machine learning models according to the redefined model structures of the multiple machine learning models and the defined assignment method; generating an optimization profile for each of the plurality of models created; Invoking a builder to create a profile definition of the network; Putting the optimization profile for each model into the profile definition; The single inference engine is created based on the network in the following manner: calling the builder to create the single inference engine based on the network and the analysis definition.

15. The video quality analysis device according to claim 14, characterized in that: The multi-task model is obtained in the following way: training a machine learning model corresponding to each of the plurality of video quality analysis tasks respectively; For each trained machine learning model, layer parameters of each machine learning model are fused to obtain each updated machine learning model.

16. The video quality analysis device according to claim 14, characterized in that: The plurality of video quality analysis tasks include at least one of the following: Noise feature analysis task, jitter feature analysis task, exposure feature analysis task, blur feature analysis task.

17. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer-executable instructions, When the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the method for creating a video quality analysis engine as described in any one of claims 1 to 5 and / or the video quality analysis method as described in any one of claims 6 to 8.

18. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the method for creating a video quality analysis engine according to any one of claims 1 to 5 and / or the video quality analysis method according to any one of claims 6 to 8.

19. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by at least one processor, the method for creating a video quality analysis engine according to any one of claims 1 to 5 and / or the method for video quality analysis according to any one of claims 6 to 8 are implemented.

Images