Image group extraction-based video quality evaluation acceleration method, acceleration system and application

Abstract

The invention discloses a video quality assessment acceleration method based on image group extraction, and the method comprises the following steps: 1, uploading an original video and a to-be-assessed video recoded based on the original video to an object storage server through a client by a user, and initiating a video quality assessment request; step 2, preprocessing and cutting the two types of videos uploaded in the step 1 to obtain a plurality of video slices; 3, performing quality evaluation on the plurality of video slices obtained by cutting, and storing evaluation results in an object storage server; and step 4, performing exception processing and merging on the quality evaluation results of the plurality of video slices to obtain the quality score of the video to be evaluated. The invention also discloses an acceleration system for realizing the acceleration method, and application of the acceleration method or the acceleration system in accelerating video quality evaluation.

Description

Technical Field

[0001] The present invention belongs to the technical field of video transcoding and reference video quality assessment, and relates to an acceleration method, an acceleration system and an application of video quality assessment based on image group extraction. Background Art

[0002] Over the past few decades, the way and scenarios of watching videos have changed dramatically, from traditional TV systems to video streaming on desktop computers, laptops and smartphones through the Internet. According to a report by Tencent, it is estimated that by 2024, streaming traffic will increase to about 74% of the entire Internet traffic. Streaming media can be roughly divided into on-demand and live streaming. Whether it is on-demand or live streaming, it needs to be converted according to the characteristics of the client device, that is, the original video must be converted according to the video format, resolution, and network bandwidth supported by the client device. This process is called video transcoding.

[0003] During the video transcoding process, video quality indicators including video content clarity, color accuracy, and distortion will change to a certain extent. Therefore, video quality assessment is essential, and this process is crucial to ensuring users' high-quality video viewing experience and promoting streaming service providers to continuously improve video quality. First, it helps ensure that the video requested by users is maintained at a high quality, allowing users to watch videos smoothly based on their real-time network and device conditions, reducing the occurrence of buffering, freezing, blurring, and other problems. This is crucial for streaming service providers because users always expect that they can get the best viewing experience. Secondly, video quality assessment can help identify technical problems or errors that have a bad impact on the entire video, such as encoding errors, bitrate or resolution abnormalities, etc. Finally, video quality assessment can help content creators and service providers determine the direction of service improvement and obtain feedback to improve users' overall viewing experience.

[0004] Video quality estimation is divided into three categories, namely full-reference video quality estimation, semi-reference video quality estimation and no-reference video quality estimation. Their meanings, advantages and disadvantages, and examples are shown in Table 1.

[0005] Table 1 Comparison of video quality estimation methods

[0006]

[0007] Both full-reference video quality estimation and semi-reference video quality estimation are based on reference video. The quality of the video under test is estimated by comparing the similarity between the reference video and the video under test. Many semi-reference video quality estimation methods are derived from the full-reference video quality estimation method. The main difference is that semi-reference video quality estimation only compares local information of the video and is often used in video transmission scenarios. No-reference video quality estimation evaluates quality by analyzing the video's own features. [1] . The advantage of having a reference video quality estimation is that it can relatively accurately evaluate the quality of the compressed video. SSIM (Structural Similarity) and PSNR (Peak signal-to-noise ratio) are two common methods for calculating video quality evaluation indicators. Their advantages are easy to understand, simple calculation, and fast operation speed; their disadvantages are that they only focus on the pixel value of the image without considering factors such as perceived color difference, clarity, sharpness, dynamic range, etc., and cannot objectively reflect the human eye's perception of video quality. Relatively speaking, VMAF (Video Multimethod Assessment Fusion), an emerging video quality evaluation method trained with a neural network model, uses multiple algorithms for quality evaluation and combines the evaluation results of different algorithms. Compared with traditional algorithms such as SSIM and PSNR, it can more accurately reflect the audience's actual perception of video quality. However, due to the higher computing power requirements of VMAF, its operation will cost higher computing power and time costs. Therefore, it is necessary to find a method that can ensure the accuracy of measuring video quality at a lower computing power and time cost.

[0008] The existing scene detection algorithms can be roughly divided into two categories: the first category is the traditional pixel-based scene recognition algorithm, and the second category is the scene detection algorithm based on deep learning. Pixel-based scene recognition algorithms include OpenCV's built-in scenedetect [4] Algorithm and ffmpeg built-in selectgt [5] The algorithm calculates the value of each frame (composed of the saturation, brightness, and hue values ​​of the pixels) and compares the value differences of adjacent frames in turn. Whenever the difference is higher than a certain threshold, the current frame can be considered a scene change frame (the picture of the current frame has changed enough compared to the picture of the previous frame). Scene detection algorithms based on deep learning include the FastR-CNN algorithm and the YOLO algorithm. These algorithms perform object detection on each frame of the video. When the appearance of some special objects or the occurrence of special behaviors are identified, the current frame is considered to be a scene change frame. Video quality assessment can effectively reduce the number of operations by intercepting each scene change frame to generate video slices.

[0009] In addition to the above, most video quality assessment services are currently deployed directly on virtual machines or containers. In this deployment mode, the service will have a high degree of flexibility, and users can customize various configurations for the service. However, in order to ensure the smooth operation of the service under different requirements, users also need to perform more manual intervention and resource planning. Considering the differences in video service quality (QoS) requirements in different application scenarios, in addition to directly deploying services on virtual machines or containers, other deployment methods such as serverless computing can be used to reduce service costs while ensuring QoS. In serverless computing, the management and operation and maintenance of servers are completed by cloud vendors, which allows users to focus more on their own business logic and improve R&D efficiency; the serverless architecture provides a relatively complete and comprehensive pay-as-you-go model, and users only need to pay according to the actual amount of resources they use. The disadvantage of serverless is that it does not perform well in scenarios with high real-time requirements. In short, in some scenarios where real-time requirements are not strong, serverless can effectively help users reduce operation and maintenance and labor costs while ensuring service quality. Summary of the invention

[0010] In order to solve the shortcomings of the prior art, the purpose of the present invention is to provide an acceleration method, acceleration system and application of video quality assessment based on image group extraction. The acceleration method in the present invention belongs to the field of full reference and semi-reference video quality estimation (reference video quality estimation), and relies on an effective scene detection method to select frames that can represent the image group, and use these frames as interception points to generate video slices.

[0011] The main technical problem solved by the present invention is how to effectively shorten the time cost of video quality evaluation while ensuring that the video quality evaluation results are highly consistent with the evaluation results of the human eye. A group of pictures is a series of continuous pictures in a video, and an image group can contain one or more video slices according to actual needs. Video slices are generated by extracting image groups, and video quality evaluation is performed on the video slices in the entire image group to evaluate the quality of the entire image group. Finally, the complete video quality to be measured is calculated using statistical methods based on the quality of each image group. A detailed description of GOP and video slices is shown below:

[0012] The picture group refers to a group of continuous pictures in a video, which consists of an I frame (a frame that can be decoded independently, called a key frame) and several B / P frames (frames that cannot be decoded independently, and the decoding of B frames and P frames depends on I frames). A group of GOPs is the basic unit accessed by video image encoders and decoders. Each video can be considered to be composed of multiple groups of GOPs.

[0013] The video slice refers to a group of continuous video frames in a GOP. The present invention does not distinguish whether the video slice has the original key frame of the picture group, because when generating the video slice, the video cutting tool will automatically create the key frame for the video slice.

[0014] In addition, the present invention also focuses on the differences in service level agreements (SLAs) for different video application scenarios. Different application scenarios have different service quality requirements for video services. For example, cloud games, live broadcasts, etc. require faster response times and concurrent processing capabilities for scenarios with higher latency, while storing and processing pre-recorded videos do not require high real-time performance. Taking into account the above scenarios in actual applications, the second core problem solved by the present invention is to select appropriate cloud deployment methods to provide services for different application scenarios, saving the economic cost of the service itself while ensuring QoS. In general, the method proposed in the present invention itself can reduce the amount of computation for video quality evaluation on the basis of effectively ensuring the results of video quality evaluation. For different application scenarios, the economic cost of the service itself can be reduced by selectively using containerized server mode and Serverless deployment services while ensuring QoS. For example, the containerized server mode can be used for usage scenarios and requirements that are more sensitive to latency, and the Serverless deployment service can be used for usage scenarios and requirements that have no requirements for latency or are not sensitive to latency.

[0015] The present invention proposes an acceleration method and system for video quality assessment based on image group extraction to meet the user's efficient and accurate video quality assessment needs in different video application scenarios. When facing delay-sensitive requirements, directly deploying services on containers can complete video quality assessment in a limited time to ensure high QoS of the service; when dealing with delay-independent requirements, the design of Serverless enables developers to focus on the development of specific business functions without spending energy on the operation and maintenance of cloud servers. In addition, the architecture that supports elastic scaling and automatic expansion and contraction can dynamically and flexibly provide customers with easy-to-use video quality assessment services on the basis of meeting application SLAs. When the user submits the access link of the video object (through which the source video and the video to be evaluated for quality can be accessed and re-encoded based on the source video) as a parameter to the client, the client sends the request to the server. The server allocates corresponding storage resources and computing resources according to the characteristics of the video, and calls the pre-deployed video cutting module, video quality assessment module and video quality analysis module for video processing. These modules align the video, detect the image group, use frame selection algorithms including YOLO, Scenedetect, etc. that can achieve the purpose of the present invention to select some frames (i.e., scene change frames) according to the range of the image group, and then cut the image group into video slices with the scene change frames as the cutoff points; apply multiple video quality estimation algorithms including SSIM, VMAF, etc. to the video slices, perform weighted summation based on the video quality evaluation results of the video slices to obtain the final video quality to be evaluated, and the quality weight of each video slice is the ratio of the number of frames in the image group to which the slice belongs to the total number of video frames.

[0016] The architecture diagram of the present invention is as follows Figure 2As shown in the figure. The client first uploads the source video and the video to be re-encoded based on the source video to be evaluated for video quality to the object storage server and obtains a response (access link of the video object), and then sends the access link together with the video quality evaluation request. When the system is deployed in the form of Serverless, the function controller (FaaSController) in the serverless system captures the request, encapsulates the request as an event, and distributes the event to the container where the corresponding cloud function is deployed. When the system is deployed directly in the container, the container directly processes the access request. The video cutting module of the container includes functions such as reading the source video and re-encoded video according to the object access link of the event, aligning the video timestamp, detecting the image group, selecting the frame selection algorithm according to the content of the image group to obtain the corresponding video slice and store it in the corresponding object storage server. The most important part of the video cutting module is to use the frame selection algorithm to select the appropriate frame for cutting. The frame selection algorithm can be implemented in many ways, generally including algorithms based on deep learning to identify key features of frames and traditional methods such as pixel-based scene recognition or simple equidistant frame extraction algorithms. The core idea of ​​the algorithm is to select a small number of frames that can represent the complete image group for quality assessment in order to obtain a quality assessment result that can represent the entire video. During the operation of the algorithm, the algorithm reads the video and finds each scene change frame. Each scene change frame can represent all frames from the current scene change frame to the next scene change frame. The result of quality assessment of all scene change frames in the image group is sufficient to represent the quality assessment result of the entire image group. The video cutting module uses the scene change frame as the interception point to generate video slices. The functions of the video quality assessment module include selecting a suitable video quality assessment algorithm to perform quality assessment on each video slice according to actual needs. After obtaining the video quality assessment result, the video quality assessment module passes the result to the video quality analysis module. The schematic diagram of the use of the video quality cutting module and the video quality assessment module is shown in the figure. Figure 3 and Figure 4 As shown in the figure, these two figures describe the schematic diagrams of applying equidistant frame extraction and applying deep learning algorithm frame extraction. The right side of the two figures describes the actual video quality assessment process and results, that is, quality assessment of all frames in the image group. Figure 2 The left side describes the selection of the 3rd and 8th frames for quality assessment based on the equidistant sampling algorithm. Figure 3The left side describes that the frame extraction algorithm based on the deep learning model algorithm extracts the 4th and 7th frames for quality assessment (the algorithm believes that these two frames are more representative of all frames of the two image groups). In the middle of each picture, the scoring results (x+y) / 2 obtained by quality assessment of some frames and the scoring results (X+Y) / 2 obtained by quality assessment of the complete frames are described. The role of the video quality analysis module is to wait for all video quality assessment modules to return results, and then perform weighted averaging of all quality assessment results to obtain the final quality result of the video and return it; the weight of the video slice is the ratio of the number of image group frames to the total number of video frames.

[0017] The process of the present invention is as follows Figure 5 As shown in the figure, the specific steps of system implementation are as follows:

[0018] Step 1: The user uploads the video to the object storage server and initiates a video quality assessment request

[0019] The client uploads the source video and the video that needs to be evaluated for quality and is re-encoded based on the source video to the object storage server, and obtains the corresponding return result (access link to the object storage). The client initiates a video quality assessment request and attaches the access link of the video in the object storage in the request. The deployment forms of the video quality assessment service include serverless deployment and / or direct container deployment and / or local direct deployment; when the video quality assessment service is deployed in Serverless form, the FaaS Controller captures the request and encapsulates it as an event, and initializes the corresponding container according to the request; when the video quality assessment service is deployed directly in the container, the container directly captures the request. The video quality assessment service of the present invention can also be directly deployed locally.

[0020] Step 2: Video cutting module pre-processes and cuts the video

[0021] The video cutting module obtains the source video and the video to be evaluated from the object storage server according to the link of the video in the object storage. The module aligns the two videos according to the timestamps and cuts out the common part. Then use the video analysis tool (such as ffmpeg [2] ,mediainfo [3] The display timestamp of the key frame of the source video and the serial number of the frame in the video are located by using open source tools such as .tsvc (such as .tsvc.com), and the corresponding image group is determined according to the timestamp and the serial number. The source video and the video to be evaluated for quality are cut into several video slices by selecting one or more frames from the image group using the video processing tool. The video cutting module records the number of video frames corresponding to each video segment and uploads all the above content to the object storage server.

[0022] Step 3: The video quality assessment module performs quality assessment on multiple video slices in parallel

[0023] The container allocates an appropriate number of video quality assessment modules according to the number of corresponding video clips. Each video quality assessment module obtains the corresponding video slice from the object storage server and uses the video quality assessment tool to assess the quality of the video slice. The specific quality assessment dimension depends on the actual application method. The applied quality assessment methods include PSNR, SSIM, VIF, VMA, etc. Different assessment methods measure different indicators. PSNR evaluates the difference in grayscale values ​​of each frame corresponding to the video. The higher the PSNR, the higher the video quality. SSIM evaluates the structural features in each frame of the video. The algorithm compares the brightness, contrast and structural similarity of the corresponding images. The higher the SSIM, the higher the video quality. VIF evaluates the ratio of reference image information (the common information of the source video image and the re-encoded video image in the HVS channel) and distorted image information (the common information of the original video image and the re-encoded video image in the distorted channel). The higher the VIF, the higher the video quality. VMAF is also a commonly used video quality evaluation method. VMAF includes three main indicators. In addition to the VIF mentioned above, there are DLM and TI. DLM evaluates detail loss, that is, the loss of useful visual information that affects the visibility of the content. TI measures the motion index in the time domain. In general, the higher the VMAF, the higher the video quality. In the actual video quality evaluation process, the appropriate video evaluation algorithm is selected according to the needs. The algorithm will evaluate the temporal information and spatial information of the video. After obtaining the quality evaluation result of each video slice, the video quality score corresponding to each video slice is stored in the object storage server.

[0024] Step 4: The video quality analysis module performs exception processing and merging of the quality results of the video slices.

[0025] The video quality analysis module obtains the video quality score and the number of video frames for each video slice from the object storage server. First, the abnormal parts that may appear in the video quality results are cleared. The abnormal video quality results come from the noise that may be generated during the image group cutting process. This is because the current video cutting tools cannot accurately cut video slices, so that the source video slices or re-encoded video slices retain more frames. The existing video quality assessment algorithm will perform a one-to-one quality assessment on all frames. Once there are extra video frames that cannot be one-to-one, the quality result will be 0, which will lead to abnormal video quality results. The process of exception handling refers to intercepting the quality results of the first n frames (n is generally 2) of each video slice and calculating the average. Then, the ratio of the number of image group frames to the total number of video frames is used as the quality weight of the video slice quality score, and the weighted score of the video slice quality score is calculated as the quality score of the quality video to be evaluated. After returning the quality score to the user, the system will release all allocated resources.

[0026] The present invention also proposes an acceleration system for realizing the above-mentioned accelerated video quality assessment method, and the acceleration system comprises:

[0027] Clients, containers, object storage servers;

[0028] The client is used to upload a video and initiate a video quality assessment request;

[0029] Processing modules including a video cutting module, a video quality assessment module and a video quality analysis module are deployed in the container to cut, assess and analyze the uploaded video;

[0030] The video cutting module is used to cut the uploaded video into video slices for subsequent module processing;

[0031] The video quality assessment module is used to perform quality assessment on the video content in the video slice through a video quality estimation algorithm to obtain a quality assessment score;

[0032] The video quality analysis module is used to remove abnormal data and perform weighted scoring to obtain the final quality score of the video to be evaluated;

[0033] The object storage server is used to store the original video and the video to be evaluated that is re-encoded based on the original video, as well as the corresponding video information and quality evaluation scores.

[0034] Furthermore, when the system is deployed in a Serverless form, the acceleration system also includes a function controller (FaaS Controller), which can capture video quality assessment requests, encapsulate the requests into events, and then distribute the events to containers.

[0035] The present invention also proposes the application of the above-mentioned video quality assessment acceleration method or acceleration system in accelerating video quality assessment.

[0036] The beneficial effects of the present invention include: The present invention proposes a video quality assessment method and system based on image group extraction. The existing technology cannot guarantee the accuracy of measuring video quality at a low computing power and time cost, and is not flexible enough in the deployment of quality assessment services. The present invention can achieve high-accuracy video quality assessment at a low time cost and economic cost through, for example, a frame selection method under a non-deep learning method and a serverless deployment or direct container deployment method. Compared with the existing technology, the evaluation time of the present invention can be reduced to less than 50% of the original, and the relative error of the evaluation is less than 1%. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 It is an illustration of the image group and video slice of the present invention.

[0039] Figure 2 It is a system architecture diagram of the present invention.

[0040] Figure 3 It is a schematic diagram of module functions under the equidistant frame extraction of the present invention.

[0041] Figure 4 It is a schematic diagram of module functions under the deep learning algorithm frame extraction of the present invention.

[0042] Figure 5 It is a lane diagram of the system of the present invention.

[0043] Figure 6 It is a reference diagram of embodiment 1 of the present invention.

[0044] Figure 7 It is a reference diagram of embodiment 2 of the present invention.

[0045] Figure 8 It is a reference diagram of embodiment 3 of the present invention.

[0046] Fig. 9 It is a reference diagram of embodiment 4 of the present invention.

[0047] Fig.10 It is a reference diagram of embodiment 5 of the present invention.

[0048] Fig.11 It is a reference diagram of Example 6 of the present invention. DETAILED DESCRIPTION

[0049] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0050] The present invention provides an acceleration method for video quality assessment based on image group extraction, the method comprising the following steps:

[0051] Step 1: The user uploads the original video and the video to be evaluated that is re-encoded based on the original video to the object storage server through the client, and initiates a video quality evaluation request;

[0052] Step 2: pre-process and cut the two types of videos uploaded in step 1 to obtain multiple video slices;

[0053] Step 3: Perform quality assessment on the multiple video slices obtained by cutting, and store the assessment results in the object storage server;

[0054] Step 4: Perform exception processing and merge the quality assessment results of multiple video slices to obtain the quality score of the quality video to be assessed.

[0055] The present invention also provides an acceleration system for implementing the acceleration method, and application of the acceleration method or the acceleration system in accelerating video quality assessment.

[0056] Specifically, the present invention designs an accelerated video quality assessment method and system based on image group extraction. Compared with the traditional video quality assessment method, the present invention can achieve a quality assessment result with a relative error of less than 1% at a time cost and economic cost of less than 50%. In the video encoding process, a new key frame will be encoded only when there is a large scene change or there are enough non-key frames from the previous key frame. Therefore, by identifying the key frames of the video, dividing the image groups of the video, and further identifying the scene change frames in the image groups, it can be ensured that whenever the scene represented by the frame changes significantly, they will be divided into different image groups, and then the video slices obtained by extracting the image groups are sufficiently representative in content. Since the frames of the video are continuous on the screen, it can be considered that the quality assessment scores of adjacent frames are also approximately continuous. By extracting the frames in the middle of each video slice, or using a machine learning algorithm to select a small number of suitable representative frames, the purpose of accelerating video quality assessment can be achieved by reducing the number of calculations.

[0057] The method proposed in the present invention belongs to a reference video quality assessment method. In general, compared with the previous reference video quality estimation algorithm, relative to PSNR and SSIM or MS-SSIM algorithm, the video quality assessment method trained by the accelerated neural network model extracted by the image group includes more factors related to vision, such as color difference, clarity, sharpness, dynamic range, etc. In addition, the algorithm also involves information in the time domain, and measures the quality of the video from the dimension of video time, which is more in line with human senses, that is, the video quality calculated by it can better represent the real feeling of the human eye. When the video quality assessment method obtained by neural network training is used in the system to evaluate the video, by intercepting continuous frames and saving them as video slices, it can be ensured that the time domain information will not be missing, thereby achieving a judgment result that is more consistent with the human eye. Compared with the video quality assessment algorithm trained by the neural network model without the accelerated image group extraction, the relative error of the video quality assessment result can be ensured to be less than 1% by extracting the image group acceleration.

[0058] Due to the complexity of neural networks, frame selection methods based on deep learning models often generate certain additional overhead (depending on the actual length of the video, the additional overhead ranges from a few seconds to tens of seconds), so non-deep learning frame selection methods can be used to achieve stable and negligible additional overhead (less than 1 second). In summary, this method achieves a significant reduction in running time while ensuring that the accuracy remains basically unchanged. In addition, the method of extracting image groups to accelerate greatly reduces the amount of calculation of the algorithm. Based on the average ratio of extracting 1 frame for every 10 frames, its calculation amount is approximately equivalent to 10% of the original. Compared with the ordinary video quality assessment method after turning on the acceleration switch, the actual running time after the acceleration of the extracted image group is reduced to less than half. Even on a machine with general configuration, video quality assessment can be completed in a relatively short time.

[0059] In different video application scenarios, deploying video quality assessment services in different ways can ensure cost savings while meeting the system service quality. In delay-independent scenarios, using serverless architecture deployment can bring more scalability to the system than volume-based deployment, thereby saving economic costs. In delay-sensitive scenarios, deploying services directly in containers can meet the high QoS requirements of the system. Because in addition to the large differences in video length and content, different video application scenarios have different service quality requirements. It is more cost-effective to choose a better deployment method to provide services based on the actual type of service, request volume, and specific video content. In specific application scenarios, the image group that has been identified by scene change frames may generate multiple video slices. The systems under both deployment methods can allocate corresponding threads for video quality assessment according to the actual number of video slices, and because there is no coupling between different image groups, the video quality assessment module can maximize parallelism, so as to achieve the purpose of completing video quality assessment for most videos in a short time.

[0060] The video quality assessment methods used in the following embodiments include traditional pixel-based and structural quality assessment methods and new deep learning-based video quality assessment methods. The former include common video quality assessment methods with low computational overhead such as PSNR and SSIM, while VMAF is the best among the latter. The video frame selection method in the embodiments includes a selection method based on video frame position and a selection method based on deep learning. The deployment form of the embodiments can be a container direct deployment method, a serverless method, and a local direct deployment method.

[0061] Since there are many combinations of deployment methods, quality assessment methods, and frame selection methods, 6 representative deployment schemes will be selected for detailed description below. Although other schemes are not introduced in detail, the deployment methods, video quality assessment methods, and frame selection methods therein are all permutations and combinations of the above. Embodiments 1 to 4 introduce the specific implementation of different video quality assessment methods and different frame selection methods in a serverless system scenario; Embodiment 5 introduces the specific implementation of the container directly deploying the service. Embodiment 6 introduces the specific implementation in a local scenario. In Embodiments 1 to 5, although only one client and one serverless system container are mentioned in the embodiment description and the corresponding schematic diagram, in actual deployment, the number of clients and serverless system containers can actually be any number.

[0062] Embodiment 1: The video quality assessment method uses a pixel-based assessment method, the frame selection method uses a non-deep learning selection method, and the serverless computing mode is deployed

[0063] like Figure 6As shown, the client client has a source video source.mp4 and a target.mp4 generated by re-encoding the source video and waiting to test the video quality. The specific implementation steps are as follows:

[0064] 1. The client uploads source.mp4 and target.mp4 to the object storage server, obtains the corresponding video access link, and sends the access link along with the video quality assessment request. FaaSController captures the request and encapsulates it as an event, and initializes the corresponding container based on the request.

[0065] 2. The video split module (splitmodule) reads the source video from the object storage server according to the access link and obtains several image groups segmented by key frames. In each image group, a non-deep learning video frame selection method is used to select several video frames that can represent the image group, and the corresponding frames are segmented into video slices.

[0066] 3. The video quality evaluation module (evaluatemodule) uses the traditional pixel-based video quality evaluation method to evaluate the video quality of the corresponding video slice to obtain the video quality score corresponding to the video slice.

[0067] 4. The video quality analysis module (analyzemodule) processes and analyzes the video quality score of each video slice, and finally obtains the video quality score of target.mp4 relative to source.mp4 and returns the result to the client.

[0068] 5. The program runs to the end. Through experiments, the actual running time of the acceleration system in this embodiment is less than half of that of the traditional method, and the relative error of the quality evaluation result is less than 1%. These optimizations come from the parallelism of the acceleration system itself.

[0069] Embodiment 2: The video quality assessment method uses a pixel-based assessment method, the frame selection method uses a deep learning-based selection method, and the serverless computing mode is deployed

[0070] like Figure 7 As shown, the client client has a source video source.mp4 and a target.mp4 generated by re-encoding the source video and waiting to test the video quality. The specific implementation steps are as follows:

[0071] 1. The client uploads source.mp4 and target.mp4 to the object storage server, obtains the corresponding video access link, and sends the access link along with the video quality assessment request. FaaSController captures the request and encapsulates it as an event, and initializes the corresponding container based on the request.

[0072] 2. The video split module (splitmodule) accesses the object storage server according to the access link to read the source video and obtains several image groups segmented by key frames. In each image group, a specific deep learning method is used to select several video frames that can represent the image group, and the corresponding frames are segmented into video slices.

[0073] 3. The video quality evaluation module (evaluatemodule) uses the traditional pixel-based video quality evaluation method to evaluate the video quality of the corresponding video slice to obtain the video quality score corresponding to the video slice.

[0074] 4. The video quality analysis module (analyzemodule) processes and analyzes the video quality score of each video slice, and finally obtains the video quality score of target.mp4 relative to source.mp4 and returns the result to the client.

[0075] 5. The program runs to the end. After experiments, the actual running time of the acceleration system in this embodiment is mostly less than 80% of the traditional method, and a small part of the actual running time exceeds the traditional method; the relative error of the quality evaluation result is less than 1%. This is because the selection method based on deep learning will generate a large additional overhead when processing long videos.

[0076] Embodiment 3: The video quality assessment method uses a deep learning-based assessment method, the frame selection method uses a non-deep learning selection method, and the serverless computing mode is deployed

[0077] like Figure 8 As shown, the client client has a source video source.mp4 and a target.mp4 generated by re-encoding the source video and waiting to test the video quality. The specific implementation steps are as follows:

[0078] 1. The client uploads source.mp4 and target.mp4 to the object storage server, obtains the corresponding video access link, and sends the access link along with the video quality assessment request. FaaSController captures the request and encapsulates it as an event, and initializes the corresponding container based on the request.

[0079] 2. The video split module (splitmodule) reads the source video from the object storage server according to the access link and obtains several image groups segmented by key frames. In each image group, a non-deep learning video frame selection method is used to select several video frames that can represent the image group, and the corresponding frames are segmented into video slices.

[0080] 3. The video quality evaluation module (evaluatemodule) uses a video quality evaluation method based on a deep learning model to evaluate the corresponding video slices to obtain the video quality score corresponding to the video slices.

[0081] 4. The video quality analysis module (analyzemodule) analyzes and aggregates the video quality scores of each video slice, and finally obtains the video quality score of target.mp4 relative to source.mp4 and returns the result to the client.

[0082] 5. The program runs to the end. Through experiments, the actual running time of the acceleration system in this embodiment is less than half of that of the traditional method, and the relative error of the quality evaluation result is less than 1%. These optimizations come from the parallelism of the acceleration system itself.

[0083] Embodiment 4: The video quality assessment method uses an assessment method based on deep learning, the frame selection method uses a selection method based on deep learning, and the serverless computing mode is deployed

[0084] like Fig. 9 As shown, the client client has a source video source.mp4 and a target.mp4 generated by re-encoding the source video and waiting to test the video quality. The specific implementation steps are as follows:

[0085] 1. The client uploads source.mp4 and target.mp4 to the object storage server, obtains the corresponding video access link, and sends the access link along with the video quality assessment request. FaaSController captures the request and encapsulates it as an event, and initializes the corresponding container based on the request.

[0086] 2. The video split module (splitmodule) accesses the object storage server according to the access link to read the source video and obtains several image groups segmented by key frames. In each image group, a specific deep learning method is used to select several video frames that can represent the image group, and the corresponding frames are segmented into video slices.

[0087] 3. The video quality evaluation module (evaluatemodule) uses a video quality evaluation model based on a deep learning model to evaluate the video quality of the corresponding video slice to obtain the video quality score corresponding to the video slice.

[0088] 4. The video quality analysis module (analyzemodule) processes and analyzes the video quality score of each video slice, and finally obtains the video quality score of target.mp4 relative to source.mp4 and returns the result to the client.

[0089] 5. The program runs to the end. After experiments, the actual running time of the acceleration system in this embodiment is mostly less than 80% of the traditional method, and a small part of the actual running time exceeds the traditional method; the relative error of the quality evaluation result is less than 1%. This is because the selection method based on deep learning will generate a large additional overhead when processing long videos.

[0090] Example 5: Direct container deployment

[0091] In order to meet the high QoS requirements of delay-sensitive video application scenarios, this example describes the implementation scenario of directly deploying services using containers. Fig.10 As shown, the client client has a source video source.mp4 and a target.mp4 generated by re-encoding the source video and waiting to test the video quality. The specific implementation steps are as follows:

[0092] 1. The client uploads source.mp4 and target.mp4 to the object storage server, obtains the corresponding video access link, and sends the access link along with the video quality assessment request. The container directly captures the request.

[0093] 2. The video split module (split module) accesses the object storage server according to the access link to read the source video and obtains several image groups divided by key frames. In each image group, a frame selection method is used to select the frames in the image group and split the frames into video slices.

[0094] 3. The video quality evaluation module (evaluatemodule) uses a video quality evaluation method to perform video quality evaluation on the corresponding video slice to obtain a video quality score corresponding to the video slice.

[0095] 4. The video quality analysis module (analyzemodule) processes and analyzes the video quality score of each video slice, and finally obtains the video quality score of target.mp4 relative to source.mp4 and returns the result to the client.

[0096] 5. The program runs to the end. When the container is directly deployed, the actual running time of the acceleration system is faster than that of the serverless system instance. This is because when the serverless system is deployed, the cold start of the container takes several hundred milliseconds. However, for longer videos, this time overhead is negligible. The relative error has nothing to do with the deployment method and has always remained below 1%.

[0097] Example 6: Specific implementation scenario deployed locally

[0098] Although the method described in this invention is very suitable for serverless scenarios, this method itself can also be deployed locally. Fig.11 As shown, the local computer has the relevant codes of the video cutting module, the quality assessment module, the quality processing module, the source video source.mp4, and the target.mp4 generated by re-encoding the source video to be tested for video quality. The specific implementation steps are as follows:

[0099] 1. Since source.mp4 and target.mp4 are stored locally and related code modules are also deployed locally, the video cutting module will directly read the local source.mp4 and target.mp4 to obtain the image group, and use the frame selection method to select the frames in the image group and cut the frames into video slices.

[0100] 2. The video quality evaluation module (evaluatemodule) uses a video quality evaluation method to perform video quality evaluation on the corresponding video slice to obtain a video quality score corresponding to the video slice.

[0101] 3. The video quality analysis module (analyzemodule) processes and analyzes the video quality scores of each video slice and finally obtains the video quality score of target.mp4 relative to source.mp4.

[0102] 4. The program runs to the end. In the local deployment environment, the actual running time of the accelerated system is also faster than that of the serverless system, and the relative error has been kept below 1%.

[0103] References

[0104] [1] Chu Jiang, Chen Qiang, Yang Xichen. A review of full reference image quality evaluation[J]. Journal of Computer Application Research, 2014, 31(01): 13-22.

[0105] [2]AboutFFmpeg.https: / / ffmpeg.org / about.html.

[0106] [3]MediaInfo. https: / / mediaarea.net / en / MediaInfo.

[0107] [4] SceneDect. https: / / www.scenedetect.com /

[0108] [5]FFmpeg-Filter. https: / / ffmpeg.org / ffmpeg-filters.html#select_002c- aselect

[0109] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. An accelerated method for video quality assessment based on image group extraction, characterized in that: The method comprises the following steps: Step 1: The user uploads the original video and the video to be evaluated that is re-encoded based on the original video to the object storage server through the client, and initiates a video quality evaluation request; Step 2: pre-process and cut the two types of videos uploaded in step 1 to obtain multiple video slices; Step 3: Perform quality assessment on the multiple video slices obtained by cutting, and store the assessment results in the object storage server; Step 4: Perform exception processing and merge the quality assessment results of multiple video slices to obtain the quality score of the quality video to be assessed.

2. The acceleration method according to claim 1, characterized in that: In step one, after the video is uploaded to the object storage server, an access link to the object storage can be obtained; the deployment forms of the video quality assessment service include serverless deployment and / or container direct deployment and / or local direct deployment; when the video quality assessment service is deployed in a serverless manner, the function controller captures the request and encapsulates it as an event, and initializes the corresponding container according to the request; when the video quality assessment service is deployed directly in the container, the container directly captures the request.

3. The acceleration method according to claim 1, characterized in that: In step 2, the preprocessing refers to aligning the original video and the video to be evaluated that is re-encoded based on the original video according to timestamps, cutting out the common parts, using a video analysis tool to locate the display timestamps of the key frames of the source video and the serial numbers of the frames in the video, and determining the corresponding image group based on the timestamps and serial numbers; the video analysis tool is an open source tool, including ffmpeg and mediainfo.

4. The acceleration method according to claim 1, characterized in that: In step 2, a frame selection algorithm is used to select scene change frames according to the range of the image group, and the original video and the video to be evaluated that is re-encoded based on the original video are respectively cut into one or more video slices with a length of 1 frame or more frames using the scene change frame as a cutoff point, and the data on the number of frames contained in the video slices and the image group are uploaded to the object storage server; the frame selection algorithms include YOLO, Scenedetect, pixel-based scene recognition frame extraction algorithm, and equidistant frame extraction algorithm.

5. The acceleration method according to claim 1, characterized in that: In step three, a video quality assessment algorithm is used to evaluate each video slice separately. The video quality assessment process will evaluate the temporal and spatial information of the video, obtain the quality assessment result of each video slice, and store the video quality score corresponding to each video slice in the object storage server.

6. The acceleration method according to claim 1, characterized in that: In step 4, the video quality assessment results stored in the object storage server are obtained, the abnormal parts in the video quality assessment results are cleared, and the weighted score of the video slice quality score is calculated as the quality score of the quality video to be evaluated based on the ratio of the number of image group frames to the total number of video frames; the abnormal part refers to the abnormal result that the video frames of the original video and the re-encoded video cannot correspond one to one due to the noise generated in the image group cutting process.

7. An acceleration system for implementing the acceleration method according to any one of claims 1 to 6, characterized in that: The acceleration system comprises: Clients, containers, object storage servers; The client is used to upload a video and initiate a video quality assessment request; Processing modules including a video cutting module, a video quality assessment module and a video quality analysis module are deployed in the container to cut, assess and analyze the uploaded video; The video cutting module is used to cut the uploaded video into video slices for subsequent module processing; The video quality assessment module is used to perform quality assessment on the video content in the video slice through a video quality estimation algorithm to obtain a quality assessment score; The video quality analysis module is used to remove abnormal data and perform weighted scoring to obtain the final quality score of the video to be evaluated; The object storage server is used to store the original video and the video to be evaluated that is re-encoded based on the original video, as well as the corresponding video information and quality evaluation scores.

8. The acceleration system according to claim 7, characterized in that: When the system is deployed in a Serverless form, the acceleration system further includes a function controller, which can capture video quality assessment requests, encapsulate the requests into events, and then distribute the events to containers.

9. Application of the acceleration method according to any one of claims 1 to 6, or the acceleration system according to claim 7 or 8 in accelerating video quality assessment.

Images