AI video stream quality detection method based on embedded device

Abstract

The invention discloses an AI video stream quality detection method based on embedded equipment, which is realized by means of embedded video detection equipment, and comprises a video de-encapsulation decoding unit, a data processing module, a post-processing module, a statistical storage module and a display module, a video stream is decapsulated by a video decapsulation decoding unit, the frame type of a video frame is obtained, only the video frame of which the frame type is I frame is decoded, and then the video frame is converted into a picture format which can be directly processed by a data processing module; the video frame processed by the video de-encapsulation decoding unit is sent to the data processing module for inspection, then analyzed and judged by the post-processing module, counted and stored in the statistical storage module, and finally displayed in the display module; according to the invention, a detection algorithm based on a YOLOv5s network model is used for video frame quality detection, manual detection can be replaced, data statistics is carried out by using scores and problem categories detected by the detection algorithm, and the overall quality of a video stream is judged.

Description

Technical Field

[0001] The present invention belongs to the technical field of video processing and relates to an AI video stream quality detection method based on embedded devices. Background Art

[0002] Traditional video streaming products require manual real-time testing of video streaming quality during the testing phase. Traditional operations have many problems. First, the labor cost is high, and testers are required to constantly check the video player screen effect. Second, during the inspection process, personnel are prone to fatigue, resulting in reduced inspection efficiency and missed inspection of some problematic screens. More importantly, personnel can only rely on subjective judgment of video streaming quality and cannot provide detailed, data-based results.

[0003] At present, some video detection solutions on the market based on edge computers, personal computers, and even server platforms are costly, difficult to deploy equipment, and consume high power when working. Summary of the invention

[0004] In order to reduce the manual workload of video stream products in the testing phase and lower the equipment requirements for video stream inspection relying on computers, the present invention studies an AI video stream quality detection method based on embedded devices.

[0005] The technical solution of the present invention is an AI video stream quality detection method based on an embedded device. The key is that the above detection method needs to be implemented with the help of an embedded video detection device. The above video detection device includes a video decapsulation decoding unit, a data processing module, a post-processing module, a statistical storage module and a display module;

[0006] The video stream to be detected is first decapsulated by the video decapsulation and decoding unit to obtain the frame type of the video frame, and only the video frame with the frame type of I frame is decoded, and then converted into a picture format that can be directly processed by the data processing module using the graphics hardware processor, and the video frames other than I frame are ignored;

[0007] The video frames processed by the video decapsulation and decoding unit are sent to the data processing module for inspection, and then analyzed and determined by the post-processing module. The quality is counted and stored in the statistical storage module, and finally the statistical results are displayed in the display module.

[0008] Specifically, obtaining the frame type of the video frame is achieved through the encoding structure, the decoding type of the video frame for the I frame is hardware decoding, and sending the video frame processed by the video decapsulation decoding unit to the data processing module is achieved through the DMA controller.

[0009] More specifically, the post-processing module analyzes and determines the quality of:

[0010] The post-processing module analyzes the similarity between the video frame features and the basic types of quality problems and obtains a similarity score: when the score is higher than the threshold, it means that the video stream has a corresponding quality problem; when the scores are lower than the preset thresholds of each type of quality problem, it means that the quality of the determined video stream is qualified;

[0011] The basic types of quality issues mentioned above include blur, mosaic, noise, screen distortion and green screen.

[0012] Furthermore, the video stream to be detected is directly pushed to the video detection device according to the video protocol or is pulled by the video push device and then pushed; the video stream to be detected can come from network transmission detection, video recording detection or storage medium transmission.

[0013] Furthermore, the above-mentioned data processing module includes a data processor storing a verification model, and the types of the above-mentioned data processor are divided into embedded neural network processors, graphics processors or floating-point processing units; the construction process of the above-mentioned verification model is divided into three steps: obtaining a training set, model training, model conversion and quantization.

[0014] Preferably, the above-mentioned acquisition of the training set is to prepare a video defect data set, and use labeimg to annotate the defects to obtain the training set; the above-mentioned model training is to perform model training on the training set based on the YOLOv5s network model to obtain the model weights for detecting video defects; the above-mentioned video defects are divided into blur, mosaic, noise, flower screen and green screen.

[0015] Furthermore, the above YOLOv5s network model is divided into an input part, a backbone part, a neck part and an output part; in the backbone part, the features of the input feature map are convolved by the C3 module to extract image features; and in the neck part, the feature map is fused.

[0016] Specifically, the above-mentioned model conversion and quantization is to convert the model into a model structure that can be supported by the data processor and perform uint8 quantization processing.

[0017] More specifically, the above statistical results are displayed in the form of charts on the WEB front-end interface.

[0018] The beneficial effects of the present invention are:

[0019] The present invention adopts a detection algorithm based on the YOLOv5s network model to perform video frame quality detection, which can replace traditional manual detection and use the scores and problem categories detected by the detection algorithm to perform data statistics to judge the overall quality of the video stream.

[0020] YOLOv5s is divided into input, Backbone, Neck, and output. In the Backbone part, YOLOv5 uses the C3 module to perform convolution operations on the features of the input feature map to extract image features at different scales, so as to better adapt to target objects of different sizes and proportions; in the Neck part, multi-scale feature maps can be fused, alleviating the phenomenon of loss of detail features due to convolution operations in the Backbone part. Such a feature transmission route is conducive to detecting quality problems on the video screen, because image quality problems are usually abnormal in a small area.

[0021] When the video stream is pushed to an embedded video detection device, the quality of each frame of the video is detected through the detection algorithm, and the classification of unqualified image quality such as blur, mosaic, noise, flower screen, green screen, etc. and the similarity score evaluation of the corresponding quality problems are realized. When scoring, the detection algorithm will give a similarity score according to the classification of the image into a certain type of quality problem. This score is compared with the set threshold to determine whether the image has a certain type of quality problem. If the overall score is lower than the preset threshold of each quality problem, it means that the image quality is qualified. Then the quality score of each frame of the image and the quality problem category information are counted and the data is visualized.

[0022] The detection algorithm deployed on the embedded device of the present invention fully considers the limited hardware resources of the embedded device itself. Therefore, the algorithm is quantized in the present invention to reduce the memory bandwidth and storage space occupied by it. At the same time, the algorithm model is pre-compiled to speed up the loading speed of the algorithm model and reduce the model size.

[0023] In terms of video stream processing, the present invention copies the video frames decoded by the hardware to the NPU through the hardware DMA controller, allowing the detection algorithm to perform inference detection to reduce CPU resources and speed up the image processing speed. The present invention converts the video frames into an image format that can be directly processed by the NPU through graphics hardware, reducing the data conversion operation of the NPU, and the video stream can be detected by detecting the video frame type and obtaining the I frame, while other types of frames can be ignored. This is because other frames are generally decoded with reference to the I frame, so if there is a quality problem with the I frame, other frames will also have it, which improves the video processing speed and can cope with high frame rate video streams. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the detection method of the present invention.

[0025] Figure 2 This is the quality inspection and analysis result of the video according to the embodiment of the present invention. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Example

[0028] S1. Build verification model:

[0029] S1-1. Get the training set:

[0030] Prepare data sets of five defect types: blur, mosaic, noise, flower screen, and green screen. Use labeimg to annotate the defects and send the prepared data sets to the model for model training.

[0031] S1-2. Model training:

[0032] The model training is performed based on the YOLOv5s network model. The YOLOv5s network model used is divided into an input part, a Backbone part, a Neck part, and an output part. In the Backbone part, the features of the input feature map are convolved through the C3 module to extract image features; in the Neck part, the feature map is fused to alleviate the phenomenon of loss of detail features due to the convolution operation in the Backbone part.

[0033] S1-3. Model transformation and quantification:

[0034] The pt format model obtained through the above training is converted into a model structure that can be supported by the data processor, and uint8 quantization is performed at the same time.

[0035] Among them, the data processor includes: an embedded neural network processor, referred to as NPU; a graphics processor, referred to as GPU; and a floating point processing unit, referred to as FPU.

[0036] This embodiment uses an NPU, and stores the above model in the NPU used.

[0037] S2, video stream pull:

[0038] Connect the embedded video push device and embedded video detection device in the same LAN;

[0039] The video pulling module in the video pushing device pulls the video stream from the network transmission detection, video recording detection or storage medium transmission, and pushes the video stream to the video detection device.

[0040] The video stream in this embodiment is derived from network transmission detection.

[0041] S3, video decapsulation and decoding:

[0042] The video decapsulation unit decapsulates the video stream received by the video detection device and obtains the frame type according to the coding structure.

[0043] If the frame type is determined to be an I frame, hardware decoding is performed, and then the graphics hardware processing module is used to convert the video frame into a picture format that can be directly processed by the NPU. The video frame is then sent to the NPU through the hardware DMA controller for detection algorithm detection;

[0044] If it is determined to be a non-I frame, it can be ignored.

[0045] S4. Data processing and post-processing:

[0046] Use NPU to detect the detection algorithm. The specific detection process is:

[0047] The detected video features are extracted in the data processing module, and the degree of proximity between the features and different types of quality problems is determined in the post-processing module, and a similarity score is obtained. If the score is higher than the set threshold, it means that the video stream has the corresponding type of quality problem;

[0048] If the overall scores are lower than the preset thresholds for each type of quality problem, it means that the quality of the video stream is qualified.

[0049] The quality problems in this embodiment are divided into five types: blur, mosaic, noise, flower screen, and green screen. The setting thresholds of different quality problems in this embodiment are shown in Table 1.

[0050] Table 1: Thresholds for different quality issues

[0051] Quality Type Setting Thresholds Vague 0.8 Mosaic 0.7 Noise 0.6 Flower screen 0.7 Green Screen 0.8

[0052] In addition, it is also possible to determine whether the video quality is good enough by detecting whether the video is a high-quality picture type and the similarity score.

[0053] S5, data statistics, storage and display:

[0054] The above analysis results, including quality scores and types of quality problems, are statistically analyzed and stored. The statistical data will also be displayed in the form of charts on the WEB front-end interface.

[0055] The detection time for a 2400Mb video is about 80 seconds. Because there are many motion scenes in the video and the push device uses a constant bit rate, there are many mosaics and the image quality is somewhat blurred. The quality inspection and analysis results of the video are shown in the attached Figure 2 .

Claims

1. An AI video stream quality detection method based on embedded devices, characterized in that: The detection method needs to be implemented with the help of an embedded video detection device, which includes a video decapsulation and decoding unit, a data processing module, a post-processing module, a statistical storage module and a display module; The video stream to be detected is first decapsulated by the video decapsulation and decoding unit to obtain the frame type of the video frame, and only the video frame with the frame type of I frame is decoded, and then converted into a picture format that can be directly processed by the data processing module using the graphics hardware processor, and the video frames other than I frame are ignored; The video frames processed by the video decapsulation and decoding unit are sent to the data processing module for inspection, and then analyzed and determined by the post-processing module. The quality is counted and stored in the statistical storage module, and finally the statistical results are displayed in the display module.

2. According to claim 1, the AI ​​video stream quality detection method based on embedded devices is characterized in that: Acquiring the frame type of the video frame is achieved through the encoding structure, the decoding type of the video frame for the I frame is hardware decoding, and sending the video frame processed by the video decapsulation decoding unit to the data processing module is achieved through the DMA controller.

3. According to the AI ​​video stream quality detection method based on embedded device in claim 1, it is characterized in that: The post-processing module analyzes and determines the quality of the following: The post-processing module analyzes the similarity between the video frame features and the basic types of quality problems and obtains a similarity score: when the score is higher than the threshold, it means that the video stream has a corresponding quality problem; when the scores are lower than the preset thresholds of each type of quality problem, it means that the quality of the determined video stream is qualified; The basic types of quality issues described include blur, mosaic, noise, screen distortion and green screen.

4. According to the AI ​​video stream quality detection method based on embedded device in claim 1, it is characterized in that: The video stream to be detected is directly pushed to the video detection device according to the video protocol or pushed after being pulled by the video push device; the video stream to be detected can come from network transmission detection, video recording detection or storage media transmission.

5. According to the AI ​​video stream quality detection method based on embedded device in claim 1, it is characterized in that: The data processing module includes a data processor storing a verification model, and the types of the data processor are divided into embedded neural network processors, graphics processors or floating-point processing units; the construction process of the verification model is divided into three steps: obtaining a training set, model training, model conversion and quantization.

6. The AI ​​video stream quality detection method based on embedded devices according to claim 5 is characterized in that: The obtaining of the training set is to prepare a video defect data set, and use labeimg to annotate the defects to obtain the training set; the model training is to train the training set based on the YOLOv5s network model to obtain the model weight for detecting video defects; the video defects are divided into blur, mosaic, noise, flower screen and green screen.

7. The AI ​​video stream quality detection method based on embedded device according to claim 6 is characterized in that: The YOLOv5s network model is divided into an input part, a backbone part, a neck part and an output part; in the backbone part, a convolution operation is performed on the features of the input feature map through a C3 module to extract image features; and in the neck part, the feature map is fused.

8. The AI ​​video stream quality detection method based on embedded device according to claim 5 is characterized in that: The model conversion and quantization are to convert the model into a model structure that can be supported by the data processor and perform uint8 quantization processing.

9. The AI ​​video stream quality detection method based on embedded devices according to claim 1 is characterized in that: The statistical results are displayed in the form of charts on the WEB front-end interface.

Images