Video analysis and processing device and video encoding and decoding system
Patent Information
- Application Number
- CN202311837103.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-28
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2043-12-28
AI Technical Summary
The existing video scene switching detection technology is unfriendly when switching scenes quickly, and it is difficult to meet the real-time encoding requirements. It is not adaptable to different video types, resulting in delays and misjudgment.
Design a video analysis processing device, including a coprocessor, a first processor and a second processor, calculates the histogram information and variance information of the video frame, and performs scene switching detection using the sum of the weighting coefficient and absolute difference, and according to Video type determines the target threshold to improve detection accuracy.
It significantly reduces the delay in video analysis processing, improves the accuracy and encoding quality of scene switching detection, and adapts to scene switching detection of different video types.
Smart Images

Figure CN117793354B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video compression, and in particular to a video analysis and processing device and a video encoding and decoding system. Background Art
[0002] Video preprocessing is the process of processing and preparing the video before video compression. It mainly analyzes and processes the video image by extracting the feature information of the video image. For example, by extracting the histogram and variance of the video, the scene change detection (SCD) of the video is completed.
[0003] The main task of video scene switching detection is to identify the transition points between scenes by analyzing the characteristics and changes of the video. The brightness / chrominance histograms of adjacent frames are extracted and their similarities are compared. When the difference in the brightness / chrominance histogram exceeds a certain threshold, it can be judged as a scene switching point. If the encoder cannot detect fast scene switching, it is not friendly to encoding compression and the video compression ratio is not high. For fast-switching scenes, by modifying the encoding frame type to I frame (I frame is a frame using intra-frame prediction), the subjective experience will be improved. Scene detection technology has been widely used in video encoding technology.
[0004] Scene detection requires statistics on the histogram and variance information of each frame. The calculation is relatively complex and causes a certain delay. For videos with relatively high resolution, it is difficult to meet the requirements of real-time encoding if only software is used; if only hardware is used, the hardware complexity and area may be increased exponentially. At the same time, it cannot adapt to all video types, and the processing results are not optimal. For example, for different video types, the threshold of the difference between the previous and next frames when the scene switches will be different, so it may cause misjudgment. Summary of the invention
[0005] The purpose of the present application is to provide a video analysis and processing device and a video encoding and decoding system, which can significantly reduce the delay when analyzing and processing the video, accurately realize scene switching detection of videos of different video types, and improve the encoding quality.
[0006] According to a first aspect of an embodiment of the present application, there is provided a video analysis and processing device, comprising: a coprocessor, a first processor, a second processor and an encoder;
[0007] The coprocessor is configured to determine a target threshold value corresponding to the target video type according to a correspondence between the target video type and a preset video type and a threshold value;
[0008] The first processor is configured to calculate the histogram information of the image after format conversion of each video frame among all video frames; all video frames include a current frame and M target reference frames, where M is a positive integer;
[0009] The coprocessor is further configured to calculate a weight coefficient of the current frame according to histogram information of the current frame and the M target reference frames, and send the weight coefficient to the second processor;
[0010] The second processor is configured to calculate an unweighted sum of absolute differences according to the histogram information of the current frame and the M target reference frames, and to calculate a weighted sum of absolute differences according to the histogram information of the current frame, the M target reference frames and the weighting coefficient;
[0011] The coprocessor is further configured to obtain a minimum value between the unweighted sum of absolute differences and the weighted sum of absolute differences, and obtain a scene switching detection result according to a comparison result between the minimum value and the target threshold;
[0012] The encoder is configured to determine a coding frame type of a current frame according to the scene switching detection result.
[0013] In one embodiment, the coprocessor is further configured to save the calculated histogram information of the current frame and N potential reference frames into a histogram queue in the coprocessor according to a first-in-first-out principle; N is a positive integer; N is greater than or equal to M;
[0014] The coprocessor is also configured to obtain the number of target reference frames and identification information of the target reference frames according to the encoding parameters and identification information of the current frame, and obtain histogram information of the current frame and M target reference frames from the histogram queue according to the identification information of the current frame and the identification information of the target reference frame.
[0015] In one embodiment, the histogram information includes a histogram average and a histogram normal mean;
[0016] The histogram average is calculated using the following formula:
[0017] HistMean=SumPixel / NumPixels;
[0018] Wherein, HistMean is the histogram mean, SumPixel is the sum of the pixel values in the current frame, and NumPixels is the number of pixels in the current frame;
[0019] The normal mean of the histogram is calculated using the following formula:
[0020]
[0021] Wherein, HistNormalMean is the normal mean of the histogram, i is the ordinal number of the bar in the histogram, L is the value of the number of bars in the histogram minus 1, Hist i is the number of occurrences of pixel value i.
[0022] In one embodiment, the weighting coefficient includes an exponential coefficient, a proportional coefficient and an offset;
[0023] The coprocessor is further configured to calculate a weighting coefficient of the current frame using the following calculation formula according to the histogram information of the current frame and the M target reference frames:
[0024] Scale = (HistNormalMean cur *(1<<denom)) / HistNormalMean ref ;
[0025] While(denom>0&&(scale>127))
[0026] {
[0027] Denom--;
[0028] Scale>>=1;
[0029] };
[0030] offset = HistMean cur ―scale*HistMean ref / 2 denom ;
[0031] Where, denom is the exponential coefficient, scale is the proportional coefficient, offset is the offset, HistNormalMean cur is the normal mean of the histogram of the current frame, HistNormalMean ref is the normal mean of the histogram of the target reference frame, HistMean cur is the histogram average of the current frame, HistMean ref is the histogram average of the target reference frame.
[0032] In one embodiment, the second processor is further configured to calculate the unweighted sum of absolute differences using the following calculation formula according to the histogram information of the current frame and the M target reference frames:
[0033]
[0034] Among them, SAD plane is the unweighted sum of absolute differences, i is the ordinal number of the bar in the histogram, L is the number of bars in the histogram minus 1, CurHist i is the number of occurrences of pixel value i in the current frame, RefHist i is the number of occurrences of pixel value i in the target reference frame;
[0035] The second processor is further configured to calculate the weighted sum of absolute differences according to the histogram information of the current frame, the M target reference frames and the weighting coefficients using the following calculation formula:
[0036]
[0037] Among them, WeightedSAD plane is the weighted sum of absolute differences.
[0038] In one embodiment, the video analysis and processing device further includes a third processor;
[0039] The third processor is configured to calculate, for each video frame in all video frames, variance information of the image after format conversion of the video frame, wherein the variance information includes a minimum variance, a maximum variance, an average variance and a variance of each macroblock of the image;
[0040] The coprocessor is further configured to obtain a correspondence between a variance threshold of a macroblock and an increment of a quantization parameter according to the minimum variance, the maximum variance, the average variance, an increment range of a quantization parameter, and a video type;
[0041] The encoder is further configured to determine, for each macroblock in the current frame, an increment of the quantization parameter of the macroblock according to the variance of the macroblock and a correspondence between a variance threshold of the macroblock and an increment of the quantization parameter.
[0042] In one embodiment, the coprocessor is further configured to obtain an initialization table according to the minimum variance, the maximum variance, the average variance, the increment range of the quantization parameter, and the video type, wherein the initialization table is used to store the corresponding relationship between the initialization ratio of the variance of the macroblock and the increment of the quantization parameter;
[0043] The coprocessor is further configured to calculate the variance threshold of each macroblock according to the initialization table by cyclically using the following calculation formula to obtain the corresponding relationship between the variance threshold of the macroblock and the increment of the quantization parameter:
[0044] threshold_var=(avg_var*s_blaq_threshold_table[j][0]) / 1024;
[0045] Among them, threshol_var is the variance threshold of the macroblock, avg_var is the average variance of the current frame, s_blaq_threshold_table[j][0] is the first column of the initialization table, and j is the row parameter of the initialization table.
[0046] In one embodiment, the coprocessor is further configured to determine a corresponding monotonic function according to the video type, and obtain the initialization table according to the minimum variance, the maximum variance, the average variance, the incremental range of the quantization parameter and the monotonic function.
[0047] In one embodiment, the video analysis and processing device further includes a memory;
[0048] The coprocessor is further configured to determine that the weighting coefficient is valid when it is determined that the weighted sum of absolute differences is less than the unweighted sum of absolute differences, and write a first image reference relationship corresponding to the weighting coefficient into the memory; the first image reference relationship includes identification information of the current frame and identification information of the M target reference frames;
[0049] The encoder is also configured to perform weighted prediction interpolation on pixel values based on the weighting coefficients when it is determined that the first image reference relationship is consistent with a stored second image reference relationship, wherein the second image reference relationship is an image reference relationship determined by the encoder based on input information.
[0050] According to a second aspect of an embodiment of the present application, a video encoding and decoding system is provided, comprising a decoding device and the above-mentioned video analysis and processing device;
[0051] The video analysis and processing device is configured to analyze and process the video before compressing the video;
[0052] The encoder is configured to encode the analysis and processing result of the video based on the analysis and processing device of the video to obtain an encoded bit stream;
[0053] The decoding device is configured to decode the encoded bitstream to obtain a video before encoding.
[0054] Compared with the prior art, the beneficial effect of the present application lies in that the video analysis and processing device, in addition to including an encoder, also includes a coprocessor, a first processor and a second processor. The coprocessor is configured to determine the target threshold corresponding to the target video type based on the correspondence between the target video type and the preset video type and the threshold. In this way, scene switching detection can be performed based on the target threshold corresponding to the target video type, thereby improving the accuracy of scene switching detection. The first processor is configured to calculate the histogram information of the image after format conversion of each video frame in all video frames, and all video frames include a current frame and M target reference frames. The coprocessor is also configured to calculate the weighting coefficient of the current frame according to the histogram information of the current frame and the M target reference frames, and send the weighting coefficient to the second processor. The second processor is configured to calculate the unweighted sum of absolute differences according to the histogram information of the current frame and the M target reference frames, and calculate the weighted sum of absolute differences according to the histogram information of the current frame and the M target reference frames and the weighting coefficient. The coprocessor is also configured to obtain the minimum value between the unweighted sum of absolute differences and the weighted sum of absolute differences, and obtain the scene switching detection result according to the comparison result of the minimum value and the target threshold value. The encoder is configured to determine the encoding frame type of the current frame according to the scene switching detection result. In the process of scene switching detection, complex calculations are completed by two hardware modules, the first processor and the second processor, which can significantly reduce delays compared to software implementations. The coprocessor can determine the corresponding threshold value according to the video type, which can improve the accuracy of scene switching detection and improve the encoding quality. In summary, the technical solution of the present application can significantly reduce delays when analyzing and processing videos, accurately implement scene switching detection for videos of different video types, and improve encoding quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 The figure is a schematic diagram of the structure of a video analysis and processing device according to an exemplary embodiment.
[0056] Figure 2 The figure is a schematic structural diagram of a video analysis and processing device according to another exemplary embodiment.
[0057] Figure 3 The figure is a schematic structural diagram of a video analysis and processing device according to another exemplary embodiment.
[0058] Figure 4 The figure is a schematic diagram showing a storage space of a BLAQ table in a memory according to an exemplary embodiment.
[0059] Figure 5 The figure is a schematic structural diagram of a video analysis and processing device according to another exemplary embodiment.
[0060] Figure 6 The diagram is a structural diagram of a video encoding and decoding system according to an exemplary embodiment. DETAILED DESCRIPTION
[0061] Unless otherwise defined, the technical terms or scientific terms used in this specification and claims shall have the usual meanings understood by persons with ordinary skills in the technical field to which the invention belongs. The specific embodiments of the present invention will be described below in conjunction with the accompanying drawings. It should be noted that in the specific description of these embodiments, in order to provide a concise description, it is impossible for this specification to provide a detailed description of all the features of the actual embodiments. Without departing from the spirit and scope of the present invention, those skilled in the art may modify and replace the embodiments of the present invention, and the resulting embodiments are also within the scope of protection of the present invention.
[0062] Video preprocessing is the process of processing and preparing the video before video compression, mainly by extracting the feature information of the video image and analyzing and processing the video image. For example, by extracting the histogram and variance information of the video, scene change detection (SCD), block level adaptive quantization (BLAQ) and weighted prediction motion compensation (WPMC) can be completed.
[0063] The main task of video scene switching detection is to identify the transition points between scenes by analyzing the characteristics and changes of the video. The brightness / chrominance histograms of adjacent frames are extracted and their similarities are compared. When the difference in the brightness / chrominance histogram exceeds a certain threshold, it can be judged as a scene switching point. If the encoder cannot detect fast scene switching, it is not friendly to encoding compression and the video compression ratio is not high. For fast-switching scenes, by modifying the encoding frame type to I frame (I frame is a frame using intra-frame prediction), the subjective experience will be improved. Scene detection technology has been widely used in video encoding technology.
[0064] Adaptive quantization aims to improve compression efficiency and maintain good visual quality by dynamically quantizing according to the content characteristics of an image or video. Different areas in an image or video may have different characteristics, such as texture, edges, flat areas, etc. Adaptive quantization dynamically adjusts the quantization step size and parameters of each block based on these local characteristics to retain important details and small quantization errors. Adaptive quantization can provide higher compression ratios and reduce distortion caused by compression. It is widely used in video coding standards (such as H.264 and HEVC), and is also used in various image compression and encoding algorithms to improve compression efficiency and visual quality. To calculate the characteristics of a macroblock, such as variance, it is necessary to traverse each pixel of the image, perform complex calculations such as logarithmic calculations based on the variance information, and then obtain dynamic quantization values for different areas.
[0065] In video compression, for videos that fade in and out, if motion prediction and motion compensation techniques are used alone, it is impossible to obtain a relatively matching reference frame or reference block, resulting in a large residual energy obtained by prediction, which is not suitable for compression. Relative to the reference frame image, the reference frame pixel value can be multiplied by a weight value (scale) and an offset (offset) to obtain a new reference frame or reference block, and the encoding effect will be greatly improved. This is weighted prediction, which is also widely used in video encoding.
[0066] Scene detection, adaptive quantization, and weighted prediction all require statistics on the histogram and variance information of each frame. The calculation is relatively complex and causes a certain delay. For videos with relatively high resolution, it is difficult to meet the requirements of real-time encoding if only software is used; if only hardware is used, the hardware complexity and area may increase exponentially. At the same time, it cannot adapt to all video types, and the processing results are not optimal. For example, for different video types, the threshold of the difference between the previous and next frames when the scene switches will be different, so it may cause misjudgment.
[0067] In order to solve the above technical problems, the present application proposes an image processing method, device, electronic device and storage medium, which can significantly reduce the delay when analyzing and processing videos, accurately realize scene switching detection and adaptive quantization of videos of different video types, and improve the encoding quality; it can greatly reduce the amount of calculation and reduce the area of hardware chips.
[0068] Figure 1 is a structural block diagram of a video analysis and processing device according to an exemplary embodiment. Figure 1The video analysis and processing device may include: a first processor 11, a second processor 12, a third processor 13, a coprocessor 14, an encoder 15, a memory 16 and a register 17.
[0069] The video analysis and processing device can be used to analyze and process the video to achieve scene change detection (SCD), adaptive quantization and weighted prediction. The video analysis and processing device is introduced below for scene change detection (SCD), adaptive quantization and weighted prediction.
[0070] 1. Scene Change Detection (SCD)
[0071] like Figure 1-2 As shown, the video analysis and processing device includes: a first processor 11, a second processor 12, a coprocessor 14 and an encoder 15.
[0072] like Figure 1-2 As shown, the first processor 11 and the second processor 12 are respectively connected to the coprocessor 14 , and the coprocessor 14 is connected to the encoder 15 .
[0073] In one embodiment, the coprocessor 14 is configured to determine the target threshold value corresponding to the target video type according to the correspondence between the target video type and the preset video type and the threshold value. The correspondence between the video type and the threshold value may be pre-stored in the coprocessor 14, and the coprocessor 14 may determine the target threshold value corresponding to the target video type according to the correspondence between the target video type and the preset video type and the threshold value. The target video type is the type of the video to be analyzed and processed. For example, if the target video type is a movie, the target threshold value may be 2, if the target video type is an animation, the target threshold value may be 1.5, and so on.
[0074] In other embodiments, the coprocessor 14 may also be configured to determine the target threshold corresponding to the target coding parameter according to the correspondence between the target coding parameter and the preset coding parameter and the threshold. For example, the coding parameter includes a continuous group of pictures size (gop_size), and the coprocessor 14 may pre-store the correspondence between the continuous group of pictures size and the threshold. The coprocessor 14 may determine the target threshold corresponding to the target continuous group of pictures size according to the correspondence between the target continuous group of pictures size and the preset continuous group of pictures size and the threshold. If the target gop_size is relatively large, the target threshold may be smaller, and if the target gop_size is relatively small, the target threshold may be larger. In this way, different continuous group of pictures sizes can be adapted to avoid misjudgment of scene switching, thereby improving coding quality.
[0075] The first processor 11 is configured to calculate the histogram information of the image for each video frame in all video frames after format conversion of the video frame; all video frames include a current frame and M target reference frames, where M is a positive integer, for example, M can be 1, 2, 7, 8, 10, 15, 16 or other positive integers.
[0076] Among them, the image format conversion of the video frame can be performed by other modules, such as by the image format conversion module. For example, an image in RGB format is converted into an image in YUV format. In the process of the image format conversion module performing format conversion on the video frame, the first processor 11 can calculate the histogram information of the image after format conversion of each video frame. The histogram information includes the histogram mean and the histogram normal mean. In the process of image format conversion of the video frame, it is necessary to operate on each pixel. Therefore, in the process of image format conversion, the calculation of the histogram information of the statistical image and the variance information of the macroblock of the image is also conveniently completed, which can reduce the frequent reading operation of the memory and reduce the delay.
[0077] In one embodiment, the histogram average is calculated using the following formula:
[0078] HistMean=SumPixel / NumPixels (1)
[0079] Among them, HistMean is the histogram mean, SumPixel is the sum of the pixel values in the current frame, and NumPixels is the number of pixels in the current frame.
[0080] The normal mean of the histogram is calculated using the following formula:
[0081]
[0082] Among them, HistNormalMean is the normal mean of the histogram, i is the ordinal number of the bar in the histogram, L is the number of bars in the histogram minus 1, Hist i is the number of occurrences of pixel value i.
[0083] If the video frame is an 8-bit image and the pixel value range is [0, 255], then the number of bars in the histogram is 256, so L = 255; if the video frame is a 10-bit image and the pixel value range is [0, 1023], then the number of bars in the histogram is 1024, so L = 1023.
[0084] The coprocessor 14 is further configured to calculate a weighting coefficient of the current frame according to histogram information of the current frame and the M target reference frames, and send the weighting coefficient to the second processor.
[0085] In one embodiment, Figure 2 As shown, the coprocessor 14 may include a parameter information acquisition module 141 , a histogram information module 142 and a first processing module 143 .
[0086] The parameter information acquisition module 141 is configured to acquire the number of target reference frames and the identification information of the target reference frame according to the encoding parameters and the identification information of the current frame. The encoding parameters include the correspondence between the current frame and the target reference frame, and the number of target reference frames. The identification information of the current frame may be the POC (picture order count) of the current frame, and the identification information of the target reference frame may be the POC of the target reference frame.
[0087] The histogram information module 142 may be configured to save the histogram information of the current frame and the N potential reference frames calculated by the first processor 11 to a histogram queue in the coprocessor according to the first-in-first-out principle; N is a positive integer, and N is greater than or equal to M. N may be, for example, 16, or other positive integers. The histogram queue includes the corresponding relationship between the identification information of the current frame and the N potential reference frames and the histogram information of the current frame and the N potential reference frames.
[0088] In one embodiment, the potential reference frame of the current frame is not necessarily the immediately previous frame, but may be separated by several frames, but the histogram information of 16 potential reference frames at most is stored in the histogram queue. If the histogram information of 17 frames (the current frame + the 16 potential reference frames that may be used) is already stored in the histogram queue, the fifo (First In First Out) method is adopted to discharge the histogram information of the first stored frame from the queue and store the latest histogram information in it.
[0089] The histogram information module 142 is further configured to obtain histogram information of the current frame and the M target reference frames from the histogram queue according to the identification information of the current frame and the identification information of the target reference frame.
[0090] In one embodiment, the weighting coefficient is a weighting coefficient for an inter-frame prediction value, and includes an exponential coefficient, a proportional coefficient, and an offset.
[0091] The first processing module 143 is configured to calculate the weighting coefficient of the current frame using the following calculation formula according to the histogram information of the current frame and the M target reference frames:
[0092] Scale = (HistNormalMean cur *(1<<denom)) / HistNormalMean ref (3)
[0093] While(denom>0&&(scale>127))
[0094] {
[0095] Denom--;
[0096] Scale>>=1;
[0097] }
[0098] offset = HistMean cur ―scale*HistMean ref / 2 denom (4)
[0099] Among them, denom is the exponential coefficient, scale is the proportional coefficient, offset is the offset, HistNormalMean cur is the normal mean of the histogram of the current frame, HistNormalMean ref is the normal mean of the histogram of the target reference frame, HistMean cur HistMean is the histogram average of the current frame. ref is the histogram average of the target reference frame.
[0100] For example, 30 / 31=0.9677 (approximate value). 0.9677 is difficult to represent in computers, so it can be represented roughly as: 0.9677=62 / 64; 62 here is scale, and 64 can be represented as 2 to the sixth power, so denom here is 6, so offset=31*62 / (2 to the sixth power)-30.
[0101] It should be noted that 0.9677 can also be approximately equal to 123 / 128. The precision of 123 / 128 is higher than that of 31*62. This is related to the standard that denom cannot exceed 7, so there is the above while loop.
[0102] Through the normal mean of the histogram of the previous frame and the target reference frame, a scale with the highest accuracy that meets the conditions (denom cannot exceed 7, scale cannot exceed 127) can be calculated, and denom can also be calculated.
[0103] The offset can also be calculated after obtaining the histogram average of the previous frame and the target reference frame and the most accurate scale and denom.
[0104] The second processor 12 is configured to calculate an unweighted sum of absolute differences based on histogram information of the current frame and the M target reference frames, and to calculate a weighted sum of absolute differences based on the histogram information of the current frame, the M target reference frames and a weighting coefficient.
[0105] In one embodiment, the second processor 12 is further configured to calculate the unweighted sum of absolute differences using the following formula according to the histogram information of the current frame and the M target reference frames:
[0106]
[0107] Among them, SAD plane is the unweighted sum of absolute differences, i is the ordinal number of the bar in the histogram, L is the number of bars in the histogram minus 1, CurHist i is the number of occurrences of pixel value i in the current frame, RefHist i is the number of occurrences of pixel value i in the target reference frame.
[0108] The histogram is a statistical calculation of the number of times each pixel value appears in a frame of an image. For example, the histogram information of the first frame is: the pixel value 1 appears 32 times, the pixel value 2 appears 40 times, ..., the pixel value 255 appears 10 times. The histogram information of the second frame is: the pixel value 1 appears 18 times, the pixel value 2 appears 20 times, ..., the pixel value 255 appears 3 times. Then the sum of the absolute differences (SAD) of the histograms of these two frames = (32-18)+(40-20)+...+(10-3).
[0109] The second processor 12 is further configured to calculate the weighted sum of absolute differences according to the histogram information of the current frame and the M target reference frames and the weighting coefficients using the following calculation formula:
[0110]
[0111] Among them, WeightedSAD plane is the weighted sum of absolute differences.
[0112] The coprocessor 14 is further configured to obtain a minimum value between the unweighted sum of absolute differences and the weighted sum of absolute differences, and obtain a scene switching detection result according to a comparison result between the minimum value and a target threshold.
[0113] In one embodiment, the first processing module 143 is further configured to compare the unweighted sum of absolute differences with the weighted sum of absolute differences, obtain the minimum value between the unweighted sum of absolute differences and the weighted sum of absolute differences, and compare the minimum value with the target threshold value. If the minimum value is greater than the target threshold value, it is determined that a scene switch occurs between the current frame and the previous video frame. If the minimum value is less than the target threshold value, it is determined that no scene switch occurs between the current frame and the previous video frame.
[0114] The second processor 12 is further configured to store the scene switching detection result in the register 17 for retrieval.
[0115] The encoder 15 is configured to determine the encoding frame type of the current frame according to the scene switch detection result.
[0116] In one embodiment, the encoder 15 obtains the scene switching detection result by reading the register 17, and determines the encoding frame type of the current frame according to the scene switching detection result. If the scene switching detection result indicates that the current frame has a scene switching, the encoder changes the encoding type of the current frame to an I frame, and if the scene switching detection result indicates that the current frame has no scene switching, the original encoding type is maintained.
[0117] When analyzing and processing the video for scene change detection (SCD), the first processor and the second processor complete the complex calculations, which can reduce system delay. At the same time, the coprocessor can determine the corresponding threshold according to the video type, which can greatly reduce the misjudgment of scene change, improve the accuracy of scene change detection, and improve the encoding quality.
[0118] In this application, a video analysis and processing method based on a combination of software and hardware significantly reduces hardware area and processing delay.
[0119] In one embodiment, Figure 1 As shown, the encoder 15 further includes a main processor 153, which is used to write the above-mentioned weighting coefficients into a bitstream. In addition, the main processor 153 is mainly used to complete some logic control and simple calculation processing, such as the microcode part it includes.
[0120] In the present application, a coprocessor can be used to determine a corresponding threshold value according to the video type, thereby reducing the amount of calculation of the main processor and enabling the main processor and the coprocessor to jointly complete the analysis and processing of the video.
[0121] 2. Adaptive Quantization
[0122] like Figure 1 and Figure 3As shown, the video analysis and processing device includes: a third processor 13, a coprocessor 14, an encoder 15 and a memory 16. The memory 16 may be a DRAM (dynamic random access memory).
[0123] like Figure 1 and Figure 3 As shown, the third processor 13 is connected to the coprocessor 14 , and the coprocessor 14 is connected to the encoder 15 .
[0124] The third processor 13 is configured to calculate the variance information of the image after format conversion for each video frame in all video frames, and the variance information includes the minimum variance, maximum variance, average variance and variance of each macroblock of the image.
[0125] Similarly, the image format conversion of the video frame can be performed by other modules, such as the image format conversion module. In the process of the image format conversion module performing format conversion on the video frame, the third processor 13 can calculate the variance information of the image after the format conversion of each video frame. In the process of the image format conversion of the video frame, each pixel needs to be operated. Therefore, the calculation of the variance information of the image is also conveniently completed in the process of the image format conversion, which can reduce the frequent reading operation of the memory and reduce the delay.
[0126] The coprocessor 14 is further configured to obtain a correspondence between a variance threshold of a macroblock and an increment of a quantization parameter according to a minimum variance, a maximum variance, an average variance, an increment range of a quantization parameter, and a video type.
[0127] In one embodiment, Figure 1 and Figure 3 As shown, the coprocessor 14 also includes an adaptive quantization information module 144, which is configured to obtain the corresponding relationship between the variance threshold of the macroblock and the increment of the quantization parameter according to the minimum variance, the maximum variance, the average variance, the increment range of the quantization parameter, and the video type. The corresponding relationship between the variance threshold of the macroblock and the increment of the quantization parameter will be stored in the memory 16 for the encoder 15 to call.
[0128] In one embodiment, the correspondence between the variance threshold of the macroblock and the increment of the quantization parameter is in the form of a table, which can be recorded as a self-adaptive quantization (BLAQ) table. The BLAQ table is a monotonically increasing table or a monotonically decreasing table. In one embodiment, different video types have different functions for establishing the table.
[0129] For example, assuming that the increment range of the quantization parameter (qp) is [-3, 5], the average variance of the current frame calculated by the third processor 13 is 100, the minimum variance is 30, and the maximum variance is 300; then, we can assume that if the variance is less than 45, the increment of the quantization parameter (qp) can be -3, if the variance is between 45 and 60, the increment of the quantization parameter (qp) can be -2, if the variance is between 60 and 85, the increment of the quantization parameter (qp) can be -1, and similarly, if the variance is between 270 and 300, the increment of the quantization parameter (qp) can be 5. In this way, several variance thresholds and corresponding increments of qp can be obtained. After establishing this table, if the variance of the current block is 55, then according to the above table, it is quickly obtained that the increment of qp of the current block is -2.
[0130] In one embodiment, the adaptive quantization information module 144 is further configured to obtain an initialization table according to the minimum variance, the maximum variance, the average variance, the increment range of the quantization parameter, and the video type, wherein the initialization table is used to store the correspondence between the initialization ratio of the variance of the macroblock and the increment of the quantization parameter. The initialization table is shown in Table 1 below.
[0131] Table 1
[0132]
[0133] static int is used to declare members as static members and to define static int variables. Int variables are integer type variables. s_blaq_threshold_table
[15] [2] is the initialization table mentioned above.
[0134] The adaptive quantization information module 144 is further configured to calculate the variance threshold of each macroblock by cyclically using the following calculation formula according to the initialization table, and obtain the corresponding relationship between the variance threshold of the macroblock and the increment of the quantization parameter:
[0135] threshold_var=(avg_var*s_blaq_threshold_table[j][0]) / 1024 (7)
[0136] Among them, threshold_var is the variance threshold of the macroblock, avg_var is the average variance of the current frame, s_blaq_threshold_table[j][0] is the first column of the initialization table, and j is the row parameter of the initialization table.
[0137] The average variance is multiplied by the initialization ratio in the initialization table to get a variance threshold. Assuming the average variance is 120, the first variance threshold is (120*64+512) / 1024, which is approximately equal to 8. The first variance threshold and qp increment of the BLAQ table are 8 and -7 respectively. The second variance threshold is (120*128+512) / 1024, which is approximately equal to 16. The second variance threshold and qp increment of the BLAQ table are 16 and -6 respectively. The third variance threshold is (120*225+512) / 1024, which is approximately equal to 27. The second variance threshold and qp increment of the BLAQ table are 27 and -5 respectively. Calculate in sequence. In this way, the increments of the initial multiple sets of variance thresholds and qp can be calculated to obtain the BLAQ table.
[0138] In one embodiment, different video types have different functions for establishing the initialization table. The adaptive quantization information module 144 is also configured to determine the corresponding monotonic function according to the video type, and obtain the initialization table according to the minimum variance, the maximum variance, the average variance, the increment range of the quantization parameter, and the monotonic function. For example, if the video type is a movie, the initialization table can be established using a one-time function, and if the video type is an animation, the initialization table can be established using a quadratic function.
[0139] In one embodiment, the BLAQ table is stored in memory 16. Figure 4 As shown, each square 41 represents 8 bits of storage space. Since a variance threshold var only needs 24 bits to store, the variance threshold var requires three squares 41. The increment Δqp of qp only needs 8 bits to store, so it needs one square 41. Since the variance threshold var and the increment Δqp of qp appear one-to-one, they can be stored in 32 bits, that is, 4 bytes.
[0140] When designing DRAM to store the BLAQ table, only 72*4Byte is needed at most. However, in practice, the length len of each frame may be different, and less than 72*4Byte is used.
[0141] The encoder 15 is further configured to determine, for each macroblock in the current frame, an increment of the quantization parameter of the macroblock according to the variance of the macroblock and a correspondence between a variance threshold of the macroblock and an increment of the quantization parameter.
[0142] For example, the correspondence between the variance threshold of the macroblock and the increment of the quantization parameter is in the BLAQ table shown in Table 2 below. Assuming that the variance of the current block is 80, since 80 is less than 100 in Table 2, the corresponding qp increment is -3. If the variance is 1000, since 1000 is between 400 and 3200 in the table, the corresponding qp increment is -1. Assuming that the variance of the current block is 11000, since 11000>10890, the qp increment is 2.
[0143] Table 2 BLAQ table
[0144] BLAQ[5][2]={
[0145] {100,-3},
[0146] {400,-2},
[0147] {3200,-1},
[0148] {6500,1},
[0149] {10890,2},
[0150] }
[0151] Compared with the traditional method, by querying the correspondence between the variance threshold of the pre-stored macroblock and the increment of the quantization parameter or obtaining the increment of the quantization parameter by looking up the table, the amount of calculation and the hardware area can be greatly reduced. At the same time, by adjusting the initialization table for storing the correspondence between the initialization ratio of the variance of the macroblock and the increment of the quantization parameter through input variables (the increment range of the quantization parameter) and the video type, etc., it can be adaptive to various types of videos to achieve the best subjective evaluation results.
[0152] 3. Weighted Prediction
[0153] like Figure 1 and Figure 5 As shown, the video analysis and processing device includes: a first processor 11, a second processor 12, a coprocessor 14, an encoder 15 and a memory 16. The memory 16 may be a DRAM (dynamic random access memory).
[0154] like Figure 1 and Figure 5 As shown, the first processor 11 and the second processor 12 are connected to the coprocessor 14 , the coprocessor 14 is connected to the encoder 15 , and the encoder 15 is connected to the memory 16 .
[0155] The first processor 11 is configured to calculate the histogram information of the image after format conversion is performed on each video frame among all video frames; all video frames include a current frame and M target reference frames.
[0156] The coprocessor 14 is further configured to calculate a weighting coefficient of the current frame according to histogram information of the current frame and the M target reference frames, and send the weighting coefficient to the second processor.
[0157] In one embodiment, Figure 5 As shown, the coprocessor 14 further includes a parameter information acquisition module 141 , a histogram information module 142 , a first processing module 143 and a second processing module 145 .
[0158] The parameter information acquisition module 141 is configured to acquire the number of target reference frames and the identification information of the target reference frame according to the encoding parameters and the identification information of the current frame. The encoding parameters include the correspondence between the current frame and the target reference frame, and the number of target reference frames. The identification information of the current frame may be the POC (picture order count) of the current frame, and the identification information of the target reference frame may be the POC of the target reference frame.
[0159] The histogram information module 142 may be configured to save the histogram information of the current frame and the N potential reference frames calculated by the first processor 11 into a histogram queue in the coprocessor according to a first-in, first-out principle.
[0160] The histogram information module 142 is further configured to obtain histogram information of the current frame and the M target reference frames from the histogram queue according to the identification information of the current frame and the identification information of the target reference frame.
[0161] The first processing module 143 is configured to calculate the weighting coefficient of the current frame using the above calculation formula (3) according to the histogram information of the current frame and the M target reference frames.
[0162] The second processor 12 is configured to calculate an unweighted sum of absolute differences based on histogram information of the current frame and the M target reference frames, and to calculate a weighted sum of absolute differences based on the histogram information of the current frame, the M target reference frames and a weighting coefficient.
[0163] The second processing module 145 is configured to determine that the weighting coefficient is valid when it is determined that the weighted sum of absolute differences is less than the unweighted sum of absolute differences, and write the first image reference relationship corresponding to the weighting coefficient into the memory. The first image reference relationship includes the identification information of the current frame and the identification information of the M target reference frames.
[0164] The second processing module 145 may also write the first image reference relationship into the register 17 .
[0165] In one embodiment, Figure 5 As shown, the encoder 15 further includes a weighted prediction module 152 .
[0166] The encoder 15 is also configured to read the first image reference relationship through the register 17, and when it is determined that the first image reference relationship is consistent with the stored second image reference relationship, write the weighting coefficient into the bitstream, and then pass the weighting coefficient to the weighted prediction module 152 through the register 17. The weighted prediction module 152 performs weighted prediction interpolation on the pixel value based on the weighting coefficient.
[0167] The second image reference relationship is an image reference relationship determined by the encoder according to input information. The input information may be preset parameters of a continuous group of pictures and other parameters used to determine the encoding order and image reference relationship during encoding.
[0168] In one embodiment, the first processing module 143 and the second processing module 145 may be combined into one module.
[0169] In one embodiment, Figure 1 and Figure 5 As shown, the encoder 15 further includes a main processor 153, which is used to write the above-mentioned weighting coefficients into a bitstream. In addition, the main processor 153 is mainly used to complete some logic control and simple calculation processing, such as a microcode part.
[0170] When analyzing and processing the video before weighted prediction, the first processor and the second processor complete the complex calculations, which can reduce system delay.
[0171] In the present application, when processing image format conversion (video color conversion (CCE)), the characteristic information of the video including the calculation of histogram information and variance information is completed at the same time, which can reduce the process of reading the original image pixel value from the memory and reduce the system delay.
[0172] In summary, this application has the following advantages:
[0173] First, compared with pure software, the more complex calculations are implemented through hardware, which can greatly improve the calculation speed and reduce coding delay. Compared with pure hardware implementation, some complex calculations are implemented through table lookup, which can greatly reduce the chip area.
[0174] Secondly, this solution implemented by software and hardware is more flexible, and different thresholds can be set according to the video type, so as to achieve the best processing effect for different types of videos.
[0175] Third, the solution of the main coprocessor jointly completing the video pre-analysis processing can reduce the calculation of the main processor, thereby reducing the encoding delay.
[0176] Finally, while completing the image color format conversion, the image feature information is also counted, thereby reducing the number of times the image pixels are read from the memory and reducing system latency.
[0177] The embodiment of the present application also proposes a video encoding and decoding system. Figure 6 As shown, the video encoding and decoding system includes a decoding device 60 and the above-mentioned video analysis and processing device 10.
[0178] The video analysis and processing device 10 is configured to analyze and process the video before compressing the video;
[0179] The encoder 15 is configured to encode the analysis and processing result of the video based on the video analysis and processing device to obtain an encoded bit stream;
[0180] The decoding device 60 is configured to decode the encoded bitstream to obtain the video before encoding.
[0181] In the present invention, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. The term "plurality" refers to two or more than two, unless otherwise clearly defined.
[0182] The above description of the embodiments is to facilitate those of ordinary skill in the art to understand and apply the present application. It is obvious that those familiar with the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without having to pay creative labor. Therefore, the present application is not limited to the embodiments herein, and improvements and modifications made by those skilled in the art based on the contents disclosed in the present application without departing from the scope and spirit of the present application are within the scope of the present application.
Claims
1. A video analysis and processing device, characterized in that: include: A coprocessor, a first processor, a second processor and an encoder; The coprocessor is configured to determine a target threshold value corresponding to the target video type according to a correspondence between the target video type and a preset video type and a threshold value; The first processor is configured to calculate the histogram information of the image after format conversion of each video frame among all video frames; all video frames include a current frame and M target reference frames, where M is a positive integer; The coprocessor is further configured to calculate a weight coefficient of the current frame according to histogram information of the current frame and the M target reference frames, and send the weight coefficient to the second processor; The second processor is configured to calculate an unweighted sum of absolute differences according to the histogram information of the current frame and the M target reference frames, and calculate a weighted sum of absolute differences according to the histogram information of the current frame, the M target reference frames and the weighting coefficient; The coprocessor is further configured to obtain a minimum value between the unweighted sum of absolute differences and the weighted sum of absolute differences, and obtain a scene switching detection result according to a comparison result between the minimum value and the target threshold; The encoder is configured to determine a coding frame type of a current frame according to the scene switching detection result.
2. The video analysis and processing device according to claim 1, characterized in that: The coprocessor is further configured to save the calculated histogram information of the current frame and N potential reference frames into a histogram queue in the coprocessor according to a first-in-first-out principle; N is a positive integer; N is greater than or equal to M; The coprocessor is also configured to obtain the number of target reference frames and identification information of the target reference frames according to the encoding parameters and identification information of the current frame, and obtain histogram information of the current frame and M target reference frames from the histogram queue according to the identification information of the current frame and the identification information of the target reference frame.
3. The video analysis and processing device according to claim 1, characterized in that: The histogram information includes a histogram average and a histogram normal mean; The histogram average is calculated using the following formula: HistMean=SumPixel / NumPixels; Wherein, HistMean is the histogram mean, SumPixel is the sum of the pixel values in the current frame, and NumPixels is the number of pixels in the current frame; The normal mean of the histogram is calculated using the following formula: Wherein, HistNormalMean is the normal mean of the histogram, i is the ordinal number of the bar in the histogram, L is the value of the number of bars in the histogram minus 1, Hist i is the number of occurrences of pixel value i.
4. The video analysis and processing device according to claim 1, characterized in that: The weighting coefficient includes an exponential coefficient, a proportional coefficient and an offset; The coprocessor is further configured to calculate a weighting coefficient of the current frame using the following calculation formula according to the histogram information of the current frame and the M target reference frames: Where, denom is the exponential coefficient, scale is the proportional coefficient, offset is the offset, HistNormalMean cur is the normal mean of the histogram of the current frame, HistNormalMean ref is the normal mean of the histogram of the target reference frame, HistMean cur is the histogram average of the current frame, HistMean ref is the histogram average of the target reference frame.
5. The video analysis and processing device according to claim 4, characterized in that: The second processor is further configured to calculate an unweighted sum of absolute differences according to the histogram information of the current frame and the M target reference frames using the following calculation formula: Among them, SAD plane is the unweighted sum of absolute differences, i is the ordinal number of the bar in the histogram, L is the number of bars in the histogram minus 1, CurHist i is the number of occurrences of pixel value i in the current frame, RefHist i is the number of occurrences of pixel value i in the target reference frame; The second processor is further configured to calculate the weighted sum of absolute differences according to the histogram information of the current frame, the M target reference frames and the weighting coefficients using the following calculation formula: Among them, WeightedSAD plane is the weighted sum of absolute differences.
6. The video analysis and processing device according to claim 1, characterized in that: Also including a third processor; The third processor is configured to calculate, for each video frame in all video frames, variance information of the image after format conversion of the video frame, wherein the variance information includes a minimum variance, a maximum variance, an average variance and a variance of each macroblock of the image; The coprocessor is further configured to obtain a correspondence between a variance threshold of a macroblock and an increment of a quantization parameter according to the minimum variance, the maximum variance, the average variance, an increment range of a quantization parameter, and a video type; The encoder is further configured to determine, for each macroblock in the current frame, an increment of the quantization parameter of the macroblock according to the variance of the macroblock and a correspondence between a variance threshold of the macroblock and an increment of the quantization parameter.
7. The video analysis and processing device according to claim 6, characterized in that: The coprocessor is further configured to obtain an initialization table according to the minimum variance, the maximum variance, the average variance, the increment range of the quantization parameter and the video type, wherein the initialization table is used to store the corresponding relationship between the initialization ratio of the variance of the macroblock and the increment of the quantization parameter; The coprocessor is further configured to calculate the variance threshold of each macroblock according to the initialization table by cyclically using the following calculation formula to obtain the corresponding relationship between the variance threshold of the macroblock and the increment of the quantization parameter: theshol_var=(avg_var*s_blaq_threshold_table[j][0]) / 1024; Among them, threshol_var is the variance threshold of the macroblock, avg_var is the average variance of the current frame, s_blaq_threshold_table[j][0] is the first column of the initialization table, and j is the row parameter of the initialization table.
8. The video analysis and processing device according to claim 7, characterized in that: The coprocessor is further configured to determine a corresponding monotonic function according to the video type, and obtain the initialization table according to the minimum variance, the maximum variance, the average variance, the increment range of the quantization parameter, and the monotonic function.
9. The video analysis and processing device according to claim 1 or 2, characterized in that: It also includes memory; The coprocessor is further configured to determine that the weighting coefficient is valid when it is determined that the weighted sum of absolute differences is less than the unweighted sum of absolute differences, and write the first image reference relationship corresponding to the weighting coefficient into the memory; The first image reference relationship includes identification information of the current frame and identification information of the M target reference frames; The encoder is also configured to perform weighted prediction interpolation on pixel values based on the weighting coefficients when it is determined that the first image reference relationship is consistent with a stored second image reference relationship, wherein the second image reference relationship is an image reference relationship determined by the encoder based on input information.
10. A video encoding and decoding system, characterized in that: A device comprising a decoding device and a video analysis and processing device as claimed in any one of claims 1 to 9; The video analysis and processing device is configured to analyze and process the video before compressing the video; The encoder is configured to encode the analysis and processing result of the video based on the analysis and processing device of the video to obtain an encoded bit stream; The decoding device is configured to decode the encoded bitstream to obtain a video before encoding.
