A scene change based intelligent bit rate control optimization method

Abstract

The application discloses a scene change-based intelligent code rate control optimization method, which comprises the following steps: acquiring a video image; performing motion estimation on video coding; performing foreground change detection to obtain a detection result; evaluating a dynamic I frame setting point according to the detection result; and judging scene change according to an internal motion vector of coding and intelligently controlling an I frame interval, so that the code rate in a specific scene can be effectively reduced, and the human eye cannot obviously feel a trailing phenomenon.

Description

Technical Field

[0001] This invention relates to the field of video encoding and decoding technology, and in particular to an intelligent bitrate control optimization method based on scene changes. Background Technology

[0002] In practical applications, video encoding typically has a fixed bitrate ceiling due to network bandwidth limitations. This is especially true in wireless mobile networks, where bitrate constraints are even more stringent, hence the usual bitrate-limited encoding. In video surveillance, real-time performance is critical. Situations often arise where the foreground of the image undergoes significant, highly intense, and prolonged changes. When these changes stabilize, motion blur often appears, sometimes persisting for an extended period.

[0003] The main solutions currently available are as follows:

[0004] a. Increases bandwidth limits; significantly increases hardware costs, resulting in lower cost-effectiveness;

[0005] b. Set the encoding bitrate to variable bitrate to reduce the fixed I-frame interval; this method can eliminate ghosting relatively quickly when bandwidth is sufficient. However, there are two problems. First, even with a fixed I-frame interval, there is still the possibility that I-frames are not refreshed in time when the scene change ends, resulting in ghosting in still images and affecting the visual experience. Second, when there are many scene changes, the increased number of I-frames leads to a further loss of bitrate control, making the ghosting problem more obvious.

[0006] c. Forcibly lowering the upper limit of encoding QP; the code control may not be able to effectively control this, resulting in frame drops in the actual network;

[0007] d. Optimize the code control algorithm; general code control algorithms have many constraints, making it difficult to significantly optimize the effect for this specific scenario. Summary of the Invention

[0008] In view of the technical problem that motion blur always occurs after the image changes stabilize in the current video encoding and decoding technology field, the present invention provides an intelligent bitrate control optimization method based on scene changes. The method judges scene changes based on the motion vector inside the encoding and intelligently controls the I-frame interval, which can effectively reduce the bitrate in specific scenes, and the human eye will not perceive obvious motion blur.

[0009] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0010] A method for intelligent bitrate control optimization based on scene changes, the method comprising the following steps:

[0011] Acquire video images;

[0012] Motion estimation is performed on the video encoding.

[0013] Perform foreground change detection to obtain information on the motion state or positional changes of foreground targets in video images;

[0014] Based on the motion state or position change information of the foreground target, it is evaluated whether the dynamic I-frame setting conditions are met. If they are met, the current frame is set as an I-frame. The dynamic I-frame setting conditions include: the foreground target enters the screen, the foreground target's motion speed exceeds a preset threshold, or the foreground target's position change exceeds a preset range.

[0015] According to one aspect of the present invention, the foreground change detection includes: using inter-frame predicted motion vectors of P frames to determine foreground changes.

[0016] According to one aspect of the present invention, the foreground change detection includes: obtaining the horizontal motion intensity and vertical motion intensity of the current image; when both the horizontal motion intensity and the vertical motion intensity are greater than the strong motion weight, the current image is judged to have strong foreground change; when both the horizontal motion intensity and the vertical motion intensity are less than the weak motion weight, the current image is judged to have weak foreground motion.

[0017] According to one aspect of the invention, the lateral motion intensity is the ratio of the sum of the lateral motion vectors of all blocks in a frame to the number of lateral motion vector blocks in a frame, and the longitudinal motion intensity is the ratio of the sum of the longitudinal motion vectors of all blocks in a frame to the number of longitudinal motion vector blocks in a frame.

[0018] According to one aspect of the invention, the method further includes setting a minimum I-frame interval.

[0019] According to one aspect of the invention, the minimum I-frame interval is set to 25.

[0020] According to one aspect of the present invention, the foreground change detection to obtain the detection result includes: determining whether the current image is in a state of change or tending to be static.

[0021] According to one aspect of the present invention, the evaluation of the dynamic I-frame set point based on the detection result includes the following steps: using P-frame encoding when the image is in a changing state, and using I-frame encoding to refresh when the image is in a static state.

[0022] According to one aspect of the present invention, the evaluation of the dynamic I-frame setting point based on the detection result includes the following steps: the first frame is coded as an I-frame by default; the frame type setting of each subsequent frame is determined based on the coded motion vector information of the previous frame.

[0023] According to one aspect of the present invention, the method includes the following steps:

[0024] Determine whether to set the current frame as an I-frame based on the initial value;

[0025] If the current frame is set as an I-frame, then the cumulative value of the encoded frames after the I-frame is counted to limit the minimum range of the I-frame interval;

[0026] Set a judgment value to determine whether there is strong foreground motion;

[0027] Set a judgment value to determine whether there is a trend of gradual change in the future;

[0028] Set a judgment value to determine whether to set an I-frame in the next frame;

[0029] If the current frame is not set to an I-frame, then the current frame is set to a P-frame.

[0030] Advantages of implementing this invention:

[0031] By judging scene changes based on the internal motion vectors of the encoding and intelligently controlling the I-frame interval, the hardware resource consumption caused by additional scene change detection algorithms can be avoided.

[0032] Adjusting the I-frame interval based on scene changes can effectively reduce the bitrate in specific scenarios (such as when the foreground of the encoded bitstream changes strongly for a long time, the VBR bitrate control cannot suppress it and will cause the encoded frame to be too large and cause frame dropping problems, while the CBR bitrate control will cause serious ghosting). The human eye will not perceive obvious ghosting.

[0033] This method detects the starting point of scene changes and then forces an I-frame refresh as the scene approaches a stop to prevent ghosting. Simultaneously, P-frame encoding is continuously used in subsequent encoding to control the bitrate. The algorithm uses two constraints to distinguish between image change and stillness, accurately detecting the degree of change or stillness to precisely determine the location for setting the I-frame, promptly eliminating ghosting without affecting human visual perception, and ensuring effective bitrate control. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart of an intelligent bitrate control optimization method based on scene changes, as described in this invention.

[0036] Figure 2 This is a schematic diagram of motion estimation according to the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] The relevant scientific terms and references used in this embodiment are explained below:

[0039] (1) Code control (code rate control);

[0040] In video coding, due to the limited bandwidth of the communication channel, it is crucial to control the number of bits output per frame and minimize image distortion within a fixed total bit count. Common methods include adjusting the frame rate, adjusting the image size, and adjusting quantization parameters.

[0041] (2) Code control mode;

[0042] There are typically two types: CBR and VBR. CBR stands for Fixed Bitrate Control, which aims to keep the bitrate at a constant value; VBR stands for Variable Bitrate Control, which allows the bitrate to fluctuate within a certain range, resulting in smoother video quality.

[0043] (3) Frame type;

[0044] I-frames: Can be fully decoded without relying on other frames;

[0045] P-frame: Uses inter-frame prediction, relying on preceding I-frames and P-frames;

[0046] B-frames: B-frames are bidirectional difference frames, meaning they record the differences between the current frame and the preceding and following frames, relying on I / P frames. Decoding them is more complex and introduces latency, making them generally unsuitable for high-real-time scenarios.

[0047] (4) Scene transformation;

[0048] The scene includes foreground and background.

[0049] Foreground: A moving object exists within a fixed shot;

[0050] Background: Camera movement causes overall changes in the image, or when the camera is stationary, other areas besides the main subject of the video experience changes.

[0051] This embodiment optimizes the motion blur phenomenon caused by significant changes in the foreground within a fixed background.

[0052] (5) Inter-frame prediction;

[0053] Inter-frame prediction involves two important concepts: motion estimation and motion compensation. Motion estimation involves finding the best corresponding block in the already encoded image for the currently coded block and calculating the offset (i.e., motion vector) of that corresponding block. Let the current frame be P and the reference frame be P'. r The current coded block is B, in P r The process of finding the best matching block with B is called motion estimation. Figure 2 As shown, B * The position of B in the image's scale is the same as that of B. The motion vector (MV) is B. r Top left corner coordinate (x r y r Subtract B * The coordinates of the top left corner (x, y) are equal to (x r -x, y r -y). B r The block is the reference block of block B, B r The pixel value is used as the predicted value for the pixels in block B. Due to the large temporal redundancy in video, the residual value to be encoded will be very small if the reference block of the reference frame is selected appropriately. Then, the motion vectors are encoded into the bitstream, and the decoder can use this information to decode the original image. Motion compensation is the process of obtaining the estimated value of the current frame based on the decoded motion vectors and the inter-frame prediction method. Motion estimation and motion compensation techniques are key methods for eliminating temporal redundancy in moving images.

[0054] like Figure 1 As shown, an intelligent bitrate control optimization method based on scene changes is proposed, the method comprising the following steps:

[0055] Step S1: Acquire video images;

[0056] Video images are acquired through video surveillance equipment, such as surveillance cameras.

[0057] Step S2: Perform motion estimation on the video code;

[0058] Motion estimation is performed on video coding through inter-frame prediction.

[0059] Step S3: Perform foreground change detection and obtain the detection results;

[0060] The foreground change detection includes using inter-frame predicted motion vectors of P-frames to determine foreground changes.

[0061] The horizontal and vertical motion intensities of the current image are obtained. When both horizontal and vertical motion intensities are greater than the strong motion weight, the foreground of the current image is considered to have strong motion. When both horizontal and vertical motion intensities are less than the weak motion weight, the foreground of the current image is considered to have weak motion. The horizontal motion intensity is the ratio of the sum of the horizontal motion vectors of all blocks in a frame to the number of horizontal motion vector blocks in a frame, and the vertical motion intensity is the ratio of the sum of the vertical motion vectors of all blocks in a frame to the number of vertical motion vector blocks in a frame.

[0062] In this embodiment, a minimum I-frame interval is set, which is 25.

[0063] Step S4: Evaluate the dynamic I-frame setpoint based on the detection results.

[0064] Determine whether the current image is in a state of flux or tending towards stillness.

[0065] P-frame encoding is used when the image is moving, and I-frame encoding is used when the image is nearing stillness. The first frame is encoded as an I-frame by default; the frame type of each subsequent frame is determined based on the encoded motion vector information of the previous frame.

[0066] In this embodiment, the specific steps include:

[0067] Determine whether to set the current frame as an I-frame based on the initial value;

[0068] If the current frame is set as an I-frame, then the cumulative value of the encoded frames after the I-frame is counted to limit the minimum range of the I-frame interval;

[0069] Set a judgment value to determine whether there is strong foreground motion;

[0070] Set a judgment value to determine whether there is a trend of gradual change in the future;

[0071] Set a judgment value to determine whether to set an I-frame in the next frame;

[0072] If the current frame is not set to an I-frame, then the current frame is set to a P-frame.

[0073] In practical applications, the following specific embodiments are provided:

[0074] (1) Regions with high motion changes in a video frame have larger MV values ​​than regions with slow movement or stillness. The magnitude of the motion vector in the predictive coding macroblock has a certain proportional relationship with the intensity of motion changes in the foreground of the video frame. Based on this rule, a method for calculating the average motion intensity of the video foreground is abstracted:

[0075] Lateral movement intensity: AverageMove x= Sum of horizontal MVs of all blocks in a frame / Number of horizontal MV blocks in a frame;

[0076] Vertical motion intensity: AverageMove y = Sum of horizontal MVs of all blocks in a frame / Number of vertical MV blocks in a frame;

[0077] When AverageMove x > StrongMoveThreshold and,

[0078] AverageMove y > StrongMoveThreshold, it is considered that the foreground motion is strong;

[0079] When AverageMove x < WeakMoveThreshold and,

[0080] AverageMove y < WeakMoveThreshold, it is considered that the foreground motion is weak and close to no obvious motion;

[0081] Experimental results show that better effects can be obtained when the strong motion weight StrongMoveThreshold = 2 and the weak motion weight WeakMoveThreshold = 1.

[0082] (2) Minimum I-frame interval limit;

[0083] To avoid excessive bandwidth consumption due to a large number of I-frames in a short period of time, there needs to be a minimum interval range for I-frames. At the same time, to prevent mosaic afterimages caused by a large number of image changes, tests have found that the human eye does not have an obvious sensory visual experience of mosaics when there is a large amount of motion in images within 1s. We set the minimum I-frame interval time to 1s; in the field of video surveillance, the frame rate is generally set to 25 or 30 fps, so the minimum I-frame interval is set to 25;

[0084] (3) Evaluate the dynamic I-frame setting point according to the foreground change detection algorithm;

[0085] Since P-frames can eliminate a large amount of redundancy through inter-frame prediction, we use P-frame coding when the motion is relatively intense, which can well control the bit rate. At the same time, to improve the human eye visual experience as much as possible, I-frames need to be inserted in a timely manner when the motion tends to be gentle to eliminate artifacts such as smear caused by insufficient instantaneous bit rate;

[0086] (4) The default encoding frame type of the first frame is I-frame. The frame type setting of each subsequent frame is determined according to the foreground change detection algorithm and the dynamic I-frame algorithm of the encoding MV information of the previous frame. The specific steps are as follows:

[0087] 1. Set initial values:

[0088] a) FrameCount = 0; / / Counts the cumulative value of encoded frames after the I-frame, used to limit the minimum range of the I-frame interval to 30.

[0089] b) Set SceneChangeFlag = 0; / / Whether there is strong foreground motion.

[0090] c) Set ChangeFinishFlag = 0; / / Whether the foreground is strong and the motion tends to be gentle.

[0091] d) Set SetKeyFrameFlag = 0; / / Whether to set an I-frame for the next frame.

[0092] 2. Algorithm processing steps per frame:

[0093] a) Before encoding begins:

[0094] i. If SetKeyFrameFlag == 1

[0095] 1. Set the current frame to an I-frame;

[0096] 2. FrameCount = 0; / / Counts the cumulative value of encoded frames after the I-frame, used to limit the minimum range of the I-frame interval to 30.

[0097] 3. Set SceneChangeFlag = 0; / / Whether to use a scene with strong foreground movement.

[0098] 4. Set ChangeFinishFlag = 0; / / Whether the foreground is strong and the motion tends to be smooth.

[0099] 5. Set SetKeyFrameFlag = 0; / / Whether to set the next frame as an I-frame. Otherwise, set the current frame as a P-frame, and FrameCount += 1.

[0100] b) After the encoded MV is generated (I-frame MV = 0):

[0101] i. If SceneChangeFlag == 1, according to the foreground change detection algorithm, detect whether the foreground motion of the image tends to be smooth. If the foreground change tends to be smooth, set ChangeFinishFlag to 1;

[0102] ii. If SceneChangeFlag == 0, detect whether there is strong motion in the image according to the foreground change detection algorithm. If there is strong motion, set SceneChangeFlag = 1;

[0103] iii. If SceneChangeFlag==1, ChangeFinishFlag=1, and FrameCount>=25, set SetKeyFrameFlag=1.

[0104] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0105] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0106] Similarly, it should be understood that, in order to simplify this disclosure and aid in understanding one or more of the various aspects of the invention, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof. However, this method of disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, wherein each claim itself is a separate embodiment of the invention.

[0107] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0108] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0109] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0110] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for intelligent bitrate control optimization based on scene changes, characterized in that, The method includes the following steps: Acquire video images and obtain the horizontal and vertical motion intensity of the current image; Motion estimation is performed on the video encoding. Foreground change detection is performed to obtain detection results. Based on the horizontal motion intensity and vertical motion intensity, they are compared with a strong motion weight and a weak motion weight, respectively. When both the horizontal motion intensity and vertical motion intensity are greater than the strong motion weight, it is determined that the foreground change of the current image is strong; when both are less than the weak motion weight, it is determined that the foreground motion of the current image is weak. The dynamic I-frame setting point is evaluated based on the detection results. If it is determined that the foreground of the current image is changing strongly, P-frame encoding is used when the image is in a changing state; if it is determined that the foreground of the current image is moving weakly, I-frame encoding is used to refresh the image when it is in a relatively static state. The method also includes setting a minimum I-frame interval.

2. The intelligent bitrate control optimization method based on scene changes according to claim 1, characterized in that, The foreground change detection includes using inter-frame predicted motion vectors of P-frames to determine foreground changes.

3. The intelligent bitrate control optimization method based on scene changes according to claim 1, characterized in that, The lateral motion intensity is the ratio of the sum of the lateral motion vectors of all blocks in a frame to the number of lateral motion vector blocks in a frame, and the longitudinal motion intensity is the ratio of the sum of the longitudinal motion vectors of all blocks in a frame to the number of longitudinal motion vector blocks in a frame.

4. The intelligent bitrate control optimization method based on scene changes according to claim 1, characterized in that, The minimum I-frame interval is set to 25.

5. The intelligent bitrate control optimization method based on scene changes according to claim 1, characterized in that, The process of evaluating the dynamic I-frame setting point based on the detection results includes the following steps: the first frame is encoded as an I-frame by default; the frame type setting for each subsequent frame is determined based on the encoded motion vector information of the previous frame.

6. The intelligent bitrate control optimization method based on scene changes according to claim 5, characterized in that, The method includes the following steps: Determine whether to set the current frame as an I-frame based on the initial value; If the current frame is set as an I-frame, then the cumulative value of the encoded frames after the I-frame is counted to limit the minimum range of the I-frame interval; Set a judgment value to determine whether there is strong foreground motion; Set a judgment value to determine whether there is a trend of gradual change in the future; Set a judgment value to determine whether to set an I-frame in the next frame; If the current frame is not set to an I-frame, then the current frame is set to a P-frame.

Images