Detection frame de-jittering method based on optimization algorithm

Abstract

The present application discloses a detection frame de-jittering method based on an optimization algorithm, and the detection frame de-jittering method comprises the following steps: inputting a video, detecting: performing object detection on each frame of the input video, tracking: using the SORT algorithm to track the detection frame, extracting all detection frames of a certain ID, where k represents the frame number, and using the Levenburg‑Marquardt algorithm to solve the minimization problem. The application is beneficial in that the second-order difference operator is used as a constraint to ensure the smoothness of the detection frames of adjacent frames. In each iteration of the optimization, the constructed matrix is ​​sparse, and the sparse algorithm is used to solve the linear equation with extremely high efficiency, which fully meets the real-time requirements. Compared with other methods, this method has good robustness. No matter how drastically the trajectory of the object changes, it can eliminate jitter well while ensuring that it will not deviate too far from the original frame.

Description

Technical Field

[0001] The present application relates to the field of detection frame de-jittering, and in particular to a detection frame de-jittering method based on an optimization algorithm. Background Art

[0002] When detecting objects in each frame of a video stream, since each frame is detected independently, for the same object, there is often a large deviation between the size and position of the detection frame in two adjacent frames. This is manifested as violent jitters of the detection frame in the entire video stream, which gives the user a poor visual experience. The detection frame trajectory needs to be stabilized to improve the user's visual experience.

[0003] The existing technical solutions mainly include Kalman filtering, low-pass filtering, and sliding window smoothing. Kalman filtering is widely used, but since the state transfer equation is difficult to model the change law of the actual state variables, and the Gaussian noise parameters are difficult to accurately determine, its smoothing effect is limited, and there is still obvious noise residue in the final result. The low-pass filtering method is simple in principle and mature in technology. However, it needs to convert the signal to the frequency domain, which is inefficient and prone to Gibbs effect, resulting in a large deviation between the smoothed detection frame and the actual trajectory. The sliding window smoothing method is simple in principle and easy to implement. However, different trajectories are highly dependent on the size parameters of the sliding window. In practical applications, it is difficult to adaptively give appropriate sliding window size parameters. Summary of the invention

[0004] In this embodiment, a detection frame de-jittering method based on an optimization algorithm is provided to solve the problem that the Kalman filter method is widely used in the prior art, but since the state transfer equation is difficult to model the change law of the actual state variables and the Gaussian noise parameters are difficult to determine accurately, its smoothing effect is limited and there is still obvious noise residual in the final result.

[0005] According to one aspect of the present application, a detection frame de-jittering method based on an optimization algorithm is provided, and the detection frame de-jittering method comprises the following steps:

[0006] (1) Input the video;

[0007] (2) Detection: Perform object detection on each frame of the input video;

[0008] (3) Tracking: Use the SORT algorithm to track the detection box;

[0009] (4) Extract all detection boxes for a certain ID Where k represents the frame number;

[0010] (5) The Levenburg-Marquardt algorithm is used to solve the minimization problem.

[0011] Furthermore, in step (2), object detection is performed on each frame of the input video to obtain a detection frame of the object of interest.

[0012] Furthermore, in step (2), each frame may obtain detection frames of multiple different objects.

[0013] Furthermore, in step (3), the SORT algorithm is used to track the detection frame, associate the detection frames of the same object in adjacent frames, and assign a detection frame ID.

[0014] Furthermore, in step (3), the detection frame of the same object finally has the same ID in the entire video stream.

[0015] Furthermore, in step (4), all detection frames of a certain ID are extracted, where k represents the frame number. Parameterized as the horizontal and vertical coordinates of the upper left and lower right corners of the detection box.

[0016] Furthermore, in step (4), For each component of , solve the minimization problem:

[0017]

[0018] st Dx=0

[0019] Where x is the detection box r of all frames k A vector composed of some components of .

[0020] Furthermore, the smoothed detection frame {r k The following two points must be met:

[0021] 1. Ensure that the result detection box r is in each frame k Compared with the original detection box The deviation is not large: that is, for the kth frame, Small enough;

[0022] 2. The position and size of the detection frame of two adjacent frames should not change much and should be smooth on the time axis. k The second-order derivative of each component with respect to time is approximately zero.

[0023] Furthermore, according to the requirements of 1 and 2, a minimization problem with linear constraints is designed: Under the condition of satisfying Dx = 0, minimize

[0024] in:

[0025] Where D is the second-order difference operator, which is equivalent to the second-order derivative in the continuous case. This constraint ensures the smoothness of the vector x, that is, it is smooth on the time axis; while minimizing This ensures that the deviation between the result and the original detection frame is as small as possible.

[0026] Furthermore, in step (5), the Levenburg-Marquardt algorithm is used to solve the minimization problem and obtain a smoothed result. k The above operation is performed on each component in to obtain the final smoothed result.

[0027] Furthermore, the final result x k satisfy:

[0028] Very small, ensuring that it does not deviate much from the original frame

[0029] Dx≈0, ensuring the smoothness between the detection frames of two adjacent frames.

[0030] Through the above embodiments of the present application, the smoothness of the detection frames of adjacent frames is guaranteed by using the second-order difference operator as a constraint. In each iteration of the optimization, the constructed matrix is ​​sparse, and the efficiency of solving linear equations using sparse algorithms is extremely high, which fully meets the real-time requirements. Compared with other methods, this method has good robustness. No matter how drastically the trajectory of the object changes, it can eliminate jitter well while ensuring that it will not deviate too far from the original frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1 This is the flow chart of this application. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] In the present application, the terms "upper", "lower", "left", "right", "front", "back", "top", "bottom", "inner", "outer", "middle", "vertical", "horizontal", "lateral", "longitudinal" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings. These terms are mainly used to better describe the present application and its embodiments, and are not used to limit the indicated devices, elements or components to have a specific orientation, or to be constructed and operated in a specific orientation.

[0036] In addition, some of the above terms may be used to express other meanings in addition to indicating orientation or positional relationship. For example, the term "on" may also be used to express a certain dependency or connection relationship in some cases. For those of ordinary skill in the art, the specific meanings of these terms in this application can be understood according to specific circumstances.

[0037] In addition, the terms "installed", "set", "provided with", "connected", "connected", and "socketed" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral structure; it can be a mechanical connection, or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be an internal connection between two devices, elements, or components. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0038] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0039] Example 1

[0040] See also Figure 1 As shown, a detection frame de-jittering method based on an optimization algorithm comprises the following steps:

[0041] (1) Input the video;

[0042] (2) Detection: Perform object detection on each frame of the input video;

[0043] (3) Tracking: Use the SORT algorithm to track the detection box;

[0044] (4) Extract all detection boxes for a certain ID Where k represents the frame number;

[0045] (5) The Levenburg-Marquardt algorithm is used to solve the minimization problem.

[0046] Furthermore, in step (2), object detection is performed on each frame of the input video to obtain a detection frame of the object of interest.

[0047] Furthermore, in step (2), each frame may obtain detection frames of multiple different objects.

[0048] Furthermore, in step (3), the SORT algorithm is used to track the detection frame, associate the detection frames of the same object in adjacent frames, and assign a detection frame ID.

[0049] Furthermore, in step (3), the detection frame of the same object finally has the same ID in the entire video stream.

[0050] Furthermore, in step (4), all detection frames of a certain ID are extracted, where k represents the frame number. Parameterized as the horizontal and vertical coordinates of the upper left and lower right corners of the detection box.

[0051] Furthermore, in step (4), For each component of , solve the minimization problem:

[0052]

[0053] s·t·Dx=0

[0054] Where x is the detection box r of all frames k A vector composed of some components of .

[0055] Furthermore, the smoothed detection frame {r k The following two points must be met:

[0056] 1. Ensure that the result detection box r is in each frame k Compared with the original detection box The deviation is not large: that is, for the kth frame, Small enough;

[0057] 2. The position and size of the detection frame of two adjacent frames should not change much and should be smooth on the time axis. k The second-order derivative of each component with respect to time is approximately zero.

[0058] Furthermore, according to the requirements of 1 and 2, a minimization problem with linear constraints is designed: Under the condition of satisfying Dx = 0, minimize

[0059] in:

[0060] Where D is the second-order difference operator, which is equivalent to the second-order derivative in the continuous case. This constraint ensures the smoothness of the vector x, that is, it is smooth on the time axis; while minimizing This ensures that the deviation between the result and the original detection frame is as small as possible.

[0061] Furthermore, in step (5), the Levenburg-Marquardt algorithm is used to solve the minimization problem and obtain a smoothed result. k The above operation is performed on each component in to obtain the final smoothing result.

[0062] Furthermore, the final result x k satisfy:

[0063] Very small, ensuring that it does not deviate much from the original frame

[0064] Bx≈0 ensures the smoothness between the detection frames of two adjacent frames.

[0065] The above method is applicable to a detection frame de-jittering method which is manifested as severe jitter of the detection frame in the video stream, which gives a poor visual experience to the user.

[0066] The benefits of this application are:

[0067] The second-order difference operator is used as a constraint to ensure the smoothness of the detection frame of adjacent frames. In each optimization iteration, the constructed matrix is ​​sparse. The sparse algorithm is highly efficient in solving linear equations and fully meets the real-time requirements. Compared with other methods, this method is very robust. No matter how drastically the object trajectory changes, it can eliminate jitter well while ensuring that it does not deviate too far from the original frame.

[0068] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A detection frame de-jittering method based on an optimization algorithm, characterized in that: The detection frame de-jittering method comprises the following steps: (1) Input the video; (2) Detection: Perform object detection on each frame of the input video; (3) Tracking: Use the SORT algorithm to track the detection box; (4) Extract all detection boxes of a certain ID Where k represents the frame number; (5) Using Levenburg-Marquardt algorithm to solve; In the step (3), the SORT algorithm is used to track the detection frame, associate the detection frames of the same object in adjacent frames, and assign a detection frame ID; In step (3), the detection frame of the same object has the same ID in the entire video stream; In the step (4), For each component of , solve the minimization problem: Where x is the detection box r of all frames k The smoothed detection box {r k The following two points must be met: a. Ensure that the result detection frame r is in each frame k Compared with the original detection box The deviation is not large: that is, for the kth frame, Small enough; b. The position and size of the detection frame of two adjacent frames should not change much and should be smooth on the time axis. Here, r k The second-order derivative of each component with respect to time is approximately zero; According to the requirements of points a and b, a minimization problem with linear constraints is designed: Under the condition of satisfying Dx = 0, minimize in: Where D is the second-order difference operator, which is equivalent to the second-order derivative in the continuous case. This constraint ensures the smoothness of the vector x, that is, it is smooth on the time axis; while minimizing This ensures that the deviation between the result and the original detection frame is as small as possible.

2. The detection frame de-jittering method based on the optimization algorithm according to claim 1, characterized in that: In the step (2), object detection is performed on each frame of the input video to obtain a detection frame of the object of interest.

3. The detection frame de-jittering method based on the optimization algorithm according to claim 1, characterized in that: In step (2), each frame will obtain detection frames of multiple different objects.

4. The detection frame de-jittering method based on the optimization algorithm according to claim 1, characterized in that: In step (4), all detection frames of a certain ID are extracted, where k represents the frame number; detection frame Parameterized as the horizontal and vertical coordinates of the upper left and lower right corners of the detection box.

5. The detection frame de-jittering method based on the optimization algorithm according to claim 1, characterized in that: In the step (5), the Levenburg-Marquardt algorithm is used to solve and obtain a smoothed result; k The above Levenburg-Marquardt operation is performed on each component in to obtain the final smoothed result.

6. The detection frame de-jittering method based on the optimization algorithm according to claim 5, characterized in that: The x k satisfy: Very small, ensuring that it does not deviate much from the original frame; Dx≈0, ensuring the smoothness between the detection frames of two adjacent frames.

Images