A method, apparatus and related medium for video image jitter detection and stabilization.

Abstract

This invention discloses a method, apparatus, and related medium for video image jitter detection and stabilization. The method includes: acquiring a target video image to be jitter detected; performing semantic segmentation on the target video image using deep learning-based image semantic segmentation technology to obtain the region of interest (ROI); extracting key points within the ROI using a corner detection algorithm; calculating the optical flow map of any two consecutive frames of the target video image using optical flow method, and obtaining the displacement information of the key points in the two consecutive frames based on the optical flow map; calculating the image translation matrix and image rotation matrix based on the displacement information to obtain jitter information; comparing the jitter information with a preset jitter threshold; and if jitter occurs in the current frame image, performing a reverse translation and / or rotation on the current frame image based on the jitter information to cancel the jitter. This invention combines deep learning semantic segmentation technology and optical flow method, which can improve the accuracy of jitter detection and achieve good jitter cancellation effect.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus and related medium for detecting and stabilizing video image jitter. Background Technology

[0002] In recent years, an increasing number of surveillance cameras have been deployed in public places, playing a vital role in security, traffic management, and regional monitoring when combined with video AI technology. However, some cameras, due to their installation at heights and exposure to wind, or in locations prone to vibration such as subways and bridges, often exhibit shaky footage, which is more pronounced in telephoto mode. This shakiness affects video viewing and observation of the monitored scene, and also significantly impacts applications requiring precise target location tracking, such as trajectory tracking and intrusion detection. Camera shake can have a substantial impact on subsequent applications. Summary of the Invention

[0003] This invention provides a method, apparatus, computer device, and storage medium for video image jitter detection and stabilization, aiming to improve the accuracy of image jitter detection and further reduce image jitter.

[0004] In a first aspect, embodiments of the present invention provide a video image jitter detection and stabilization method, including:

[0005] The target video image to be detected is acquired, and the target video image is semantically segmented using deep learning-based image semantic segmentation technology to obtain the region of interest of the target video image.

[0006] Key points in the region of interest are extracted using a corner detection algorithm;

[0007] The optical flow diagram of any two consecutive frames in the target video image is calculated using the optical flow method, and the displacement information of the key point in the two consecutive frames is obtained based on the optical flow diagram.

[0008] The image translation matrix and image rotation matrix are calculated based on the displacement information, and the jitter information of the current frame image relative to the previous frame image is obtained.

[0009] The jitter information is compared with a preset jitter threshold, and the jitter information is used to determine whether the current frame image is jittering.

[0010] If the current frame image is jittered, the current frame image is translated and / or rotated in reverse according to the jitter information to counteract the jitter.

[0011] Secondly, embodiments of the present invention provide a video image jitter detection and stabilization device, comprising:

[0012] The semantic segmentation unit semantically acquires the target video image to be jitter detected, and uses deep learning-based image semantic segmentation technology to perform semantic segmentation on the target video image to obtain the region of interest of the target video image;

[0013] The key point extraction unit extracts key points in the region of interest using a corner detection algorithm.

[0014] The displacement information acquisition unit semantically calculates the optical flow map of any two consecutive frames in the target video image using the optical flow method, and obtains the displacement information of the key point in the two consecutive frames based on the optical flow map.

[0015] The jitter information acquisition unit calculates the image translation matrix and image rotation matrix based on the displacement information, and obtains the jitter information of the current frame image relative to the previous frame image.

[0016] The jitter determination unit semantically compares the jitter information with a preset jitter threshold, and determines whether the current frame image is jittering based on the comparison result;

[0017] A jitter cancellation unit is used to perform reverse translation and / or rotation on the current frame image according to the jitter information if jitter occurs in the current frame image, so as to cancel the jitter.

[0018] Thirdly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video image jitter detection and stabilization method as described in the first aspect.

[0019] Fourthly, embodiments of the present invention provide a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the video image jitter detection and stabilization method as described in the first aspect.

[0020] This invention provides a method, apparatus, computer device, and storage medium for video image jitter detection and stabilization. The method includes: acquiring a target video image to be jitter detected, and performing semantic segmentation on the target video image using a deep learning-based image semantic segmentation technique to obtain a region of interest (ROI); extracting key points in the ROI using a corner detection algorithm; calculating an optical flow graph for any two consecutive frames of the target video image using an optical flow method, and obtaining displacement information of the key points in the two consecutive frames based on the optical flow graph; calculating an image translation matrix and an image rotation matrix based on the displacement information, and obtaining jitter information of the current frame image relative to the previous frame image; comparing the jitter information with a preset jitter threshold, and determining whether the current frame image is jittered based on the comparison result; if the current frame image is jittered, performing a reverse translation and / or rotation on the current frame image based on the jitter information to counteract the jitter. This invention combines deep learning semantic segmentation technology to extract high-quality scene key points, and uses optical flow to track these key points. Based on the displacement information of the key points, the jitter information of the image is calculated, thereby achieving the function of jitter detection and cancellation. The method provided by this invention has fast calculation speed, high jitter detection accuracy, and good jitter cancellation effect, and can be applied to a variety of scenarios. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a video image jitter detection and stabilization method according to an embodiment of the present invention;

[0023] Figure 2 This invention provides a schematic diagram of a sub-process of a video image jitter detection and stabilization method according to an embodiment of the present invention;

[0024] Figure 3 A schematic block diagram of a video image jitter detection and stabilization device is provided in this embodiment of the invention;

[0025] Figure 4 This invention provides a schematic block diagram of a video image jitter detection and stabilization device. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0028] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0029] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0030] Please see below. Figure 1 , Figure 1 The flowchart of a video image jitter detection and stabilization method provided in an embodiment of the present invention specifically includes steps S101 to S106.

[0031] S101. Obtain the target video image to be subjected to jitter detection, and use deep learning-based image semantic segmentation technology to perform semantic segmentation on the target video image to obtain the region of interest of the target video image.

[0032] S102. Extract key points in the region of interest using a corner detection algorithm;

[0033] S103. Calculate the optical flow map of any two consecutive frames in the target video image using the optical flow method, and obtain the displacement information of the key point in the two consecutive frames based on the optical flow map;

[0034] S104. Calculate the image translation matrix and image rotation matrix based on the displacement information, and obtain the jitter information of the current frame image relative to the previous frame image;

[0035] S105. Compare the jitter information with a preset jitter threshold, and determine whether the current frame image is jittering based on the comparison result;

[0036] S106. If the current frame image is jittered, the current frame image is translated and / or rotated in the reverse direction according to the jitter information to counteract the jitter.

[0037] In this embodiment, firstly, semantic segmentation of the acquired target video image is performed using deep learning-based image semantic segmentation technology to obtain regions of interest (ROIs) containing key points. Then, key points within these ROIs are extracted using a corner detection algorithm. Furthermore, optical flow is used to calculate the displacement information of the key points over two consecutive frames. Subsequently, the image translation and rotation matrices of the target video image are calculated based on this displacement information, thereby obtaining the jitter information of the current frame image. Based on this jitter information, it can be determined whether jitter has occurred in the current frame image, and if jitter is determined to have occurred, it can be counteracted through a reverse operation.

[0038] This invention combines deep learning semantic segmentation technology to extract high-quality key points in surveillance scenes, uses optical flow to track these key points, and calculates the translation and rotation of the image based on the displacement of the key points, thereby achieving the function of jitter detection and cancellation. The method provided in this embodiment has a fast calculation speed, good jitter detection and cancellation effect, and can be applied to a variety of scenarios, especially security scenarios where video surveillance is carried out through cameras or other cameras.

[0039] In one embodiment, step S101 includes:

[0040] The labeled training images are used to train the deep learning-based image semantic segmentation technology to construct a deep learning-based semantic segmentation model.

[0041] The target video image is semantically segmented using the deep learning-based semantic segmentation model.

[0042] The semantic segmentation results are filtered and selected according to the preset feature regions, and the selected results are used as the regions of interest.

[0043] In this embodiment, deep learning-based image semantic segmentation technology is used to perform semantic segmentation on target video images to identify different types of objects such as sky, roads, trees, and buildings. To obtain a high-precision image semantic segmentation model, a sufficient number of training images, such as traffic scene images, are first collected. Regions representing the sky, roads, trees, and vehicles in the training images are manually labeled and segmented. Then, a deep learning-based semantic segmentation model is constructed. The input to this model is a traffic scene image that has undergone preprocessing operations such as normalization and scaling, and the output is a two-dimensional image of the same size as the input. The value of each pixel in the image is the category number of the corresponding pixel in the input image. Finally, manually labeled images with segmentation annotations are used to train the semantic segmentation model, improving its prediction accuracy. Furthermore, to ensure the quality of keypoint extraction, after the above training is completed, the segmentation results output by the deep learning-based semantic segmentation model are filtered. For example, regions lacking image texture and color features, such as the sky and roads, are filtered out, as are moving targets such as vehicles and trees. Finally, regions with obvious corner points, texture, and color features, such as buildings and signs, are selected as regions of interest.

[0044] In one embodiment, step S102 includes:

[0045] A sliding window is set in the region of interest using the Harris corner detection algorithm;

[0046] Calculate the average intensity change of each pixel in the sliding window in a specified direction;

[0047] The average intensity change is compared with a predetermined intensity threshold, and when the average intensity change is greater than the predetermined intensity threshold, the pixel corresponding to the average intensity change is taken as the key point.

[0048] In this embodiment, key points are extracted using the Harris corner detection algorithm. Of course, in other embodiments, algorithms such as Shi-Tomasi corner detection and FAST corner detection can also be used. Taking Harris corner detection as an example, a small window (i.e., the sliding window) is first placed on the image, and the average intensity change of each pixel in the sliding window in certain predetermined directions is calculated. If the average intensity change in all directions is greater than a predetermined intensity threshold, the corresponding pixel can be determined as a key point. Preferably, to avoid camera watermarks, lens dirt, etc., being detected as key points or key points in low-quality image areas lacking stable features being extracted, which could lead to calculation deviations in subsequent steps, key points can be extracted only from the target video image within the region of interest, or the region outside the region of interest can be filled with black before extraction.

[0049] In one embodiment, step S103 includes:

[0050] The sparse optical flow method is used to calculate the optical flow map of key points in two consecutive frames of images, so as to directly obtain the displacement information of the key points;

[0051] Alternatively, the dense optical flow method can be used to calculate the optical flow map of two consecutive frames, and the displacement information of the key points can be extracted from the optical flow of the two consecutive frames.

[0052] The displacement information includes the displacement direction and pixel distance value of the key point.

[0053] In this embodiment, optical flow maps are calculated using the optical flow method for two consecutive image frames. From the optical flow map, the direction and distance of movement of each pixel in the previous frame relative to the corresponding pixel in the next frame (i.e., the current frame image) can be obtained, thus yielding the coordinates of the keypoint's displacement relative to the first moment in the current frame. The calculation of the optical flow map can be performed using either sparse optical flow methods (including but not limited to LK optical flow) to calculate only the optical flow at the keypoints, or dense optical flow methods (including but not limited to Farneback optical flow) to calculate the optical flow of the entire image, and then taking the displacement direction and pixel distance values ​​of the keypoints.

[0054] In one embodiment, step S104 includes:

[0055] The image translation matrix is ​​calculated using the averaging method according to the following formula:

[0056]

[0057] In the formula, Δ represents the image translation matrix. This represents the x-coordinate of the i-th keypoint in the previous frame. This represents the ordinate of the i-th keypoint in the previous frame. This represents the x-coordinate of the i-th keypoint in the current frame image. This represents the x-coordinate of the i-th keypoint in the current frame image, where i represents the i-th keypoint and n represents the total number of keypoints;

[0058] The image rotation matrix is ​​calculated using the averaging method according to the following formula:

[0059]

[0060] In the formula, R(θ) represents the image rotation matrix, and θ represents the average rotation angle.

[0061] This embodiment calculates the translation and rotation of the current frame image relative to the previous frame image based on the displacement of key points in the image, using either the least squares method, the averaging method, or the reprojection error minimization method, thus obtaining the image translation and rotation matrices. Assuming a total of n key points are extracted, and the coordinates of the i-th key point in the previous frame image are... The coordinates in the current frame image are This embodiment uses the averaging method as an example to provide the following formulas for calculating the image translation matrix and the image rotation matrix:

[0062]

[0063]

[0064] Wherein, Δ represents the image translation matrix, which can also be understood as the translation information of the current frame image relative to the previous frame image, and R(θ) represents the image rotation matrix, which can also be understood as the rotation information of the current frame image relative to the previous frame image.

[0065] In one embodiment, the jitter information includes displacement jitter information and rotational jitter information, wherein the displacement jitter information includes displacement component jitter information;

[0066] Step S105 includes:

[0067] The displacement jitter information, displacement component jitter information, and rotational jitter information are compared with their respective preset jitter thresholds.

[0068] If the displacement jitter information, displacement component jitter information, and / or rotation jitter information are higher than a preset jitter threshold, then it is determined that the current frame image is jittering.

[0069] If the displacement jitter information, displacement component jitter information, and rotation jitter information are all below the preset jitter threshold, then it is determined that the current frame image is not jittering.

[0070] Step S106 includes:

[0071] Perform reverse translation and / or rotation on all pixels in the current frame image; or

[0072] Perform reverse translation and / or rotation on key points in the current frame image.

[0073] This embodiment sets a jitter threshold based on the displacement and rotation of pixels in the current frame image. This jitter threshold can correspond to the displacement jitter information, displacement component jitter information, and rotation jitter information included in the jitter information. Here, the displacement component jitter information can be understood as the displacement information of a pixel in a certain direction in space. When the displacement jitter information, displacement component jitter information, or rotation jitter information is greater than the preset jitter threshold, the current frame image is determined to be in a jitter state; otherwise, the current frame image is determined to be in a stable state. When in a jitter state, this embodiment uses two methods to achieve jitter cancellation: if jitter cancellation and stabilization of the image are to be performed, all pixels in the current frame image can be translated and rotated in reverse according to the jitter information; if jitter cancellation is to be performed on targets of interest in the image (such as vehicles, pedestrians, landmarks, etc. whose positions need to be obtained after jitter removal), the position calculation result can be translated and rotated separately when calculating the specific target position. Of course, in some application scenarios, the influence of rotation can be ignored in order to reduce calculation time.

[0074] In one embodiment, such as Figure 2 As shown, the video image jitter detection and stabilization method further includes steps S201 to S204.

[0075] S201. Extract SIFT feature points from any two consecutive frames of the target video image, and record the key point positions and descriptors corresponding to the SIFT feature points;

[0076] S202. The SIFT feature points in two consecutive frames of images are matched using a feature point matching algorithm, and jitter correction information is calculated for the key points of the successfully matched SIFT feature points.

[0077] S203. Compare the jitter correction information with the jitter information;

[0078] S204. If the result of the difference comparison exceeds the preset difference threshold, the key points are recalculated to perform timed correction on the key points.

[0079] In this embodiment, to prevent error accumulation or calculation deviation, key points can be corrected periodically. Normally, key points only need to be extracted once, but in this embodiment, to prevent key point drift, loss, or occlusion, key points are corrected periodically. Specifically, new key points can be periodically extracted using the same method as described above for jitter detection calculation. Alternatively, a time-consuming but more accurate feature point matching algorithm can be used. This algorithm calculates the image translation and rotation based on the displacement of feature points to obtain the jitter correction information, and then compares it with the jitter information obtained through optical flow. If the difference between the two exceeds a preset difference threshold, the key points are recalculated. For example, SIFT feature points can be extracted from two consecutive frames of images, and the corresponding keypoint positions and descriptors (the descriptor of a SIFT feature point is usually a 128-dimensional vector) can be recorded. All SIFT feature points in the two consecutive frames can be matched according to the Euclidean distance of the descriptors (the closer the distance, the more similar; a distance less than a specified threshold indicates a successful match). Then, the displacement and rotation of the jitter are calculated based on the keypoint positions of the successfully matched feature points. This is compared with the jitter information calculated by the optical flow method to determine whether the keypoints need to be recalculated. It should be noted that the scope of keypoints described in this embodiment is broad. Generally speaking, any point of interest can be called a keypoint, while feature points generally refer to points with specific attributes. Feature points generally have position (i.e., keypoint, emphasizing positional attributes) and descriptive information (i.e., descriptor, which is usually a multi-dimensional vector).

[0080] Figure 3 This is a schematic block diagram of a video image jitter detection and stabilization device 300 provided in an embodiment of the present invention. The device 300 includes:

[0081] The semantic segmentation unit 301 semantically acquires the target video image to be jitter detected, and uses deep learning-based image semantic segmentation technology to perform semantic segmentation on the target video image to obtain the region of interest of the target video image;

[0082] Key point extraction unit 302 extracts key points in the region of interest using a corner detection algorithm;

[0083] The displacement information acquisition unit 303 calculates the optical flow map of any two consecutive frames in the target video image using the optical flow method, and obtains the displacement information of the key point in the two consecutive frames based on the optical flow map.

[0084] The jitter information acquisition unit 304 calculates the image translation matrix and image rotation matrix based on the displacement information, and obtains the jitter information of the current frame image relative to the previous frame image.

[0085] The jitter determination unit 305 compares the jitter information with a preset jitter threshold and determines whether the current frame image is jittering based on the comparison result.

[0086] The jitter cancellation unit 306 is used to perform reverse translation and / or rotation on the current frame image according to the jitter information if jitter occurs in the current frame image, so as to cancel the jitter.

[0087] In one embodiment, the semantic segmentation unit 301 includes:

[0088] The model training unit is used to acquire labeled training images to train the deep learning-based image semantic segmentation technology in order to construct a deep learning-based semantic segmentation model.

[0089] An image segmentation unit is used to perform semantic segmentation on the target video image using the deep learning-based semantic segmentation model;

[0090] The filtering unit is used to filter the semantic segmentation results according to the preset feature regions, and to use the filtered results as the region of interest.

[0091] In one embodiment, the key point extraction unit 302 includes:

[0092] A window setting unit is used to set a sliding window in the region of interest using the Harris corner detection algorithm;

[0093] An average value calculation unit is used to calculate the average intensity change of each pixel in the sliding window in a specified direction;

[0094] The key point selection unit is used to compare the average intensity change with a predetermined intensity threshold, and when the average intensity change is greater than the predetermined intensity threshold, to select the pixel corresponding to the average intensity change as the key point.

[0095] In one embodiment, the displacement information acquisition unit 303 includes:

[0096] The first optical flow map calculation unit is used to calculate the optical flow map of key points in two consecutive frames of images using the sparse optical flow method, so as to directly obtain the displacement information of the key points.

[0097] The second optical flow graph calculation unit is used to calculate the optical flow graph of two consecutive frames of images or to use the dense optical flow method, and to extract the displacement information of the key points in the optical flow of the two consecutive frames of images.

[0098] The displacement information includes the displacement direction and pixel distance value of the key point.

[0099] In one embodiment, the jitter information acquisition unit 304 includes:

[0100] The translation matrix calculation unit is used to calculate the image translation matrix using the averaging method according to the following formula:

[0101]

[0102] In the formula, Δ represents the image translation matrix. This represents the x-coordinate of the i-th keypoint in the previous frame. This represents the ordinate of the i-th keypoint in the previous frame. This represents the x-coordinate of the i-th keypoint in the current frame image. This represents the x-coordinate of the i-th keypoint in the current frame image, where i represents the i-th keypoint and n represents the total number of keypoints;

[0103] A rotation matrix calculation unit is used to calculate the image rotation matrix using an averaging method according to the following formula:

[0104]

[0105] In the formula, R(θ) represents the image rotation matrix, and θ represents the average rotation angle.

[0106] In one embodiment, the jitter information includes displacement jitter information and rotational jitter information, wherein the displacement jitter information includes displacement component jitter information;

[0107] The jitter determination unit 305 includes:

[0108] The information comparison unit is used to compare the displacement jitter information, displacement component jitter information and rotation jitter information with the corresponding preset jitter thresholds respectively;

[0109] The first determination unit is used to determine that the current frame image is jittering if the displacement jitter information, displacement component jitter information and / or rotation jitter information are higher than a preset jitter threshold.

[0110] The second determination unit is used to determine that the current frame image does not jitter if the displacement jitter information, displacement component jitter information and rotation jitter information are all not higher than a preset jitter threshold.

[0111] The jitter cancellation unit 306 includes:

[0112] The first inverse unit is used to perform inverse translation and / or rotation on all pixels in the current frame image; or

[0113] The second reverse unit is used to reverse translate and / or rotate key points in the current frame image.

[0114] In one embodiment, such as Figure 4 As shown, the video image jitter detection and stabilization device 300 further includes:

[0115] The feature point extraction unit 401 is used to extract SIFT feature points from any two consecutive frames of the target video image and record the key point positions and descriptors corresponding to the SIFT feature points.

[0116] The feature point matching unit 402 is used to match SIFT feature points in two consecutive frames of images using a feature point matching algorithm, and to calculate jitter correction information for the key points of the successfully matched SIFT feature points.

[0117] The difference comparison unit 403 is used to compare the jitter correction information with the jitter information.

[0118] The key point correction unit 404 is used to recalculate the key points if the result of the difference comparison exceeds the preset difference threshold, so as to perform timed correction on the key points.

[0119] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0120] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed, can perform the steps provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0121] This invention also provides a computer device, which may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps provided in the above embodiments. Of course, the computer device may also include various network interfaces, power supplies, and other components.

[0122] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0123] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method of video image shake detection and stabilization, the method comprising: receiving a video image; determining a shake vector for the video image; and stabilizing the video image based on the shake vector. The method comprises the following steps: acquiring a target video image to be subjected to jitter detection, and performing semantic segmentation on the target video image by using an image semantic segmentation technology based on deep learning to acquire a region of interest of the target video image; extracting key points in the region of interest by using a corner detection algorithm; calculating an optical flow map of any two continuous frames of the target video image by using an optical flow method, and acquiring displacement information of the key points in the two continuous frames according to the optical flow map; calculating an image translation matrix and an image rotation matrix according to the displacement information, and obtaining jitter information of a current frame of image relative to a previous frame of image; comparing the jitter information with a preset jitter threshold, and judging whether the current frame of image is subjected to jitter according to a comparison result; if the current frame of image is subjected to jitter, performing inverse translation and / or rotation on the current frame of image according to the jitter information to offset the jitter. The method further comprises the following steps: extracting SIFT feature points from any two continuous frames of the target video image, and recording positions of the key points corresponding to the SIFT feature points and descriptors of the SIFT feature points; matching the SIFT feature points in the two continuous frames by using a feature point matching algorithm, and calculating jitter correction information of the key points of the SIFT feature points that are successfully matched; differentially comparing the jitter correction information with the jitter information; if a differential comparison result exceeds a preset differential threshold, re-calculating the key points to perform timing correction on the key points.

2. The method of video image shake detection and stabilization of claim 1, wherein, The step of acquiring a target video image to be subjected to jitter detection, and performing semantic segmentation on the target video image by using an image semantic segmentation technology based on deep learning to acquire a region of interest of the target video image comprises the following steps: training the image semantic segmentation technology based on deep learning by using labeled training images to construct a semantic segmentation model based on deep learning; performing semantic segmentation on the target video image by using the semantic segmentation model based on deep learning; filtering and screening the semantic segmentation result according to preset feature regions, and taking a screening result as the region of interest.

3. The method of video image shake detection and stabilization of claim 1, wherein, The step of extracting key points in the region of interest by using a corner detection algorithm comprises the following steps: setting a sliding window in the region of interest by using a Harris corner detection algorithm; calculating average values of intensity changes of each pixel point in the sliding window in a specified direction; comparing the average values of intensity changes with a predetermined intensity threshold, and taking a pixel point corresponding to the average value of intensity changes as the key point when the average value of intensity changes is greater than the predetermined intensity threshold.

4. The method of video image shake detection and stabilization of claim 1, wherein, The step of calculating an optical flow map of any two continuous frames of the target video image by using an optical flow method, and acquiring displacement information of the key points in the two continuous frames according to the optical flow map comprises the following steps: calculating the optical flow map of the key points in the two continuous frames by using a sparse optical flow method to directly acquire the displacement information of the key points; or calculating the optical flow map of the two continuous frames by using a dense optical flow method, and extracting the displacement information of the key points in the optical flow of the two continuous frames; wherein the displacement information comprises a displacement direction and a pixel distance value of the key points.

5. The method of video image shake detection and stabilization of claim 1, wherein, The image translation matrix and the image rotation matrix are calculated according to the displacement information, and jitter information of a current frame image relative to a previous frame image is obtained. The image translation matrix is calculated by using an average method according to the following formula: ； wherein, denotes the image translation matrix, denotes the horizontal coordinate of the i-th key point in the previous frame image, denotes the vertical coordinate of the i-th key point in the previous frame image, denotes the horizontal coordinate of the i-th key point in the current frame image, denotes the horizontal coordinate of the i-th key point in the current frame image, i denotes the i-th key point, and n denotes the total number of key points. The image rotation matrix is calculated by using an average method according to the following formula: ； wherein denotes the image rotation matrix, denotes the average rotation angle, wherein .

6. A video image jitter detection and stabilization apparatus, characterized by, The method comprises the following steps: A semantic segmentation unit acquires a target video image to be subjected to jitter detection, and performs semantic segmentation on the target video image by using an image semantic segmentation technology based on deep learning, so as to acquire a region of interest of the target video image; A key point extraction unit acquires key points in the region of interest by using a corner point detection algorithm; A displacement information acquisition unit acquires displacement information of the key points in two continuous frames of the target video image by calculating an optical flow map of the two continuous frames of the target video image by using an optical flow method; A jitter information acquisition unit calculates an image translation matrix and an image rotation matrix according to the displacement information, and obtains jitter information of a current frame image relative to a previous frame image; A jitter judgment unit compares the jitter information with a preset jitter threshold, and judges whether the current frame image is subjected to jitter according to a comparison result; A jitter offset unit offsets jitter by performing reverse translation and / or rotation on the current frame image according to the jitter information if the current frame image is subjected to jitter. The method further comprises the following steps: A feature point extraction unit extracts SIFT feature points from any two continuous frames of the target video image, and records positions of the key points corresponding to the SIFT feature points and descriptors of the SIFT feature points; A feature point matching unit matches the SIFT feature points in the two continuous frames by using a feature point matching algorithm, and calculates jitter correction information of the key points of the SIFT feature points that are successfully matched; A difference comparison unit compares the jitter correction information with the jitter information; A key point correction unit re-calculates the key points to correct the key points in time if a result of the difference comparison exceeds a preset difference threshold.

7. A computer device, comprising: The computer program is stored on the computer readable storage medium and is executed by the processor to implement the video image jitter detection and stabilization method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium and is executed by the processor to implement the video image jitter detection and stabilization method according to any one of claims 1 to 5.

Images