A multi-scene video jitter detection method based on video surveillance

Abstract

The present invention relates to a multi-scene video jitter detection method based on video surveillance, comprising: acquiring images of two adjacent frames of a conference monitoring screen; searching and matching feature points of the acquired two frames of images; if the number of matched feature points is less than a set threshold, it is judged that the venue is switched, and no screen jitter detection is performed at this time; if the number of matched feature points is greater than the set threshold, it is judged that the venue is not switched, and the row and column projections of the two frames of images are calculated respectively, and then the row and column projections are respectively correlated, and the extreme value in the correlation value curve is taken as the displacement between the two frames, and the vector absolute value is the displacement size; when the displacement distance is greater than the set threshold, it is judged that the video has jittered. The present invention can accurately and efficiently determine whether the video screen has jittered for video conferencing. The present invention can adapt to multi-scene monitoring scenarios, avoid the misjudgment of jitter detection caused by scene conversion, have a small amount of calculation, good detection effect, and a wider range of application scenarios.

Description

Technical Field

[0001] The invention relates to a multi-scene video jitter detection method based on video monitoring, and belongs to the technical field of computer vision. Background Art

[0002] The monitoring system has become an indispensable part of our daily life. Its importance in banks, shopping malls, companies, schools, residential areas, public transportation and other fields is self-evident. With the construction of cities and the development of companies, the monitoring system plays an increasingly important role. Especially with the outbreak of the COVID-19 pandemic in recent years, in order to meet the needs of epidemic prevention and avoid large gatherings of people in different regions, the application of multi-venue monitoring systems has become more and more extensive. The multi-venue monitoring system refers to the monitoring images of multiple branch venues being transmitted to the main monitoring image in turn to meet the requirements of polling in each venue. In order to ensure the quality of the monitoring video, a series of problem detections need to be performed on the polling video, and video jitter detection is a very critical part of it.

[0003] Chinese patent document CN106385580B discloses a video jitter detection method based on image grayscale distribution characteristics, which mentions a video picture jitter detection method, the method is: step 1: intercept two adjacent frames of images in the video; step 2: convert the intercepted two frames of images into grayscale space; step 3: count the grayscale value of each row and each column of the previous frame of the image, calculate the row grayscale mean and row grayscale variance value and the column grayscale mean and column grayscale variance value; step 4: count the grayscale value of each row and each column of the next frame of the image, calculate the row grayscale mean and row grayscale variance value and the column grayscale mean and column grayscale variance value; step Step 5: Perform a row hypothesis test on the row grayscale mean and row grayscale variance of the two frames before and after calculated in step 3 and step 4 to obtain a row test factor; Step 6: Perform a column hypothesis test on the column grayscale mean and column grayscale variance of the two frames before and after calculated in step 3 and step 4 to obtain a column test factor; Step 7: Compare the row test factor and column test factor calculated in step 5 and step 6 with the given threshold respectively. If one of them exceeds the threshold, the previous frame is determined to be a jitter frame; Step 8: Count the proportion of jitter frames in the entire video. If it exceeds the set jitter threshold, the video is determined to be jittered. This method mainly uses the grayscale features of the image. By judging the row and column grayscale mean and variance of the two frames before and after, it determines whether the video screen is jittered. It is characterized by small calculation amount and good real-time performance. However, this method has high limitations and is not applicable to the current multi-venue monitoring scene. The scene switching of the screen will cause the grayscale features of the image to change dramatically, which directly leads to misjudgment of the algorithm. Summary of the invention

[0004] In order to overcome the deficiencies in the prior art, the present invention proposes a multi-scene video jitter detection method based on video surveillance to solve the problem that jitter detection is currently mostly applied to a single scene.

[0005] The technical solution of the present invention is:

[0006] A multi-venue video jitter detection method based on video surveillance includes the following steps:

[0007] Step 1: Detect feature points of the two acquired images using the Shi-Tomasi corner detection method;

[0008] Step 2: Determine whether a venue switch has occurred by matching corner points; that is, if the number of matched corner points is less than the set threshold, it is determined that the venue has been switched, and no image jitter detection is performed at this time; if the number of matched corner points is greater than the set threshold, it is determined that the venue has not been switched, and the process proceeds to step 3;

[0009] Step 3: Determine whether the image is jittering by the grayscale features of the image; that is: calculate the row and column projections of the two frames of the image respectively, then perform correlation operations on the row and column projections respectively, take the extreme value in the correlation value curve as the displacement between the two frames, and the absolute value of the vector is the displacement size; when the displacement distance is greater than the set threshold, it is determined that the video is jittering.

[0010] Preferably, according to the present invention, the specific implementation steps of step 1 include:

[0011] Step 1.1: In a Cartesian coordinate system with the upper left corner of the captured video frame image as the origin, the x-axis to the right, and the y-axis downward, when the window moves in both the x and y directions, calculate the pixel value change E(u, v) inside the window;

[0012] Step 1.2: For each window, calculate the corresponding corner point response function R;

[0013] Step 1.3 sets a threshold value and performs a threshold judgment on the calculated corner point response function. If R>threshold, it means that the window corresponds to a corner point feature and the pixel corresponds to a corner point. Otherwise, ignore this pixel point and detect the next pixel point.

[0014] Further preferably, the specific implementation steps of step 1.1 include:

[0015] Let the center of a window be located at any position (x, y) of the grayscale image of any frame of the video acquired by the video surveillance. The grayscale value of the pixel at this position is I(x, y). If the window moves a small displacement u and v in the x and y directions respectively to a new position (x+u, y+v), the grayscale value of the pixel at this position is I(x+u, y+v). I(x+u, y+v)-I(x, y) refers to the change in the grayscale value caused by the movement of the window.

[0016] Define ω(x, y) as the window function at position (x, y) to represent the weight of each pixel in the window, and set the weight of all pixels in the window to 1; or, set ω(x, y) to a Gaussian distribution with the center of the window as the origin; if the pixel at the center of the window is a corner point, the weight coefficient of the center of the window is set to 1, indicating that the center of the window contributes more to the grayscale change; and the farther away from the center of the window, i.e. the corner point, the smaller the grayscale change of these points, and the weight coefficient is set to a smaller value, set to 01, and the farther away from the corner point, the closer the weight coefficient is to 0, indicating that these points contribute less to the grayscale change;

[0017] The formula for obtaining the pixel value change E(u, v) inside the window is shown in formula (I):

[0018]

[0019] In formula (I), I x and I y is the partial differential of I, which is the gradient map in the x and y directions in the image.

[0020] The matrix M is:

[0021] The right side of the arrow is the result of diagonalizing the real symmetric matrix. R is the corner point response function, which does not affect the change components in the two orthogonal directions.

[0022] After diagonalization, the change components in two orthogonal directions are extracted, which is λ 1 and λ 2 ,λ 1 and λ 2 are the two eigenvalues ​​of the matrix M.

[0023] Further preferably, the specific implementation steps of step 1.2 include:

[0024] Directly use the smaller eigenvalue as the corner point response function R, as shown in formula (II):

[0025] R=min(λ 1 ,λ 2 ) (II)

[0026] In formula (II), λ 1 and λ 2 is the eigenvalue of the matrix M.

[0027] Further preferably, in step 1.3, threshold=15.

[0028] Preferably, according to the present invention, the specific implementation steps of step 2 include:

[0029] Intercept two adjacent frames of the video, the previous frame is defined as the reference frame, and the next frame is defined as the current frame. Take a corner point of the reference frame image, and find the two corner points in the current frame image that are closest to the reference frame feature point in Euclidean distance. Among these two corner points, if the ratio of the Euclidean distance of the corner point closest to the reference frame corner point divided by the Euclidean distance of the second closest corner point is less than the threshold T, then the two corner points are matched successfully, otherwise, the two corner points are not matched successfully;

[0030] Repeat this operation until all corner points detected in step 1 are matched, and T is set to 0.40.6;

[0031] If the matching degree of all corner points of the two frames reaches more than 90%, then the venue switching does not occur; otherwise, it is determined that the venue switching occurs.

[0032] Further preferably, the Euclidean distance ρ of the current frame corner point to the reference frame corner point is 1 The formula for obtaining is shown in formula (III):

[0033]

[0034] In formula (III), x 1 ,y 1 are the row and column coordinates of the reference frame corner point, x 2 ,y 2 The row and column coordinates of the corresponding corner point in the current frame;

[0035] The Euclidean distance ρ between the current frame corner and the reference frame corner 2 The formula for obtaining is shown in formula (IV):

[0036]

[0037] In formula (IV), x 3 ,y 3 The row and column coordinates of the corner point of the current frame that is next closest to the corner point of the reference frame;

[0038] The formula for calculating the ratio is shown in formula (V):

[0039]

[0040] Preferably, according to the present invention, the specific implementation steps of step 3 include:

[0041] Step 3.1: Calculate the total pixel value of each row of the image Row k (i), as shown in formula (VI):

[0042]

[0043] In formula (VI), k represents the k-th frame image, i represents the i-th row, and m represents the number of columns of the image;

[0044] Step 3.2: Calculate the average gray value Row of the entire image k , as shown in formula (VII):

[0045] Row k =[∑Row k (i))] / n (VII)

[0046] In formula (VII), n represents the total number of rows in the image;

[0047] Step 3.3: Use the total pixel value of each row of the image Row k (i) Subtract the average grayscale value of the image row k , get the row correction value Rowproject for the projection value of the k-th frame image k (i), as shown in formula (VIII):

[0048] Rowproject k (i) = Row k (i)-Row k (VIII)

[0049] Step 3.4: Calculate the total pixel value Col for each column of the image k (j), as shown in formula (IX):

[0050]

[0051] In formula (IX), k represents the k-th frame image, j represents the j-th row, and n represents the total number of rows in the image;

[0052] Step 3.5: Calculate the column average gray value Col of the entire image k , as shown in formula (X):

[0053] Col k =[∑Col k (j))] / m(X)

[0054] In formula (X), m represents the total number of columns of the image;

[0055] Step 3.6: Use the total pixel value Col of each column of the image k (j) Subtract the column average gray value Col of the image k , get the column correction value Colproject for the projection value of the k-th frame image k (j), as shown in formula (XI):

[0056] Colproject k (j) = Col k (j)-Col k (XI)

[0057] Using the row and column projection value correction values ​​calculated in step 3.3 and step 3.6, draw the row and column grayscale projection curves of the image;

[0058] Step 3.7: After calculating the row and column grayscale projection curves of the current frame and the reference frame respectively, perform cross-correlation operations on the row and column grayscale projection curves of the current frame and the reference frame respectively, take the extreme value in the correlation value curve as the displacement of the image frame, take the absolute value of its vector as the displacement size, and the cross-correlation operation calculation formula is shown in Formula (XII) and Formula (XIII):

[0059]

[0060]

[0061] In formula (XII) and formula (XIII), R x (w) and R y (v) respectively represent the calculation formulas for the relevant operations on the processed row and column projections, Col pre (j) is the grayscale projection value of the jth column of the current frame image, Col ref (j) is the grayscale projection value of the jth column of the reference frame, p and q are the search lengths of the current frame relative to the reference frame on one side of the image;

[0062] w and v are within the range of values, so that R x (w) and R y The minimum value of (v) is defined as w min and v min , then the horizontal and vertical displacement vectors of the current frame relative to the reference frame are shown in equations (XIV) and (XV) respectively:

[0063] d x =m+1-w min (XIV)

[0064] d y =n+1-v min (XV)

[0065] In formula (XIV) and formula (XV), d x Indicates the horizontal displacement of the current frame compared to the reference frame, where d y Indicates the vertical displacement of the current frame compared to the reference frame;

[0066] When x or y The value is greater than the set threshold T 1 When the video image is shaken, it is determined that the video image is shaken; otherwise, it is determined that the video image is not shaken.

[0067] Further preferably, according to different applicable scenarios, T 1 Take 5-30.

[0068] The beneficial effects of the present invention are:

[0069] The present invention provides a multi-scene video jitter detection method based on video surveillance. The method can effectively identify whether a video screen has been switched through corner point matching and detection, thereby being able to adapt to multi-scene monitoring scenarios, avoiding misjudgment of jitter detection due to scene switching, and then determining whether the video screen has jittered through the grayscale features of the image. This invention improves the traditional single-scene video jitter detection mode, making it more widely applicable, more anti-interference capable, and with better detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a flow chart of a multi-venue video jitter detection method based on video monitoring of the present invention;

[0071] Figure 2 This is a schematic diagram of corner point matching when the video scene does not switch;

[0072] Figure 3 This is a schematic diagram of corner point matching when the video scene switches;

[0073] Figure 4(a) shows a video scene. Figure 1 ;

[0074] FIG4( b ) is a schematic diagram of a column projection of the image of FIG4( a );

[0075] FIG4( c ) is a schematic diagram of a line projection of the image of FIG4( a );

[0076] Figure 5(a) is a schematic diagram of a video scene Figure 2 ;

[0077] FIG5( b ) is a schematic diagram of a column projection of the image of FIG5( a );

[0078] FIG5(c) is a schematic diagram of a line projection of the image in FIG5(a). DETAILED DESCRIPTION

[0079] The present invention will be further described below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0080] Example 1

[0081] A multi-venue video jitter detection method based on video surveillance detects feature points through the Shi-Tomasi corner point detection method, then determines whether the venue switching occurs through feature point matching, and finally determines whether the image jitters through the grayscale features of the image. Figure 1 As shown, the steps include:

[0082] Step 1: Detect feature points (corner points) of the two acquired frames using the Shi-Tomasi corner point detection method;

[0083] Step 2: Determine whether a venue switch has occurred by matching corner points; that is, if the number of matched corner points is less than the set threshold, it is determined that the venue has been switched, and no image jitter detection is performed at this time; if the number of matched corner points is greater than the set threshold, it is determined that the venue has not been switched, and the process proceeds to step 3;

[0084] Step 3: Determine whether the image is jittering by the grayscale features of the image; that is: calculate the row and column projections of the two frames of the image respectively, then perform correlation operations on the row and column projections respectively, take the extreme value in the correlation value curve as the displacement between the two frames, and the absolute value of the vector is the displacement size; when the displacement distance is greater than the set threshold, it is determined that the video is jittering.

[0085] Example 2

[0086] The difference between the multi-venue video jitter detection method based on video surveillance described in Example 1 is that:

[0087] When a window moves in the smooth area of ​​the image, the image grayscale does not change; when the window moves in the direction of the edge, the image grayscale does not change; when the window moves at the corner point, it will cause a significant change in the image grayscale. Shi-Tomas corner point detection uses this intuitive physical phenomenon to determine whether it is a corner point by the degree of change of the window in various directions. The specific implementation steps of step 1 include:

[0088] Step 1.1: In a Cartesian rectangular coordinate system with the upper left corner of the captured video frame image as the origin, the x-axis to the right, and the y-axis downward, when the window moves in both the x and y directions, calculate the pixel value change E(u, v) inside the window; specifically:

[0089] Let the center of a window be located at any position (x, y) of the grayscale image of any frame of the video acquired by the video surveillance. The grayscale value of the pixel at this position is I(x, y). If the window moves a small displacement u and v in the x and y directions respectively to a new position (x+u, y+v), the grayscale value of the pixel at this position is I(x+u, y+v). I(x+u, y+v)-I(x, y) refers to the change in the grayscale value caused by the movement of the window.

[0090] Define ω(x, y) as the window function at position (x, y) to represent the weight of each pixel in the window, and set the weight of all pixels in the window to 1; or, set ω(x, y) to a Gaussian distribution (bivariate normal distribution) with the center of the window as the origin; if the pixel at the center of the window is a corner point, then the grayscale value of the center of the window changes very strongly before and after the window moves, so the weight coefficient of the center of the window is set to 1, indicating that the center of the window contributes more to the grayscale change; and the farther away from the center of the window, that is, the corner point, the smaller the grayscale change of these points, the weight coefficient is set smaller, set to 01, and the farther away from the corner point, the closer the weight coefficient is to 0, indicating that these points contribute less to the grayscale change;

[0091] The formula for the change in pixel grayscale value caused by the window moving in each direction (u, v) is as follows:

[0092]

[0093] For a corner point, E(u, v) will be very large. Therefore, the above function can be maximized to get the corner point in the image. Using the above function to calculate E(u, v) will be very slow. Therefore, the Taylor expansion (only first order) can be used to get an approximate form of this formula.

[0094] The Taylor expansion formula for two dimensions is:

[0095] T(x,y)≈f(u,v)+(xu)f x (u,v)+(yv)f y (u,v)+…

[0096] Applying the above formula to I(u+x, y+v) yields:

[0097] I(x+u,y+v)≈I(x,y)+uI x +vI y

[0098] Among them I x and I y is the partial differential of I, which is the gradient map in the x and y directions in the image.

[0099]

[0100] Next, continue to deduce:

[0101]

[0102] Take u and v out and get the final form: The formula for calculating the pixel value change E(u, v) inside the window is shown in formula (I):

[0103]

[0104] In formula (I), I x and I y is the partial differential of I, which is the gradient map in the x and y directions in the image.

[0105] The matrix M is:

[0106] The right side of the arrow is the result of diagonalizing the real symmetric matrix. R is the corner point response function, which does not affect the change components in the two orthogonal directions.

[0107] After diagonalization, the change components in two orthogonal directions are extracted, which is λ 1 and λ 2 ,λ 1 and λ 2 are the two eigenvalues ​​of the matrix M;

[0108] Step 1.2: For each window, calculate the corresponding corner point response function R;

[0109] The final form of E(u, v) has been obtained by the derivation of step 1.1. Next, we need to use the eigenvalue to find the windows that will cause large changes in grayscale values. The stability of the corner point is related to the smaller eigenvalue of the matrix M, so the smaller eigenvalue is directly used as the corner point response function R, as shown in formula (II):

[0110] R=min(λ 1 ,λ 2 ) (II)

[0111] In formula (II), λ 1 and λ 2 is the eigenvalue of the matrix M;

[0112] Step 1.3: The result of Shi-Tomas corner detection is a grayscale image with a corner response function R. Set a threshold (this method uses threshold = 15) and perform a threshold judgment on the calculated corner response function. If R>threshold, it means that the window corresponds to a corner feature and the pixel corresponds to a corner. Otherwise, ignore this pixel and detect the next pixel.

[0113] The specific implementation steps of step 2 include:

[0114] Intercept two adjacent frames of the video, the previous frame is defined as the reference frame, and the next frame is defined as the current frame. Take a corner point of the reference frame image, and find the two corner points in the current frame image that are closest to the reference frame feature point in Euclidean distance. Among these two corner points, if the ratio of the Euclidean distance of the corner point closest to the reference frame corner point divided by the Euclidean distance of the second closest corner point is less than the threshold T, then the two corner points are matched successfully, otherwise, the two corner points are not matched successfully;

[0115] Repeat this operation until all corner points detected in step 1 are matched, and T is set to 0.40.6;

[0116] If the matching degree of all corner points of the two frames reaches more than 90%, then the venue switching does not occur; otherwise, it is determined that the venue switching occurs.

[0117] The Euclidean distance ρ between the current frame corner and the reference frame corner 1 The formula for obtaining is shown in formula (III):

[0118]

[0119] In formula (III), x 1 ,y 1 are the row and column coordinates of the reference frame corner point, x 2 ,y 2 The row and column coordinates of the corresponding corner point in the current frame;

[0120] The Euclidean distance ρ between the current frame corner and the reference frame corner 2 The formula for obtaining is shown in formula (IV):

[0121]

[0122] In formula (IV), x 3 ,y 3 The row and column coordinates of the corner point of the current frame that is next closest to the corner point of the reference frame;

[0123] The formula for calculating the ratio is shown in formula (V):

[0124]

[0125] The most notable feature of video jitter is that there will be an overall displacement between frames. After the displacement is detected, further logic is used to determine whether the video is jittering. Therefore, basically the jitter of the video is centered around how to detect this displacement. The present invention calculates the row and column displacement based on the grayscale features of the video screen image, and then determines whether the video screen is jittering. The specific implementation steps of step 3 include:

[0126] Step 3.1: Calculate the total pixel value of each row of the image Row k (i), as shown in formula (VI):

[0127]

[0128] In formula (VI), k represents the k-th frame image, i represents the i-th row, and m represents the number of columns of the image;

[0129] Step 3.2: Calculate the average gray value Row of the entire image k , as shown in formula (VII):

[0130] Row k =[∑Row k (i))] / n (VII)

[0131] In formula (VII), n represents the total number of rows in the image;

[0132] Step 3.3: Use the total pixel value of each row of the image Row k (i) Subtract the average grayscale value of the image row k , get the row correction value Rowproject for the projection value of the k-th frame image k (i), as shown in formula (VIII):

[0133] Rowproject k (i) = Row k (i)-Row k (VIII)

[0134] Step 3.4: Calculate the total pixel value Col for each column of the image k (j), as shown in formula (IX):

[0135]

[0136] In formula (IX), k represents the k-th frame image, j represents the j-th row, and n represents the total number of rows in the image;

[0137] Step 3.5: Calculate the column average gray value Col of the entire image k, as shown in formula (X):

[0138] Col k =[∑Col k (j))] / m (X)

[0139] In formula (X), m represents the total number of columns of the image;

[0140] Step 3.6: Use the total pixel value Col of each column of the image k (j) Subtract the column average gray value Col of the image k , get the column correction value Colproject for the projection value of the k-th frame image k (j), as shown in formula (XI):

[0141] Colproject k (j) = Col k (j)-Col k (XI)

[0142] Using the row and column projection value correction values ​​calculated in step 3.3 and step 3.6, draw the row and column grayscale projection curves of the image;

[0143] Step 3.7: After calculating the row and column grayscale projection curves of the current frame and the reference frame respectively, perform cross-correlation operations on the row and column grayscale projection curves of the current frame and the reference frame respectively, take the extreme value in the correlation value curve as the displacement of the image frame, take the absolute value of its vector as the displacement size, and the cross-correlation operation calculation formula is shown in Formula (XII) and Formula (XIII):

[0144]

[0145]

[0146] In formula (XII) and formula (XIII), R x Ry(w) and Ry(v) represent the calculation formulas for the related operations on the processed row and column projections, respectively. pre (j) is the grayscale projection value of the jth column of the current frame image, Col ref (j) is the grayscale projection value of the jth column of the reference frame, p and q are the search lengths of the current frame relative to the reference frame on one side of the image;

[0147] w and v are within the range of values, so that R x (w) and R y The minimum value of (v) is defined as w min and v min , then the horizontal and vertical displacement vectors of the current frame relative to the reference frame are shown in equations (XIV) and (XV) respectively:

[0148] d x =m+1-w min (XIV)

[0149] d y =n+1-v min (XV)

[0150] In formula (XIV) and formula (XV), d x Indicates the horizontal displacement of the current frame compared to the reference frame, where d y Indicates the vertical displacement of the current frame compared to the reference frame;

[0151] When x or y The value is greater than the set threshold T 1 When the video image is shaken, it is determined that the video image is shaken, otherwise, it is determined that the video image is not shaken. According to different applicable scenarios, T 1 Take 5-30.

[0152] Figure 2 This is a schematic diagram of corner point matching when the video scene does not switch; Figure 3 is a schematic diagram of corner point matching when the video scene switches; Figure 2 , Figure 3 It can be seen that when the monitoring scene has not changed, the corner points of the two frames of images captured from the video are basically completely matched, while when the monitoring scene has changed, the corner points of the two frames of images captured from the video are only partially matched. The degree of matching of the corner points of the two frames of images is used to determine whether the video scene has switched.

[0153] Figure 4(a) shows a video scene. Figure 1 ; FIG4(b) is a schematic diagram of the column projection of the image of FIG4(a); FIG4(c) is a schematic diagram of the row projection of the image of FIG4(a); FIG5(a) is a schematic diagram of a video scene Figure 2 ; Figure 5(b) is a schematic diagram of the column projection of the image in Figure 5(a); Figure 5(c) is a schematic diagram of the row projection of the image in Figure 5(a). After the row and column projection diagrams of the image are obtained, the row and column projections are respectively correlated using the formula in step 3, and the extreme value in the correlation value curve is taken as the displacement between the two frames, and the absolute value of the vector is the displacement size. When the displacement distance is greater than the set threshold, it is determined that the video is jittering.

Claims

1. A multi-venue video jitter detection method based on video surveillance, characterized in that: The steps include: Step 1: Detect feature points of the two acquired images using the Shi-Tomasi corner detection method; Step 2: Determine whether a venue switch has occurred by matching corner points; that is, if the number of matched corner points is less than the set threshold, it is determined that the venue has been switched, and no image jitter detection is performed at this time; if the number of matched corner points is greater than the set threshold, it is determined that the venue has not been switched, and the process proceeds to step 3; Step 3: Determine whether the image is jittering by the grayscale features of the image; that is, respectively calculate the row and column projections of the two frames of the image, then perform correlation operations on the row and column projections, take the extreme value in the correlation value curve as the displacement between the two frames, and the absolute value of the vector is the displacement size; when the displacement distance is greater than the set threshold, it is determined that the video is jittering; The specific implementation steps of step 3 include: Step 3.1: Calculate the total pixel value of each row of the image Row k (i), as shown in formula (VI): In formula (VI), k represents the k-th frame image, i represents the i-th row, and m represents the number of columns of the image; Step 3.2: Calculate the average gray value Row of the entire image k , as shown in formula (VII): Row k =[∑Row k (i))] / n (VII) In formula (VII), n represents the total number of rows in the image; Step 3.3: Use the total pixel value of each row of the image Row k (i) Subtract the average grayscale value of the image row k , get the row correction value Rowproject for the projection value of the k-th frame image k (i), as shown in formula (VIII): Rowproject k (i)=Row k (i)-Row k (VIII) Step 3.4: Calculate the total pixel value Col for each column of the image k (j), as shown in formula (IX): In formula (IX), k represents the k-th frame image, j represents the j-th row, and n represents the total number of rows in the image; Step 3.5: Calculate the column average gray value Col of the entire image k , as shown in formula (X): Col k =[∑Col k (j))] / m (X) In formula (X), m represents the total number of columns of the image; Step 3.6: Use the total pixel value Col of each column of the image k (j) Subtract the column average gray value Col of the image k , get the column correction value Colproject for the projection value of the k-th frame image k (j), as shown in formula (XI): Colproject k (j)=Col k (j)-Col k (XI) Using the row and column projection value correction values ​​calculated in step 3.3 and step 3.6, draw the row and column grayscale projection curves of the image; Step 3.7: After calculating the row and column grayscale projection curves of the current frame and the reference frame respectively, perform cross-correlation operations on the row and column grayscale projection curves of the current frame and the reference frame respectively, take the extreme value in the correlation value curve as the displacement of the image frame, take the absolute value of its vector as the displacement size, and the cross-correlation operation calculation formula is shown in Formula (XII) and Formula (XIII): In formula (XII) and formula (XIII), R x (w) and R y (v) respectively represent the calculation formulas for the relevant operations on the processed row and column projections, Col pre (j) is the grayscale projection value of the jth column of the current frame image, Col ref (j) is the grayscale projection value of the jth column of the reference frame, p and q are the search lengths of the current frame relative to the reference frame on one side of the image; w and v are within the range of values, so that R x (w) and R y The minimum value of (v) is defined as w min and v min , then the horizontal and vertical displacement vectors of the current frame relative to the reference frame are shown in equations (XIV) and (XV) respectively: d x =m+1-w min (XIV) d y =n+1-v min (XV) In formula (XIV) and formula (XV), d x Indicates the horizontal displacement of the current frame compared to the reference frame, where d y Indicates the vertical displacement of the current frame compared to the reference frame; When x or y When the value of is greater than the set threshold value T1, it is determined that the video picture is shaken, otherwise, it is determined that the video picture is not shaken.

2. According to the method for detecting multi-venue video jitter based on video surveillance according to claim 1, it is characterized in that: The specific implementation steps of step 1 include: Step 1.1: In a Cartesian coordinate system with the upper left corner of the captured video frame image as the origin, the x-axis to the right, and the y-axis downward, when the window moves in both the x and y directions, calculate the pixel value change E(u,v) inside the window; Step 1.2: For each window, calculate the corresponding corner point response function R; Step 1.3: Set a threshold value and perform a threshold judgment on the calculated corner point response function. If R>threshold, it means that the window corresponds to a corner point feature and the pixel corresponds to a corner point. Otherwise, ignore this pixel and detect the next pixel.

3. The multi-venue video jitter detection method based on video surveillance according to claim 2 is characterized in that: Step 1.1 The specific implementation steps include: Let the center of a window be located at any position (x, y) of the grayscale image of any frame of the video acquired by the video surveillance. The grayscale value of the pixel at this position is I(x, y). If the window moves a small displacement u and v in the x and y directions respectively to a new position (x+u, y+v), the grayscale value of the pixel at this position is I(x+u, y+v). I(z+u, y+v)-I(x, y) refers to the change in the grayscale value caused by the window movement. Define ω(x,y) as the window function at position (x,y) to represent the weight of each pixel in the window, and set the weight of all pixels in the window to 1; or set ω(x,y) to a Gaussian distribution with the center of the window as the origin; if the pixel at the center of the window is a corner point, the weight coefficient of the center of the window is set to 1, indicating that the center of the window contributes more to the grayscale change; and the farther away from the center of the window, i.e. the corner point, the smaller the grayscale change of these points is, the weight coefficient is set smaller, set to 0-1, and the farther away from the corner point, the closer the weight coefficient is to 0, indicating that these points contribute less to the grayscale change; The formula for obtaining the pixel value change E(u,v) inside the window is shown in formula (I): In formula (I), I x and I y is the partial differential of I, which is the gradient map in the x and y directions in the image. The matrix M is: The right side of the arrow is the result of diagonalizing the real symmetric matrix. R is the corner point response function, which does not affect the change components in the two orthogonal directions. After diagonalization, the change components in two orthogonal directions are extracted, namely λ1 and λ2, which are the two eigenvalues ​​of the matrix M.

4. The multi-venue video jitter detection method based on video surveillance according to claim 2 is characterized in that: Step 1.2 The specific implementation steps include: Directly use the smaller eigenvalue as the corner point response function R, as shown in formula (II): R=min(λ1,λ2) (II) In formula (II), λ1 and λ2 are the eigenvalues ​​of matrix M.

5. The multi-venue video jitter detection method based on video surveillance according to claim 2 is characterized in that: In step 1.3, threshold=15.

6. The multi-venue video jitter detection method based on video surveillance according to claim 1 is characterized in that: The specific implementation steps of step 2 include: Intercept two adjacent frames of the video, the previous frame is defined as the reference frame, and the next frame is defined as the current frame. Take a corner point of the reference frame image, and find the two corner points in the current frame image that are closest to the reference frame feature point in Euclidean distance. Among these two corner points, if the ratio of the Euclidean distance of the corner point closest to the reference frame corner point divided by the Euclidean distance of the second closest corner point is less than the threshold T, then the two corner points are matched successfully, otherwise, the two corner points are not matched successfully; Repeat this operation until all corner points detected in step 1 are matched, and T is set to 0.4-0.6; If the matching degree of all corner points of the two frames reaches more than 90%, then the venue switching does not occur; otherwise, it is determined that the venue switching occurs.

7. The multi-venue video jitter detection method based on video surveillance according to claim 6 is characterized in that: The formula for calculating the Euclidean distance ρ1 of the current frame corner point to the reference frame corner point is shown in formula (III): In formula (III), x1 and y1 are the row and column coordinates of the corner point of the reference frame, and x2 and y2 are the row and column coordinates of the corresponding corner point of the current frame; The formula for calculating the Euclidean distance ρ2 of the current frame corner point to the reference frame corner point is shown in formula (IV): In formula (IV), x3 and y3 are the row and column coordinates of the corner point of the current frame that is next closest to the corner point of the reference frame; The formula for calculating the ratio is shown in formula (V):

8. The multi-venue video jitter detection method based on video surveillance according to claim 7 is characterized in that: Depending on the application scenario, T1 ranges from 5 to 30.

Images