A method for locating fireworks points based on video surveillance

Abstract

The present invention discloses a method for locating fireworks points based on video surveillance, comprising the following steps: taking a frame of image from a video surveillance device at equal intervals per second to generate a continuous image sequence, dividing the image sequence into blocks according to time rules to form image blocks, and calculating the signal-to-noise ratio of each image; according to the change sequence of the signal-to-noise ratio, setting a threshold by a background difference method; according to the signal-to-noise ratio of the image and the size of the threshold, finding the changed image block and determining it as a suspected smoke image; extracting the LBP feature vector of the suspected smoke image, inputting the trained SVM classification model, classifying the suspected smoke image, and distinguishing between smoke images and non-smoke images; using a single-point positioning method, calculating the position of the smoke image, and combining forestry class data to determine the coordinates of the fireworks point. The method disclosed by the present invention uses video surveillance to identify smoke areas, avoids confidentiality issues, and uses deep learning to identify smoke, which can improve recognition accuracy.

Description

Technical Field

[0001] The invention relates to a method for locating fireworks points based on video monitoring. Background Art

[0002] The resources in the forest are one of our important natural resources, playing an important role in human life and social development. They can maintain biodiversity, keep soil and water stable, and maintain the carbon balance of nature. Therefore, protecting forests from fires has significant economic and social benefits. At present, the ground monitoring of fires has been greatly improved by combining human defense and technical defense. However, how to accurately determine the location of the fire in the later stage of the fire has become one of the next technologies that need to be focused on. At present, the main methods used are artificial observation positioning, double-point cross positioning, laser ranging positioning, and high-precision DEM data perspective analysis positioning.

[0003] Manual observation requires familiarity with the terrain of the monitoring area. During forest fire prevention, wireless communication equipment, handheld GPS, compass and other equipment must be brought every day to observe the fire and determine the location of the fire. The horizontal position can be determined but the distance depth cannot be accurately determined. The accuracy of double-point cross positioning can be guaranteed, but it cannot be achieved at the edge of the monitoring area, and at least two monitoring devices are required to overlap to achieve it. This improvement in positioning accuracy is based on reducing equipment coverage, resulting in large investments. Laser ranging positioning will increase equipment costs and has a short effective distance, which cannot meet long-distance positioning requirements. Using high-precision DEM data can achieve high-precision spatial perspective analysis and calculation of fire point locations, but it cannot effectively avoid confidentiality issues. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a method for locating fireworks points based on video surveillance, so as to achieve the purpose of using video surveillance to identify smoke areas, avoid confidentiality issues, and use deep learning to identify smoke and improve recognition accuracy.

[0005] To achieve the above object, the technical solution of the present invention is as follows:

[0006] A method for locating fireworks points based on video surveillance comprises the following steps:

[0007] Step 1: Take one frame of image from the video surveillance equipment installed on the tower at equal intervals per second to generate a continuous image sequence, divide the image sequence into blocks according to time rules to form image blocks, and calculate the signal-to-noise ratio of each image;

[0008] Step 2: according to the change sequence of signal-to-noise ratio, the threshold is set by background difference method;

[0009] Step 3: According to the signal-to-noise ratio of the image and the size of the threshold, the changed image block is found and determined to be a suspected smoke image;

[0010] Step 4: extract the LBP feature vector of the suspected smoke image, input it into the trained SVM classification model, classify the suspected smoke image, and distinguish between smoke images and non-smoke images;

[0011] Step 5: Using the single-point positioning method, the position of the smoke image is calculated based on the angle at which the video surveillance equipment captures the smoke image, and the coordinates of the fireworks point are determined.

[0012] In the above scheme, the video surveillance device has 12 preset positions, each preset position stays for 20 seconds, and 4 minutes is a cycle to complete 360° detection, so 20 seconds is set as the time window.

[0013] In the above scheme, in step 1, the calculation formula of the signal-to-noise ratio SNR is as follows:

[0014]

[0015] In the formula, x and y represent the row and column where the pixel is located, respectively, and N x and N y They represent the total number of rows and columns in the calculation area respectively. f(x,y) is the pixel value of the reference frame image at the pixel point (x,y). f′(x,y) represents the pixel value of the current frame image at the corresponding pixel point (x,y). The SNR calculation result is a value in dB.

[0016] In the above scheme, the specific method of step 2 is as follows: select the signal-to-noise ratio of an image sequence of 20 consecutive frames, wherein the maximum value is used as the upper threshold limit and the minimum value is used as the lower threshold limit.

[0017] In the above scheme, the specific method of step three is as follows: if the signal-to-noise ratio of the area to be detected exceeds the set threshold, the detection value of the area to be detected is marked as 1, otherwise it is marked as 0, so as to generate a continuous binary detection sequence; within the time window, four consecutive detection values ​​form a detection code. If the detection code 0111 appears, it means that the time when the change is detected, and the area to be detected is determined as a suspected smoke area.

[0018] In the above scheme, in step 4, the calculation method of the LBP feature vector is as follows:

[0019] Select pixels in a 3×3 neighborhood, compare the grayscale value of the central pixel in the neighborhood with the grayscale values ​​of the 8 neighboring pixels. If the grayscale value of the neighboring pixel is greater than that of the central pixel, the position of the central pixel is marked as 1, otherwise it is 0; the grayscale values ​​of the 8 neighboring pixels are combined into a binary number and converted into a decimal number, which is the LBP value of the central pixel, and the texture distribution characteristics of the image are obtained:

[0020]

[0021]

[0022] In the formula, g i is the gray value of the neighborhood pixel; g c is the gray value of the central pixel, p and r are the number of neighborhood pixels and the neighborhood radius respectively, p=8, r=1; the neighborhood radius is the Euclidean distance between the central pixel and the neighborhood pixel; I represents the lower bound of the function Σ, and S(x) is the sign function.

[0023] In the above scheme, in step 4, when training the SVM classification model, the cross-validation method is used to obtain the optimal parameters of the model, and the kernel function is set to the radial basis kernel function, which maps the data nonlinearly to a high-dimensional space and processes the nonlinear relationship between features and their attributes.

[0024] In the above scheme, in step 4, the evaluation indicators for classifying the suspected smoke area include the following:

[0025] P=T P / (T P +F P )

[0026] R=T P / (T P +F N )

[0027] F 1 =2*P*R / (P+R)

[0028] Where: P is the precision, which indicates the proportion of samples predicted as smoke that are actually smoke; R is the recall, which indicates the proportion of samples that are actually smoke that are accurately predicted as smoke; F 1 is the harmonic mean of precision and recall; T P Indicates the number of image blocks predicted to be smoke and actually smoke; F P Indicates the number of image blocks predicted to be smoke but actually not smoke; F N Indicates the number of image blocks predicted to be non-smoke but actually are smoke.

[0029] In the above scheme, the specific calculation process of step five is as follows:

[0030] Taking the video surveillance device C as the starting point, a ray is extended along the optical axis direction. The ray intersects with the mountain surface in the geographic space. The intersection point E is the smoke point to be located, that is, the location of the captured smoke image.

[0031] Let the coordinates of point C in the geodetic coordinate system be (X c , Y c , Z c ), can be directly measured by GPS; the coordinates of point E are (X E , Y E , Z E ), according to the geometric relationship in the model:

[0032]

[0033] Where l is the distance between the video surveillance device C and the smoke point E, α is the angle between the video surveillance device C and the horizontal plane when the smoke image is taken, and β is the angle between the video surveillance device C and the true north direction when the smoke image is taken;

[0034] Then, according to the average tree height in the forest class data, the actual position of the corresponding fire point (X, Y, Z) is calculated:

[0035]

[0036] Among them, H is the regional average tree height in the forest small class data.

[0037] Through the above technical solution, the method for locating fireworks points based on video monitoring provided by the present invention has the following beneficial effects:

[0038] 1. The present invention uses images captured by video surveillance equipment to identify fireworks points, and does not require manual entry into forest areas for on-site inspections, thus saving labor costs and greatly improving monitoring and identification efficiency.

[0039] 2. The present invention uses the signal-to-noise ratio of the image and the size of the threshold to find the changed image blocks and determine them as suspected smoke images; then the LBP feature vector of the suspected smoke image is input into the trained SVM classification model to classify the suspected smoke image and distinguish between smoke images and non-smoke images; the smoke recognition accuracy is high and the algorithm is accurate.

[0040] 3. The present invention uses video images and forestry class data to locate the location of fireworks and fire points, without using confidential geographic information data, effectively avoiding confidentiality issues, and avoiding errors caused by inaccurate mountain elevation and underreporting problems caused by the inability of some thermal cores to identify fire scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0042] Figure 1 A flow chart of a method for locating fireworks points based on video monitoring disclosed in an embodiment of the present invention;

[0043] Figure 2 A simulation diagram for determining the coordinates of the fireworks point. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0045] The present invention provides a method for locating fireworks points based on video monitoring, such as Figure 1 As shown, the following steps are included:

[0046] Step 1: Take one frame of image from the video surveillance equipment installed on the tower at equal intervals per second to generate a continuous image sequence, divide the image sequence into blocks according to time rules to form image blocks, and calculate the signal-to-noise ratio of each image;

[0047] The video surveillance equipment has 12 preset positions, each of which stays at 20 seconds. A 360° detection is completed in 4 minutes, so 20 seconds is set as the time window. The video surveillance equipment includes high-point PTZ or ball camera.

[0048] The calculation formula of signal-to-noise ratio SNR is as follows:

[0049]

[0050] In the formula, x and y represent the row and column where the pixel is located, respectively, and N x and N y They represent the total number of rows and columns in the calculation area respectively. f(x,y) is the pixel value of the reference frame image at the pixel point (x,y). f′(x,y) represents the pixel value of the current frame image at the corresponding pixel point (x,y). The SNR calculation result is a value in dB.

[0051] Step 2: according to the change sequence of signal-to-noise ratio, the threshold is set by using background difference method;

[0052] When the reference frame is the background frame, the SNR is obtained m , which indicates the absolute change of the current frame relative to the background frame; when the reference frame is the previous frame, the SNR n, represents the relative change between two consecutive frames. The signal-to-noise ratio of the image sequence of 20 consecutive frames is selected, where the maximum value is used as the upper threshold and the minimum value is used as the lower threshold.

[0053] Step 3: According to the signal-to-noise ratio of the image and the size of the threshold, the changed image block is found and determined to be a suspected smoke image;

[0054] If the signal-to-noise ratio of the area to be detected exceeds the upper or lower threshold, the detection value of the area to be detected is marked as 1, otherwise it is marked as 0, so as to generate a continuous binary detection sequence; within the time window, four consecutive detection values ​​form a detection code. If the detection code 0111 appears, it means that the time when the change is detected, and the area to be detected is determined as a suspected smoke area.

[0055] Step 4: extract the LBP feature vector of the suspected smoke image, input it into the trained SVM classification model, classify the suspected smoke image, and distinguish between smoke images and non-smoke images;

[0056] The hardware environment of the experiment is a 64-bit Windows 10 operating system desktop computer, with an Intel(R) Core(TM) i7-2600 CPU@3.40GHz, 4GB memory, and Intel(R) HD Graphics graphics card. Using the Microsoft VisualStudio 2017 development platform, the python 3.6 programming language and the opencv3.4.5 open source function library are used to implement the detection algorithm and obtain the suspected smoke area classification results.

[0057] The calculation method of LBP feature vector is as follows:

[0058] Select pixels in a 3×3 neighborhood, compare the grayscale value of the central pixel in the neighborhood with the grayscale values ​​of the 8 neighboring pixels. If the grayscale value of the neighboring pixel is greater than that of the central pixel, the position of the central pixel is marked as 1, otherwise it is 0. The grayscale values ​​of the 8 neighboring pixels are combined into a binary number and converted into a decimal number, which is the LBP value of the central pixel. The texture distribution characteristics of the image are obtained:

[0059]

[0060]

[0061] In the formula, g i is the gray value of the neighborhood pixel; g cis the gray value of the central pixel, p and r are the number of neighborhood pixels and the neighborhood radius respectively, p=8, r=1; the neighborhood radius is the Euclidean distance between the central pixel and the neighborhood pixel; I represents the lower bound of the function Σ, and S(x) is the sign function.

[0062] Support vector machine (SVM) achieves correct classification of positive and negative samples by establishing an optimal classification surface to maximize the interval between two types of samples. It has low computational complexity and helps solve problems such as small sample learning, nonlinearity, and high-dimensional pattern recognition. The forest fire smoke area in all videos is selected as the positive sample, and the typical non-smoke area includes clouds, cars, pedestrians, etc. as negative samples.

[0063] When training the SVM classification model, the cross-validation method is used to obtain the optimal parameters of the model. The kernel function is set to the radial basis kernel function, which maps the data nonlinearly to a high-dimensional space and processes the nonlinear relationship between features and their attributes. The penalty factor C is 100 and the coefficient μ is 0.001.

[0064] The evaluation indicators for classifying suspected smoke areas include the following:

[0065] P=T P / (T P +F P )

[0066] R=T P / (T P +F N )

[0067] F 1 =2*P*R / (P+R)

[0068] Where: P is the precision, which indicates the proportion of samples predicted as smoke that are actually smoke; R is the recall, which indicates the proportion of samples that are actually smoke that are accurately predicted as smoke; F 1 is the harmonic mean of precision and recall; T P Indicates the number of image blocks predicted to be smoke and actually smoke; F P Indicates the number of image blocks predicted to be smoke but actually not smoke; F N Indicates the number of image blocks predicted to be non-smoke but actually are smoke.

[0069] Step 5: Using the single-point positioning method, the position of the smoke image is calculated based on the angle at which the video surveillance equipment captures the smoke image, and the coordinates of the fireworks point are determined.

[0070] like Figure 2As shown in the figure, C is a video surveillance device installed on the iron tower, the tetrahedron is a mountain within the monitoring range of the surveillance device, and Ow-XwYwZw is the geodetic coordinate system. The origin of Ow-XwYwZw is translated to the video surveillance device C to establish the auxiliary coordinate system C-XYZ.

[0071] Taking the video surveillance device C as the starting point, a ray is extended along the optical axis direction. The ray intersects with the mountain surface in the geographic space. The intersection point E is the smoke point to be located, that is, the location of the captured smoke image.

[0072] Let the coordinates of point C in the geodetic coordinate system be (X c , Y c , Z c ), can be directly measured by GPS; the coordinates of point E are (X E , Y E , Z E ), according to the geometric relationship in the model:

[0073]

[0074] Where l is the distance between the video surveillance device C and the smoke point E, α is the angle between the video surveillance device C and the horizontal plane when the smoke image is taken, and β is the angle between the video surveillance device C and the true north direction when the smoke image is taken;

[0075] Then, according to the average tree height in the forest class data, the actual position of the corresponding fire point (X, Y, Z) is calculated:

[0076]

[0077] Among them, H is the regional average tree height in the forest small class data.

[0078] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for locating fireworks points based on video surveillance. It is characterized in that The steps include: Step 1: Take one frame of image from the video surveillance equipment installed on the tower at equal intervals per second to generate a continuous image sequence, divide the image sequence into blocks according to time rules to form image blocks, and calculate the signal-to-noise ratio of each image; Step 2: according to the change sequence of signal-to-noise ratio, the threshold is set by background difference method; Step 3: According to the signal-to-noise ratio of the image and the size of the threshold, the changed image block is found and determined to be a suspected smoke image; Step 4: extract the LBP feature vector of the suspected smoke image, input it into the trained SVM classification model, classify the suspected smoke image, and distinguish between smoke images and non-smoke images; Step 5: Using the single-point positioning method, the position of the smoke image is calculated according to the angle at which the video surveillance equipment captures the smoke image, and the coordinates of the fireworks point are determined; In step 1, the calculation formula of the signal-to-noise ratio SNR is as follows: In the formula, x and y represent the row and column where the pixel is located, respectively, and N x and N y They represent the total number of rows and columns in the calculation area, respectively. f(x,y) is the pixel value of the reference frame image at the pixel point (x,y). f′(x,y) represents the pixel value of the current frame image at the corresponding pixel point (x,y). The SNR calculation result is a value in dB. The specific method of step three is as follows: if the signal-to-noise ratio of the area to be detected exceeds the set threshold, the detection value of the area to be detected is marked as 1, otherwise it is marked as 0, so as to generate a continuous binary detection sequence; within the time window, four consecutive detection values ​​form a detection code. If the detection code 0111 appears, it means that the time of the change is detected, and the area to be detected is determined as a suspected smoke area; In step 4, the calculation method of the LBP feature vector is as follows: Select pixels in a 3×3 neighborhood, compare the grayscale value of the central pixel in the neighborhood with the grayscale values ​​of the 8 neighboring pixels. If the grayscale value of the neighboring pixel is greater than that of the central pixel, the position of the central pixel is marked as 1, otherwise it is 0. The grayscale values ​​of the 8 neighboring pixels are combined into a binary number and converted into a decimal number, which is the LBP value of the central pixel. The texture distribution characteristics of the image are obtained: In the formula, g i is the gray value of the neighborhood pixel; g c is the gray value of the central pixel, p and r are the number of neighborhood pixels and the neighborhood radius respectively, p=8, r=1; the neighborhood radius is the Euclidean distance between the central pixel and the neighborhood pixel; I represents the lower bound of the function Σ, and S(x) is the sign function.

2. A method for locating fireworks points based on video surveillance according to claim 1, It is characterized in that The video surveillance device has 12 preset positions, each of which stays at 20 seconds. A cycle of 4 minutes completes 360° detection, so 20 seconds is set as the time window.

3. A method for locating fireworks points based on video monitoring according to claim 1, It is characterized in that The specific method of step 2 is as follows: select the signal-to-noise ratio of an image sequence of 20 consecutive frames, wherein the maximum value is used as the upper threshold limit, and the minimum value is used as the lower threshold limit.

4. A method for locating fireworks points based on video surveillance according to claim 1, It is characterized in that In step 4, when training the SVM classification model, the cross-validation method is used to obtain the optimal parameters of the model, and the kernel function is set to the radial basis kernel function, which maps the data nonlinearly to a high-dimensional space and processes the nonlinear relationship between features and their attributes.

5. A method for locating fireworks points based on video monitoring according to claim 1, It is characterized in that In step 4, the evaluation indicators for classifying suspected smoke areas include the following: P=T P / (T P +F P ) R=T P / (T P +F N ) F 1 =2*P*R / (P+R) Where: P is the precision, which indicates the proportion of samples predicted as smoke that are actually smoke; R is the recall, which indicates the proportion of samples that are actually smoke that are accurately predicted as smoke; F 1 is the harmonic mean of precision and recall; T P Indicates the number of image blocks predicted to be smoke and actually smoke; F P Indicates the number of image blocks predicted to be smoke but actually not smoke; F N Indicates the number of image blocks predicted to be non-smoke but actually are smoke.

6. A method for locating fireworks points based on video monitoring according to claim 1, It is characterized in that The specific calculation process of Step Five is as follows: Starting from the video surveillance device C, a ray is extended along the optical axis direction, and this ray intersects with the mountain surface in the geographical space. The intersection point E is the smoke point to be located, that is, the position where the captured smoke image is located; Let the coordinates of point C in the geodetic coordinate system be (X c ,Y c ,Z c ), can be directly measured by GPS; the coordinates of point E are (X E ,Y E ,Z E ), according to the geometric relationship in the model: where l is the distance between the video surveillance device C and the smoke point E, α is the angle between the video surveillance device C and the horizontal plane when capturing the smoke image, and β is the angle between the video surveillance device C and the due north direction when capturing the smoke image; then, based on the regional average tree height in the forest land plot data, the true position (X, Y, Z) of the corresponding fire point is obtained: where H is the regional average tree height in the forest land plot data.

Images