A high-altitude object throwing warning system and method based on visual sensing and trajectory prediction

Abstract

The present invention discloses a high-altitude object throwing warning system and method based on visual sensing and trajectory prediction, the method comprising: acquiring images through binocular camera video monitoring; applying a convolutional neural network algorithm trained on the basis of inception v4 at the receiving end to identify the category of high-altitude falling objects; the system remains in standby mode when there is no threat mode; when there is a threat mode, while identifying the falling object, a mathematical model established by analyzing its possible force mode is used to jointly predict the data received by the dual cameras using the Kalman filter method, predict the coordinates of the object landing point and transmit it; the sound and light warning module that receives the coordinates quickly processes the data, uses a striking laser light to mark the landing point before the falling object lands, and uses a sound alarm to remind pedestrians, and uses a blocking net to intercept and prevent pedestrians from being injured. The present invention can timely detect and warn the situation of high-altitude objects, ensure personnel safety, and reduce the risk of accidents.

Description

Technical Field

[0001] The present invention relates to the field of security monitoring, and in particular to a high-altitude object throwing warning system and method based on visual sensing and trajectory prediction. Background Art

[0002] Modern skyscrapers are often built hundreds of meters high, and residents' irregular living habits and lack of public safety awareness can all lead to such incidents. Objects dropped from tens to hundreds of meters high can reach a speed of 30 to 45 meters per second. For people walking on the street or downstairs, the threat is huge, and it is extremely difficult to trace the dropped objects afterwards.

[0003] In order to eliminate potential safety hazards, the promulgation of laws and regulations is important, and technological innovation and design are also indispensable. Existing technical means are in urgent need of improvement and development. The society is increasingly calling for a technology that can provide early warning and prevention.

[0004] At present, there are very few research solutions at home and abroad that focus on the detection of objects thrown from high places.

[0005] In China, a multi-camera group is commonly used to monitor the residents online, and the cameras are cross-deployed to mark the range of the parachuted objects and eliminate blind spots. This is the most traditional high-altitude parachuted object detection solution, and its shortcomings are also obvious, that is, it does not have the ability to identify and capture dynamic parachuted objects in the video, cannot predict their landing point and direction, and cannot effectively respond to the falling events that have occurred to reduce the harm to people.

[0006] According to foreign literature, the foreign countries still use traditional cameras and snapshots to deal with high-altitude objects, and reduce the occurrence of such incidents by holding the perpetrators accountable and recording them. Some inventors use cloud networking to track the number of high-altitude objects thrown to determine the corresponding punishment measures, but lack corresponding early warning and emergency measures after the accident.

[0007] Therefore, a high-altitude object throwing warning system based on visual sensing and trajectory prediction is needed to achieve efficient and timely object throwing detection and intelligent warning. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide a high-altitude object throwing warning system and method based on visual sensing and trajectory prediction in view of the defects in the prior art.

[0009] The technical solution adopted by the present invention to solve the technical problem is:

[0010] The present invention provides a high-altitude object throwing warning method based on visual sensing and trajectory prediction, the method comprising the following steps:

[0011] Step 1: Obtain images through binocular camera video monitoring, and perform morphological processing on the initial images using grayscale and Gaussian filtering methods;

[0012] Step 2: Data is transmitted through the 5G communication module, and the receiving end uses a convolutional neural network algorithm trained on the basis of Inception v4 to identify the categories of high-altitude falling objects and divide them into two modes: non-threatening and threatening;

[0013] Step 3: In the non-threatening mode, the system remains in standby mode to reduce false triggering caused by environmental factors or minor disturbances, and reduce energy and system losses;

[0014] When entering the threat mode, while identifying falling objects, the Kalman filter method is used to jointly predict the data received by the dual cameras through the mathematical model established by analyzing the possible force mode, and the coordinates of the object's landing point are predicted and transmitted;

[0015] Step 4: The sound and light warning module that receives the coordinates quickly processes the data, uses a conspicuous laser light to mark the landing point before the falling object hits the ground, and uses a sound alarm to remind pedestrians, and uses a blocking net to intercept the falling object to prevent pedestrians from being injured.

[0016] Furthermore, the method for processing the image in step 1 of the present invention includes:

[0017] The RGB color three-channel image is converted into a single-channel grayscale image and Gaussian blur is performed. The template operation is introduced to achieve the Gaussian blur effect. The template formula is:

[0018]

[0019] Among them, M(I,J) is the value of the matrix element in the corresponding template. The pixel weighted operation of the template coverage area is determined by moving the constructed template to replace the value of the pixel at the center of the image. Then, the creatBackgroundSubstractorKNN algorithm API of OpenCV is used to process the original data stream to obtain the difference image after subtracting the background. Further processing is performed on the basis of the difference image, including digital image processing methods such as grayscale and threshold processing. Finally, a connected domain analysis is performed on the optimized image to obtain its maximum contour, and video data for detecting parabolic events is obtained.

[0020] Furthermore, in step 2 of the present invention, the convolutional neural network algorithm retrained based on inception v4 identifies the category of high-altitude falling objects, and the method is:

[0021] Based on the retraining on the basis of Inception v4, according to the theory of CNN, transfer learning is used in image recognition. Each filter is regarded as a small feature capturer. According to the weights of the model trained by Inception v4, the top softmax layer of Inception v4 is constructed, and retrained using a training set adapted to the usage environment.

[0022] Furthermore, when entering the threat mode in step 3 of the present invention, the mathematical model established while identifying the falling object includes:

[0023] The information collection chooses to obtain the top view and side view of the parabola trajectory, which are provided by cameras in two directions respectively; the special trajectory point information selected is the three-dimensional coordinates and time (x0, y0, z0, t0) of the parabola throwing point, the three-dimensional coordinates and time of the highest point (x1, y1, z1, t1), and the three-dimensional coordinates and time of a point in the descent process (x2, y2, z2, t2); in order to simplify the calculation, the model is simplified on the basis of decomposing the motion trajectory vector into three planes x, y, and z for separate solutions; the x, y, and z planes are determined as follows: the plane directly opposite the building is the y plane, the ground is the z plane, and any side is the x plane;

[0024] ①Side view trajectory analysis:

[0025] For high-altitude parabolic objects, the side view is understood as a parabola-like trajectory, and the vertical coordinate analysis is performed on it. If the orientation is positive, the force analysis of the object is performed during the rising stage:

[0026] ma z =-mg-v z 2 λ

[0027] Where f = v z 2 λ is air resistance, let mg = c 2 λ, then we have

[0028]

[0029] Arrange and accumulate to get

[0030]

[0031] Calculation and sorting

[0032]

[0033] Finally, the integral of the vertical velocity is the landing height and the formula is:

[0034]

[0035] Get the landing time t' of the parabola;

[0036] Then analyze the y coordinate, taking the direction of parabolic motion as positive, and perform force analysis on it:

[0037] ma=-v y 2 λ

[0038] Where “-” indicates the direction of force, f = v y 2 λ is the magnitude of the air resistance, and the deformation integral is:

[0039]

[0040] get

[0041]

[0042] By inserting the data of the collected trajectory points, we can get v y1 and v y2 About v yo The expression of

[0043]

[0044] ma=-v y 2 λ

[0045] The three equations can be immediately solved to get v y Regarding the expression of t, the landing time t' has been obtained in the vertical direction analysis, so the y coordinate of the final landing point is

[0046]

[0047] Among them, v x is the speed of the object in the x-axis direction, v y is the speed of the object in the y-axis direction, v z is the speed of the object in the z-axis direction, (x, y, z) is the coordinate of the object's position, c is a constant, λ is the air resistance coefficient, m is the mass of the object, g is the acceleration of gravity, f is the air resistance, and (X, Y) is the horizontal and vertical coordinates of the landing point;

[0048] ②Top view trajectory analysis:

[0049] During the throwing process, the gravity will not change the horizontal trajectory of the object, and the air resistance is always opposite to the direction of motion. Therefore, the trajectory of the object will not deviate in the horizontal dimension. Therefore, the trajectory when viewed from above should be a line segment. The linear relationship between the parabola coordinates x and y can be obtained based on the data of the starting point (x0, y0, z0, t0) and the highest point (x1, y1, z1, t1):

[0050]

[0051] By substituting the y-coordinate Y of the landing point obtained from the side view, the x-coordinate X of the landing point can be obtained, and the three-dimensional coordinates and time (X, Y, Z, t) of the landing point can be obtained, so as to achieve the purpose of predicting the landing point. After the coordinates of the object's landing point are predicted, they can be transmitted.

[0052] Furthermore, in step 3 of the present invention, a Kalman filter method is used for joint prediction, and the method is:

[0053] Preparation process:

[0054] Calculate the error between the actual position and the estimated position at time i

[0055]

[0056] is the predicted value at time i, and X(i) is the actual value at time i. After the error is calculated, the covariance matrix can be obtained based on the error calculation:

[0057]

[0058] Modeling implementation:

[0059] First, establish the time update equation:

[0060]

[0061] Where U(i) is the input quantity, that is, the input location information, is the predicted value at time i; A is a state transfer matrix, B is a gain matrix; W(i) is a set noise;

[0062] Establish the error correlation matrix P to determine whether the estimated value is accurate:

[0063]

[0064] Q is a covariance matrix of system noise, which is a constantly changing value; P(i) is the covariance matrix of the estimation error. is the covariance matrix of the forecast errors;

[0065] The above is the prediction process; after calculating the predicted value at the corresponding moment, the predicted value is corrected; this process is to compare the measured value with the true value and further update the data;

[0066]

[0067] in is the measurement margin, Z(i) is the measurement value;

[0068] Calculate the necessary Kalman gains:

[0069] Kalman filtering itself is a minimum mean square error, and the mean square error is the trace of P(i), so the formula for finding the prediction error covariance is traced to obtain a K k The equation for the optimal estimate of K k The value makes the trace of P(i) the smallest. For this equation, both sides of K k Derivative, and then use the matrix differential operation rules to make the differential result zero to get K k The calculation expression is:

[0070] K k =P′(i)H T (HP′(i)H T +R) -1

[0071] The Kalman gain can be used to obtain and update the estimated value. For the estimated value at time i, it can usually be processed in this way, that is, using the Kalman coefficient to perform an operation to obtain the Kalman prediction value

[0072]

[0073] This completes an update of the Kalman estimate using the estimate and the measurement margin;

[0074] Then update the covariance matrix, and derive a covariance with the prediction error represented by the Kalman gain:

[0075] P(i)=(IK k H)P′(i)

[0076] Through continuous iteration and recursion, we can finally calculate P(n) and X(n) to complete a complete estimation of each position on the trajectory.

[0077] The model updates the position information of the target object at any time according to its situation, and obtains its corrected trajectory through effective loop detection. The relevant data information is obtained through simulation of Matlab tools to obtain the landing area of ​​the target object.

[0078] Furthermore, the method of step 4 of the present invention is:

[0079] After receiving the predicted coordinates of the object's landing point, the signal data emission and transmission module will transmit the data instructions to the single-chip microcomputer system. The single-chip microcomputer box controls the pan-tilt head. The strong light on the pan-tilt head will project a strong red light on the landing area and a buzzer alarm will sound. Before the falling object hits the ground, a conspicuous laser light will be used to mark the landing point and a sound alarm will be used to remind pedestrians. A net will be used to intercept the object to prevent it from being hurt, thereby protecting the personal and property safety of pedestrians to the greatest extent. When the warning is completed, the execution program will cause the pan-tilt head to automatically initialize and return to its original position and continue to be in standby mode.

[0080] The present invention provides a high-altitude object throwing warning system based on visual sensing and trajectory prediction, the system comprising:

[0081] The camera module uses a binocular camera to obtain video frames of the parabolic monitoring area and preprocess the video frames to obtain video data for detecting parabolic events; the preprocessing includes: using grayscale and Gaussian filtering methods to perform morphological processing on the initial image;

[0082] The communication module uses a 5G communication module to send the images collected by the camera module to the moving object target tracking detection module at the receiving end; and transmit the landing point prediction results and instructions to the network alarm module and the warning light source projection control module;

[0083] The moving object target tracking and detection module is used to apply the convolutional neural network algorithm retrained on the basis of Inception v4 to identify the categories of high-altitude falling objects and divide them into two modes: non-threatening and threatening. In the non-threatening mode, the system remains in standby mode to reduce false triggering caused by environmental factors or small disturbances, and reduce energy and system losses. When entering the threatening mode, while identifying the falling object, the mathematical model established by analyzing its possible force mode is used to jointly predict the data received by the dual cameras using the Kalman filter method, and the coordinates of the object's landing point are predicted and transmitted;

[0084] The network alarm module is used to generate a corresponding sound alarm when receiving a signal of a falling object;

[0085] The early warning light source projection control module is used to mark and warn of the landing point of falling objects, divide the warning range of this system into areas with sufficiently high accuracy, pre-set the corresponding pan / tilt control parameters for each area in the system, and feedback the predicted landing point coordinates after the algorithm completes the processing of the image. The required parameters are obtained by table lookup, and laser lights are used for illumination to complete the early warning of pedestrians.

[0086] The beneficial effects produced by the present invention are:

[0087] 1. Detect the source of high-altitude projectiles, establish a mathematical model of the three-dimensional projectile motion, predict the trajectory and landing point of the falling object, and issue a warning.

[0088] 2. Use high-speed cameras to capture the situation in the building, and use neural network calculations to monitor and measure the whereabouts of objects. At the same time, the falling object data and video data are saved locally, leaving effective physical evidence of objects thrown from high places.

[0089] 3. Determine the type of parabolic objects based on deep learning, predict the magnitude of the impact, and issue corresponding warnings so that pedestrians can make correct responses and avoid serious casualties caused by falling objects to the greatest extent possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0091] Figure 1 It is a schematic diagram of the overall design framework of a high-altitude object throwing warning system based on visual sensing and trajectory prediction of the present invention (two-dimensional flow chart);

[0092] Figure 2 This is an example of single target tracking using background subtraction in a moving car surveillance video resource.

[0093] Figure 3 An example of using background subtraction to track multiple targets in moving car surveillance video resources;

[0094] Figure 4 The learning curves are the loss and accuracy of the training set and the loss and accuracy of the test set. DETAILED DESCRIPTION

[0095] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0096] See also Figure 1 , Figure 1 The figure is a schematic diagram of the steps of a high-altitude object throwing warning method based on visual sensing and trajectory prediction in the first embodiment of the present invention. In the embodiment of the present invention, a high-altitude object throwing warning method based on visual sensing and trajectory prediction comprises:

[0097] After obtaining the parabolic zone video images recorded by the dual cameras, the images are preprocessed, including: converting the RGB color three-channel image into a single-channel grayscale image, and performing Gaussian blur. The template operation is introduced to achieve the effect of Gaussian filtering. The template formula is:

[0098]

[0099] Where M(I,J) is the value of the matrix element in the corresponding template. The weighted operation of the pixels in the template coverage area is determined by the movement of the constructed template to replace the value of the pixel at the center of the image, thereby achieving the purpose of Gaussian filtering. Then, the background subtraction method is used to obtain the background-subtracted image of the moving target. The background subtraction method is similar to the inter-frame difference method, but the background subtraction is obtained by subtracting the original image from its continuously updated background. It is a way to obtain the contour of the moving target in the image using the differential relationship. At the same time, this processing method of subtracting the background to a certain extent can effectively improve and crack the "false target interference" caused by the brightness change of the frame difference method. The background subtraction method is relatively simple to calculate. Since there is no moving target in the background image and there is a moving target in the current image, the two images are subtracted, and it is obvious that the complete moving target can be extracted, which solves the problem that the target extracted by the inter-frame difference method contains "holes".

[0100] The use of background subtraction to achieve target detection mainly includes four steps: background modeling, background updating, target detection, and post-processing. Among them, background modeling and background updating are the core issues in background subtraction. The quality of background model establishment directly affects the effect of target detection. Background modeling is to construct a model that can represent the "background" through mathematical methods. The most ideal way to obtain the background is to obtain a frame of "pure" image as the background without a moving target. Figure 2 , Figure 3 They are examples of single target tracking and multi-target tracking in moving car surveillance video resources using background subtraction method.

[0101] Then, the original data stream is processed using the creatBackgroundSubstractorKNN algorithm API of OpenCV to obtain the image after background subtraction. Further processing is performed on the basis of the differential image, including grayscale, threshold processing and other basic digital image processing techniques. Finally, a connected domain analysis is performed on the optimized image to obtain its maximum contour. Video data for detecting parabolic events is obtained.

[0102] After capturing the video data of the object throwing event, the trained convolutional neural network is used to judge the moving object. According to the theory of CNN, transfer learning can be used in image recognition, and each filter is a small feature capturer. Google's DeepMind has trained Inception V4 to recognize 1,000 objects, but the objects they trained cannot meet the requirements of high-altitude throwing in this project. For example, it is impossible to throw a car at a high-altitude object.

[0103] Therefore, the present invention directly uses the weights of the model trained by Inception V4, constructs the top softmax layer of Inception V4 by itself, and retrains using its own training set.

[0104] The training platform of this neural network is the GPU version of tensorflow2.0, and the Adam gradient method is used to implement gradient descent.

[0105] The training process is monitored using tensorboard. During the first 10 epochs of training, the learning curves of the loss and accuracy of the training set and the loss and accuracy of the test set are as follows: Figure 4 shown.

[0106] After using the algorithm to judge the data, if it is determined that there is no parabolic object, it will enter the non-threatening mode. In the non-threatening mode, the system remains in standby state to reduce the false triggering caused by environmental factors or small disturbances, and reduce the loss of energy and the system itself. After it is determined that a parabolic object is found, the mathematical model established by analyzing its possible force mode is used to jointly predict the data received by the dual cameras using the Kalman filter method. The calculation of the landing point benefits from the prediction of the parabola trajectory, and needs to take into account the influence of wind speed and damping.

[0107] The kinematic and mechanical analysis, solution and establishment of the above mathematical model are as follows:

[0108] (1) Modeling environment description

[0109] Resistance description: Since the model aims to solve the trajectory and landing point of high-altitude objects as accurately as possible, the influencing factors in practice should be fully considered during the establishment of the model. Resistance is a factor that cannot be ignored. This factor makes the parabola problem cannot be simply described and solved by the parabola model, so the air resistance problem is also worth considering

[0110] Description of problem analysis method: In real problems, the distribution of cameras is wide enough, and the data of two cameras can be used to determine the throwing time and real-time three-dimensional coordinates of the thrown object. By integrating the simulation calculations of the two aspects, a complete model of high-altitude throwing can be obtained.

[0111] (2) Environmental assumptions

[0112] Assumption 1: During the falling process of a parabolic object, various influencing factors (such as the shape and posture of the object) will not change with the height of the falling object.

[0113] Assumption 2: The buoyancy of the air will not affect the object during its fall. (The mass of the object is large enough to make the air resistance negligible.)

[0114] Assumption 3: There will be no uncertain factors (such as birds, other objects, etc.) that will have any impact on the trajectory of the parabola during its fall.

[0115] (3) Symbols used in the simulation process and their meanings

[0116]

[0117]

[0118] Table 1 Symbols used in the simulation process and their meanings

[0119] (4) Model establishment and solution

[0120] The information collection chooses to obtain the top view and side view of the parabola trajectory, which are provided by cameras in two directions respectively; the special trajectory point information selected is the three-dimensional coordinates and time (x0, y0, z0, t0) of the parabola throwing point, the three-dimensional coordinates and time of the highest point (x1, y1, z1, t1), and the three-dimensional coordinates and time of a point in the descent process (x2, y2, z2, t2); in order to simplify the calculation, the model is simplified on the basis of decomposing the motion trajectory vector into three planes x, y, and z for separate solutions; the x, y, and z planes are determined as follows: the plane directly opposite the building is the y plane, the ground is the z plane, and any side is the x plane;

[0121] ①Side view trajectory analysis:

[0122] For high-altitude parabolic objects, the side view is understood as a parabola-like trajectory, and the vertical coordinate analysis is performed on it. If the orientation is positive, the force analysis of the object is performed during the rising stage:

[0123] ma z =-mg-v z 2 λ

[0124] Where f = v z 2 λ is air resistance, let mg = c 2 λ, then we have

[0125]

[0126] Arrange and accumulate to get

[0127]

[0128] Calculation and sorting

[0129]

[0130] Finally, the integral of the vertical velocity is the landing height and the formula is:

[0131]

[0132] Get the landing time t' of the parabola;

[0133] Then analyze the y coordinate, taking the direction of the parabola's motion as positive, and perform force analysis on it:

[0134] ma=-v y 2 λ

[0135] Where “-” indicates the direction of force, f = v y 2 λ is the magnitude of the air resistance, and the deformation integral is:

[0136]

[0137] get

[0138]

[0139] By inserting the data of the collected trajectory points, we can get v y1 and v y2 About v yo The expression of

[0140]

[0141] ma=-v y 2 λ

[0142] The three equations can be immediately solved to get v y Regarding the expression of t, the landing time t' has been obtained in the vertical direction analysis, so the y coordinate of the final landing point is

[0143]

[0144] Among them, v x is the speed of the object in the x-axis direction, v y is the speed of the object in the y-axis direction, v z is the speed of the object in the z-axis direction, (x, y, z) is the coordinate of the object's position, c is a constant, λ is the air resistance coefficient, m is the mass of the object, g is the acceleration of gravity, f is the air resistance, and (X, Y) is the horizontal and vertical coordinates of the landing point;

[0145] ②Top view trajectory analysis:

[0146] During the throwing process, the gravity will not change the horizontal trajectory of the object, and the air resistance is always opposite to the direction of motion. Therefore, the trajectory of the object will not deviate in the horizontal dimension. Therefore, the trajectory when viewed from above should be a line segment. The linear relationship between the parabola coordinates x and y can be obtained based on the data of the starting point (x0, y0, z0, t0) and the highest point (x1, y1, z1, t1):

[0147]

[0148] By substituting the y-coordinate Y of the landing point obtained from the side view, the x-coordinate X of the landing point can be obtained, and the three-dimensional coordinates and time (X, Y, Z, t) of the landing point can be obtained, so as to achieve the purpose of predicting the landing point. After the coordinates of the object's landing point are predicted, they can be transmitted.

[0149] (5) The influence of wind speed on motion trajectory

[0150] When an object is falling, the wind speed may affect its trajectory and landing point. If the falling object is very heavy, the effect of wind speed on it is smaller than the effect of air resistance and can be ignored. If the wind speed is high and the object is light, the effect of wind speed on the falling trajectory of the object cannot be ignored.

[0151] There are two solutions for modeling after wind speed becomes an influencing factor of parabolic motion. One is to project wind speed onto three planes in three-dimensional space like the trajectory of parabolic motion. This solution will result in a large amount of calculation and is different from the previous modeling. The other is to regard wind speed as a moving three-dimensional space and parabolic motion as a parabolic motion relative to the above three-dimensional space. In comparison, this solution can systematically migrate the previous landing point calculation system and is simpler to calculate than the previous solution.

[0152] (6) Modeling evaluation

[0153] The model can directly calculate the final landing point of the parabola and can meet the requirements of practical problems for landing point prediction. However, since the model does not take emergencies into consideration and some parts of the problem are optimized, there are still certain errors. Therefore, the warning range is increased by expanding the judged landing point to achieve a higher effectiveness of landing point judgment.

[0154] After the modeling is completed, an equation for the projectile's landing point is obtained based on the dual-camera landing point motion modeling data. The landing point equation can be expanded to obtain the motion trajectory of the projectile, and the position of the projectile can be inferred from the landing point and the motion equation. In order to make the model for predicting the landing point more perfect, a loop and feedback process is added on the basis of the landing point, so that it can be continuously corrected during the projectile's motion to increase the stability and robustness of the system.

[0155] To this end, the Kalman filter algorithm is introduced. Kalman filtering is widely used and powerful. It can be used to estimate the past and current states of the signal, and can estimate and infer the future motion trajectory based on the existing state to correct the landing point.

[0156] Kalman filtering mainly completes two processes, namely, prediction and correction. Its basic idea is to use the minimum mean square error as the best estimation criterion, use the estimated value of the previous moment and the observed value of the current moment to jointly estimate and update the state variable, calculate the estimated value of the current moment, and establish the system equation and observation equation according to the algorithm to estimate the minimum mean square error of the signal to be processed.

[0157] Preparation process:

[0158] Calculate the error between the actual position and the estimated position at time i

[0159]

[0160] is the predicted value at time i, and X(i) is the actual value at time i. After the error is calculated, the covariance matrix can be calculated based on the error:

[0161]

[0162] Modeling implementation:

[0163] First, establish the time update equation:

[0164]

[0165] Where U(i) is the input quantity, that is, the input location information, is the predicted value at time i; A is a state transfer matrix, B is a gain matrix; W(i) is a set noise;

[0166] Establish the error correlation matrix P to determine whether the estimated value is accurate:

[0167]

[0168] Q is a covariance matrix of system noise, which is a constantly changing value; P(i) is the covariance matrix of the estimation error. is the covariance matrix of the forecast errors;

[0169] The above is the prediction process; after calculating the predicted value at the corresponding moment, the predicted value is corrected; this process is to compare the measured value with the true value and further update the data;

[0170]

[0171] in is the measurement margin, Z(i) is the measurement value;

[0172] Calculate the necessary Kalman gains:

[0173] Kalman filtering itself is a minimum mean square error, and the mean square error is the trace of P(i), so the formula for finding the prediction error covariance is a trace about K. k The equation for the optimal estimate of K k The value makes the trace of P(i) the smallest. For this equation, both sides of K k Derivative, and then use the matrix differential operation rules to make the differential result zero to get K k The calculation expression is:

[0174] K k =P′(i)H T (HP′(i)H T +R) -1

[0175] The Kalman gain can be used to obtain and update the estimated value. For the estimated value at time i, it can usually be processed in this way, that is, using the Kalman coefficient to perform an operation to obtain the Kalman prediction value

[0176]

[0177] This completes an update of the Kalman estimate using the estimate and the measurement margin;

[0178] Then update the covariance matrix, and derive a covariance with the prediction error represented by the Kalman gain:

[0179] P(i)=(IK k H)P′(i)

[0180] Kalman's process of predicting the trajectory of a projectile can be summarized as follows:

[0181] Step 1: First, P(0) and X(0) are known. Then P′(1) is calculated from P(0). Then K1 is calculated from P′(1). With these parameters, X(1) can be estimated by combining the observed values. Then K1 is used to update P(1).

[0182] 2 steps, and then the next update process is to calculate P′(2) from P(1), and then calculate K2 from P′(2). With these parameters, X(2) can be estimated in combination with the observed values, and then K2 is used to update P(2).

[0183] Through continuous iteration and recursion, we can finally calculate P(n) and X(n) to complete a complete estimation of each position on the trajectory.

[0184] The model updates the position information of the target object at any time according to its situation, and obtains its corrected trajectory through effective loop detection. The relevant data information is obtained through simulation of Matlab tools to obtain the landing area of ​​the target object.

[0185] After receiving the simulation results, the signal data transmission module will transmit the data instructions to the single-chip computer system. The single-chip computer box controls the PTZ. The strong light on the PTZ will project a strong red light to the landing area, and the buzzer will sound an alarm. Before the falling object hits the ground, a striking laser light will be used to mark the landing point and a sound alarm will be used to remind pedestrians. A blocking net will be used to intercept the falling object to prevent pedestrians from being hurt. This will maximize the protection of pedestrians' personal and property safety. When the warning is completed, the execution program will automatically initialize the PTZ and return to its original position, and continue to be in standby mode.

[0186] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0187] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0188] The above is a description of the present invention. For those skilled in the art, according to the concept of the embodiments of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A high-altitude object throwing warning method based on visual sensing and trajectory prediction, characterized in that: The method comprises the following steps: Step 1: Obtain images through binocular camera video monitoring, and perform morphological processing on the initial images using grayscale and Gaussian filtering methods; Step 2: Data is transmitted through the 5G communication module, and the receiving end uses a convolutional neural network algorithm trained on the basis of Inception v4 to identify the categories of high-altitude falling objects and divide them into two modes: non-threatening and threatening; Step 3: In the non-threatening mode, the system remains in standby mode to reduce false triggering caused by environmental factors or minor disturbances, and reduce energy and system losses; When entering the threat mode, while identifying falling objects, the Kalman filter method is used to jointly predict the data received by the dual cameras through the mathematical model established by analyzing the possible force mode, and the coordinates of the object's landing point are predicted and transmitted; Step 4: The sound and light warning module that receives the coordinates quickly processes the data, uses a striking laser light to mark the landing point before the falling object hits the ground, and uses a sound alarm to remind pedestrians, and uses a blocking net to intercept pedestrians to prevent them from being hurt; In step 3, the Kalman filter method is used for joint prediction, and the method is: Preparation process: Calculate the error between the actual position and the estimated position at time i is the predicted value at time i, and X(i) is the actual value at time i. After the error is calculated, the covariance matrix can be obtained based on the error calculation: Modeling implementation: First, establish the time update equation: Where U(i) is the input quantity, that is, the input location information, is the predicted value at time i; A is a state transfer matrix, B is a gain matrix; W(i) is a set noise; Establish the error correlation matrix P to determine whether the estimated value is accurate: Q is a covariance matrix of system noise, which is a constantly changing value; P(i) is the covariance matrix of the estimation error. is the covariance matrix of the forecast errors; The above is the prediction process; after calculating the predicted value at the corresponding moment, the predicted value is corrected; this process is to compare the measured value with the true value and further update the data; in is the measurement margin, Z(i) is the measurement value; Calculate the necessary Kalman gains: Kalman filtering itself is a minimum mean square error, and the mean square error is the trace of P(i), so the formula for finding the prediction error covariance is traced to obtain a The equation for the optimal estimate of The value makes the trace of P(i) the smallest. Take the derivative, and then use the matrix differential operation rules to set the differential result to zero to obtain The calculation expression is: The obtained Kalman gain can be used to obtain and update the estimated value. For the estimated value at time i, it can usually be processed in this way, that is, using the Kalman coefficient to perform an operation to obtain the Kalman prediction value : This completes an update of the Kalman estimate using the estimate and the measurement margin; Then update the covariance matrix, and derive a covariance with the prediction error represented by the Kalman gain: Through continuous iteration and recursion, we can finally calculate P(n) and X(n) to complete a complete estimation of each position on the trajectory. The model is used to update the position information of the target object at any time according to its situation, and its corrected trajectory is obtained through effective loop detection. The relevant data information is obtained through simulation of Matlab tools to obtain the landing area of ​​the target object.

2. The high-altitude object throwing warning method based on visual sensing and trajectory prediction according to claim 1 is characterized in that: The method for processing the image in step 1 includes: The RGB color three-channel image is converted into a single-channel grayscale image and Gaussian blur is performed. The template operation is introduced to achieve the Gaussian blur effect. The template formula is: Among them, M(I,J) is the value of the matrix element in the corresponding template. The pixel weighted operation of the template coverage area is determined by moving the constructed template to replace the value of the pixel at the center of the image. Then, the creatBackgroundSubstractorKNN algorithm API of OpenCV is used to process the original data stream to obtain the difference image after subtracting the background. Further processing is performed on the basis of the difference image, including digital image processing methods such as grayscale and threshold processing. Finally, a connected domain analysis is performed on the optimized image to obtain its maximum contour, and video data for detecting parabolic events is obtained.

3. The high-altitude object throwing warning method based on visual sensing and trajectory prediction according to claim 1 is characterized in that: In step 2, the convolutional neural network algorithm retrained based on inception v4 identifies the category of high-altitude falling objects, and the method is: Based on the retraining on the basis of Inception v4, according to the theory of CNN, transfer learning is used in image recognition. Each filter is regarded as a small feature capturer. According to the weights of the model trained by Inception v4, the top softmax layer of Inception v4 is constructed, and retrained using a training set adapted to the usage environment.

4. The high-altitude object throwing warning method based on visual sensing and trajectory prediction according to claim 1 is characterized in that: When entering the threat mode in step 3, the mathematical model established while identifying the falling object includes: The information collection selects the top view and side view of the parabola trajectory, which are provided by cameras in two directions respectively; the special trajectory point information selected is the three-dimensional coordinates and time of the parabola motion throwing point ( ), the highest point three-dimensional coordinates and time ( ), the three-dimensional coordinates and time of a certain point during the descent ( ); In order to simplify the calculation, the model is simplified on the basis of decomposing the motion trajectory vector into three planes x, y, and z for separate solutions; the x, y, and z planes are determined as follows: the plane directly facing the building is the y plane, the ground is the z plane, and any side is the x plane; ①Side view trajectory analysis: For high-altitude parabolic objects, the side view is understood as a parabola-like trajectory, and the vertical coordinate analysis is performed on it. If the orientation is positive, the force analysis of the object is performed during the rising stage: in is the air resistance, , then Arrange and accumulate to get Calculation and sorting Finally, the integral of the vertical velocity is the landing height and the formula is: Get the landing time of the parabola ; Then analyze the y coordinate, taking the direction of the parabola's motion as positive, and perform force analysis on it: The "-" indicates the direction of force. is the magnitude of the air resistance, and the deformation integral is obtained get Bringing in the data of the collected trajectory points, we can get and about The expression of The triple combination can be solved immediately to Regarding the expression of t, the landing time has been obtained in the vertical direction analysis: , so the y coordinate of the final landing point is in, is the speed of the object in the x-axis direction, is the speed of the object in the y-axis direction, is the speed of the object in the z-axis direction, (x, y, z) is the coordinate of the object's position, c is a constant, λ is the air resistance coefficient, m is the mass of the object, g is the acceleration of gravity, f is the air resistance, and (X, Y) is the horizontal and vertical coordinates of the landing point; ②Top view trajectory analysis: Since the gravity will not change the horizontal trajectory of the object during the throwing process, and the air resistance is always opposite to the direction of motion, the trajectory of the object will not deviate in the horizontal dimension. Therefore, the trajectory when viewed from above should be a line segment. ) and the highest point ( ) data can obtain the linear relationship between the parabolic coordinates x and y: By substituting the y-coordinate Y of the landing point obtained from the side view, the x-coordinate X of the landing point can be obtained, and the three-dimensional coordinates and time (X, Y, Z, t) of the landing point can be obtained, so as to achieve the purpose of predicting the landing point. After the coordinates of the object's landing point are predicted, they can be transmitted.

5. The high-altitude object throwing warning method based on visual sensing and trajectory prediction according to claim 1 is characterized in that: The method of step 4 is: After receiving the predicted coordinates of the object's landing point, the signal data emission and transmission module will transmit the data instructions to the single-chip microcomputer system. The single-chip microcomputer box controls the pan-tilt head. The strong light on the pan-tilt head will project a strong red light on the landing area and a buzzer alarm will sound. Before the falling object hits the ground, a conspicuous laser light will be used to mark the landing point and a sound alarm will be used to remind pedestrians. A net will be used to intercept the object to prevent it from being hurt, thereby protecting the personal and property safety of pedestrians to the greatest extent. When the warning is completed, the execution program will cause the pan-tilt head to automatically initialize and return to its original position and continue to be in standby mode.

6. A high-altitude object throwing warning system based on visual sensing and trajectory prediction, used to implement the high-altitude object throwing warning method based on visual sensing and trajectory prediction as described in any one of claims 1-5, characterized in that: The system includes: The camera module uses a binocular camera to obtain video frames of the parabolic monitoring area and preprocess the video frames to obtain video data for detecting parabolic events; the preprocessing includes: using grayscale and Gaussian filtering methods to perform morphological processing on the initial image; The communication module uses a 5G communication module to send the images collected by the camera module to the moving object target tracking detection module at the receiving end; and transmit the landing point prediction results and instructions to the network alarm module and the warning light source projection control module; The moving object target tracking and detection module is used to apply the convolutional neural network algorithm retrained on the basis of Inception v4 to identify the categories of high-altitude falling objects and divide them into two modes: non-threatening and threatening. In the non-threatening mode, the system remains in standby mode to reduce false triggering caused by environmental factors or small disturbances, and reduce energy and system losses. When entering the threatening mode, while identifying the falling object, the mathematical model established by analyzing its possible force mode is used to jointly predict the data received by the dual cameras using the Kalman filter method, and the coordinates of the object's landing point are predicted and transmitted; The network alarm module is used to generate a corresponding sound alarm when receiving a signal of a falling object and alert people nearby; The early warning light source projection control module is used to mark and warn of the landing point of falling objects, divide the warning range of this system into areas with sufficiently high accuracy, pre-set the corresponding pan / tilt control parameters for each area in the system, and feedback the predicted landing point coordinates after the algorithm completes the processing of the image. The required parameters are obtained by table lookup, and laser lights are used for illumination to complete the early warning of pedestrians.

7. A high-altitude object throwing warning device based on visual sensing and trajectory prediction, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, each step in the method according to any one of claims 1 to 5 is implemented.

Images