A dynamic visual monitoring method and device for high-altitude falling objects

Abstract

The application discloses a kind of dynamic visual monitoring method and device for high-altitude falling object, mainly by high-altitude falling object detector and trajectory predictor two parts consist of, cover two functions of high-altitude falling object event detection based on event camera, high-altitude falling object trajectory prediction, can be realized in certain complex environment for the accurate identification and trajectory tracking of falling object target.Because the event camera itself high dynamic range, low delay etc., compared with the monitoring method based on traditional optical camera, the present application can be applicable to more complex real environment and bring higher monitoring ability.Project results can provide a new solution for the current high-altitude falling object monitoring supervision problem, and bring greater social and economic benefits to society, have important practical significance.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a dynamic visual monitoring method and device for objects falling from heights. Background Technology

[0002] Objects fall at extremely high speeds. Anomaly detection and trajectory prediction of falling objects from heights can help in early warning and liability determination for accidents involving falling objects, which is crucial for maintaining public order. In recent years, target detection algorithms for high-speed moving objects have been based on RGB images captured by traditional optical cameras. However, traditional cameras inherently suffer from information loss in the temporal dimension due to their imaging mechanism, leading to inaccurate detection results. Summary of the Invention

[0003] This invention provides a dynamic visual monitoring method and device for objects falling from heights, which solves or at least partially solves the technical problem of low accuracy of detection results in the prior art.

[0004] To address the aforementioned technical problems, the first aspect of this invention provides a dynamic visual monitoring method for objects falling from heights, comprising:

[0005] Pre-construct object detection networks and trajectory prediction networks;

[0006] The target detection network and trajectory prediction network are trained separately to obtain the trained target detection network and trajectory prediction network;

[0007] Event point data in the scene to be monitored are collected using an event camera, and preprocessed and framed to form an event frame image;

[0008] The event framed image of the scene to be monitored is input into the trained target detection network to obtain the detection result. The detection result includes the target category and the target location information. The target location information and the current time in the detection result are combined into a tuple and stored in the detection result buffer. The event point data used is stored in the event data buffer.

[0009] When the data stored in the detection result buffer reaches 4 frames, the event point data of the corresponding time period in the data buffer is framed to form visual input information. The minimum bounding rectangle is calculated for the tuple corresponding to the 4 frames of data in the detection result buffer, and the position input information is formed according to the position information in the detection result and the size information of the minimum bounding rectangle.

[0010] Visual input information and positional input information are fed into the trained trajectory prediction network to obtain the trajectory prediction result of the detected target.

[0011] In one implementation, the target detection network uses the YOLOv7 algorithm network structure.

[0012] In one implementation, the trajectory prediction network includes a DNN module, a CNN module, and a fully connected layer. The first DNN module is used to extract features from the location input information to obtain spatial location information. The CNN module is used to extract features from the visual input information to obtain image semantic information. The extracted image semantic information is reshaped and then connected with the spatial location information. The result is then input into the second DNN module for feature fusion. The fused features are then input into the fully connected layer to obtain the trajectory prediction result.

[0013] In one implementation, when training the target detection network, the training data used are preprocessed and framed image data of event point data collected by an event camera under different high-altitude falling object scenarios.

[0014] In one implementation, an event camera is used to acquire event point data in the scene to be monitored, and preprocessing and framing operations are performed to construct an event frame image, including:

[0015] Use an event camera to collect event point data in the scene to be monitored;

[0016] Spatial registration and timestamp registration are performed on the collected event point data;

[0017] Given a normalized timestamp, the event stream is divided into N segments. Event data points with a time length of 1 / N around the timestamp are selected and accumulated to obtain a data format of H×W×2, where H, W, and N represent the height, width, and number of segments of the image, respectively, and 2 indicates the positive and negative poles. This data is used to reflect the degree of light intensity change at each pixel and the edge information of moving objects in the scene. Given a normalized timestamp, the timestamp of the last event at each pixel is retained, also resulting in H×W×2 data. This data is the event frame image, used to obtain the motion information of objects.

[0018] In one implementation, each detection target has a corresponding detection result buffer and a data buffer, and each detection result in the detection result buffer corresponds to one frame of data; the minimum bounding rectangle is calculated for the tuple corresponding to the four frames of data in the detection result buffer, and the position input information is constructed based on the position information in the detection result and the size information of the minimum bounding rectangle, including:

[0019] Obtain the midpoint coordinates of the target position information of four consecutive detection results in the detection result buffer;

[0020] Calculate the minimum bounding rectangle of the target, and obtain the height and width of the minimum bounding rectangle;

[0021] The midpoint coordinates of the target location information, the height and width of the minimum bounding rectangle, and the corresponding time are constructed into a tuple (x, y, h, w, t) and used as the location input information.

[0022] In one implementation, when visual input information and position input information are input into the trained trajectory prediction network for trajectory prediction, each input is a tuple constructed from 4 frames of detection results and the corresponding frame image data. After a trajectory prediction is completed, the tuple with the earliest timestamp is deleted from the detection result buffer, and the event data before the timestamp of the tuple is removed from the data buffer. The above steps are repeated to finally obtain the trajectory of the falling object.

[0023] Based on the same inventive concept, a second aspect of the present invention provides a dynamic visual monitoring device for objects falling from heights, comprising:

[0024] Network building blocks are used to pre-build object detection networks and trajectory prediction networks;

[0025] The network training module is used to train the target detection network and the trajectory prediction network separately, so as to obtain the trained target detection network and trajectory prediction network;

[0026] The acquisition and preprocessing module is used to acquire event point data in the scene to be monitored using the event camera, and to perform preprocessing and framing operations to form event frame images.

[0027] The target detection module is used to input the event framed image of the scene to be monitored into the trained target detection network to obtain the detection results. The detection results include the target category and the target location information. The target location information and the current time in the detection results are combined into a tuple and stored in the detection result buffer. The event point data used is stored in the event data buffer.

[0028] The data processing module is used to frame the event point data of the corresponding time period in the data buffer to form visual input information when the data stored in the detection result buffer reaches 4 frames, calculate the minimum bounding rectangle of the tuple corresponding to the 4 frames of data in the detection result buffer, and form position input information based on the position information in the detection result and the size information of the minimum bounding rectangle.

[0029] The trajectory prediction module is used to input visual input information and position input information into the trained trajectory prediction network to obtain the trajectory prediction result of the detected target.

[0030] Based on the same inventive concept, a third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the method described in the first aspect.

[0031] Based on the same inventive concept, a fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0032] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0033] This invention provides a dynamic visual monitoring method for objects falling from heights. It pre-constructs a target detection network and a trajectory prediction network, and trains both networks separately. When monitoring a scene, an event camera collects event point data, which is then preprocessed and framed. The framed images of the events in the scene are input into the trained target detection network to obtain detection results. When the data in the detection result buffer reaches four frames, the event point data for the corresponding time period in the buffer are framed to form visual input information. The minimum bounding rectangle is calculated for the tuples corresponding to the four frames in the detection result buffer, and position input information is constructed based on the position information in the detection results and the size of the minimum bounding rectangle. Finally, the visual input information and position input information are input into the trained trajectory prediction network to obtain the trajectory prediction result of the detected target. Firstly, due to the high dynamic range and low latency of the event camera itself, the method of this invention is applicable to more complex real-world environments and provides higher monitoring capabilities compared to monitoring methods based on traditional optical cameras. Furthermore, this invention constructs and trains a target detection network for target detection and a trajectory prediction network for trajectory prediction. After the target detection network detects the target, the detection result data and the event point data used are stored in corresponding buffers to construct the positional input information and visual input information for the trajectory prediction network. Finally, the trajectory prediction network fuses the two types of information to predict the trajectory, which can greatly improve the accuracy of target detection and trajectory prediction, thereby improving the monitoring effect. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a diagram illustrating the event framing effect of an embodiment of the present invention.

[0036] Figure 2 This is a network structure diagram of the detection algorithm in an embodiment of the present invention;

[0037] Figure 3 This is a diagram showing the target detection effect in an embodiment of the present invention;

[0038] Figure 4 This is a network structure diagram of the trajectory predictor in an embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram of the visual input in the trajectory predictor in an embodiment of the present invention;

[0040] Figure 6 This is a schematic diagram of the position input tuple in the trajectory predictor in an embodiment of the present invention;

[0041] Figure 7 This is an overall flowchart of an embodiment of the present invention;

[0042] Figure 8 A schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present invention;

[0043] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0044] This invention proposes a method for target detection and trajectory prediction using event cameras. Leveraging the high temporal resolution, high dynamic range, and low power consumption of event cameras, it effectively eliminates interference in complex and ever-changing environments, accurately detecting falling objects from heights without revealing excessive information. This invention utilizes event point framing to construct image-like structures (framed images) and employs convolutional neural network-based detection and trajectory prediction algorithms to complete the tasks of detecting and predicting falling objects from heights. The invention mainly consists of two parts: a falling object detector (target detection network) and a trajectory predictor (trajectory prediction network), encompassing two functions: falling object event detection and falling object trajectory prediction based on event cameras. It enables accurate identification and trajectory tracking of falling objects in complex environments. Due to the high dynamic range and low latency of event cameras, compared to monitoring methods based on traditional optical cameras, this invention is applicable to more complex real-world environments and offers higher monitoring capabilities. The project's results provide a novel solution to the current problem of monitoring and regulating falling objects from heights, bringing significant social and economic benefits and possessing important practical significance.

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Example 1

[0047] This invention provides a dynamic visual monitoring method for objects falling from heights, comprising:

[0048] S1: Pre-build the target detection network and trajectory prediction network;

[0049] S2: Train the target detection network and the trajectory prediction network separately to obtain the trained target detection network and trajectory prediction network;

[0050] S3: Use the event camera to collect event point data in the scene to be monitored, and perform preprocessing and framing operations to form an event frame image;

[0051] S4: Input the event framed image of the scene to be monitored into the trained target detection network to obtain the detection results. The detection results include the target category and the target location information. Store the target location information and the current time in the detection results as a tuple in the detection result buffer. Store the event point data used in the event data buffer.

[0052] S5: When the data stored in the detection result buffer reaches 4 frames, the event point data of the corresponding time period in the data buffer is framed to form visual input information. The minimum bounding rectangle is calculated for the tuple corresponding to the 4 frames of data in the detection result buffer, and the position input information is formed according to the position information in the detection result and the size information of the minimum bounding rectangle.

[0053] S6: Input the visual input information and position input information into the trained trajectory prediction network to obtain the trajectory prediction result of the detected target.

[0054] Please see Figure 7 This is an overall flowchart of an embodiment of the present invention.

[0055] Specifically, an object detection network is used to detect objects, and a trajectory prediction network is used to predict trajectories. Both networks are trained and evaluated separately, and then used for object detection and trajectory prediction in a scenario involving falling objects from heights.

[0056] S3 is used to acquire video data in the scene to be monitored. Event point data refers to a series of events output by the event camera when the brightness of the scene changes, also known as an event stream.

[0057] The detection results include the target's category, classification confidence, and target location information. Since it is a single-target detection, only the target location information is used in combination with the current time to form a tuple, which is then stored in the detection result buffer B_detect.

[0058] 1. The detection results obtained by the detection network are categorized into tuples and stored in the detection result cache area B. detect In the process, the event point data used is processed through the event data buffer B. event Store it.

[0059] 2. When the data in the B_detect buffer reaches four frames (each frame corresponds to one detection result, i.e., there are four detection results), the event data in the B_event buffer during this period is framed using time-surface. At the same time, the minimum bounding rectangle is calculated based on the tuples in B_detect. The framed image of the event data in the B_event buffer is cropped and resized. Then, the resized image and the four tuples are input into the trajectory prediction network for prediction to obtain the predicted trajectory and future coordinates.

[0060] 3. After a trajectory prediction is completed, from B... detect Top out the tuple with the earliest timestamp, and start from B event Delete event data prior to the timestamp of this tuple;

[0061] The above steps 1 to 3 are repeated, and the trajectory of the falling target can be obtained in the end.

[0062] This invention can perform target detection and trajectory prediction for high-speed falling objects, and the overall system performance is stable within a certain error range.

[0063] In one implementation, the target detection network uses the YOLOv7 algorithm network structure.

[0064] Specifically, please see Figure 2 , is a network structure diagram of the detection algorithm in this embodiment of the invention.

[0065] In one implementation, the trajectory prediction network includes a DNN module, a CNN module, and a fully connected layer. The first DNN module is used to extract features from the location input information to obtain spatial location information. The CNN module is used to extract features from the visual input information to obtain image semantic information. The extracted image semantic information is reshaped and then connected with the spatial location information. The result is then input into the second DNN module for feature fusion. The fused features are then input into the fully connected layer to obtain the trajectory prediction result.

[0066] Please see Figure 4 This is a network structure diagram of the trajectory predictor in an embodiment of the present invention. The trajectory prediction network is an original and first-proposed network structure in this invention. It consists of convolutional layers and fully connected layers, and utilizes the strong fitting ability of neural networks combined with the semantic information and spatial location information of the input image to fit the motion trajectory of the object.

[0067] In one implementation, when training the target detection network, the training data used are preprocessed and framed image data of event point data collected by an event camera under different high-altitude falling object scenarios.

[0068] In the actual implementation process, the datasets created all used balls as the objects of motion.

[0069] For the detection network, the selected detection algorithm was used for training. Event point data from different high-altitude falling object scenarios were collected, and the model's performance was evaluated using preprocessed and framed image data. The detection results are as follows: Figure 3 As shown.

[0070] For the trajectory prediction network, the visual input and position input are cropped and selected based on the first 4 ground truth bounding boxes (detection box ground truth, which is used to supervise the detection results). The midpoint coordinates of the 4 inputs are randomly perturbed. The trajectory ground truth is the manually labeled trajectory points. The temporal resolution of the labeled trajectory points is higher than that of the prediction. When the temporal resolution of the two is not an integer multiple, bilinear interpolation is used for selection. This constitutes the dataset of the trajectory prediction network.

[0071] In one implementation, an event camera is used to acquire event point data in the scene to be monitored, and preprocessing and framing operations are performed to construct an event frame image, including:

[0072] Use an event camera to collect event point data in the scene to be monitored;

[0073] Spatial registration and timestamp registration are performed on the collected event point data;

[0074] Given a normalized timestamp, the event stream is divided into N segments. Event data points with a time length of 1 / N around the timestamp are selected and accumulated to obtain a data format of H×W×2, where H, W, and N represent the height, width, and number of segments of the image, respectively, and 2 indicates the positive and negative poles. This data is used to reflect the degree of light intensity change at each pixel and the edge information of moving objects in the scene. Given a normalized timestamp, the timestamp of the last event at each pixel is retained, also resulting in H×W×2 data. This data is the event frame image, used to obtain the motion information of objects.

[0075] Please see Figure 1 This is a diagram illustrating the event framing effect of an embodiment of the present invention.

[0076] Specifically, preprocessing includes spatial registration and timestamp registration, which are performed manually. The framed image is a time-surface image, also known as an event-framed image, which is obtained by storing the timestamp of the last event for each pixel, given a normalized timestamp. This data is used to obtain motion information of objects.

[0077] In one implementation, each detection target has a corresponding detection result buffer and a data buffer, and each detection result in the detection result buffer corresponds to one frame of data; the minimum bounding rectangle is calculated for the tuple corresponding to the four frames of data in the detection result buffer, and position input information is constructed based on the position information in the detection result and the size information of the minimum bounding rectangle, including;

[0078] Obtain the midpoint coordinates of the target position information of four consecutive detection results in the detection result buffer;

[0079] Calculate the minimum bounding rectangle of the target, and obtain the height and width of the minimum bounding rectangle;

[0080] The midpoint coordinates of the target location information, the height, width and height of the minimum bounding rectangle, and the corresponding time are constructed into a tuple (x, y, h, w, t) and used as the location input information.

[0081] The structure of the trajectory prediction network is as follows Figure 4 As shown, its input is divided into visual input and position input, and the output is trajectory parameters and coordinates. The visual input is a time-surface image cropped based on the first four detection results, such as... Figure 5 As shown; the position input consists of the midpoint coordinates, height, width, and corresponding normalized time tuples (x, y, h, w, t) of four consecutive detection results, as shown. Figure 6As shown. Assume the trajectory consists of a line, a parabola, and a segmented line containing one bounce, i.e., C(t) = c0 + c1min(2t,1) + c2(2t,1). 2 +c3max(2t-1,0),st0≤t≤1, where c k ∈R 2 If t is the normalized timestamp, then the output fitted trajectory contains 8 parameters (four coefficients from c0 to c3, each coefficient is a two-dimensional vector, so there are 8 parameters).

[0082] In one implementation, when visual input information and position input information are input into the trained trajectory prediction network for trajectory prediction, each input is a tuple constructed from 4 frames of detection results and the corresponding frame image data. After a trajectory prediction is completed, the tuple with the earliest timestamp is deleted from the detection result buffer, and the event data before the timestamp of the tuple is removed from the data buffer. The above steps are repeated to finally obtain the trajectory of the falling object.

[0083] Specifically, each time a detection result is obtained through the detector, a tuple is generated and stored in cache B. detect In this process, the time t in the tuple is accumulated with the input. Different targets have different buffers, and the matching of consecutive frames for different targets is obtained based on the intersection-over-union ratio (IoU). The event point data used is passed through a buffer B. event Store it. When B detect When four frames are accumulated in the buffer, time-surface framing is performed on the events within this time period. The minimum bounding rectangle of the tuple of detection results corresponding to the four frames of data is calculated. The framed images are cropped and resized. Then, the resized image and the tuple consisting of the position information of the detection results and the size of the minimum bounding rectangle are used as visual input and position input, respectively, and fed into the trajectory prediction network for prediction to obtain the predicted trajectory and future coordinates. After one trajectory prediction is completed, from B... detect Top out the tuple with the earliest timestamp, and start from B event Delete event data prior to the timestamp of the tuple. Repeat the above steps until the trajectory of the falling object (detected target) is obtained.

[0084] In summary, the event camera-assisted high-speed moving object target detection and trajectory prediction method of this invention compensates for the information loss in the temporal dimension of traditional optical cameras by utilizing the high temporal resolution of event cameras. Using event histograms, information on the degree of light intensity change at each pixel and the edge information of moving objects in the scene are obtained to construct a frame image. A detection algorithm is used to construct a detector, trained using event stream data from the dataset images, and the frame image is used for performance testing. A trajectory predictor composed of CNN and DNN networks is used to predict the trajectory of falling objects. Finally, qualitative and quantitative analysis of the target detection results shows that the method described in this application, under the premise of low power consumption and privacy protection, achieves superior target detection results compared to RGB image target detection results.

[0085] Table 1. Comparison of experimental results for target detection using RGB images and the method of this application.

[0086]

[0087] Precision: accuracy.

[0088] Recall: the rate of recall.

[0089] mAP0.5: When IoU is set to 0.5, the AP of all images in each class is calculated, and then the average of all classes is calculated, which is mAP.

[0090] mAP0.5:0.95: Average mAP at different IoU thresholds (from 0.5 to 0.95, in steps of 0.05) (0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95).

[0091] Example 2

[0092] Based on the same inventive concept, this embodiment provides a dynamic visual monitoring device for objects falling from heights, comprising:

[0093] Network building blocks are used to pre-build object detection networks and trajectory prediction networks;

[0094] The network training module is used to train the target detection network and the trajectory prediction network separately, so as to obtain the trained target detection network and trajectory prediction network;

[0095] The acquisition and preprocessing module is used to acquire event point data in the scene to be monitored using the event camera, and to perform preprocessing and framing operations to form event frame images.

[0096] The target detection module is used to input the event framed image of the scene to be monitored into the trained target detection network to obtain the detection results. The detection results include the target category and the target location information. The target location information and the current time in the detection results are combined into a tuple and stored in the detection result buffer. The event point data used is stored in the event data buffer.

[0097] The data processing module is used to frame the event point data of the corresponding time period in the data buffer to form visual input information when the data stored in the detection result buffer reaches 4 frames, calculate the minimum bounding rectangle of the tuple corresponding to the 4 frames of data in the detection result buffer, and form position input information based on the position information in the detection result and the size information of the minimum bounding rectangle.

[0098] The trajectory prediction module is used to input visual input information and position input information into the trained trajectory prediction network to obtain the trajectory prediction result of the detected target.

[0099] Since the device described in Embodiment 2 of this invention is the same device used in implementing the dynamic visual monitoring method for falling objects from heights in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this device based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All devices used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0100] Example 3

[0101] Based on the same inventive concept, please refer to Figure 8 The present invention also provides a computer-readable storage medium 300 having a computer program 311 stored thereon, which, when executed, implements the method described in Embodiment 1.

[0102] Since the computer-readable storage medium described in Embodiment 3 of this invention is the same computer-readable storage medium used in implementing the dynamic visual monitoring method for falling objects from heights in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer-readable storage medium based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer-readable storage media used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0103] Example 4

[0104] Based on the same inventive concept, this application also provides a computer device, such as... Figure 9 As shown, it includes a memory 401, a processor 402, and a computer program 403 stored in the memory and executable on the processor. When the processor executes the program, it implements the method in Embodiment 1.

[0105] Since the computer device described in Embodiment 4 of this invention is the same computer device used to implement the dynamic visual monitoring method for falling objects from heights in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer device based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer devices used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0106] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0107] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0109] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations to the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A dynamic visual monitoring method for high-altitude falling objects, characterized in that, The application relates to a method for detecting and predicting the trajectory of a falling object in a scene, and belongs to the field of computer vision. The method comprises the following steps: pre-building a target detection network and a trajectory prediction network; training the target detection network and the trajectory prediction network respectively to obtain the trained target detection network and the trained trajectory prediction network; collecting event point data in a scene to be monitored by using an event camera, and performing preprocessing and framing operations to form an event framed image; inputting the event framed image in the scene to be monitored into the trained target detection network to obtain a detection result, the detection result comprising a target category and position information of the target, storing the position information of the target in the detection result and a current time into a detection result buffer in a multi-tuple form, and storing the event point data used into an event data buffer; when the data stored in the detection result buffer reaches 4 frames, performing framing on the event point data corresponding to the time period in the data buffer to form visual input information, calculating a minimum bounding rectangle of the multi-tuple corresponding to the 4 frames of data in the detection result buffer, and forming position input information according to the position information in the detection result and the size information of the minimum bounding rectangle; 2. The dynamic visual fall-to-height monitoring method of claim 1, wherein, inputting the visual input information and the position input information into the trained trajectory prediction network to obtain a trajectory prediction result of the detected target.

3. The dynamic visual fall-to-height monitoring method of claim 1, wherein, The target detection network adopts the network structure of a YOLOV7 algorithm.

4. The dynamic visual fall-to-height monitoring method of claim 1, wherein, The trajectory prediction network comprises a DNN module, a CNN module and a full connection layer, the first DNN module is used for extracting spatial position information from the position input information, the CNN module is used for extracting image semantic information from the visual input information, the extracted image semantic information is reshaped and then connected with the spatial position information, and then the connection result is input into the second DNN module for feature fusion, the fused features are input into the full connection layer, and finally the trajectory prediction result is obtained.

5. The dynamic visual fall-to-height monitoring method of claim 1, wherein, When the target detection network is trained, the training data is image data obtained by preprocessing and framing on event point data collected by using an event camera in different falling object scenes. The event point data in the scene to be monitored is collected by using an event camera, and preprocessing and framing operations are performed to form an event framed image, which comprises the following steps: collecting the event point data in the scene to be monitored by using an event camera; performing spatial registration and time stamp registration on the collected event point data; 6. The dynamic visual fall-to-height monitoring method of claim 1, wherein, dividing the event stream into N segments according to a given normalized time stamp, selecting event point data with a time length of 1 / N around the time stamp for accumulation, obtaining data in the form of HxWx2, wherein H, W and N respectively represent the height and width of the image and the number of segments, and 2 represents the positive and negative polarities, the data is used for reflecting the light intensity change degree information of each pixel point and the edge information of the moving object in the scene; and retaining the time stamp of the last event of each pixel according to the given normalized time stamp, also obtaining data in the form of HxWx2, which is the event framed image and is used for obtaining the motion information of the object. Each detection target has a corresponding detection result buffer and data buffer, each detection result in the detection result buffer corresponds to one frame of data; the minimum bounding rectangle of the multi-tuple corresponding to the 4 frames of data in the detection result buffer is calculated, and the position input information is formed according to the position information in the detection result and the size information of the minimum bounding rectangle, which comprises Obtaining the midpoint coordinates of the target position information of four continuous detection results in the detection result buffer; Calculating the minimum bounding rectangle of the target, and obtaining the height and width of the minimum bounding rectangle; The midpoint coordinates of the target position information, the height and width of the minimum bounding rectangle, and the corresponding time are constructed into a multivariate group (x, y, h, w, t) as position input information.

7. The dynamic visual fall-to-height monitoring method of claim 1, wherein, When the visual input information and the position input information are input into the trained trajectory prediction network for trajectory prediction, each time, a multivariate group constructed by four frames of detection results and corresponding frame image data are input. When one trajectory prediction is completed, the multivariate group with the earliest timestamp is deleted from the detection result buffer, and the event data before the timestamp of the multivariate group is deleted from the data buffer. The above steps are repeated to finally obtain the trajectory of the falling detection object.

8. A dynamic visual monitoring device for high altitude falling objects, characterized by, Comprise: a network construction module for pre-construction of a target detection network and a trajectory prediction network; a network training module for training the target detection network and the trajectory prediction network respectively to obtain the trained target detection network and the trained trajectory prediction network; a collection and preprocessing module for collecting event point data in a to-be-monitored scene by using an event camera, and performing preprocessing and framing operations to construct event frame images; a target detection module for inputting the event frame images in the to-be-monitored scene into the trained target detection network to obtain detection results, the detection results including target categories and target position information, and constructing a multivariate group of the target position information in the detection results and the current time into a detection result buffer, and storing the collected event point data into an event data buffer; a data processing module for performing framing on the event point data in the corresponding time period in the data buffer to obtain visual input information when the data stored in the detection result buffer reaches four frames, calculating the minimum bounding rectangle of the multivariate group corresponding to the four frames of data in the detection result buffer, and constructing position input information according to the position information in the detection results and the size information of the minimum bounding rectangle; a trajectory prediction module for inputting the visual input information and the position input information into the trained trajectory prediction network to obtain the trajectory prediction result of the detection target.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed to implement the method of any one of claims 1 to 7.

10. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method of any one of claims 1 to 7 when executing the program. The processor implements the method of any one of claims 1 to 7 when executing the program.

Images