Method, device and medium for identifying objects thrown from high altitude

Abstract

The present invention discloses a method, device and medium for identifying objects thrown from a high altitude, which belongs to the field of image processing technology and is used to solve the technical problem of low detection accuracy of existing methods for identifying objects thrown from a high altitude in special scenarios. The method comprises: determining the edge features of a suspicious moving target in consecutive frames in a video stream; performing time-series processing on the edge features corresponding to the consecutive frames to obtain edge time-series features corresponding to the suspicious moving target; and performing frame difference processing on the edge features to obtain edge frame difference features corresponding to the suspicious moving target; inputting the edge time-series features and the edge frame difference features into a backbone network to identify whether the suspicious moving target is an object thrown from a high altitude. The above method is used to achieve accurate identification of objects thrown from a high altitude in special scenarios such as rainy and snowy weather and interference from flying insects.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device and medium for identifying objects thrown from a high altitude. Background Art

[0002] Among the various common uncivilized behaviors currently occurring in cities, the social harm caused by throwing objects from high-rise buildings is the most serious. Throwing objects from high-rise buildings can cause significant harm to pedestrians, and limited monitoring methods make it difficult to effectively hold victims accountable. Detecting objects thrown from high places typically requires video surveillance equipment to sample building images. Then, through target detection, object tracking, trajectory estimation, and other methods, suspected objects are identified, ultimately determining the floor where the object was thrown. However, in rainy or snowy weather, object detection can mistakenly identify raindrops, snow crystals, and other objects as objects, as well as inaccurately detect flying insects in front of the camera as objects. This can lead to numerous false alarms and reduce the accuracy of the object detection system.

[0003] Patent CN113139478A proposes a method, device, electronic device, and storage medium for detecting objects dropped from high altitudes. The method first acquires surveillance video of objects dropped from high altitudes and parses the video to obtain multiple frames. The multiple frames are then used to obtain inter-frame difference maps, which are then compressed to obtain compressed inter-frame difference maps. A feature matrix is ​​then synthesized based on the compressed inter-frame difference maps. Finally, the feature matrix is ​​input into a binary convolutional neural network classifier to obtain the detection results of objects dropped from high altitudes. This method uses parsed video to obtain inter-frame difference maps. However, due to the large size of the original video images, direct differential processing consumes a large amount of computing resources. Furthermore, the method feeds the feature matrix synthesized from the inter-frame difference maps into a classifier for discrimination, resulting in generalization differences in special scenarios such as raindrops and flying insects.

[0004] Patent CN 114332777B proposes a method for detecting high-altitude parabolic objects. The method first obtains the difference images of each image channel between the current and previous frames of surveillance video and merges them into a first frame difference image of the current frame. This frame difference image is then integrated according to a first period. The difference image between the first integral image of the current frame and the first integral image of the previous frame is then used as the second frame difference image of the current frame. This is then weighted and superimposed according to a second period. Finally, the second integral image of the current frame is used to determine whether a high-altitude parabolic object has occurred. This method integrates multiple frame difference images and then weights the integral images. This process consumes significant computing resources for processing the original video frames. While this frame difference image integration method works well for large, obvious parabolic objects, it lacks accuracy for special scenarios such as the irregular motion of small objects.

[0005] Patent CN112686186A proposes a deep learning-based method for identifying high-altitude objects. The method first acquires a surveillance video sequence and uses frame subtraction to obtain an image of the target area. The high-altitude object sample data is then trained on a ResNet50 model with an attention-based SE module. Finally, the probability of a high-altitude object being thrown is compared with a preset threshold to determine whether the target is a high-altitude object. This method uses frame subtraction to distinguish the target area's overlapping coordinates from the foreground and background. This method is less adaptable to fine-grained, irregularly moving targets, resulting in low detection accuracy.

[0006] In summary, existing methods for identifying objects thrown from high places are not very accurate in special scenarios such as rain, snow, and flying insects. This makes it difficult to further determine the floor and exact location of objects thrown from high places in such special scenarios. Furthermore, existing algorithms for identifying objects thrown from high places are computationally intensive and resource-intensive. Summary of the Invention

[0007] The purpose of this application is to provide a method, device and medium for identifying objects thrown from high places, which are used to solve the following technical problems: the existing methods for identifying objects thrown from high places have low detection accuracy for special scenarios.

[0008] To solve the above technical problems, this application provides a method for identifying objects thrown from high altitudes. The specific technical solutions are as follows:

[0009] Determine the edge features of a suspicious moving target in consecutive frames in a video stream; perform time sequence processing on the edge features corresponding to the consecutive frames to obtain edge time sequence features corresponding to the suspicious moving target; and perform frame difference processing on the edge features to obtain edge frame difference features corresponding to the suspicious moving target; input the edge time sequence features and the edge frame difference features into a backbone network to identify whether the suspicious moving target is a high-altitude parabolic object.

[0010] In one possible implementation, determining the edge features of a suspicious moving target in a video stream in consecutive frames specifically includes: extracting the region of interest (ROI) of consecutive frames corresponding to the suspicious moving target in the video stream; merging the consecutive frame ROIs into a frame image in chronological order; inputting the merged image into an edge frame difference model, and extracting the edge features of the consecutive frame ROIs through an edge detection module of the edge frame difference model.

[0011] In one possible implementation, extracting the region of interest (ROI) of consecutive frames corresponding to the suspicious moving target in the video stream specifically includes: determining the ROI of the i-th frame in the consecutive frames; and segmenting and extracting corresponding ROIs in the i+1 to i+n frames of the consecutive frames according to coordinate position information corresponding to the ROI of the i-th frame to obtain the consecutive frame ROI; wherein i and n are both positive integers.

[0012] In one possible implementation, before determining the ROI of the i-th frame in the continuous frames, the method further includes: determining the image background of the i-th frame image using a background modeling method, and determining the moving target in the i-th frame image based on the image background and the i-th frame image; filtering the moving target using a trajectory filtering method to determine the suspicious moving target in the i-th frame image, wherein, in the ROI of the i-th frame, the area perpendicular to the ground direction is larger than the area parallel to the ground direction.

[0013] In one possible implementation, frame difference processing is performed on the edge features to obtain edge frame difference features corresponding to the suspicious moving target, specifically including: calculating the absolute values ​​of the frame differences between the edge features of the first frame ROI and the edge features of the subsequent frame ROIs in the edge features of the consecutive frames ROIs; and performing time series processing on the absolute values ​​of the frame differences through a concat function to obtain the edge frame difference features corresponding to the suspicious moving target.

[0014] In one possible implementation, after inputting the edge timing features and the edge frame difference features into the backbone network, the method further includes: inputting the edge timing features and the edge frame difference features into the backbone network of the edge frame difference model; obtaining the output features of the backbone network of the edge frame difference model; processing the output features through the AdaptiveAvgPool2D function to obtain feature variance; and determining the contrast loss of the edge frame difference model based on the feature variance.

[0015] In one possible implementation, after obtaining the output features of the backbone network of the edge frame difference model, the method further includes: processing the output features through the AdaptiveAvgPool2D function to obtain a feature vector, and inputting the feature vector into the fully connected layer of the edge frame difference model; obtaining the category probability output by the fully connected layer through the softmax function; when the category probability is misclassified, determining the loss weight according to the feature variance, and determining the comparative loss of the edge frame difference model through the loss weight and the category probability.

[0016] In one possible implementation, a loss weight is determined based on the feature variance, and the contrast loss of the edge frame difference model is determined by the loss weight and the category probability, specifically including: obtaining the maximum and minimum values ​​of the feature variance; adding 1 to the difference between the maximum and minimum values ​​to determine the loss weight; calculating the product of the category probability and the category probability after taking the log function, and calculating the final product between the loss weight and the product; and determining the final product as the contrast loss of the edge frame difference model.

[0017] In one possible implementation, after obtaining the category probability output by the fully connected layer through the softmax function, the method further includes: determining whether the category probability classification is correct; and multiplying the category probability by the category probability after taking the log function to determine the contrast loss of the edge frame difference model.

[0018] In one possible implementation, before inputting the merged image into the edge frame difference model, the method further includes: collecting videos of high-altitude parabolic objects in rainy and snowy weather and / or videos of high-altitude parabolic objects during periods when flying insects are prone to appear, and determining the i-th frame to i+n-th frame images corresponding to the moving target in the high-altitude parabolic object video; wherein the i-th frame is the frame in which the moving target first appears, and the moving target includes one or more of a high-altitude parabolic object target, a rain and snow target, and a flying insect target; determining the continuous frame ROIs corresponding to the i-th frame to i+n-th frame images, and performing time series processing on the continuous frame ROIs to synthesize time series inputs; determining the time series input corresponding to the high-altitude parabolic object target as a positive sample, and determining the time series input corresponding to the rain and snow target and the flying insect target as negative samples; and inputting the positive sample and the negative sample into the edge frame difference model for training.

[0019] The present application also provides a device for identifying objects thrown from high places, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, so that the at least one processor can execute a method for identifying objects thrown from high places as described above.

[0020] The present application also provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute a method for identifying objects thrown from a high altitude as described above.

[0021] The present invention provides a method, device and medium for identifying high-altitude objects, which have the following beneficial effects: edge timing features and edge frame difference features of continuous frame ROIs are extracted through an edge frame difference model, and then these two features are input into a backbone network for classification to identify high-altitude objects, which can effectively improve the detection rate and recognition accuracy of high-altitude objects in special scenes such as rainy and snowy weather and flying insects. At the same time, when training the aforementioned edge frame difference model, the inter-frame temporal feature variance is weighted to expand the classification boundary by constructing a contrast loss, which can effectively distinguish high-altitude objects from samples such as raindrops and flying insects that are prone to false detection, effectively improving the generalization ability of the model and improving the detection accuracy. Moreover, by performing regional segmentation on the continuous frame video images of the video stream, the temporal input of the continuous frame ROI synthetic edge frame difference model is obtained, which can also effectively reduce the amount of calculation and save resource usage. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0023] Figure 1 A flow chart of a method for identifying objects thrown from a high altitude provided in an embodiment of the present application;

[0024] Figure 2 A schematic diagram of a region of interest in consecutive frames provided in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of edge features provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of edge timing characteristics provided in an embodiment of the present application;

[0027] Figure 5 A schematic diagram of a frame difference processing result provided in an embodiment of the present application;

[0028] Figure 6 A schematic diagram of an edge frame difference feature provided in an embodiment of the present application;

[0029] Figure 7 A schematic diagram of a comparative loss calculation process provided in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of the structure of an edge frame difference model provided in an embodiment of the present application;

[0031] Figure 9 A flow chart of another method for identifying objects thrown from a high altitude provided in an embodiment of the present application;

[0032] Figure 10 A schematic diagram of the structure of a device for identifying objects thrown from a high place provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the technical solutions in this application, the following will clearly and completely describe the technical solutions in the embodiments of this application in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0034] The present application embodiment provides a method for identifying objects thrown from a high altitude, such as Figure 1 As shown, the method in the embodiment of the present application includes at least the following execution steps:

[0035] Step 101: Determine edge features of a suspicious moving target in consecutive frames in a video stream.

[0036] The method for identifying objects thrown from high places provided in the embodiment of the present application first processes the video stream, which can be a surveillance video stream corresponding to the period when objects thrown from high places occur, or a real-time surveillance video stream of the area where objects thrown from high places occur. The video stream is image-cut according to the frame number to obtain continuous frame video images corresponding to the video stream. Thereafter, the continuous frame video images are modeled using a background modeling method / technique. After modeling, the stationary part or the unchanged part in the continuous frame video images is determined as the background. In this way, the moving target in the continuous frame video images can be determined, that is, the changed part in the continuous frame video images can be determined.

[0037] Since moving targets such as high-altitude objects have certain motion trajectories, the embodiment of the present application adopts a trajectory filtering method / technology to filter the aforementioned moving targets, screen out suspicious moving targets among the moving targets, that is, screen out moving targets that may be high-altitude objects, and then find the frame where the suspicious moving target first appears and the frame where it last appears in the continuous frame video image, and then process the video image between these two frames as follows: in the frame where the suspicious moving target first appears, determine the coordinate position corresponding to the suspicious moving target, where the coordinate position can be a pixel position coordinate, and then, with the pixel position coordinate as the center, expand a preset number of pixels upward and downward in the aforementioned first-appearing frame, At the same time, a preset number of pixels can be expanded to the left and right. After the expansion is completed, the frame where the suspicious moving target first appears is segmented and intercepted according to the expanded pixels to achieve regional segmentation, and the region of interest ROI corresponding to the frame where the suspicious moving target first appears is obtained. In addition, the frames where the suspicious moving target subsequently appears are also segmented in the same segmentation method to obtain the region of interest ROI of consecutive frames. The same segmentation method here can be based on the coordinate position information corresponding to the ROI segmented out of the frame where the suspicious moving target first appears, and the subsequent frame images are segmented and processed. This can ensure that the segmented consecutive frame ROIs have the same background. In this case, the motion trajectory of the suspicious moving target can be more clearly reflected.

[0038] It should be noted that, in the above continuous frame ROIs, the area perpendicular to the ground is larger than the area parallel to the ground, so that the segmented ROI can better adapt to the longitudinal trajectory changes of the suspicious moving target.

[0039] For example, the frame where a suspicious moving target first appears is recorded as frame i. If the suspicious moving target appears in frames i through i+n, and the pixel position coordinates of the suspicious moving target are determined to be (x, y) in frame i, then the pixel position coordinates are used as the center point, and the image is expanded 48 pixels upward, 48 pixels downward, and 16 pixels left and right respectively. The so-called expansion actually starts from the center point and counts a certain number of pixels upward, downward, left, and right, thus extending the image by a certain number of pixels to determine the region of interest. When taking 6 consecutive frames, the image size of the ROI after stitching the consecutive frames is 96x96, which is convenient for subsequent model training.

[0040] Figure 2 This is a schematic diagram of a region of interest in a continuous frame provided in an embodiment of the present application. Figure 2 As described above, the frames corresponding to the six regions of interest are continuous from left to right, and the background parts of the six regions of interest do not change. The suspicious moving target ( Figure 2 The white dots in the figure can form a motion trajectory from left to right in these six regions of interest.

[0041] It should be noted that the aforementioned background modeling method and trajectory filtering method can be implemented through existing technical means, and will not be elaborated in detail in the embodiments of the present application.

[0042] Furthermore, after obtaining the continuous frame ROIs, the continuous frame ROIs are merged and input into the edge frame difference model. Merging the input here means splicing the continuous frame ROIs according to the number of previous and subsequent frames to obtain a single image, which is then input into the edge frame difference model. It should be noted that segmenting the continuous frame video images to obtain the continuous frame ROIs and then merging the continuous frame ROIs into the edge frame difference model avoids the traditional approach of directly inputting video frame images into the model, reducing the model's computational workload and conserving resources. Furthermore, merging the continuous frame ROIs also simplifies the model's input process.

[0043] Afterwards, the edge features of the continuous frame ROI, that is, edge information, are extracted through the edge detection module of the model. Specifically, the edge detection module first divides the input of the edge frame difference model according to the number of frames through the split operation. After the segmentation is completed, it is weighted by the Sobel operator in the x-direction and the y-direction, and the square sum is calculated. Finally, the square root is taken to obtain the edge features of the continuous frame ROI.

[0044] In one or more possible implementations of the embodiments of the present application, the edge detection module may extract edge features of ROIs of consecutive frames using the following algorithm:

[0045]

[0046]

[0047]

[0048]

[0049] Among them, Ops x 、Ops y Represents the Sobel operator in the x direction and y direction respectively; split means splitting the input of the model according to fs to obtain the image fragment f i , and f i =[f1,f2,...f n ], fs is the number of frames of continuous frame ROI, c is the number of channels; e i is the edge feature of ROI in consecutive frames, and e i =[e1,e2,...e n ].

[0050] Figure 3 A schematic diagram of edge features provided in an embodiment of the present application is shown as follows: Figure 3 As shown in FIG, from left to right, the edge features corresponding to the ROIs of consecutive frames are represented. By extracting the edge features, the trajectory of the suspicious moving target can be better seen.

[0051] Step 102: Perform time sequence processing on the edge features corresponding to the consecutive frames to obtain edge time sequence features corresponding to the suspicious moving target; and perform frame difference processing on the edge features to obtain edge frame difference features corresponding to the suspicious moving target.

[0052] After obtaining the edge features corresponding to the ROI of consecutive frames, the edge temporal features corresponding to the suspicious moving target are calculated using the following algorithm:

[0053] feature edge =concat(e1,e2,...e n )

[0054] Among them, feature edge is the edge time series feature. The concat function is used to combine the edge feature e i Connect the elements in .

[0055] Figure 4 A schematic diagram of edge timing characteristics provided in an embodiment of the present application is shown as follows: Figure 4 As shown in the figure, the edge timing features are actually connected / joined together from left to right by the edge features.

[0056] Furthermore, after obtaining the edge features of the continuous frame ROI, it is also necessary to use the edge features to calculate the edge frame difference features corresponding to the suspicious moving target through the frame difference processing module of the edge frame difference model. Specifically, first calculate the absolute value of the frame difference between the edge features of the first frame and the edge features of each subsequent frame in the continuous frame ROI. Then, the obtained multiple frame difference absolute values ​​are connected through the concat function to obtain the edge frame difference features corresponding to the suspicious moving target.

[0057] In one or more possible implementations of the embodiments of the present application, the frame difference processing module may calculate the edge frame difference features corresponding to the suspicious moving target using the following algorithm:

[0058]

[0059]

[0060] in, is the absolute value of the frame difference between the first frame and the fs frame of the continuous frame ROI, e1 is the edge feature of the first frame, e fs is the edge feature of the fs frame, feature diff is the edge frame difference feature.

[0061] Figure 5 A schematic diagram of a frame difference processing result provided in an embodiment of the present application is shown in FIG. Figure 5 In the figure, from left to right, they represent the difference between the edge features of the first frame and the edge features of each subsequent frame. Figure 6 A schematic diagram of edge frame difference characteristics provided in an embodiment of the present application is shown as follows: Figure 6 As shown in FIG, the edge frame difference feature is essentially obtained by connecting / splicing the absolute values ​​of the frame differences.

[0062] Step 103: Input the edge timing features and the edge frame difference features into a backbone network to identify whether the suspicious moving target is a high-altitude parabolic object.

[0063] In one or more possible implementations of the embodiments of the present application, the specificity of continuous frame ROI processing of high-altitude objects causes the objects in the picture to move rapidly from top to bottom, which has a predictable linear trajectory compared to other disordered moving targets such as raindrops and flying insects. Therefore, in the embodiments of the present application, the aforementioned extracted edge timing features and edge frame difference features are input into the backbone network of the edge frame difference model together for feature extraction and classification. Finally, it can be determined whether the suspicious moving target is a high-altitude object based on the classification results. In one example of the present application, the backbone network adopts MobileNetV3.

[0064] It should be noted that the aforementioned edge timing features and edge frame difference features are input into the backbone network together. The edge timing features and edge frame difference features can be first spliced ​​according to the image channel, and then input into the backbone network after the splicing is completed. Splicing according to the image channel here can be simply understood as stacking the edge timing features and edge frame difference features on top of each other. This allows the two features to be fused, making the motion trajectory shown in the two features clearer. At the same time, inputting these two features into the backbone network together can also enable the backbone network to learn two types of features at the same time. Using the two types of features as the basis for distinguishing high-altitude projectiles can better ensure recognition accuracy.

[0065] It should also be noted that in the embodiment of the present application, the edge frame difference feature and the edge timing feature are selected to be input into the backbone network because the edge frame difference feature and the edge timing feature can describe the trajectory change trend of the suspicious moving target in the continuous frame image from two perspectives. At the same time, the edge frame difference feature obtained by frame difference processing can well highlight the motion trajectory of the suspicious moving target. Therefore, these two features can well reflect the position changes of false detection targets such as high-altitude parabolic targets, raindrops and / or flying insects in the time and space state, so they can well filter out false detection targets in many difficult scenarios, which is also something that traditional features cannot achieve.

[0066] Furthermore, before the aforementioned edge frame difference model is used, it must be trained. In one example of the present application, the training process of the edge frame difference model is as follows:

[0067] First, obtain training samples: collect N different high-altitude object dropping surveillance video sequences. These high-altitude object dropping videos can include videos of high-altitude object dropping in rainy and snowy weather and / or during periods when flying insects are most likely to appear. The ROI region from frame i to frame i+n, where the moving target appears, is synthesized into a time-series input. The time-series input here refers to a single image formed by merging the ROI regions, where high-altitude object dropping targets are positive samples and rainy and snowy targets and flying insect targets are negative samples (positive sample labels are 1, negative sample labels are 0).

[0068] Next, we fed positive and negative samples into the model for training. Based on the Pytorch deep learning framework, we fed the time series samples into the network and pre-trained it for 100 epochs using the AdamW optimizer (with batch_size = 128 and lr = 0.01). We then fine-tuned it for 200 epochs using the SGD optimizer (with batch_size = 32 and lr = 0.001). CrossEntropy and ContrastLoss were used as classification losses during training.

[0069] After the training is completed, the edge frame difference model used above is obtained.

[0070] In one or more possible implementations of the present invention, real-time processing of video image frames to determine whether objects have been thrown from a high altitude can lead to confusion by extracting suspicious motion trajectories due to the constantly changing location of the target. Furthermore, simple time-series frame feature extraction cannot fully distinguish nonlinear trajectories from those of objects thrown from a high altitude in complex scenes such as rain, snow, and flying insects.

[0071] Therefore, in the embodiment of the present application, a contrast loss ContrastLoss is constructed during the above-mentioned model training process to process the relationship between the edge frame differences from the i-th frame to the i+n-th frame. During the feature extraction process, the originally similar samples remain similar in the feature space after dimensionality reduction. On the contrary, the samples with differences between each frame gradually become more different in the feature space during the scale dimensionality reduction and feature extraction process. By adjusting the sample output by ContrastLoss during the gradient backpropagation process, the degree of inter-frame matching can be better expressed to extract the semantic information of the temporal frame difference of the network.

[0072] Specifically, the output features of the backbone network of the edge frame difference model are obtained and the contrast loss is calculated using the output features. Figure 7 A schematic diagram of a contrast loss calculation process provided in an embodiment of the present application is shown in FIG. Figure 7 As shown in Figure 2, after the edge timing features and edge frame difference features are input into the backbone network, the backbone network outputs a feature map, which is the aforementioned output feature. Figure 7 It can be seen that the dimension of the feature map is 6×6.

[0073] First, the output features are processed by the AdaptiveAvgPool2D((1, 6)) function to obtain a feature vector of dimension (N, 6), and the feature variance is calculated using this feature vector as follows:

[0074]

[0075] Among them, f idx is the element in the eigenvector, N is the number of elements, μ is the mean of the eigenvector, var f is the characteristic variance calculated based on the eigenvector.

[0076] Secondly, the output features are processed by the AdaptiveAvgPool2D((1, 1)) function to obtain a feature vector of dimension (N, 1). This feature vector is used as the input of the fully connected layer. The fully connected layer outputs the category probability through softmax as follows:

[0077]

[0078] Among them, f(z i ) is the class probability, is the category of softmax output.

[0079] Finally, the contrast loss of the edge frame difference model is calculated by the following formula:

[0080] weight=(max(var f )-min(var f ))+1

[0081]

[0082] Among them, weight is the loss weight, Loss contrast is the final contrast loss.

[0083] From the above formula, we can see that when the aforementioned category probability is incorrectly judged, that is, when prob=wrong, the difference between the maximum and minimum values ​​of the feature variance will be added to 1 to determine the loss weight. After that, the loss weight is multiplied by the category probability and its log function as the final contrast loss. When the aforementioned category probability is correctly judged, that is, when prob=right, the category probability and its log function product are directly used as the final contrast loss. That is, Figure 7 The CE in represents the product of the category probability and its log function.

[0084] Therefore, this application constructs ContrastLoss, which brings the feature contrast differences in batch samples into the loss backpropagation process of calculation errors during the model learning process, increases the parabolic feature discrimination during the model learning process, and thus can improve the discrimination accuracy of the edge frame difference model.

[0085] Figure 8 This is a structural diagram of an edge frame difference model provided in an embodiment of the present application. Figure 8 As shown in the figure, after the continuous frame ROIs are merged and input into the model, the edge features of the continuous frames will be extracted, and then the edge features will be processed to obtain the edge frame difference features and edge timing features, and then the edge frame difference features and edge timing features will be input into the backbone network together, and the classification output will be obtained through the backbone network.

[0086] Figure 9 This is another flow chart of a method for identifying objects thrown from a high altitude provided in an embodiment of the present application. Figure 9As shown, first, continuous frame images are obtained in the real-time video stream, and then each frame image is processed using background modeling technology to determine the moving target. Then, the feasible parabolic area, that is, trajectory filtering technology, is used to filter the moving target and determine the suspicious parabolic objects. The target segmentation is performed from the i-th frame to the i+n-th frame where the suspicious parabolic objects exist. After the segmentation is completed, the target is merged and input into the trained edge frame difference model. The model is used to identify the suspicious parabolic objects. If it is determined to be a suspicious parabolic object, a parabolic warning is issued. If not, the suspicious parabolic objects continue to be monitored in real time.

[0087] It should also be noted that the above-mentioned edge frame difference model can be used to extract the edge timing features and edge frame difference features of suspicious moving targets in the video stream. The backbone network of the model can then use these two features to effectively identify high-altitude objects in special scenarios such as rain, snow, and flying insects, thereby improving the accuracy of identifying high-altitude objects. In this way, on the premise of accurately identifying high-altitude objects, the motion trajectory of high-altitude objects in the video stream can be used to calculate the floor where the high-altitude objects occurred, thereby achieving accurate positioning of the floor where the high-altitude objects occurred, and making it easier for victims to be held accountable.

[0088] In addition, the present invention also provides a device for identifying objects thrown from a high altitude, the structure of which is as follows: Figure 10 shown.

[0089] Figure 10 This is a schematic diagram of the structure of a device for identifying objects thrown from a high altitude provided in an embodiment of the present application. Figure 10 As shown, the device 1000 specifically includes: at least one processor 1001; and a memory 1003 that is communicatively connected to the at least one processor 901 (connected via a bus 1002); wherein the memory 1003 stores instructions that can be executed by the at least one processor 1001, so that the at least one processor 1001 can execute a method for identifying high-altitude objects as described in the above embodiment.

[0090] In one or more possible implementations of the embodiments of the present application, the aforementioned processor is used to execute: determining the edge features of a suspicious moving target in consecutive frames in a video stream; performing time sequence processing on the edge features corresponding to the consecutive frames to obtain the edge time sequence features corresponding to the suspicious moving target; and performing frame difference processing on the edge features to obtain the edge frame difference features corresponding to the suspicious moving target; inputting the edge time sequence features and the edge frame difference features into a backbone network to identify whether the suspicious moving target is a high-altitude parabolic object.

[0091] In addition, an embodiment of the present application further provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute a method for identifying objects thrown from a high altitude as described above.

[0092] In one or more possible implementations of the embodiments of the present application, the aforementioned executable computer instructions are configured to execute, determine the edge features of a suspicious moving target in consecutive frames in a video stream; perform timing processing on the edge features corresponding to the consecutive frames to obtain the edge timing features corresponding to the suspicious moving target; and perform frame difference processing on the edge features to obtain the edge frame difference features corresponding to the suspicious moving target; input the edge timing features and the edge frame difference features into a backbone network to identify whether the suspicious moving target is a high-altitude parabolic object.

[0093] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device and computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0094] The foregoing description describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0095] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for identifying objects thrown from a high altitude, characterized in that: The method comprises: Determine the edge features of suspicious moving targets in consecutive frames in a video stream; Performing time sequence processing on edge features corresponding to the consecutive frames to obtain edge time sequence features corresponding to the suspicious moving target; and performing frame difference processing on the edge features to obtain edge frame difference features corresponding to the suspicious moving target; Inputting the edge temporal features and the edge frame difference features into the backbone network of the edge frame difference model to identify whether the suspicious moving target is a high-altitude parabolic object; The method also includes: determining the contrast loss of the edge frame difference model, including obtaining the category probability output by the fully connected layer of the edge frame difference model, and when the category probability classification is correct, determining the category probability and the category probability after taking the log function as the contrast loss of the edge frame difference model; when the category probability classification is incorrect, determining the second product between the first product and the loss weight as the contrast loss of the edge frame difference model, wherein the loss weight is obtained by adding one to the difference between the maximum and minimum values ​​of the feature variance obtained after the output feature processing of the backbone network.

2. A method for identifying objects thrown from a high altitude according to claim 1, characterized in that: Determining edge features of a suspicious moving target in consecutive frames in a video stream specifically includes: Extract the region of interest (ROI) of consecutive frames corresponding to suspicious moving targets in the video stream; Merging the consecutive frame ROIs into one image in time sequence; The merged image is input into an edge frame difference model, and the edge features of the continuous frame ROIs are extracted through an edge detection module of the edge frame difference model.

3. A method for identifying objects thrown from a high altitude according to claim 2, characterized in that: The step of extracting the region of interest (ROI) of consecutive frames corresponding to the suspicious moving target in the video stream specifically includes: Determine the ROI of the i-th frame in the continuous frames; According to the coordinate position information corresponding to the ROI of the i-th frame, corresponding ROIs are segmented and extracted in the i+1-th frame to the i+n-th frame of the continuous frames to obtain the continuous frame ROI; wherein i and n are both positive integers.

4. A method for identifying objects thrown from a high altitude according to claim 3, characterized in that: Before determining the ROI of the i-th frame in the continuous frames, the method further includes: Determine the image background of the i-th frame image by using a background modeling method, and determine the moving target in the i-th frame image based on the image background and the i-th frame image; The moving target is filtered by a trajectory filtering method to determine a suspicious moving target in the i-th frame image, wherein, in the ROI of the i-th frame, the area perpendicular to the ground direction is larger than the area parallel to the ground direction.

5. A method for identifying objects thrown from a high altitude according to claim 1, characterized in that: Performing frame difference processing on the edge features to obtain edge frame difference features corresponding to the suspicious moving target specifically includes: Among the edge features of the consecutive frame ROIs, calculating the absolute values ​​of the frame differences between the edge features of the first frame ROI and the edge features of the subsequent frame ROIs; The frame difference absolute value is subjected to time sequence processing by a concat function to obtain edge frame difference features corresponding to the suspicious moving target.

6. A method for identifying objects thrown from a high altitude according to claim 1, characterized in that: After inputting the edge temporal features and the edge frame difference features into a backbone network of an edge frame difference model, the method further includes: Obtaining output features of the backbone network of the edge frame difference model; The output features are processed by the AdaptiveAvgPool2D function to obtain feature variance; A contrast loss of the edge frame difference model is determined based on the feature variance.

7. A method for identifying objects thrown from a high altitude according to claim 6, characterized in that: After obtaining the output features of the backbone network of the edge frame difference model, the method further includes: Processing the output features by using the AdaptiveAvgPool2D function to obtain a feature vector, and inputting the feature vector into the fully connected layer of the edge frame difference model; Obtain the category probability output by the softmax function of the fully connected layer.

8. A method for identifying objects thrown from a high altitude according to claim 2, characterized in that: Before inputting the merged image into the edge frame difference model, the method further includes: Collect videos of objects thrown from high altitudes in rainy or snowy weather and / or during periods when flying insects are likely to appear, and determine the i-th to i+n-th frames corresponding to moving targets in the videos of objects thrown from high altitudes; wherein the i-th frame is the frame in which the moving target first appears, and the moving target includes one or more of the following: objects thrown from high altitudes, rain or snow, and flying insects; Determine the continuous frame ROIs corresponding to the i-th frame to the i+n-th frame image, and perform time sequence processing on the continuous frame ROIs to synthesize a time sequence input; Determine the time series input corresponding to the high-altitude parabolic target as a positive sample, and determine the time series input corresponding to the rain and snow target and the flying insect target as a negative sample; The positive samples and the negative samples are input into the edge frame difference model for training.

9. A device for identifying objects thrown from a high altitude, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, so that the at least one processor can execute the method for identifying objects thrown from high altitude according to any one of claims 1-8.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to execute a method for identifying objects thrown from a high altitude according to any one of claims 1-8.

Images