High-altitude throwing monitoring method and device, electronic equipment and readable storage medium
By improving the multi-layer convolutional neural network and hybrid long short-term memory network model to extract features from video images of objects thrown from high-rise buildings, and combining them with building structure information, the problem of not being able to monitor and track objects thrown from high-rise buildings in real time in existing technologies has been solved, and efficient source tracing of objects thrown from high-rise buildings has been achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA MOBILE M2M
- Filing Date
- 2026-01-20
- Publication Date
- 2026-06-09
AI Technical Summary
Existing technologies are unable to flexibly cope with the complex and ever-changing urban environment and the behavior of objects being thrown from high-rise buildings, and cannot effectively monitor and track such behavior in real time.
An improved multilayer convolutional neural network model and a hybrid long short-term memory network model were used to extract features from video images. Combined with building structure information, the trajectory and launch position of objects thrown from high altitudes were analyzed.
It enables accurate real-time monitoring and tracking of objects thrown from high-rise buildings, improving the accuracy of tracing the source of such objects.
Smart Images
Figure CN122176618A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of image processing technology, specifically relating to a method, device, electronic equipment, and readable storage medium for monitoring objects thrown from heights. Background Technology
[0002] Throwing objects from high-rise buildings has become a serious public safety hazard in modern urban environments, especially in areas with a high density of tall buildings, where its harmfulness and complexity have increased significantly. Among related technologies, high-rise object throwing detection systems typically employ a rule-based approach, identifying throwing behavior by setting fixed monitoring areas and specific detection thresholds, and relying on physical models and accurate measurements of the object's speed, angle, and time to determine the originating floor.
[0003] However, these methods are neither flexible enough to cope with the complex and ever-changing urban environment and the behavior of throwing objects from high-rise buildings, and the fixed rules also lack adaptability, making it impossible to effectively monitor and track the behavior of throwing objects from high-rise buildings in real time. Summary of the Invention
[0004] The purpose of this application is to provide a method, device, electronic device, and readable storage medium for monitoring objects thrown from heights, which can solve the problem of the inability to effectively monitor and track objects thrown from heights in real time.
[0005] In a first aspect, embodiments of this application provide a method for monitoring objects thrown from heights. The method includes: acquiring video images of a monitored area and structural information of buildings; extracting features from the video images using an improved multilayer convolutional neural network model and a hybrid long short-term memory network model to obtain a comprehensive feature vector corresponding to the video images; wherein the comprehensive feature vector includes spatial features and temporal features; determining the trajectory information of the objects thrown from heights in the monitored area based on the comprehensive feature vector corresponding to the video images; and determining the starting position of the objects thrown from heights based on the trajectory information of the objects thrown from heights and the structural information of the buildings.
[0006] Secondly, embodiments of this application provide a high-altitude object throwing monitoring device, which includes: an acquisition module for acquiring video images of a monitored area and structural information of buildings; a feature extraction module for extracting features from the video images using an improved multilayer convolutional neural network model and a hybrid long short-term memory network model to obtain a comprehensive feature vector corresponding to the video images; wherein the comprehensive feature vector includes spatial features and temporal features; a first determination module for determining the trajectory information of the high-altitude object thrown in the monitored area based on the comprehensive feature vector corresponding to the video images; and a second determination module for determining the starting position of the high-altitude object thrown based on the trajectory information of the high-altitude object and the structural information of the buildings.
[0007] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.
[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.
[0009] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the method described in the first aspect.
[0010] In a sixth aspect, embodiments of this application provide a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including a program or instructions, which, when executed, implement the steps of the method described in the first aspect.
[0011] In this embodiment, by acquiring video images of the monitored area and structural information of buildings, the video images are used to extract features through an improved multilayer convolutional neural network model and a hybrid long short-term memory network model to obtain a comprehensive feature vector corresponding to the video images. Based on the improved multilayer convolutional neural network model and the hybrid long short-term memory network model, the acquired video images are processed to analyze the spatial and temporal characteristics of objects thrown from heights, thereby obtaining the trajectory information of the objects. Then, based on the trajectory information of the objects and the structural information of the buildings, the relative relationship between the objects and the buildings is determined to identify the starting point of the objects. This embodiment differs from relatively fixed identification methods by utilizing an improved multilayer convolutional neural network model and a hybrid long short-term memory network model, which have adaptive learning capabilities and the ability to identify diverse high-altitude object-throwing behaviors. This allows for accurate analysis of video images showing high-altitude object-throwing behavior, and by combining the structural information of the buildings to analyze the trajectory of the objects and their relative relationship with surrounding buildings, thereby improving the accuracy of tracking and tracing the source of objects thrown from heights. Attached Figure Description
[0012] Figure 1 This is a flowchart illustrating a method for monitoring objects thrown from heights provided in an embodiment of this application; Figure 2 This is a flowchart illustrating another method for monitoring objects thrown from heights provided in this application embodiment; Figure 3aThis is a schematic diagram of the structure of a high-altitude object throwing monitoring system provided in an embodiment of this application; Figure 3b This is a flowchart illustrating another method for monitoring objects thrown from heights provided in this application embodiment; Figure 4 This is a schematic diagram of the structure of a high-altitude object throwing monitoring device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0013] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0014] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0015] The high-altitude object throwing monitoring method, device, electronic equipment, and readable storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.
[0016] Figure 1 This is a flowchart illustrating a method for monitoring objects thrown from heights provided in an embodiment of this application. This method can be executed by an electronic device. See also... Figure 1 The method may include the following steps.
[0017] Step 102: Obtain video images of the monitored area and structural information of the buildings.
[0018] The video images of the monitored area are captured in real time by cameras installed on and around the buildings within the monitored area. The structural information of the buildings can be pre-entered and stored by staff for later use, or it can be captured by the aforementioned cameras and calculated.
[0019] Step 104: Extract features from the video image using an improved multi-layer convolutional neural network model and a hybrid long short-term memory network model to obtain a comprehensive feature vector corresponding to the video image; wherein, the comprehensive feature vector includes spatial features and temporal features.
[0020] Among these improvements, the multi-layer convolutional neural network model introduces a dynamic convolutional kernel mechanism. This means that the attention weights of this improved multi-layer convolutional neural network model can be dynamically adjusted, for example, by adjusting the attention weights according to the video images at different time steps. The hybrid long short-term memory network model can adopt a bidirectional long short-term based network structure and introduce a temporal self-attention mechanism. In other words, the attention weights of this hybrid long short-term memory network model can be calculated and determined based on the correlation between video images at different time steps.
[0021] Step 106: Determine the trajectory information of the object thrown from a height in the monitoring area based on the comprehensive feature vector corresponding to the video image.
[0022] The comprehensive feature vector corresponding to the video image includes both spatial and temporal features. Spatial features characterize the spatial dimensions of each object thrown from a height in the video image, such as its location. Temporal features characterize the temporal dimensions of each object thrown from a height in the video image, such as when it appeared. By combining and analyzing the spatial and temporal features, the state of the object thrown from a height at what time and place can be determined. Correlating the states of objects thrown from a height at multiple time steps yields their trajectory information.
[0023] Step 108: Determine the launching position of the object based on the trajectory information of the object thrown from a height and the structural information of the building.
[0024] The structural information of a building may include the number of floors, floor height, relative position between buildings, and building shape.
[0025] In this embodiment, by acquiring video images of the monitored area and structural information of buildings, the video images are used to extract features through an improved multilayer convolutional neural network model and a hybrid long short-term memory network model to obtain a comprehensive feature vector corresponding to the video images. Based on the improved multilayer convolutional neural network model and the hybrid long short-term memory network model, the acquired video images are processed to analyze the spatial and temporal characteristics of objects thrown from heights, thereby obtaining the trajectory information of the objects. Then, based on the trajectory information of the objects and the structural information of the buildings, the relative relationship between the objects and the buildings is determined to identify the starting point of the objects. This embodiment differs from relatively fixed identification methods by utilizing an improved multilayer convolutional neural network model and a hybrid long short-term memory network model, which have adaptive learning capabilities and the ability to identify diverse high-altitude object-throwing behaviors. This allows for accurate analysis of video images showing high-altitude object-throwing behavior, and by combining the structural information of the buildings to analyze the trajectory of the objects and their relative relationship with surrounding buildings, thereby improving the accuracy of tracking and tracing the source of objects thrown from heights.
[0026] In one implementation, step 104 above, which extracts features from the video image using an improved multi-layer convolutional neural network model and a hybrid long short-term memory network model to obtain a comprehensive feature vector corresponding to the video image, may include the following steps.
[0027] Step 1041: Input the video image into the improved multilayer convolutional neural network model for feature extraction to obtain the spatial features corresponding to the video image; wherein, the improved multilayer convolutional neural network model has different attention weight matrices for the video image at different time steps.
[0028] In some embodiments, the video image can be preprocessed first, and image denoising, enhancement, and scaling can be performed on the video image using an adaptive filter based on an attention mechanism, so that the video image can adapt to the input size of the improved multilayer convolutional neural network model.
[0029] The improved multilayer convolutional neural network model introduces a dynamic convolutional kernel mechanism. Specifically, it calculates the attention weight matrix corresponding to the video image at different time steps, enabling the improved multilayer convolutional neural network model to extract features based on the attention weight matrix of that time step. Similarly, it calculates the attention weight matrix corresponding to the video image at other time steps, enabling the improved multilayer convolutional neural network model to extract features based on the attention weight matrix of that other time step. The attention weight matrix of the improved multilayer convolutional neural network model can be dynamically adjusted based on the video image at different time steps, thereby optimizing the feature extraction effect in complex and diverse environments.
[0030] Step 1042: Input the spatial features corresponding to the video image into the hybrid long short-term memory network model for feature extraction to obtain the temporal features corresponding to the video image; wherein, the hybrid long short-term memory network has different attention weight matrices for the video images at different time steps.
[0031] In this hybrid long short-term memory (HSM) network model, the states of the neural network layers are updated according to different time steps, and spatial features are processed through neural network layers with states corresponding to different time steps. The HSM network model also introduces a temporal self-attention mechanism, which calculates the attention weight matrix corresponding to each time step based on the correlation between video images at different time steps. This allows the HSM network model to extract features from the video images at each time step based on this attention weight matrix, thereby optimizing the feature extraction effect.
[0032] Step 1043: Combine the spatial features and the temporal features to obtain the comprehensive feature vector.
[0033] In this embodiment, the improved multi-layer convolutional neural network model introduces a dynamic convolutional kernel mechanism, which optimizes the spatial feature extraction effect and improves the accuracy of spatial features. The hybrid long short-term memory network model introduces a temporal self-attention mechanism, which optimizes the temporal feature extraction effect and improves the accuracy of temporal features. Therefore, the comprehensive feature vector obtained based on the combination of spatial and temporal features also has high accuracy, and further analysis of this comprehensive feature vector can obtain more accurate trajectory information.
[0034] In one implementation, step 106 above, which determines the trajectory information of the object thrown from a height in the monitoring area based on the comprehensive feature vector corresponding to the video image, may include the following steps.
[0035] Step 1061: For the video image at time step t, classify the comprehensive feature vector corresponding to the video image through a graph convolutional network model to determine the state information of the high-altitude object at time step t.
[0036] The status information of objects thrown from high-rise buildings can include the location, type, and confidence level of the objects.
[0037] Step 1062: Determine the state information of the object thrown from the height at time step t+1 based on the state information of the object thrown from the height at time step t and the comprehensive feature vector corresponding to the video image at time step t.
[0038] In some embodiments, the comprehensive feature vector corresponding to the video image at time step t in this step can also be weighted and summed with the attention weight at time step t to obtain a weighted feature vector. The state increment of the object thrown from the high altitude is calculated based on the weighted feature vector or the comprehensive feature vector. Then, the state information of the object thrown from the high altitude at time step t+1 is determined based on the state increment of the object thrown from the high altitude and the state information of the object thrown from the high altitude at time step t.
[0039] Step 1063: Based on the state information of the object thrown from the high altitude at time step t and the state information of the object thrown from the high altitude at time step t+1, determine the trajectory information of the object thrown from the high altitude.
[0040] In this embodiment, the state information of the object thrown from a height at each time step is determined by analyzing the comprehensive feature vectors corresponding to the video images at each time step. The state information of the object thrown from a height at time step t is determined by classifying the comprehensive feature vectors corresponding to the video images using a graph convolutional network model. The state of the object thrown from a height at time step t+1 is determined by combining the state of the object thrown from a height at time step t with the comprehensive feature vectors corresponding to the video images at time step t. By combining the state information of the object thrown from a height at each time step, the trajectory of the object thrown from a height within a certain area over a period of time can be obtained, thus achieving the tracking of the object thrown from a height.
[0041] In one implementation, after determining the trajectory information of the object thrown from a height as described above, the method may further include the following steps.
[0042] Step 1071: Obtain the environmental information of the monitored area at time step t and the initial state information of the object thrown from the height.
[0043] The environmental information may include factors that affect the motion of objects thrown from a height, such as the gravitational acceleration and air resistance of the current monitored area. The initial state information of the object thrown from a height may include initial velocity and initial position. For example, the initial velocity can be estimated from the displacement and time interval of the object thrown from a height over two or more consecutive time steps, or a default initial value can be used and corrected based on the trajectory information determined in real time.
[0044] Step 1072: Based on the kinematic and physical model of the high-altitude projectile, determine the predicted state information of the high-altitude projectile at time step t according to the environmental information and the initial state information.
[0045] Step 1073: Based on the Kalman filter algorithm, update the state information of the high-altitude object at time step t according to the predicted state information of the high-altitude object at time step t.
[0046] In this embodiment of the application, in order to improve the accuracy of tracking the trajectory of objects thrown from high altitudes, environmental information of the monitoring area and the initial state information of the objects thrown from high altitudes are introduced. Using a kinematic and physical model suitable for objects thrown from high altitudes, the predicted state information of the objects thrown from high altitudes at time step t is calculated. Then, based on the Kalman filter algorithm, the state information of the objects thrown from high altitudes at time step t is updated according to the calculated predicted state information. The state information of the objects thrown from high altitudes at time step t+1 is also updated in the same way, thereby realizing the updating of the trajectory information of the objects thrown from high altitudes and optimizing the determined trajectory information of the objects thrown from high altitudes.
[0047] In one implementation, after extracting features from the video image using an improved multilayer convolutional neural network model and a hybrid long short-term memory network model in step 104 to obtain the comprehensive feature vector corresponding to the video image, the method may further include the following steps.
[0048] Step 1051: Obtain sensor data of the object thrown from a height; wherein the sensor data includes acceleration data and angular velocity data.
[0049] Step 1052: Based on the complementary filtering algorithm, determine the corrected attitude information of the high-altitude projectile according to the acceleration data and the angular velocity data; Step 1053: Update the comprehensive feature vector corresponding to the video image based on the corrected posture information of the high-altitude object throwing.
[0050] In this embodiment, after extracting features from the video image using an improved multi-layer convolutional neural network model and a hybrid long short-term memory network model to obtain the comprehensive feature vector corresponding to the video image, sensor data of objects thrown from a height are acquired and, based on a complementary filtering algorithm, the advantages of acceleration data and angular velocity data are combined to generate corrected attitude information of the objects thrown from a height. The comprehensive feature vector is then updated according to the corrected attitude information, so that the comprehensive feature vector used to determine the trajectory of the objects thrown from a height includes the features extracted by the improved multi-layer convolutional neural network model and the hybrid long short-term memory network model, as well as the corrected attitude information. Thus, sensor data also participates in determining the trajectory of objects thrown from a height, achieving more accurate tracking of objects thrown from a height.
[0051] In one implementation, step 108, which determines the launching position of the object based on the trajectory information of the object thrown from a height and the structural information of the building, may include the following steps.
[0052] Step 1081: Determine the boundary area of each floor in the building based on the structural information of the building.
[0053] The boundary area of each floor refers to the area in the coordinate system of the video image of the monitored area where the spatial range corresponding to the facade of each floor may be subject to high-altitude object throwing. The upper and lower boundaries of this boundary area are determined by the height of each floor, and the left and right boundaries are determined by the relative position of each floor in the video image.
[0054] Step 1082: Determine whether the trajectory of the object thrown from a height falls within the boundary area of the floor based on the trajectory information of the object thrown from a height.
[0055] Specifically, when an object is thrown from a height, its trajectory will cross the spatial area corresponding to the exterior facade of a certain floor at a certain moment. However, if the object lands within the boundary area of a certain floor at a particular moment, it indicates that the object has crossed from the inside to the outside of that floor, thus confirming that the object originated from that floor. Furthermore, if the boundary areas of two adjacent floors simultaneously meet certain conditions, or if the two boundary areas overlap, the spatial features extracted by the improved multi-convolutional neural network model can be combined for a comprehensive judgment to improve the accuracy of determining the origin of the thrown object.
[0056] Step 1083: If the trajectory of the object thrown from a height falls into the boundary area of the floor, determine the floor as the starting point of the object being thrown from the height.
[0057] In this embodiment of the application, based on the determined trajectory information of the object thrown from a height, and combined with the boundary area of each floor determined by the structural information of the building, the relative relationship between the object thrown from a height and each floor of the building during its movement is determined, that is, whether the trajectory of the object thrown from a height falls into the boundary area of a certain floor. If the trajectory of the object thrown from a height falls into the boundary area of a certain floor, then that floor can be determined as the source floor of the object thrown from a height, that is, the starting position of the object thrown from a height is determined.
[0058] In one implementation, after determining the origin of the object being thrown from a height, a real-time alarm mechanism is triggered and recorded. Relevant personnel can then take appropriate action. For example, for staff, this could involve alerting pedestrians within a certain area where the object landed, handling the object at the landing point, and issuing warnings and issuing appropriate measures to the person who threw the object. The alarm could take the form of sending messages to the terminal devices held by relevant personnel, or providing audio-visual alerts within the monitored area.
[0059] Figure 2 This is a flowchart illustrating another method for monitoring objects thrown from heights provided in this application, which can be executed by electronic equipment. See also... Figure 2 The method may include the following steps.
[0060] Step 201: Collect video images of the monitored area in real time using a camera.
[0061] Step 202: Input the video image into the improved multi-layer convolutional neural network model to extract the spatial features of the object thrown from the height. Extract the temporal features of the object thrown from the height through the hybrid long short-term memory network model. Combine the spatial and temporal features of the object thrown from the height to obtain a comprehensive feature vector.
[0062] Specifically, step 202 may include the following steps: S21-S24.
[0063] S21, the video image is preprocessed using an attention-based adaptive filter to determine the pixel values of the filtered video image.
[0064] Among them, the filtered video image pixel values .in, The pixel values of the input video image. For the weights of the adaptive filter, The attention weights can be calculated using the following formula: ; ;in, This is the attention query weight.
[0065] S22, inputting the filtered video image pixel values into an improved multi-layer convolutional neural network model to extract the spatial features of high-altitude parabolic objects can be done as follows: .
[0066] in, For the spatial features of the output, The input is the filtered video image pixel value. It is the convolution kernel.
[0067] Among these improvements, the multi-layer convolutional neural network model introduces a dynamic convolutional kernel mechanism, meaning that the attention weights of this improved multi-layer convolutional neural network model can be dynamically adjusted. ; ; in, The convolution kernel at time step t, Here is the attention weight matrix. and For a trainable weight matrix, For the current spatial characteristics, and This is a bias term.
[0068] In other embodiments, extracting the spatial features of objects thrown from a height can be achieved by: .
[0069] in, For the spatial features of the output, For adaptive attention weights: ; in, .
[0070] In this embodiment, multi-level feature extraction is performed on the input video image, and the features obtained from each layer are sequentially input into the next convolutional process: The final output spatial features include features processed by multiple convolutions, thus optimizing the extraction of spatial features.
[0071] in, For activation function, This represents the convolution operation. is the convolution kernel of the (l+1)th layer.
[0072] S23, Update the state of the neural network layers in the hybrid Long Short Term Memory (LSTM) model according to different time steps: ; ; ; in, and The outputs of the forward and backward LSTMs are respectively. }and For output gate, and This represents the state of a neural network layer.
[0073] The hybrid long short-term memory network model introduces a temporal self-attention mechanism, which calculates the attention weights for each time step based on the correlation between video images at different time steps. ; in, Let be the attention weights of time step t for time step i. For attention scoring function: ; in, For a trainable weight matrix, and These are the LSTM outputs for time step t and data packet i, respectively. This is a bias term.
[0074] S24, Comprehensive Feature Vector ,in, For spatial features, It is a time-related feature.
[0075] Step 203: Update the comprehensive feature vector of the high-altitude projectile by combining the accelerometer and gyroscope.
[0076] Specifically, step 203 may include the following steps: S31-S33.
[0077] S31 collects acceleration and angular velocity data of objects thrown from high altitudes.
[0078] S32, based on the complementary filtering algorithm, determines the corrected attitude information of high-altitude projectiles based on acceleration and angular velocity data.
[0079] The corrected attitude information includes the corrected acceleration. : ;in, These are the filter coefficients. The corrected acceleration at time step t-1. For time step.
[0080] S33, Update the comprehensive feature vector: ;in, The updated composite feature vector includes the composite feature vector. Corrected acceleration data and angular velocity data .
[0081] Step 204: Classify the comprehensive feature vector using a graph convolutional network model to determine the state information of the high-altitude parabola at time step t.
[0082] The graph convolutional network model consists of l+1 neural network layers, each containing multiple neurons and activation functions. .
[0083] Step 205: Determine the weighted feature vector based on the attention weight and comprehensive feature vector at each time step; determine the state information of the high-altitude projectile at time step t+1 based on the weighted feature vector and the state information of the high-altitude projectile at time step t; and determine the trajectory information of the high-altitude projectile based on the state information of the high-altitude projectile at each time step.
[0084] Specifically, step 205 may include the following steps: S51-S53.
[0085] S51, based on a multi-head attention mechanism, determines the attention weights at each time step. .
[0086] S52, the attention weights at each time step are summed with the current comprehensive feature vector to obtain the weighted feature vector: ;in, For weighted eigenvectors, Let i be the comprehensive feature vector at time step i.
[0087] S53, determine the state information of the high-altitude parabola at time step t+1: ;in, The state increment is determined by the weighted eigenvector. The calculation shows that: ;in, and For trainable parameters, It can be an activation function or an identity mapping.
[0088] Step 206: Update the trajectory information of the object thrown from a height by combining environmental information and the initial state information of the object.
[0089] Specifically, step 206 may include the following steps: S61-S64.
[0090] S61, acquire environmental information of the monitored area and initial status information of objects thrown from heights.
[0091] The environmental information includes factors that affect the motion of objects thrown from heights, such as gravitational acceleration and air resistance in the monitored area. The initial state information of the thrown object can include its initial velocity and initial position.
[0092] S62, based on the kinematic and physical model of high-altitude projectiles, calculates the predicted state information of the high-altitude projectile at time step t based on environmental information and initial state information.
[0093] The kinematic and physical models are shown below: Where m is the mass of the object thrown from a height. For position vectors, As an external force, It is the acceleration due to gravity. It is a unit vector. For air resistance, This is the velocity vector.
[0094] The predicted state information of the high-altitude projectile at time step t can be calculated using the following formula: .
[0095] S63, based on the Kalman filter algorithm, updates the state information of the high-altitude projectile at time step t according to the predicted state information of the high-altitude projectile at time step t: ; ; in, This is the updated state. To predict the state, To obtain environmental and initial state information, For the observation matrix, For Kalman gain, To predict covariance, It is an identity matrix.
[0096] S64, based on the updated status information of objects thrown from high altitudes, updates the trajectory information of objects thrown from high altitudes.
[0097] Step 207: Determine the floor from which the object was thrown based on the trajectory information of the object and the structural information of the building.
[0098] Specifically, step 207 may include the following steps: S71-S73.
[0099] S71, Determine the boundary area of each floor in the building based on the building's structural information.
[0100] S72, determine whether the trajectory of an object thrown from a height falls within the boundary area of a certain floor based on the trajectory information of the object thrown from a height.
[0101] The following formula can be used to determine this: in, Let t be the position of the high-altitude parabola at time step t in the trajectory information. Let i be the boundary region of the i-th floor.
[0102] S73, when the trajectory of an object thrown from a height falls within the boundary area of a certain floor, it is determined that the object originated from that floor.
[0103] Step 208: After determining the floor from which the object was thrown from a height, trigger the real-time alarm mechanism and record the time, location, trajectory, and source floor of the object-throwing incident.
[0104] This application utilizes cameras installed around buildings to capture real-time video images related to objects thrown from heights within the monitored area. The captured video images are then processed in real-time using an improved multi-layer convolutional neural network model and a hybrid long short-term memory network model, enabling multi-level extraction of the spatial and temporal features of the thrown objects. Furthermore, the spatial and temporal features are combined to form a comprehensive feature vector, which is then updated using sensor data, improving the accuracy of the comprehensive feature vector and providing a solid foundation for accurately determining the trajectory of objects thrown from heights.
[0105] Furthermore, after analyzing the comprehensive feature vector to determine the trajectory information of objects thrown from high altitudes, the trajectory information is updated by combining the real-time environmental information of the monitoring area and the initial state information of the objects thrown from high altitudes with the Kalman filter algorithm, which further improves the accuracy and real-time performance of trajectory calculation.
[0106] In addition, after identifying the floor from which the object was thrown, a real-time alarm mechanism was introduced, enabling timely response to such incidents.
[0107] This application also provides a high-altitude object throwing monitoring system, such as... Figure 3a As shown, the high-altitude object throwing monitoring system 300 may include: a camera 31, a gyroscope 32, an accelerometer 33, a processor 34, and an alarm 35. The camera 31, gyroscope 32, accelerometer 33, and alarm 35 are communicatively connected to the processor 34.
[0108] The camera 31, gyroscope 32, accelerometer 33 and alarm 35 may include multiple units, which can be reasonably set based on the structure and range of the monitored area.
[0109] Figure 3b This is a flowchart illustrating another method for monitoring objects thrown from heights provided in this application, applicable to, for example... Figure 3aThe high-altitude object throwing monitoring system shown herein includes the following steps: A camera 31 transmits video images of the monitored area to a processor 34. A gyroscope 32 transmits measured angular velocities to the processor 34. An accelerometer 33 transmits measured accelerations to the processor 34. The processor 34 processes the received video images and sensor data. It extracts features from the video images using an improved multi-layer convolutional neural network model and a hybrid long short-term memory network model to determine the comprehensive feature vector corresponding to the video images. It uses a complementary filtering algorithm to determine the corrected attitude information of the object thrown from the high altitude using the sensor data, updating the comprehensive feature vector. The updated comprehensive feature vector is analyzed to identify the trajectory information of the object thrown from the high altitude. This trajectory information is then updated by combining environmental information of the monitored area and the initial state information of the object thrown from the high altitude, further improving the accuracy of determining the trajectory. Based on the trajectory information of the object thrown from the high altitude and the structural information of the building, it is determined whether the trajectory of the object intersects with the boundary areas of each floor in the building, i.e., whether the position of the object at a certain moment in its trajectory falls into the boundary area of a certain floor. If so, that floor is determined as the source floor of the object thrown from the high altitude. The processor 34 sends an alarm signal to the alarm 35 to trigger the alarm mechanism and records the time, location, trajectory, and floor of the object thrown from a height. The height-throwing object monitoring system can accurately identify the trajectory of objects thrown from heights and respond promptly to such incidents.
[0110] In one exemplary embodiment, the high-altitude object throwing monitoring system provided in this application captured a video image of an object being thrown from a high-rise building in real time at 9 PM one evening using cameras installed around the buildings. The video showed an object being thrown from a window of a high-rise residential building and falling rapidly. The high-altitude object throwing monitoring system immediately initiated real-time processing and analysis of the object.
[0111] First, the high-altitude object throwing detection system processes video images using an improved multi-layer convolutional neural network model and a hybrid long short-term memory network model to identify and detect objects thrown from heights. The detection results show that the object is a beverage bottle, initially positioned above the video frame. Next, the system performs multi-level feature extraction, analyzing the spatial and temporal characteristics of the object in detail, and optimizes the feature extraction effect by incorporating a dynamic convolutional kernel mechanism.
[0112] Secondly, the high-altitude object throwing monitoring system analyzes the comprehensive feature vector obtained after feature extraction and, combined with the attention weight at each time step, determines the trajectory of the object. Then, by incorporating initial state information and environmental information, the trajectory of the object is updated to improve its accuracy.
[0113] Furthermore, the high-altitude object throwing monitoring system judges the trajectory of the object and the boundary area of each floor in the building, determines that the object's trajectory falls into the boundary area of the fifteenth floor, records the time, location, trajectory and floor of the high-altitude object throwing behavior, and triggers a real-time alarm mechanism so that relevant personnel can take timely measures to deal with the high-altitude object throwing behavior.
[0114] Table 1 compares the high-altitude object throwing monitoring method provided in this application with traditional methods:
[0115] The high-altitude object throwing monitoring method provided in this application significantly improves real-time performance, accuracy, and response speed. It efficiently and accurately processes video images by utilizing an improved multi-layer convolutional neural network model and a hybrid long short-term memory network model, and updates the data in conjunction with sensor data, providing an effective foundation for determining the trajectory of objects thrown from heights. Furthermore, it updates the initially determined trajectory by combining environmental information and the initial state information of the thrown object, further improving the accuracy of trajectory determination. When determining the source floor of the thrown object, it incorporates the building's structural information, taking into account the complexity and diversity of buildings, enabling accurate identification of the source floor.
[0116] It should be noted that the high-altitude object throwing detection method provided in this application embodiment can be executed by a high-altitude object throwing detection device, or a control module in the high-altitude object throwing detection device for executing the high-altitude object throwing detection method. This application embodiment uses the execution method of a high-altitude object throwing detection device as an example to illustrate the high-altitude object throwing detection device provided in this application embodiment.
[0117] Figure 4 This is a schematic diagram of the structure of a high-altitude object throwing monitoring device provided in an embodiment of this application. See also... Figure 4 The device 400 may include: an acquisition module 41, a feature extraction module 42, a first determination module 43, and a second determination module 44.
[0118] The system includes: an acquisition module 41 for acquiring video images of the monitored area and structural information of the buildings; a feature extraction module 42 for extracting features from the video images using an improved multi-layer convolutional neural network model and a hybrid long short-term memory network model to obtain a comprehensive feature vector corresponding to the video images; wherein the comprehensive feature vector includes spatial features and temporal features; a first determination module 43 for determining the trajectory information of the object thrown from a height in the monitored area based on the comprehensive feature vector corresponding to the video images; and a second determination module 44 for determining the starting and throwing positions of the object thrown from a height based on the trajectory information of the object thrown from a height and the structural information of the buildings.
[0119] In one implementation, the feature extraction module 42 can be specifically used to: input the video image into the improved multi-layer convolutional neural network model for feature extraction to obtain the spatial features corresponding to the video image; wherein, the improved multi-layer convolutional neural network model has different attention weight matrices for the video image at different time steps; input the spatial features corresponding to the video image into the hybrid long short-term memory network model for feature extraction to obtain the temporal features corresponding to the video image; wherein, the hybrid long short-term memory network has different attention weight matrices for the video image at different time steps; and combine the spatial features and the temporal features to obtain the comprehensive feature vector.
[0120] In one implementation, the first determining module 43 can be specifically used to: classify the comprehensive feature vector corresponding to the video image at time step t using a graph convolutional network model to determine the state information of the object thrown from the height at time step t; determine the state information of the object thrown from the height at time step t+1 based on the state information of the object thrown from the height at time step t and the comprehensive feature vector corresponding to the video image at time step t; and determine the trajectory information of the object thrown from the height based on the state information of the object thrown from the height at time step t and the state information of the object thrown from the height at time step t+1.
[0121] In one implementation, the aforementioned device 400 may further include an update module, configured to acquire environmental information of the monitored area at time step t and initial state information of the object thrown from a height; determine predicted state information of the object at time step t based on the kinematic and physical model of the object, according to the environmental information and the initial state information; and update the state information of the object at time step t based on the predicted state information of the object at time step t using a Kalman filter algorithm.
[0122] In one implementation, the update module described above can also be used to: acquire sensor data of the object thrown from a height; wherein the sensor data includes acceleration data and angular velocity data; determine the corrected attitude information of the object thrown from a height based on the acceleration data and the angular velocity data using a complementary filtering algorithm; and update the comprehensive feature vector corresponding to the video image based on the corrected attitude information of the object thrown from a height.
[0123] In one implementation, the second determining module 44 can be specifically used to: determine the boundary area of each floor in the building based on the structural information of the building; determine whether the trajectory of the object thrown from the height falls into the boundary area of the floor based on the trajectory information of the object thrown from the height; and determine the floor as the starting position of the object thrown from the height if the trajectory of the object falls into the boundary area of the floor.
[0124] The high-altitude object-throwing monitoring device in this application embodiment can be an electronic device or a component of an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.
[0125] The high-altitude object throwing monitoring device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.
[0126] The high-altitude object throwing monitoring device provided in this application embodiment can achieve... Figures 1 to 2 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.
[0127] Based on the same technical concept, this application also provides an electronic device for performing the above-described method for detecting objects thrown from heights. Figure 5 This is a schematic diagram of the structure of an electronic device to implement the various embodiments of this application. The electronic device can vary significantly due to differences in configuration or performance, and may include a processor 501, a communication interface 502, a memory 503, and a communication bus 504. The processor 501, communication interface 502, and memory 503 communicate with each other via the communication bus 504. The processor 501 can call a computer program stored in the memory 503 and executable on the processor 501 to perform the various steps of the above-described embodiments of the high-altitude object throwing monitoring method, achieving the same technical effects. To avoid repetition, further details are omitted here.
[0128] It should be noted that the electronic devices in the embodiments of this application include servers, terminals, or other devices besides terminals. For example, automobiles, robots, and handheld devices.
[0129] The above electronic device structure does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or arrange them differently. For example, an input unit may include a Graphics Processing Unit (GPU) and a microphone, and a display unit may use a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar display panels. User input units include at least one of a touch panel and other input devices. A touch panel is also called a touchscreen. Other input devices may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be elaborated further here.
[0130] Memory can be used to store software programs and various data. Memory can primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area can store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, memory can include volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DRRAM).
[0131] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly handles operations related to the operating system, user interface, and applications, while the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor.
[0132] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described high-altitude object throwing monitoring method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.
[0133] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.
[0134] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described high-altitude object throwing monitoring method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0135] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0136] This application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes a program or instructions. When the program or instructions are executed, they implement the various processes of the above-described high-altitude object throwing monitoring method embodiments and can achieve the same technical effects. To avoid repetition, they will not be described again here.
[0137] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0139] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. A method for monitoring objects thrown from heights, characterized in that, include: Acquire video images of the monitored area and structural information of the buildings; The video image is subjected to feature extraction using an improved multi-layer convolutional neural network model and a hybrid long short-term memory network model to obtain a comprehensive feature vector corresponding to the video image; wherein, the comprehensive feature vector includes spatial features and temporal features; The trajectory information of the object thrown from a height in the monitored area is determined based on the comprehensive feature vector corresponding to the video image. The launching position of the object is determined based on the trajectory information of the object thrown from a height and the structural information of the building.
2. The method according to claim 1, characterized in that, The step of extracting features from the video image using an improved multi-layer convolutional neural network model and a hybrid long short-term memory network model to obtain a comprehensive feature vector corresponding to the video image includes: The video image is input into the improved multilayer convolutional neural network model for feature extraction to obtain the spatial features corresponding to the video image; wherein, the improved multilayer convolutional neural network model has different attention weight matrices for the video image at different time steps; The spatial features corresponding to the video image are input into the hybrid long short-term memory network model for feature extraction to obtain the temporal features corresponding to the video image; wherein, the hybrid long short-term memory network has different attention weight matrices for the video image at different time steps. The spatial features and the temporal features are combined to obtain the comprehensive feature vector.
3. The method according to claim 1, characterized in that, Determining the trajectory information of the object thrown from a height in the monitored area based on the comprehensive feature vector corresponding to the video image includes: For the video image at time step t, the comprehensive feature vector corresponding to the video image is classified through a graph convolutional network model to determine the state information of the high-altitude object at time step t. Based on the state information of the object thrown from a height at time step t and the comprehensive feature vector corresponding to the video image at time step t, the state information of the object thrown from a height at time step (t+1) is determined. Based on the state information of the object thrown from the height at time step t and the state information of the object thrown from the height at time step t+1, the trajectory information of the object thrown from the height is determined.
4. The method according to claim 3, characterized in that, After determining the trajectory information of the object thrown from a height, the method further includes: Obtain the environmental information of the monitored area at time step t and the initial state information of the object thrown from the height. Based on the kinematic and physical model of the high-altitude projectile, the predicted state information of the high-altitude projectile at time step t is determined according to the environmental information and the initial state information. Based on the Kalman filter algorithm, the state information of the object thrown from the height at time step t is updated according to the predicted state information of the object thrown from the height at time step t.
5. The method according to claim 1, characterized in that, After extracting features from the video image using an improved multi-layer convolutional neural network model and a hybrid long short-term memory network model to obtain the comprehensive feature vector corresponding to the video image, the method further includes: Acquire sensor data of the object thrown from a height; wherein the sensor data includes acceleration data and angular velocity data; Based on the complementary filtering algorithm, the corrected attitude information of the high-altitude projectile is determined according to the acceleration data and the angular velocity data; The comprehensive feature vector corresponding to the video image is updated based on the corrected posture information of the object thrown from a height.
6. The method according to claim 1, characterized in that, Determining the launching position of the object based on its trajectory information and the building's structural information includes: The boundary area of each floor in the building is determined based on the structural information of the building; Based on the trajectory information of the object thrown from a height, determine whether the trajectory of the object falls within the boundary area of the floor. If the trajectory of the object thrown from a height falls within the boundary area of the floor, the floor is determined as the starting point of the object being thrown from the height.
7. A high-altitude object throwing monitoring device, characterized in that, include: The acquisition module is used to acquire video images of the monitored area and structural information of the buildings. The feature extraction module is used to extract features from the video image using an improved multi-layer convolutional neural network model and a hybrid long short-term memory network model to obtain a comprehensive feature vector corresponding to the video image; wherein, the comprehensive feature vector includes spatial features and temporal features; The first determining module is used to determine the trajectory information of the object thrown from a height in the monitoring area based on the comprehensive feature vector corresponding to the video image; The second determining module is used to determine the launching position of the object based on the trajectory information of the object thrown from the height and the structural information of the building.
8. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the high-altitude object throwing monitoring method as described in any one of claims 1 to 6.
9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the high-altitude object throwing monitoring method as described in any one of claims 1 to 6.
10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including programs or instructions that, when executed, implement the steps of the high-altitude object throwing monitoring method as described in any one of claims 1 to 6.