Safety risk analysis method and system based on intelligent camera

Abstract

The invention provides a security risk analysis method and system based on an intelligent camera. The method comprises the following steps: acquiring a real-time video stream from a monitoring area through an intelligent camera, and performing target detection on each frame of image in the real-time video stream to obtain a target list; the target list comprises a first detection result of the target object detected in each frame of image; the cloud server receives a target list sent by each intelligent camera; associating the first detection result of the target object according to the spatial position of each intelligent camera and the corresponding target list to obtain a second detection result; according to the second detection result of each target object, carrying out tracking analysis on the motion track of the target object to obtain behavior information and track information of the target object; when it is judged that the behavior information and the track information of the target object meet early warning conditions, an alarm message is sent to terminal equipment; and the terminal equipment displays the alarm information, so that abnormal behaviors and dangerous events can be identified timely and accurately.

Description

Technical Field

[0001] The present application relates to data processing technology, and in particular to a security risk analysis method and system based on smart cameras. Background Art

[0002] A smart camera is a type of imaging device that integrates advanced computing power and intelligent algorithms. It plays a major role in improving the efficiency and accuracy of monitoring and management, and is widely used in security monitoring, traffic management, environmental monitoring, industrial production and other fields.

[0003] With the development of intelligent technology, the security risks faced in security monitoring are becoming increasingly complex and diverse. Especially in the detection of abnormal behavior and dangerous events, traditional monitoring systems often have problems such as low detection accuracy, high false alarm rate, and difficulty in adapting to complex scenarios.

[0004] Therefore, how to apply smart cameras to security monitoring management and accurately identify abnormal behaviors and dangerous events has become a problem that needs to be urgently solved in the current field of security monitoring technology. Summary of the invention

[0005] The present application provides a security risk analysis method and system based on a smart camera, which is used to perform target detection on the real-time video stream collected by the smart camera, and track and analyze the motion trajectory of the target object according to the detection result to obtain the behavior information and trajectory information of the target object, and issue an alarm message when it is judged that the behavior information and trajectory information of the target object meet the warning conditions, so as to timely and accurately identify abnormal behaviors and dangerous events, and provide timely alarms and emergency responses.

[0006] In a first aspect, the present application provides a security risk analysis method based on a smart camera, which is applied to a security protection system, wherein the security protection system includes a plurality of smart cameras, a cloud server, and a terminal device arranged at different locations in a monitoring area; the terminal device and each of the smart cameras are communicatively connected to the cloud server; the method includes: The smart camera collects real-time video stream from the monitoring area; The smart camera performs target detection on each frame of the real-time video stream to obtain a target list; the target list includes a first detection result of the target object detected in each frame of the image, and the first detection result includes at least bounding box information, a confidence score and a category label; The cloud server receives the target list sent by each of the smart cameras; The cloud server associates the first detection result of the target object according to the spatial position of each of the smart cameras and the corresponding target list to obtain a second detection result; The cloud server tracks and analyzes the motion trajectory of the target object according to the second detection result of each target object to obtain behavior information and trajectory information of the target object; When the cloud server determines that the behavior information and trajectory information of the target object meet the warning conditions, the cloud server sends an alarm message to the terminal device; The terminal device displays the alarm information.

[0007] In the above scheme, target detection is performed on the real-time video stream collected by the smart camera, and the movement trajectory of the target object is tracked and analyzed according to the detection result to obtain the behavior information and trajectory information of the target object. When it is judged that the behavior information and trajectory information of the target object meet the warning conditions, an alarm information is issued, thereby being able to timely and accurately identify abnormal behaviors and dangerous events.

[0008] In a possible design of the first aspect, the smart camera performs target detection on each frame of the real-time video stream to obtain a target list, including: The smart camera performs feature extraction processing on each frame of the image to obtain a first feature map corresponding to each frame of the image; The smart camera estimates the position and size of the target object in the first feature map according to the predefined anchor frame to obtain an estimation result; The smart camera filters the prediction results according to preset conditions to obtain the first detection result.

[0009] In the above scheme, the intelligent camera performs feature extraction processing on each frame of the image to obtain the corresponding first feature map, and obtains the first detection result of the target object from the first feature map according to the predefined anchor frame, thereby achieving efficient target detection.

[0010] In a possible design of the first aspect, the smart camera estimates the position and size of the target object in the first feature map according to a predefined anchor frame to obtain an estimation result, including: The smart camera divides the first feature map into a plurality of grid units; For each of the grid units, the smart camera predicts the position and size of the target object in the grid unit according to the predefined anchor frame to obtain the prediction result; the prediction result includes a prediction bounding box, a prediction category and a prediction confidence; The smart camera removes the prediction results corresponding to the overlapping prediction boundary boxes, and screens out the prediction results whose prediction confidence is less than the confidence threshold, to obtain the estimation result.

[0011] In the above scheme, by dividing the first feature map into multiple grid units, the prediction result of the target object is obtained from each grid unit respectively, and then through deduplication and screening operations, a more accurate estimation result of the target object can be obtained, and then a more accurate first detection result can be obtained.

[0012] In a possible design of the first aspect, the cloud server associates the first detection result of the target object according to the spatial position of each of the smart cameras and the corresponding target list to obtain the second detection result, including: The cloud server constructs a cost matrix of M✖N according to the first detection result; wherein N represents the data of the target object included in the target list of one of the smart cameras, and M represents the data of the target object included in the target list of another of the smart cameras; each element in the cost matrix represents the associated cost between the two target objects; The cloud server obtains an initial matching solution according to the cost matrix; The cloud server obtains the optimal matching solution by adding paths and reducing paths according to the initial matching solution; The cloud server determines the target objects that can be associated with each other according to the optimal matching solution; The cloud server associates the first detection results of the target objects that can be associated with each other to obtain the second detection result.

[0013] In the above scheme, the cloud server constructs a cost matrix according to the first detection results of each smart camera, and obtains the optimal matching scheme of the target object according to the cost matrix, thereby associating the first detection results of target objects that can be related to each other to form an overall target tracking result, thereby ensuring the continuity and accuracy of target tracking across cameras.

[0014] In a possible design of the first aspect, the cloud server tracks and analyzes the motion trajectory of the target object according to the second detection result of each target object to obtain behavior information and trajectory information of the target object, including: The cloud server determines the motion trajectory of the target object according to the second detection result of each target object; The cloud server determines the behavior information and trajectory information of the target object according to the movement and posture of the target object in the motion trajectory.

[0015] In the above scheme, the cloud server can jointly determine the motion trajectory of the target object based on the second detection results of each target object, and then determine the behavior information and trajectory information of the target object based on the movement and posture of the target object in the motion trajectory, thereby ensuring that the behavior information and trajectory information of the target object can be accurately tracked and identified in complex scenarios, so as to facilitate timely and accurate identification of abnormal behaviors and dangerous events based on the behavior information and trajectory information.

[0016] In a possible design of the first aspect, when the cloud server determines that the behavior information and the trajectory information of the target object meet the warning condition, sending an alarm message to the terminal device includes: When the cloud server determines that the behavior information belongs to abnormal behavior, or the trajectory information belongs to an abnormal position, the cloud server sends the alarm information to the terminal device.

[0017] In the above scheme, when it is determined that the target object has abnormal behavior and dangerous events, an alarm can be provided in time so that relevant staff can take emergency measures in time.

[0018] In a possible design of the first aspect, the method further includes: The multiple smart cameras are synchronously demarcated and calibrated to ensure that the fields of view of the multiple smart cameras are correctly aligned in space and time.

[0019] In the above solution, the cloud server ensures that the data captured by all smart cameras are synchronized by correctly aligning the fields of view of multiple smart cameras in space and time, thereby enabling the cloud server to achieve accurate target tracking based on the synchronized data.

[0020] In a second aspect, the present application provides a security protection system, comprising: a plurality of smart cameras, a cloud server and a terminal device arranged at different locations in a monitoring area; the terminal device and each of the smart cameras are communicatively connected to the cloud server; The smart camera collects real-time video stream from the monitoring area; The smart camera performs target detection on each frame of the real-time video stream to obtain a target list; the target list includes a first detection result of the target object detected in each frame of the image, and the first detection result includes at least bounding box information, a confidence score and a category label; The cloud server receives the target list sent by each of the smart cameras; The cloud server associates the first detection result of the target object according to the spatial position of each of the smart cameras and the corresponding target list to obtain a second detection result; The cloud server tracks and analyzes the motion trajectory of the target object according to the second detection result of each target object to obtain behavior information and trajectory information of the target object; When the cloud server determines that the behavior information and trajectory information of the target object meet the warning conditions, the cloud server sends an alarm message to the terminal device; The terminal device displays the alarm information.

[0021] In a possible design of the second aspect, the smart camera performs feature extraction processing on each frame of the image to obtain a first feature map corresponding to each frame of the image; The smart camera estimates the position and size of the target object in the first feature map according to the predefined anchor frame to obtain an estimation result; The smart camera filters the prediction results according to preset conditions to obtain the first detection result.

[0022] In a possible design of the second aspect, the smart camera divides the first feature map into a plurality of grid units; For each of the grid units, the smart camera predicts the position and size of the target object in the grid unit according to the predefined anchor frame to obtain the prediction result; the prediction result includes a prediction bounding box, a prediction category and a prediction confidence; The smart camera removes the prediction results corresponding to the overlapping prediction boundary boxes, and screens out the prediction results whose prediction confidence is less than the confidence threshold, to obtain the estimation result.

[0023] In a possible design of the second aspect, the cloud server constructs a cost matrix of M✖N according to the first detection result; wherein N represents data of target objects included in the target list of one of the smart cameras, and M represents data of target objects included in the target list of another of the smart cameras; each element in the cost matrix represents an association cost between two of the target objects; The cloud server obtains an initial matching solution according to the cost matrix; The cloud server obtains the optimal matching solution by adding paths and reducing paths according to the initial matching solution; The cloud server determines the target objects that can be associated with each other according to the optimal matching solution; The cloud server associates the first detection results of the target objects that can be associated with each other to obtain the second detection result.

[0024] In a possible design of the second aspect, the cloud server determines a motion trajectory of the target object according to the second detection result of each target object; The cloud server determines the behavior information and trajectory information of the target object according to the movement and posture of the target object in the motion trajectory.

[0025] In a possible design of the second aspect, when the cloud server determines that the behavior information belongs to abnormal behavior, or the trajectory information belongs to an abnormal location, the cloud server sends the alarm information to the terminal device.

[0026] In a possible design of the second aspect, the cloud server synchronously demarcates and calibrates the multiple smart cameras to ensure that the fields of view of the multiple smart cameras are correctly aligned in space and time.

[0027] In a third aspect, the present application provides an electronic device, including: processor; and, A memory, configured to store executable instructions of the processor; The processor is configured to perform any possible method described in the first aspect by executing the executable instructions.

[0028] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement any possible method described in the first aspect.

[0029] The security risk analysis method and system based on smart cameras provided in the present application collect real-time video streams from the monitoring area through smart cameras, and perform target detection on each frame of the real-time video stream to obtain a target list; the target list includes a first detection result of the target object detected in each frame of the image; the cloud server receives the target list sent by each smart camera; the first detection result of the target object is associated with the spatial position of each smart camera and the corresponding target list to obtain a second detection result; according to the second detection result of each target object, the motion trajectory of the target object is tracked and analyzed to obtain the behavior information and trajectory information of the target object; when it is judged that the behavior information and trajectory information of the target object meet the warning conditions, an alarm message is sent to the terminal device; the terminal device displays the alarm information, so that abnormal behavior and dangerous events can be identified in a timely and accurate manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0031] Figure 1 This is a flowchart of a security risk analysis method based on a smart camera according to an exemplary embodiment of the present application; Figure 2 It is a flow chart of a method for performing target detection on each frame image in a real-time video stream in an embodiment of the present application; Figure 3 is a flowchart of a method for associating a first detection result in an embodiment of the present application; Figure 4 is a structural schematic diagram of a safety protection system according to an exemplary embodiment of the present application; Figure 5 It is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application.

[0032] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0034] As a security monitoring tool based on visual perception, smart cameras play an increasingly important role in modern security management. However, with the development of intelligent technology, the security risks it faces are becoming increasingly complex and diverse. Especially in the detection of abnormal behavior and dangerous events, traditional monitoring systems often have problems such as low detection accuracy, high false alarm rate, and difficulty in adapting to complex scenarios.

[0035] The development of security risk analysis methods based on smart cameras aims to improve the detection effect and accuracy of abnormal behaviors and dangerous events by combining advanced image processing technology, deep learning algorithms and real-time data analysis capabilities. The application of this technology in the monitoring system can not only effectively respond to various security threats, but also provide timely warnings and decision-making support for security management departments.

[0036] However, traditional rule-based anomaly detection methods usually rely on predefined rules and thresholds, which are difficult to adapt to complex and dynamic real-world scenarios. In addition, traditional methods often rely on manually designed features and models, which are difficult to adapt to the diversity and complexity of data, thus limiting the accuracy and real-time performance of detection.

[0037] This application aims to solve the problem of how to apply smart cameras to security monitoring management and accurately identify abnormal behaviors and dangerous events.

[0038] Specifically, the conception of this application revolves around the following core points: Target detection: By setting up multiple smart cameras at different locations in the monitoring area, each smart camera performs target detection on the real-time video stream it collects through a preset target detection model, and obtains a target list including the first detection result of the target object detected in each frame of the image. This requires the model to accurately and efficiently locate and identify the target object.

[0039] Integrate data from multiple smart cameras: The cloud server associates the first detection results of multiple smart cameras according to their spatial positions to form an overall target tracking result, which facilitates a more comprehensive analysis of the motion trajectory of the target object and ensures the continuity and accuracy of target tracking across cameras.

[0040] Abnormal detection and real-time alarm: The cloud server analyzes the movement trajectory of the target object and detects abnormal behavior or dangerous events in real time. Once abnormal behavior or dangerous events are detected, an alarm message will be issued immediately to prompt relevant personnel to carry out further emergency treatment.

[0041] In summary, the concept of this application is to improve the detection effect and accuracy of abnormal behavior and dangerous events. Through target detection, integration of multiple smart camera data, anomaly detection and real-time alarm, abnormal behavior and dangerous events can be identified more timely and accurately. The application of this technology in the monitoring system can not only effectively respond to various security threats, but also provide timely warnings and decision-making support for security management departments.

[0042] The security risk analysis method based on smart camera provided in this application is described in detail below with reference to the accompanying drawings.

[0043] Figure 1 FIG. 1 is a flow chart of a security risk analysis method based on a smart camera according to an exemplary embodiment of the present application. Figure 1 As shown, the security risk analysis method based on the smart camera provided in this embodiment includes: S101. The smart camera collects real-time video streams from the monitoring area.

[0044] S102, the intelligent camera performs target detection on each frame image in the real-time video stream to obtain a target list; the target list includes a first detection result of the target object detected in each frame image, and the first detection result includes at least bounding box information, a confidence score and a category label.

[0045] S103: The cloud server receives the target list sent by each smart camera.

[0046] S104: The cloud server associates the first detection result of the target object according to the spatial position of each smart camera and the corresponding target list to obtain a second detection result.

[0047] S105. The cloud server tracks and analyzes the motion trajectory of each target object according to the second detection result of each target object to obtain behavior information and trajectory information of the target object.

[0048] S106. When the cloud server determines that the behavior information and trajectory information of the target object meet the warning conditions, it sends an alarm message to the terminal device.

[0049] S107: The terminal device displays the alarm information.

[0050] The following is a detailed description of each of the above steps: S101. The smart camera collects real-time video streams from the monitoring area.

[0051] The smart camera-based security risk analysis method of this embodiment can be applied to a security protection system, which includes multiple smart cameras, a cloud server and a terminal device arranged at different locations in a monitoring area; the terminal device and each smart camera are communicatively connected to the cloud server.

[0052] In this step, the smart camera can monitor the scene in the monitoring area in real time and collect the real-time video stream in the monitoring area.

[0053] The field of view of multiple smart cameras may or may not overlap. Through the real-time video streams collected by smart cameras set at different locations, more comprehensive video data of the monitored area can be obtained.

[0054] It should be noted that before smart cameras collect real-time video streams from the monitoring area, multiple smart cameras need to be synchronously calibrated and calibrated to ensure that the fields of view of multiple smart cameras are correctly aligned in space and time.

[0055] Specifically, spatial alignment may include camera intrinsic and extrinsic calibration, coordinate system calibration, and lens distortion correction. Temporal alignment may include hardware-level time synchronization or software-level timestamp synchronization.

[0056] Among them, the camera intrinsic parameters include focal length, aperture size, etc. These parameters directly affect the projection of the image. By using a calibration plate or a specific calibration object, the intrinsic parameters of each smart camera can be obtained to ensure the correspondence between the pixel coordinates in the image and the physical units in the actual space. The camera extrinsic parameters include the position and orientation of the camera. Determining the position and orientation of each camera (i.e., external parameters) is usually achieved using a calibration plate and multi-view images so that images taken by different cameras can be converted to a common coordinate system.

[0057] Coordinate system 1 refers to converting the data of all smart cameras into a unified global coordinate system. Usually a fixed reference coordinate system is selected, such as a ground coordinate system or a specific coordinate system inside a building.

[0058] Lens distortion correction refers to the distortion that may exist in the lens of each smart camera, such as radial distortion and tangential distortion. By using a distortion correction algorithm, such as the correction method provided in the OpenCV library, the distortion in the image can be corrected to make the image more accurate in space.

[0059] Hardware-level time synchronization means that for systems that require high-precision time synchronization, such as high-speed moving target tracking, hardware devices may be needed to achieve time synchronization of smart cameras. This can be achieved by using a dedicated synchronization clock or GPS timing.

[0060] Software-level timestamp synchronization means that if hardware-level time synchronization is not feasible, the timestamp of each camera can be synchronized at the software level through the Network Time Protocol (NTP) or other time synchronization protocols to ensure that the timestamps of each camera are accurately aligned when capturing images.

[0061] In this way, the cloud server can ensure that the data captured by all smart cameras are synchronized by correctly aligning the fields of view of multiple smart cameras in space and time, thereby enabling the cloud server to achieve accurate target tracking based on the synchronized data.

[0062] S102, the intelligent camera performs target detection on each frame image in the real-time video stream to obtain a target list; the target list includes a first detection result of the target object detected in each frame image, and the first detection result includes at least bounding box information, a confidence score and a category label.

[0063] In this step, each smart camera performs target detection on each frame of the real-time video stream it collects, and obtains a target list including a first detection result of the target object detected in each frame of the image. The target object may be different people, different objects, different animals, etc. that appear in the video.

[0064] When smart cameras detect targets in each frame of a real-time video stream, they can use modern deep learning technology, for example, target detection algorithms such as the YOLO (You Only Look Once) algorithm, the Faster R-CNN algorithm, and the SSD (Single Shot MultiBox Detector) algorithm to achieve efficient target detection.

[0065] In one embodiment, Figure 2 As shown, the smart camera uses a real-time target detection algorithm to detect targets in each frame of the real-time video stream. The process of obtaining a target list may include: S1021. The intelligent camera performs feature extraction processing on each frame of the image to obtain a first feature map corresponding to each frame of the image.

[0066] Before performing feature extraction processing on each frame of image, the smart camera may first adjust the size of each input frame of image to a fixed size, for example, 416x416 pixels, and perform normalization processing.

[0067] Exemplarily, the real-time target detection algorithm can be composed of a convolutional neural network and use Darknet as a base network, which contains multiple convolutional layers and pooling layers to extract features of the image.

[0068] The smart camera processes each frame of the image through a convolutional neural network to obtain a first feature map, which retains various levels of semantic information in each frame of the image.

[0069] S1022. The intelligent camera estimates the position and size of the target object in the first feature map according to the predefined anchor frame to obtain an estimation result.

[0070] Among them, predefined anchor boxes are mainly used to solve the bounding box regression problem in target detection. Specifically, when predicting the target bounding box, predefined anchor boxes are used to estimate the position and size of the target object.

[0071] In this step, when the smart camera estimates the position and size of the target object in the first feature map, the first feature map can be divided into multiple grid units. For example, the first feature map can be divided into 13x13, 26x26, and 52x52 grid units. This embodiment does not specifically limit the size of the grid unit division. Each grid unit is responsible for predicting a set of bounding boxes, categories, and confidence levels.

[0072] For each grid cell, the smart camera predicts the position and size of the target object in the grid cell based on the predefined anchor frame to obtain the prediction result; the prediction result includes the predicted bounding box, predicted category and predicted confidence. Among them, the predicted bounding box is mainly defined by 5 main parameters: the center coordinates of the bounding box (x, y), the width and height of the bounding box (w, h), and the confidence score. Each bounding box is also accompanied by a category prediction vector, which is used to represent the probability that the object in the bounding box belongs to each category.

[0073] After obtaining the prediction results, the smart camera removes the prediction results corresponding to the overlapping prediction bounding boxes, and screens out the prediction results whose prediction confidence is less than the confidence threshold to obtain the estimated results.

[0074] Specifically, the smart camera can use a non-maximum suppression algorithm to reduce overlapping bounding boxes and filter out prediction results whose prediction confidence is less than a confidence threshold to obtain a qualified prediction bounding box, as well as the corresponding prediction category and prediction confidence.

[0075] S1023. The intelligent camera screens the prediction results according to preset conditions to obtain a first detection result.

[0076] In this way, by dividing the first feature map into multiple grid units, the prediction result of the target object is obtained from each grid unit respectively, and then through deduplication and screening operations, a more accurate estimation result of the target object can be obtained, and then a more accurate first detection result can be obtained.

[0077] In an example, suppose that for a frame of an image from a smart camera, the object detection model outputs the following list of objects: Target object 1: The bounding box information is (x_min1, y_min1, x_max1, y_max1); the confidence is 0.92; the category label is pedestrian.

[0078] Target object 2: The bounding box information is (x_min2, y_min2, x_max2, y_max2); the confidence level is 0.85; the category label is vehicle.

[0079] This shows that two target objects, namely pedestrians and vehicles, are found in this frame of the smart camera. The bounding box information of each target object can be used to draw a bounding box in the image and further tracking or recognition tasks. The confidence of each target object is used to indicate the probability that the target object is a pedestrian or a vehicle.

[0080] S103. Each smart camera sends a target list to the cloud server.

[0081] Correspondingly, the cloud server receives the target list sent by each smart camera.

[0082] In a possible implementation, in addition to the first detection result, the target list may also include identification information of the smart camera, such as the code of the smart camera, the physical address, port, and identity of the smart camera. This embodiment does not limit the specific type of identification information.

[0083] S104: The cloud server associates the first detection result of the target object according to the spatial position of each smart camera and the corresponding target list to obtain a second detection result.

[0084] In this step, by associating the first detection result of the target object, for example, the detection results of different cameras can be mapped to the global coordinate system by a spatial alignment method. In this way, the data from different smart cameras can be fused to ensure the continuity and accuracy of target tracking across cameras.

[0085] In one embodiment, Figure 3 As shown, the process of the cloud server associating the first detection result of the target object according to the spatial position of each smart camera and the corresponding target list may include: S1041. The cloud server constructs a cost matrix of M✖N based on the first detection result; wherein N represents the data of the target object included in the target list of one smart camera, and M represents the data of the target object included in the target list of another smart camera; each element in the cost matrix represents the association cost between the two target objects.

[0086] The association cost may be based on the similarity of the feature vectors as a distance metric, for example, using the Euclidean distance to calculate the similarity between two target objects, or using the cosine similarity to calculate the similarity between two target objects.

[0087] S1042. The cloud server obtains an initial matching solution based on the cost matrix.

[0088] After constructing the cost matrix, the cloud server can first initialize the cost matrix. During initialization, some preprocessing is performed on the cost matrix, for example, the cost matrix is ​​converted into a form that is convenient for processing, such as ensuring that all costs are non-negative. If necessary, the rows and columns can also be normalized to ensure that the algorithm can run at the minimum cost.

[0089] S1043. The cloud server obtains the optimal matching solution by adding paths and reducing paths according to the initial matching solution.

[0090] In practical applications, the basic idea of ​​finding the optimal matching solution is to modify the edges in the cost matrix to minimize the total associated cost. This process mainly involves two processes: adding paths and reducing paths. These processes enable the optimal matching solution to be found on the cost matrix to minimize the total associated cost.

[0091] Among them, adding paths may include staggered paths and augmentation steps. Staggered paths: By alternating the edges in the matching, paths containing more targets are added to the matching. This can be achieved in the augmentation path stage of the Hungarian algorithm, that is, looking for paths on the cost matrix that can improve the current matching. Augmentation step: Find unmatched target pairs and try to connect them into a staggered path. In this way, the current number of matching can be increased, thereby gradually approaching the optimal solution.

[0092] The method of reducing the path may include modifying the cost matrix. Specifically, modifying the cost matrix means reducing the cost in the current matching in order to find a better matching solution when the matching is incomplete. This includes reducing the weight or cost on the path so that the total matching cost is smaller.

[0093] In practice, in the augmentation path stage, you can try to find an interleaved path that can increase the number of currently matched target pairs. In the cost matrix modification stage, at each step, you may need to modify the edges in the cost matrix to find a better match. This includes adjusting the cost of unmatched target pairs or adjusting the cost of matched target pairs to make the total cost smaller. Repeat the augmentation path and cost matrix modification operations until the optimal matching solution is found.

[0094] S1044. The cloud server determines target objects that can be associated with each other based on the optimal matching solution.

[0095] S1045. The cloud server associates the first detection results of target objects that can be associated with each other to obtain a second detection result.

[0096] S105. The cloud server tracks and analyzes the motion trajectory of each target object according to the second detection result of each target object to obtain behavior information and trajectory information of the target object.

[0097] In this step, the cloud server determines the motion trajectory of the target object based on the second detection result of each target object; the cloud server determines the behavior information and trajectory information of the target object based on the motion and posture of the target object in the motion trajectory.

[0098] Once the target object is detected, the cloud server can perform tracking analysis. In this process, motion tracking algorithms, such as Kalman filter algorithms, can be used to predict the position and direction of movement of the target object.

[0099] Based on the motion trajectory of the target object, the behavior information of the target object can be extracted, such as activity mode and posture, speed change, dwell time, etc. The trajectory information describes the motion path of the target object over a period of time.

[0100] S106. When the cloud server determines that the behavior information and trajectory information of the target object meet the warning conditions, it sends an alarm message to the terminal device.

[0101] In this step, when the cloud server determines that the behavior information belongs to abnormal behavior or the trajectory information belongs to abnormal location, it sends an alarm message to the terminal device.

[0102] In one possible implementation, the behavior information can be compared with the preset abnormal behavior information, and when the similarity is greater than a first similarity threshold, it is determined that the behavior information belongs to abnormal behavior. Similarly, the trajectory information can be compared with the preset abnormal position, and when the similarity is greater than a second similarity threshold, it is determined that the trajectory information belongs to an abnormal position.

[0103] S107: The terminal device displays the alarm information.

[0104] The alarm information is used to notify relevant personnel of detected abnormal behavior or dangerous events. Optionally, the alarm information includes but is not limited to: intrusion alarm, motion detection alarm, perimeter alarm, fire alarm, etc. These alarm information can be sent to relevant personnel via SMS, sound, visual prompts, email, etc., so that necessary response measures can be taken in time to ensure safety and protection.

[0105] Among them, intrusion alarm refers to the alarm triggered when unauthorized entry or crossing of the set boundary is detected. Motion detection alarm refers to the alarm triggered when the smart camera detects unauthorized movement of people or objects. Perimeter alarm refers to the alarm triggered when abnormal behavior or object entry is detected within the set monitoring range. Fire alarm refers to the alarm issued when signs of fire are detected.

[0106] The technical solution of this embodiment is to collect real-time video streams from the monitoring area through smart cameras, and perform target detection on each frame of the real-time video stream to obtain a target list; the target list includes a first detection result of the target object detected in each frame of the image; the cloud server receives the target list sent by each smart camera; the first detection result of the target object is associated with the spatial position of each smart camera and the corresponding target list to obtain a second detection result; according to the second detection result of each target object, the motion trajectory of the target object is tracked and analyzed to obtain the behavior information and trajectory information of the target object; when it is determined that the behavior information and trajectory information of the target object meet the warning conditions, an alarm message is sent to the terminal device; the terminal device displays the alarm information, so that abnormal behavior and dangerous events can be identified in a timely and accurate manner.

[0107] Figure 4 FIG. 1 is a schematic diagram of a safety protection system according to an exemplary embodiment of the present application. Figure 4 As shown, the security protection system 400 provided in this embodiment may include: multiple smart cameras 410, a cloud server 420 and a terminal device 430 arranged at different locations in the monitoring area; the terminal device 430 and each smart camera 410 are communicatively connected to the cloud server 420.

[0108] The smart camera 410 collects real-time video streams from the monitoring area.

[0109] The smart camera 410 performs target detection on each frame image in the real-time video stream to obtain a target list; the target list includes a first detection result of the target object detected in each frame image, and the first detection result includes at least bounding box information, a confidence score and a category label.

[0110] The cloud server 420 receives the target list sent by each smart camera 410 .

[0111] The cloud server 420 associates the first detection result of the target object according to the spatial position of each smart camera 410 and the corresponding target list to obtain a second detection result.

[0112] The cloud server 420 tracks and analyzes the motion trajectory of each target object according to the second detection result of each target object to obtain the behavior information and trajectory information of the target object.

[0113] When the cloud server 420 determines that the behavior information and trajectory information of the target object meet the warning conditions, it sends an alarm message to the terminal device 430.

[0114] The terminal device 430 displays the alarm information.

[0115] In one embodiment, the smart camera 410 performs feature extraction processing on each frame of image to obtain a first feature map corresponding to each frame of image.

[0116] The smart camera 410 estimates the position and size of the target object in the first feature map according to the predefined anchor frame to obtain an estimation result.

[0117] The smart camera 410 screens the prediction results according to preset conditions to obtain a first detection result.

[0118] In one embodiment, the smart camera 410 divides the first feature map into a plurality of grid units. For each grid unit, the smart camera 410 predicts the position and size of the target object in the grid unit according to the predefined anchor frame to obtain a prediction result; the prediction result includes a prediction bounding box, a prediction category, and a prediction confidence; the smart camera 410 removes the prediction results corresponding to the overlapping prediction bounding boxes, and screens out the prediction results whose prediction confidence is less than the confidence threshold to obtain an estimation result.

[0119] In one embodiment, the cloud server 420 constructs a cost matrix of M✖N based on the first detection result; wherein N represents the data of the target object included in the target list of one smart camera 410, and M represents the data of the target object included in the target list of another smart camera 410; each element in the cost matrix represents the association cost between two target objects; the cloud server 420 obtains an initial matching scheme based on the cost matrix; the cloud server 420 obtains an optimal matching scheme based on the initial matching scheme by adding and reducing paths; the cloud server 420 determines target objects that can be associated with each other based on the optimal matching scheme; the cloud server 420 associates the first detection results of the target objects that can be associated with each other to obtain a second detection result.

[0120] In one embodiment, the cloud server 420 determines the motion trajectory of the target object based on the second detection result of each target object; the cloud server 420 determines the behavior information and trajectory information of the target object based on the motion and posture of the target object in the motion trajectory.

[0121] In one embodiment, when the cloud server 420 determines that the behavior information belongs to abnormal behavior, or the trajectory information belongs to an abnormal location, it sends an alarm message to the terminal device 430.

[0122] In one embodiment, the cloud server 420 performs synchronous calibration and demarcation on the multiple smart cameras 410 to ensure that the fields of view of the multiple smart cameras 410 are properly aligned in space and time.

[0123] The security protection system of this embodiment can detect the behavior information and trajectory information of the target object in the monitoring area in real time, and send an alarm message to the terminal device when the behavior information and trajectory information meet the warning conditions, thereby realizing timely and accurate identification of abnormal behaviors and dangerous events, effectively responding to various security threats, and providing timely warnings and decision-making support for security management departments.

[0124] Figure 5 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application. Figure 5 As shown, an electronic device 500 provided in this embodiment includes: a processor 501 and a memory 502; wherein: The memory 502 is used to store computer programs, and the memory may also be a flash memory.

[0125] The processor 501 is used to execute the execution instructions stored in the memory to implement each step in the above method. For details, please refer to the relevant description in the above method embodiment.

[0126] Optionally, the memory 502 may be independent or integrated with the processor 501 .

[0127] When the memory 502 is a device independent of the processor 501, the electronic device 500 may further include: The bus 503 is used to connect the memory 502 and the processor 501 .

[0128] This embodiment further provides a readable storage medium, in which a computer program is stored. When at least one processor of an electronic device executes the computer program, the electronic device executes the methods provided in the above-mentioned various implementation modes.

[0129] This embodiment also provides a program product, which includes a computer program stored in a readable storage medium. At least one processor of the electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device implements the methods provided in the above various embodiments.

[0130] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0131] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A security risk analysis method based on smart cameras, characterized in that: Applied to a security protection system, the security protection system includes multiple smart cameras, a cloud server and terminal devices arranged at different locations in the monitoring area; The terminal device and each of the smart cameras are connected to the cloud server for communication; the method comprises: The smart camera collects real-time video stream from the monitoring area; The smart camera performs target detection on each frame of the real-time video stream to obtain a target list; the target list includes a first detection result of the target object detected in each frame of the image, and the first detection result includes at least bounding box information, a confidence score and a category label; The cloud server receives the target list sent by each of the smart cameras; The cloud server associates the first detection result of the target object according to the spatial position of each of the smart cameras and the corresponding target list to obtain a second detection result; The cloud server tracks and analyzes the motion trajectory of the target object according to the second detection result of each target object to obtain behavior information and trajectory information of the target object; When the cloud server determines that the behavior information and trajectory information of the target object meet the warning conditions, the cloud server sends an alarm message to the terminal device; The terminal device displays the alarm information.

2. The security risk analysis method based on smart camera according to claim 1 is characterized in that: The smart camera performs target detection on each frame of the real-time video stream to obtain a target list, including: The smart camera performs feature extraction processing on each frame of the image to obtain a first feature map corresponding to each frame of the image; The smart camera estimates the position and size of the target object in the first feature map according to the predefined anchor frame to obtain an estimation result; The smart camera filters the prediction results according to preset conditions to obtain the first detection result.

3. The security risk analysis method based on smart camera according to claim 2 is characterized in that: The smart camera estimates the position and size of the target object in the first feature map according to the predefined anchor frame to obtain an estimation result, including: The smart camera divides the first feature map into a plurality of grid units; For each of the grid units, the smart camera predicts the position and size of the target object in the grid unit according to the predefined anchor frame to obtain the prediction result; the prediction result includes a prediction bounding box, a prediction category and a prediction confidence; The smart camera removes the prediction results corresponding to the overlapping prediction boundary boxes, and screens out the prediction results whose prediction confidence is less than the confidence threshold, to obtain the estimation result.

4. The security risk analysis method based on smart camera according to claim 1 is characterized in that: The cloud server associates the first detection result of the target object according to the spatial position of each of the smart cameras and the corresponding target list to obtain a second detection result, including: The cloud server constructs a cost matrix of M✖N according to the first detection result; wherein N represents the data of the target object included in the target list of one of the smart cameras, and M represents the data of the target object included in the target list of another of the smart cameras; each element in the cost matrix represents the associated cost between the two target objects; The cloud server obtains an initial matching solution according to the cost matrix; The cloud server obtains the optimal matching solution by adding paths and reducing paths according to the initial matching solution; The cloud server determines the target objects that can be associated with each other according to the optimal matching solution; The cloud server associates the first detection results of the target objects that can be associated with each other to obtain the second detection result.

5. The security risk analysis method based on smart camera according to claim 1 is characterized in that: The cloud server tracks and analyzes the motion trajectory of the target object according to the second detection result of each target object to obtain behavior information and trajectory information of the target object, including: The cloud server determines the motion trajectory of the target object according to the second detection result of each target object; The cloud server determines the behavior information and trajectory information of the target object according to the movement and posture of the target object in the motion trajectory.

6. The security risk analysis method based on smart camera according to claim 1 is characterized in that: When the cloud server determines that the behavior information and trajectory information of the target object meet the warning conditions, the cloud server sends an alarm message to the terminal device, including: When the cloud server determines that the behavior information belongs to abnormal behavior, or the trajectory information belongs to an abnormal position, the cloud server sends the alarm information to the terminal device.

7. The security risk analysis method based on smart camera according to claim 1 is characterized in that: The method further comprises: The multiple smart cameras are synchronously demarcated and calibrated to ensure that the fields of view of the multiple smart cameras are correctly aligned in space and time.

8. A safety protection system, characterized in that: include: Multiple smart cameras, cloud servers and terminal devices set up in different locations in the monitoring area; The terminal device and each of the smart cameras are communicatively connected to the cloud server; The smart camera collects real-time video stream from the monitoring area; The smart camera performs target detection on each frame of the real-time video stream to obtain a target list; the target list includes a first detection result of the target object detected in each frame of the image, and the first detection result includes at least bounding box information, a confidence score and a category label; The cloud server receives the target list sent by each of the smart cameras; The cloud server associates the first detection result of the target object according to the spatial position of each of the smart cameras and the corresponding target list to obtain a second detection result; The cloud server tracks and analyzes the motion trajectory of the target object according to the second detection result of each target object to obtain behavior information and trajectory information of the target object; When the cloud server determines that the behavior information and trajectory information of the target object meet the warning conditions, the cloud server sends an alarm message to the terminal device; The terminal device displays the alarm information.

9. An electronic device, characterized in that: include: processor; as well as, A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images