Video monitoring method and apparatus

Abstract

The application discloses a video monitoring method and device. The method comprises the following steps: collecting a video frame image of a pan-tilt camera in a current visual angle range at each collection time; analyzing the video frame image by using a multi-target tracking algorithm to obtain a first position detection box of each to-be-tracked object in the video frame image at the current collection time and a first position prediction box of each to-be-tracked object at a next collection time; determining a target visual angle range comprising each first position detection box and each first position prediction box; and adjusting the current visual angle range of the pan-tilt camera to the target visual angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera. The application solves the technical problem that the camera lacks flexibility in collecting video and the target tracking effect is poor in the related video monitoring scheme.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video monitoring, in particular to a video monitoring method and device. BACKGROUND

[0002] Visual intercom pan-tilt cameras combine video recording functions and real-time intercom technology, providing not only high-definition video monitoring but also remote intercom through built-in two-way audio systems. Users can easily view real-time images and communicate instantly with people at the camera's location through mobile applications or computers. In the home, such cameras can be used for remote care, home security monitoring, and communication with family members; in commercial settings, they are often used to monitor the safety of employees, customers, or materials, improving management efficiency and security. Some advanced models are also equipped with intelligent recognition functions such as face recognition or motion detection, further enhancing security capabilities.

[0003] However, traditional visual intercom pan-tilt cameras have some limitations in multi-target tracking: for simultaneous tracking of multiple targets, problems such as tracking loss and false tracking may occur, especially when there is occlusion, intersection, or similar appearance between targets, making it difficult for traditional tracking algorithms to accurately identify and track each target; during tracking, how to automatically adjust the camera's field of view based on the target's position and motion to ensure that the image within a certain range is presented in the frame is still a challenge. Existing systems do not perform well in this regard, which may result in some targets being outside the frame or a large number of invalid areas in the frame, reducing monitoring effectiveness and information integrity; when the other party needs to focus on multiple targets, it is unable to automatically adjust the display scale based on the distribution and importance of the targets, making it difficult for users to clearly see the details of all targets during audio and video communication or affecting the understanding of the overall situation due to excessively large or small frames.

[0004] To address the above problems, no effective solutions have been proposed so far. SUMMARY

[0005] The embodiments of the present application provide a video monitoring method and device to at least solve the technical problems of lack of flexibility in camera video collection and poor target tracking effect in related video monitoring solutions.

[0006] According to an aspect of the embodiments of the present application, a video monitoring method is provided, comprising: at each collection time, collecting a video frame image of a PTZ camera in a current view angle range; analyzing the video frame image by using a multi-target tracking algorithm to obtain a first position detection box of each to-be-tracked object in the video frame image at the current collection time and a first position prediction box of each to-be-tracked object at a next collection time; determining a target view angle range including each first position detection box and each first position prediction box; and adjusting the current view angle range of the PTZ camera to the target view angle range by adjusting the rotation angle and / or focal length of the PTZ camera.

[0007] Optionally, the multi-target tracking algorithm is a simple online real-time tracking SORT algorithm including a target detection algorithm, a Kalman filtering algorithm and a Hungarian algorithm, and the analyzing of the video frame image by using the multi-target tracking algorithm to obtain the position detection box of each to-be-tracked object in the video frame image at the current collection time and the position prediction box of each to-be-tracked object at the next collection time comprises: analyzing the video frame image by using the target detection algorithm to obtain a second position detection box of each object; obtaining a second position prediction box of each object determined by using the Kalman filtering algorithm at a previous collection time; matching each second position detection box and each second position prediction box by using the Hungarian algorithm, and determining the first position detection box of each to-be-tracked object at the current collection time according to a matching result; and predicting the first position prediction box of each to-be-tracked object at the next collection time according to the first position detection box of each to-be-tracked object by using the Kalman filtering algorithm.

[0008] Optionally, the matching of each second position detection box and each second position prediction box by using the Hungarian algorithm and the determination of the first position detection box of each to-be-tracked object at the current collection time according to a matching result comprises: calculating an intersection over union between each second position detection box and each second position prediction box respectively, and constructing a cost matrix according to the obtained multiple intersection over unions; solving a target matching scheme with minimum cost of the cost matrix by using the Hungarian algorithm, wherein the target matching scheme includes multiple sets of matched second position detection boxes and second position prediction boxes, and the intersection over union between each set of matched second position detection box and second position prediction box is greater than a preset threshold; for each set of matched second position detection box and second position prediction box, determining that an object corresponding to the second position detection box and the second position prediction box is an existing to-be-tracked object, and determining the second position detection box as the first position detection box of the to-be-tracked object; for a second position detection box that is not successfully matched, determining that an object corresponding to the second position detection box is a new to-be-tracked object, and determining the second position detection box as the first position detection box of the to-be-tracked object; and if the position prediction box corresponding to any object at multiple collection times is not successfully matched, the object is no longer tracked.

[0009] Optionally, the Kalman filtering algorithm is used to predict the first position prediction frame of each to-be-tracked object at the next collection time according to the first position detection frame of each to-be-tracked object, including: for a new to-be-tracked object, generating a tracker of the to-be-tracked object according to the first position detection frame of the to-be-tracked object; for an existing to-be-tracked object, updating the tracker of the to-be-tracked object according to the first position detection frame of the to-be-tracked object; for each to-be-tracked object, predicting the state of the tracker of the to-be-tracked object at the next collection time according to the state of the tracker of the to-be-tracked object at the current collection time, to obtain the first position prediction frame of the to-be-tracked object at the next collection time.

[0010] Optionally, the target view range including each first position detection frame and each first position prediction frame is determined, including: determining a minimum bounding box including each first position detection frame and each first position prediction frame; and magnifying the minimum bounding box by a preset proportion to obtain the target view range.

[0011] Optionally, in a case where the current view range of the pan-tilt camera cannot be adjusted to the target view range by adjusting the rotation angle and / or focal length of the pan-tilt camera, a first rotation angle is determined at which the to-be-tracked objects included in the view range of the pan-tilt camera are the most when the focal length of the pan-tilt camera is the largest, wherein for any to-be-tracked object, the to-be-tracked object is determined to be included in the view range of the pan-tilt camera only when the first position detection frame and the first position prediction frame of the to-be-tracked object are both included in the view range of the pan-tilt camera; or, a multi-target tracking algorithm is used to determine the types of each to-be-tracked object at the current collection time, and a preset mapping relationship table is used to determine the priority levels of each to-be-tracked object, a second rotation angle is determined at which the sum of the priority levels of all to-be-tracked objects included in the view range of the pan-tilt camera is the highest when the focal length of the pan-tilt camera is the largest, wherein the mapping relationship table stores the priority levels corresponding to objects of different types; the focal length of the pan-tilt camera is adjusted to the maximum, and the rotation angle of the pan-tilt camera is adjusted to the first rotation angle or the second rotation angle.

[0012] Optionally, in response to a tracking instruction for a target to-be-tracked object, a minimum bounding box including the first position detection frame and the first position prediction frame of the target to-be-tracked object is determined; the minimum bounding box is magnified by a preset proportion to obtain a tracking view range; and the current view range of the pan-tilt camera is adjusted to the tracking view range by adjusting the rotation angle and / or focal length of the pan-tilt camera.

[0013] According to another aspect of the embodiments of the present application, a video monitoring device is also provided, comprising: a collection module, configured to collect a video frame image of a PTZ camera in a current view angle range at each collection time; an analysis module, configured to analyze the video frame image by using a multi-target tracking algorithm to obtain a first position detection box of each to-be-tracked object in the video frame image at the current collection time and a first position prediction box of each to-be-tracked object at a next collection time; a view angle determination module, configured to determine a target view angle range including each first position detection box and each first position prediction box; and an adjustment module, configured to adjust the current view angle range of the PTZ camera to the target view angle range by adjusting a rotation angle and / or a focal length of the PTZ camera.

[0014] According to another aspect of the embodiments of the present application, a computer program product is also provided, comprising: a computer program, wherein the computer program is executed by a processor to implement the video monitoring method described above.

[0015] According to another aspect of the embodiments of the present application, an electronic device is also provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the video monitoring method described above by using the computer program.

[0016] In the embodiments of the present application, at each collection time, a video frame image of a PTZ camera in a current view angle range is collected; the video frame image is analyzed by using a multi-target tracking algorithm to obtain a first position detection box of each to-be-tracked object in the video frame image at the current collection time and a first position prediction box of each to-be-tracked object at a next collection time; a target view angle range including each first position detection box and each first position prediction box is determined; and the current view angle range of the PTZ camera is adjusted to the target view angle range by adjusting a rotation angle and / or a focal length of the PTZ camera. In this way, the video frame image is analyzed by using a multi-target tracking algorithm (such as the SORT algorithm) to obtain a target position detection box at the current collection time and to predict a target position prediction box at the next collection time, the accuracy of multi-target tracking is improved, the problems of tracking loss and false tracking are reduced, the target view angle range is determined by comprehensively considering the current detection box and the prediction box, it is ensured that all to-be-tracked objects are within the field of view, the integrity and the clarity of the monitoring picture are ensured, the current view angle range is adjusted to the target view angle range by adjusting the rotation angle and / or the focal length of the PTZ camera, it is ensured that all targets are always within the field of view, and the technical problems that the camera lacks flexibility in collecting video and the target tracking effect is poor in the related video monitoring scheme are solved. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0018] Figure 1 is a flow diagram of an optional video monitoring method according to an embodiment of the application;

[0019] Figure 2 is a structural diagram of an optional video monitoring device according to an embodiment of the application;

[0020] Figure 3 is a structural diagram of an optional electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0021] In order to enable persons skilled in the art to better understand the application scheme, the technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only a part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by persons skilled in the art without creative work should fall within the protection scope of the application.

[0022] It should be noted that the terms "first", "second", and the like in the specification and claims of the application and the drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to include only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, method, product or device.

[0023] In order to better understand the embodiments of the application, first, the following translation explanation is given for some nouns or terms appearing in the description of the embodiments of the application:

[0024] SORT (Simple Online and Realtime Tracking) algorithm: an online real-time multi-target tracking algorithm. It mainly uses Kalman filter to predict the position of the target object in the future frame, and uses IOU (Intersection over Union) as a measurement index to establish the relationship between targets, and realizes the tracking of multiple targets. The core of the SORT algorithm includes the following modules: Kalman filter, used for target state prediction and update; Hungarian algorithm, used for data association, that is, establishing the association relationship of the target between the current frame and the previous frame; detection module, the SORT algorithm uses a target detector (such as Faster R-CNN) to detect the position and bounding box of the target object in each frame.

[0025] Embodiment 1

[0026] According to an embodiment of the present application, a video monitoring method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0027] Figure 1 is a flowchart of a video monitoring method according to an embodiment of the present application, as shown in Figure 1 The method comprises the following steps:

[0028] Step S102, at each collection time, collect the video frame image of the pan-tilt camera in the current view range;

[0029] Step S104, analyze the video frame image using a multi-target tracking algorithm to obtain a first position detection box of each to-be-tracked object in the current collection time and a first position prediction box of each to-be-tracked object in the next collection time;

[0030] Step S106, determine a target view range including each first position detection box and each first position prediction box;

[0031] Step S108, adjust the current view range of the pan-tilt camera to the target view range by adjusting the rotation angle and / or focal length of the pan-tilt camera.

[0032] The steps of the video monitoring method will be described below in combination with a specific implementation process.

[0033] In order to solve the technical problems of lack of flexibility in video collection of the camera and poor target tracking effect in related video monitoring schemes, the present application proposes the following way to improve the problem:

[0034] Firstly, at each collection time, a video frame image within a current view range of the pan-tilt camera is collected; a multi-target tracking algorithm is used to analyze the video frame image, to obtain a first position detection box of each to-be-tracked object in the video frame image at the current collection time, and a first position prediction box of each to-be-tracked object at a next collection time.

[0035] Optionally, the multi-target tracking algorithm is a simple online real-time tracking SORT algorithm including a target detection algorithm, a Kalman filtering algorithm, and a Hungarian algorithm.

[0036] The multi-target tracking algorithm is used to analyze the video frame image, to obtain a position detection box of each to-be-tracked object in the video frame image at the current collection time, and a position prediction box of each to-be-tracked object at the next collection time, mainly including the following steps:

[0037] S1, a target detection algorithm is used to analyze the video frame image, to obtain a second position detection box of each object;

[0038] S2, a second position prediction box of each object determined by using the Kalman filtering algorithm at a previous collection time is obtained;

[0039] S3, the Hungarian algorithm is used to match each second position detection box and each second position prediction box, and a first position detection box of each to-be-tracked object at the current collection time is determined according to a matching result;

[0040] S4, the Kalman filtering algorithm is used to predict a first position prediction box of each to-be-tracked object at the next collection time according to the first position detection box of each to-be-tracked object.

[0041] The target detection algorithm can apply Faster R-CNN or YOLO algorithm, etc., to obtain a detection result of the current video frame. Assuming that YOLO is used for target detection, an output of YOLO is usually a list containing multiple detection results, each detection result contains the following information: a class label (Class Label) representing a detected target category, such as “car”, “person”, etc.; a confidence score (Confidence Score) representing a credibility of the detection result, which is usually a floating point number between 0 and 1; a bounding box representing a position of the detected target in the image, which is usually represented by four coordinate values, namely a left upper corner (x1, y1) and a right lower corner (x2, y2), or a center point (x, y) and a width (w) and a height (h), that is, a position detection box of each to-be-tracked object in the video frame image at the current collection time, and a position prediction box of each to-be-tracked object at the next collection time.

[0042] In the method, the Hungarian algorithm is used to match each second position detection frame and each second position prediction frame, and the first position detection frame of each to-be-tracked object at the current collection moment is determined according to the matching result. The following steps can be used: the intersection-over-union (IoU) between each second position detection frame and each second position prediction frame is calculated respectively, and a cost matrix is constructed according to the obtained multiple IoUs; the Hungarian algorithm is used to solve the target matching scheme with the minimum cost of the cost matrix, wherein the target matching scheme includes multiple matched second position detection frames and second position prediction frames, and the IoU between each matched second position detection frame and second position prediction frame is greater than a preset threshold; for each matched second position detection frame and second position prediction frame, the object corresponding to the second position detection frame and the second position prediction frame is determined as an existing to-be-tracked object, and the second position detection frame is determined as the first position detection frame of the to-be-tracked object; for the second position detection frame that is not matched successfully, the object corresponding to the second position detection frame is determined as a new to-be-tracked object, and the second position detection frame is determined as the first position detection frame of the to-be-tracked object; and a general formula of the IoU is as follows:

[0043]

[0044] In the formula, the IoU represents the intersection-over-union, the intersection set of two frames is represented by ∩, the union set of two frames is represented by ∪, the value range of the IoU is from 0 to 1, and the greater the value is, the higher the overlapping degree of the two frames is. For each second position detection frame D i and each second position prediction frame T j , the IoU value IoU ij is calculated.

[0045] The Hungarian algorithm is used to solve the optimal matching problem between the detection frame and the tracker, and a cost matrix is constructed according to the IoU between the detection frame and the prediction frame, and the matching scheme with the minimum cost is found.

[0046] 1-IoU ij is used as the element of the cost matrix C, because the Hungarian algorithm is usually used to find the minimum cost matching, and a high IoU value (that is, a high degree of overlapping of two regions) should correspond to a low cost, therefore, C[i][j] = 1-IoU ij , and the cost matrix C is input into the Hungarian algorithm.

[0047] The Hungarian algorithm finds the minimum cost matching through a series of row and column subtraction operations (referred to as row reduction and column reduction), and possible marking and searching steps. The output of the algorithm is a set of matching pairs, each second position detection frame D i is assigned to a second position prediction frame T j , so that the total cost (that is, the sum of the costs of all matching pairs) is minimum.

[0048] The Kalman filtering algorithm is used to predict the first position prediction box of each to-be-tracked object at the next collection time according to the first position detection box of each to-be-tracked object. The following steps can be used: for a new to-be-tracked object, a tracker of the to-be-tracked object is generated according to the first position detection box of the to-be-tracked object; for an existing to-be-tracked object, the tracker of the to-be-tracked object is updated according to the first position detection box of the to-be-tracked object; for each to-be-tracked object, the state of the tracker of the to-be-tracked object at the next collection time is predicted according to the state of the tracker of the to-be-tracked object at the current collection time, and the first position prediction box of the to-be-tracked object at the next collection time is obtained.

[0049] For example, for each matched detection box D i and prediction box P j , if the corresponding object of D i and P j is an existing to-be-tracked object, D i is determined as the first position detection box of the to-be-tracked object; if the matched detection box D i is not matched successfully, the corresponding object of D i is determined as a new to-be-tracked object, and D i is determined as the first position detection box of the new to-be-tracked object.

[0050] The target view range including each first position detection box and each first position prediction box is determined; the minimum bounding box including each first position detection box and each first position prediction box is determined; the minimum bounding box is enlarged by a preset proportion to obtain the target view range.

[0051] By using the above method, after the target view range is determined, the current view range of the pan-tilt camera is adjusted to the target view range by adjusting the rotation angle and / or focal length of the pan-tilt camera.

[0052] In the case where the current view range cannot be adjusted to the target view range by adjusting the rotation angle and / or focal length of the pan-tilt camera, the following strategy is used to determine the optimal rotation angle:

[0053] In a case where the current view angle range of the pan-tilt camera cannot be adjusted to the target view angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera, a first rotation angle is determined at which the pan-tilt camera has the largest view angle range and includes the most to-be-tracked objects at the maximum focal length of the pan-tilt camera, wherein for any to-be-tracked object, the to-be-tracked object is determined to be included in the view angle range of the pan-tilt camera only when the first position detection frame and the first position prediction frame of the to-be-tracked object are both included in the view angle range of the pan-tilt camera; or a type of each to-be-tracked object at the current collection moment is determined by using a multi-target tracking algorithm, and a priority level of each to-be-tracked object is determined according to a preset mapping relationship table, and a second rotation angle is determined at which the pan-tilt camera has the largest view angle range and includes all to-be-tracked objects with the highest total priority level at the maximum focal length of the pan-tilt camera, wherein the mapping relationship table stores priority levels corresponding to different types of objects; the focal length of the pan-tilt camera is adjusted to the maximum, and the rotation angle of the pan-tilt camera is adjusted to the first rotation angle or the second rotation angle.

[0054] Through the above adjustment strategy, even in a case where the view angle range cannot be adjusted by a conventional method, the to-be-tracked objects can be ensured to be included in the view angle range of the pan-tilt camera as much as possible, or important objects are preferentially included.

[0055] In response to the tracking instruction for the target to-be-tracked object, a minimum bounding box including the first position detection frame and the first position prediction frame of the target to-be-tracked object is determined; the minimum bounding box is enlarged by a preset proportion to obtain a tracking view angle range; and the current view angle range of the pan-tilt camera is adjusted to the tracking view angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera.

[0056] Specifically, in response to the tracking instruction for the target to-be-tracked object, a minimum bounding box including the first position detection frame and the first position prediction frame of the target to-be-tracked object is first determined, which contains the predicted positions of the target object at the current and next moments, ensuring that the target object is always within the field of view. The minimum bounding box is enlarged by a preset proportion to obtain a tracking view angle range. The enlargement proportion can be set according to actual needs, and the purpose is to ensure that there is a certain buffer area around the target object to prevent the target object from suddenly exceeding the field of view during movement. The current view angle range of the pan-tilt camera is adjusted to the tracking view angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera. This step ensures that the camera can focus on the target object and maintain the stability and clarity of the target object in the field of view.

[0057] In the embodiment of the present application, at each collection time, a video frame image within a current view angle range of the pan-tilt camera is collected; a multi-target tracking algorithm is used to analyze the video frame image to obtain a first position detection box of each to-be-tracked object in the current collection time and a first position prediction box of each to-be-tracked object in the next collection time; a target view angle range including each first position detection box and each first position prediction box is determined; and the current view angle range of the pan-tilt camera is adjusted to the target view angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera. In this way, the multi-target tracking algorithm (such as the SORT algorithm) is used to analyze the video frame image to obtain the target position detection box in the current collection time and predict the target position prediction box in the next collection time, which improves the accuracy of multi-target tracking, reduces the problems of tracking loss and false tracking, determines the target view angle range by comprehensively considering the current detection box and the prediction box, ensures that all to-be-tracked objects are within the field of view, guarantees the integrity and clarity of the monitoring picture, adjusts the rotation angle and / or focal length of the pan-tilt camera to adjust the current view angle range to the target view angle range, and ensures that all targets are always within the field of view, thereby avoiding that part of the targets are out of the picture range, and further solving the technical problems of lack of flexibility in camera video collection and poor target tracking effect in related video monitoring schemes.

[0058] Embodiment 2

[0059] According to the embodiments of the present application, a video monitoring device for implementing the video monitoring method in Embodiment 1 is also provided, as shown in Figure 2 The video monitoring device at least includes a collection module 21, an analysis module 22, a view angle determination module 23, and an adjustment module 24, wherein:

[0060] The collection module 21 is configured to collect a video frame image within a current view angle range of the pan-tilt camera at each collection time.

[0061] The analysis module 22 is configured to use a multi-target tracking algorithm to analyze the video frame image to obtain a first position detection box of each to-be-tracked object in the current collection time and a first position prediction box of each to-be-tracked object in the next collection time.

[0062] The view angle determination module 23 is configured to determine a target view angle range including each first position detection box and each first position prediction box.

[0063] The adjustment module 24 is configured to adjust the current view angle range of the pan-tilt camera to the target view angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera.

[0064] The functions of each module of the video monitoring device will be described below in combination with the specific implementation process.

[0065] The acquisition module acquires a video frame image of the PTZ camera within a current view range at each acquisition time; the analysis module analyzes the video frame image by using a multi-target tracking algorithm to obtain a first position detection box of each to-be-tracked object in the video frame image at the current acquisition time and a first position prediction box of each to-be-tracked object at a next acquisition time.

[0066] Optionally, the multi-target tracking algorithm is a simple online real-time tracking (SORT) algorithm including a target detection algorithm, a Kalman filtering algorithm, and a Hungarian algorithm.

[0067] The analysis of the video frame image by using the multi-target tracking algorithm to obtain the position detection box of each to-be-tracked object in the video frame image at the current acquisition time and the position prediction box of each to-be-tracked object at the next acquisition time mainly includes the following steps:

[0068] S1, analyzing the video frame image by using a target detection algorithm to obtain a second position detection box of each object;

[0069] S2, obtaining a second position prediction box of each object determined by using a Kalman filtering algorithm at a previous acquisition time;

[0070] S3, matching each second position detection box and each second position prediction box by using a Hungarian algorithm, and determining a first position detection box of each to-be-tracked object at the current acquisition time according to a matching result;

[0071] S4, predicting a first position prediction box of each to-be-tracked object at the next acquisition time according to the first position detection box of each to-be-tracked object by using a Kalman filtering algorithm.

[0072] In the method, the Hungarian algorithm is used to match each second position detection frame and each second position prediction frame, and the first position detection frame of each to-be-tracked object at the current collection moment is determined according to a matching result. The following steps can be used: the intersection-over-union between each second position detection frame and each second position prediction frame is calculated respectively, and a cost matrix is constructed according to the obtained multiple intersection-overs-union; the Hungarian algorithm is used to solve a target matching scheme with minimum cost of the cost matrix, wherein the target matching scheme includes multiple matched second position detection frames and second position prediction frames, and the intersection-over-union between each matched second position detection frame and second position prediction frame is greater than a preset threshold; for each matched second position detection frame and second position prediction frame, the object corresponding to the second position detection frame and the second position prediction frame is determined as an existing to-be-tracked object, and the second position detection frame is determined as the first position detection frame of the to-be-tracked object; for the second position detection frame that is not matched successfully, the object corresponding to the second position detection frame is determined as a new to-be-tracked object, and the second position detection frame is determined as the first position detection frame of the to-be-tracked object; and a general formula of the intersection-over-union is as follows:

[0073]

[0074] In the formula, IoU represents the intersection-over-union, ∩ represents the intersection region of two frames, ∪ represents the union region of two frames, the value range of the intersection-over-union is from 0 to 1, and the greater the value is, the higher the overlapping degree of two frames is. For each second position detection frame D i and each second position prediction frame P j , the IoU value IoU(D i , P j ) is calculated.

[0075] According to the calculated multiple intersection-overs-union, a cost matrix C is constructed, and the element C ij of the cost matrix represents the cost between each second position detection frame D i and each second position prediction frame P j , which is usually defined as 1-IoU(D i , P j ). The higher the intersection-over-union is, the lower the cost is. The Hungarian algorithm is used to solve a target matching scheme with minimum cost of the cost matrix. The Hungarian algorithm can find a matching mode such that the sum of all matching costs is minimum.

[0076] The Kalman filtering algorithm is used to predict the first position prediction frame of each to-be-tracked object at the next collection time according to the first position detection frame of each to-be-tracked object. The following steps can be used: for a new to-be-tracked object, a tracker of the to-be-tracked object is generated according to the first position detection frame of the to-be-tracked object; for an existing to-be-tracked object, the tracker of the to-be-tracked object is updated according to the first position detection frame of the to-be-tracked object; for each to-be-tracked object, the state of the tracker of the to-be-tracked object at the next collection time is predicted according to the state of the tracker of the to-be-tracked object at the current collection time, and the first position prediction frame of the to-be-tracked object at the next collection time is obtained.

[0077] For example, for each matched detection frame D i and prediction frame P j , if the corresponding object of D i and P j is an existing to-be-tracked object, D i is determined as the first position detection frame of the to-be-tracked object; if the matched detection frame D i is not matched successfully, the corresponding object of D i is determined as a new to-be-tracked object, and D i is determined as the first position detection frame of the new to-be-tracked object.

[0078] As an optional implementation, the view angle determination module determines the target view angle range including the first position detection frame and the first position prediction frame, which can include: determining the minimum bounding box of each first position detection frame and each first position prediction frame; and magnifying the minimum bounding box by a preset proportion to obtain the target view angle range.

[0079] In the above manner, after the target view angle range is determined, the current view angle range of the pan-tilt camera is adjusted to the target view angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera.

[0080] Optionally, in the case where the current view angle range cannot be adjusted to the target view angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera, the following strategy is used to determine the optimal rotation angle:

[0081] In a case where the current view angle range of the pan-tilt camera cannot be adjusted to the target view angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera, a first rotation angle is determined, at which the pan-tilt camera has the largest view angle range and includes the most to-be-tracked objects, when the focal length of the pan-tilt camera is maximum, wherein for any to-be-tracked object, the to-be-tracked object is determined to be included in the view angle range of the pan-tilt camera only when the first position detection frame and the first position prediction frame of the to-be-tracked object are both included in the view angle range of the pan-tilt camera; or a type of each to-be-tracked object at the current collection moment is determined by using a multi-target tracking algorithm, and a priority level of each to-be-tracked object is determined according to a preset mapping relationship table, and a second rotation angle is determined, at which the pan-tilt camera has the largest view angle range and includes all to-be-tracked objects with the highest total priority level, when the focal length of the pan-tilt camera is maximum, wherein the mapping relationship table stores priority levels corresponding to different types of objects; the focal length of the pan-tilt camera is adjusted to be maximum, and the rotation angle of the pan-tilt camera is adjusted to be the first rotation angle or the second rotation angle.

[0082] Through the above adjustment strategy, even in a case where the view angle range cannot be adjusted by a conventional method, the to-be-tracked objects can be ensured to be included in the view angle range of the pan-tilt camera as much as possible, or important objects are preferentially included.

[0083] In response to the tracking instruction for the target to-be-tracked object, a minimum bounding box including the first position detection frame and the first position prediction frame of the target to-be-tracked object is determined; the minimum bounding box is enlarged by a preset proportion to obtain a tracking view angle range; and the current view angle range of the pan-tilt camera is adjusted to the tracking view angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera.

[0084] Specifically, in response to the tracking instruction for the target to-be-tracked object, a minimum bounding box including the first position detection frame and the first position prediction frame of the target to-be-tracked object is first determined, which contains the predicted positions of the target object at the current and next moments, ensuring that the target object is always within the field of view. The minimum bounding box is enlarged by a preset proportion to obtain a tracking view angle range. The enlargement proportion can be set according to actual needs, and the purpose is to ensure that there is a certain buffer area around the target object to prevent the target object from suddenly exceeding the field of view during movement. The current view angle range of the pan-tilt camera is adjusted to the tracking view angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera. This step ensures that the camera can focus on the target object and maintain the stability and clarity of the target object in the field of view.

[0085] It should be noted that the modules in the video monitoring device in the embodiments of the present application correspond one by one to the implementation steps of the video monitoring method in Embodiment 1. Since Embodiment 1 has been described in detail, the details not embodied in this embodiment can be referred to Embodiment 1, and will not be described in more detail here.

[0086] Embodiment 3

[0087] According to the embodiments of the present application, a computer program product is also provided, which includes a computer program. When the computer program is executed by a processor, the video monitoring method in Embodiment 1 is implemented.

[0088] According to the embodiments of the present application, a non-volatile storage medium is also provided, which includes a stored computer program. The device in which the non-volatile storage medium is located executes the video monitoring method in Embodiment 1 by running the computer program.

[0089] According to the embodiments of the present application, a processor is also provided, which is used to run a computer program. When the computer program is run, the video monitoring method in Embodiment 1 is executed.

[0090] According to the embodiments of the present application, an electronic device is also provided, which includes a memory and a processor. The memory stores a computer program, and the processor is configured to execute the video monitoring method in Embodiment 1 by the computer program.

[0091] Specifically, when the computer program is run, the following steps are implemented: at each collection time, collecting a video frame image of the PTZ camera in a current view angle range; using a multi-target tracking algorithm to analyze the video frame image, obtaining a first position detection box of each to-be-tracked object in the video frame image at the current collection time, and a first position prediction box of each to-be-tracked object at the next collection time; determining a target view angle range including each first position detection box and each first position prediction box; and adjusting the current view angle range of the PTZ camera to the target view angle range by adjusting the rotation angle and / or focal length of the PTZ camera.

[0092] As an optional implementation, the electronic device can exist in the form of a mobile terminal, a computer terminal or a similar computing device. Figure 3 A hardware structure block diagram of an electronic device for implementing the video monitoring method is shown. As shown in FIG. 6, the electronic device includes a processor 601, a memory 602, a communication interface 603, and a display 604. Figure 3As shown, the electronic device 30 may include one or more processors 302 (shown as 302a, 302b, ..., 302n in the figure) (processor 302 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 304 for storing data, and a transmission device 306 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 3 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, electronic device 30 may also include... Figure 3 The more or fewer components shown, or having the same Figure 3 The different configurations shown.

[0093] It should be noted that the aforementioned one or more processors 302 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element of the electronic device 30. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0094] The memory 304 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the video surveillance method in this embodiment. The processor 302 executes various functional applications and data processing by running the software programs and modules stored in the memory 304, thereby implementing the aforementioned vulnerability detection method for the application. The memory 304 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 304 may further include memory remotely located relative to the processor 302, and these remote memories can be connected to the electronic device 30 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0095] The transmission device 306 is configured to receive or send data via a network. The network can include a wireless network provided by a communication provider of the electronic device 30. In one example, the transmission device 306 includes a network interface controller (NIC) that can be connected to other network devices through a base station to communicate with the Internet. In one example, the transmission device 306 can be a radio frequency (RF) module that is configured to communicate with the Internet wirelessly.

[0096] The display can be a liquid crystal display (LCD) that is touch screen type, for example, which can enable a user to interact with a user interface of the electronic device 30.

[0097] The above-mentioned embodiment numbers are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0098] In the above-mentioned embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0099] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of units can be a logical function division, and actual implementation can have another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0100] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0101] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0102] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0103] The above is only the preferred embodiment of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. A video surveillance method, characterized in that, include: At each acquisition moment, video frame images of the PTZ camera within the current field of view are acquired; The video frame image is analyzed using a multi-object tracking algorithm to obtain the first position detection box of each trackable object in the video frame image at the current acquisition time, and the first position prediction box of each trackable object at the next acquisition time. The multi-object tracking algorithm is a simple online real-time tracking (SORT) algorithm, which includes an object detection algorithm, a Kalman filter algorithm, and a Hungarian algorithm. The analysis process includes: analyzing the video frame image using the object detection algorithm to obtain the second position detection boxes of multiple objects; obtaining the second position prediction boxes of each object determined by the Kalman filter algorithm at the previous acquisition time; calculating the intersection-union ratio (IUU) between each second position detection box and each second position prediction box, and constructing a cost matrix based on the obtained IUU ratios; and using the Hungarian algorithm to solve for the target matching scheme that minimizes the cost matrix. The target matching scheme includes multiple sets of matched second position detection boxes and second position prediction boxes, and the IUU between each set of matched second position detection boxes and second position prediction boxes is greater than a preset threshold. For each set of matched second position detection boxes and second position prediction boxes... A second position detection box and a second position prediction box are used to determine that the objects corresponding to the second position detection box and the second position prediction box are existing objects to be tracked, and the second position detection box is determined to be the first position detection box of the object to be tracked. For second position detection boxes that fail to match, the objects corresponding to the second position detection boxes are determined to be new objects to be tracked, and the second position detection box is determined to be the first position detection box of the object to be tracked. If the position prediction boxes corresponding to any object fail to match at multiple acquisition times, the object is no longer tracked. For new objects to be tracked, a tracker for the object to be tracked is generated based on the first position detection box of the object to be tracked. For existing objects to be tracked, the tracker for the object to be tracked is updated based on the first position detection box of the object to be tracked. For each object to be tracked, the state of the tracker for the object to be tracked at the next acquisition time is predicted based on the state of the tracker for the object to be tracked at the current acquisition time, and the first position prediction box of the object to be tracked at the next acquisition time is obtained. Determine the target view range including each of the first location detection boxes and each of the first location prediction boxes; By adjusting the rotation angle and / or focal length of the PTZ camera, the current field of view of the PTZ camera is adjusted to the target field of view.

2. The method according to claim 1, characterized in that, Determining the target view range, including each of the first location detection boxes and each of the first location prediction boxes, includes: Determine the minimum bounding box that includes each of the first location detection boxes and each of the first location prediction boxes; The target viewing angle range is obtained by scaling up the minimum bounding box by a preset ratio.

3. The method according to claim 1, characterized in that, The method further includes: In cases where it is impossible to adjust the current field of view of the pan-tilt camera to the target field of view by adjusting the rotation angle and / or focal length of the pan-tilt camera, A first rotation angle is determined to maximize the number of tracked objects included within the pan-tilt camera's field of view when the camera's focal length is at its maximum. Specifically, for any tracked object, the pan-tilt camera's field of view is considered to include the tracked object only if both the first position detection box and the first position prediction box of the tracked object are simultaneously included within the pan-tilt camera's field of view; or... The type of each object to be tracked at the current acquisition time is determined by a multi-target tracking algorithm, and the priority level of each object to be tracked is determined according to a preset mapping table. The second rotation angle is determined when the focal length of the pan-tilt camera is at its maximum, so that the sum of the priority levels of all objects to be tracked within the field of view of the pan-tilt camera is the highest. The mapping table stores the priority levels corresponding to different types of objects. Adjust the focal length of the PTZ camera to its maximum, and adjust the rotation angle of the PTZ camera to either the first rotation angle or the second rotation angle.

4. The method according to claim 1, characterized in that, The method further includes: In response to a tracking instruction for a target object, a minimum bounding box including the first location detection box and the first location prediction box of the target object is determined; The minimum bounding box is enlarged by a preset ratio to obtain the tracking view range; By adjusting the rotation angle and / or focal length of the PTZ camera, the current field of view of the PTZ camera is adjusted to the tracking field of view.

5. A video surveillance device, characterized in that, include: The acquisition module is used to acquire video frame images of the PTZ camera within the current field of view at each acquisition moment; The analysis module is used to analyze the video frame image using a multi-target tracking algorithm to obtain the first position detection box of each trackable object in the video frame image at the current acquisition time, and the first position prediction box of each trackable object at the next acquisition time. The multi-target tracking algorithm is a simple online real-time tracking (SORT) algorithm including a target detection algorithm, a Kalman filter algorithm, and a Hungarian algorithm. The analysis process includes: analyzing the video frame image using a target detection algorithm to obtain the second position detection boxes of multiple objects; obtaining the second position prediction boxes of each object determined by the Kalman filter algorithm at the previous acquisition time; calculating the intersection-union ratio (IUU) between each second position detection box and each second position prediction box, and constructing a cost matrix based on the obtained IUU ratios; and using the Hungarian algorithm to solve for a target matching scheme that minimizes the cost matrix. The target matching scheme includes multiple sets of matched second position detection boxes and second position prediction boxes, and the IUU between each set of matched second position detection boxes and second position prediction boxes is greater than a preset threshold. For each set of matched second position detection boxes and second position prediction boxes... The second position detection box and the second position prediction box are matched. The objects corresponding to the second position detection box and the second position prediction box are determined to be existing objects to be tracked. The second position detection box is determined to be the first position detection box of the object to be tracked. For the second position detection box that is not matched, the object corresponding to the second position detection box is determined to be a new object to be tracked. The second position detection box is determined to be the first position detection box of the object to be tracked. If the position prediction box corresponding to any object is not matched at multiple acquisition times, the object is no longer tracked. For the new object to be tracked, a tracker for the object to be tracked is generated based on the first position detection box of the object to be tracked. For the existing object to be tracked, the tracker for the object to be tracked is updated based on the first position detection box of the object to be tracked. For each object to be tracked, the state of the tracker for the object to be tracked at the next acquisition time is predicted based on the state of the tracker for the object to be tracked at the current acquisition time, and the first position prediction box of the object to be tracked at the next acquisition time is obtained. The viewpoint determination module is used to determine the target viewpoint range including each of the first position detection boxes and each of the first position prediction boxes; The adjustment module is used to adjust the current field of view of the PTZ camera to the target field of view by adjusting the rotation angle and / or focal length of the PTZ camera.

6. A computer program product, characterized in that, include: A computer program, wherein when executed by a processor, the computer program implements the video surveillance method according to any one of claims 1 to 4.

7. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the video surveillance method of any one of claims 1 to 4 through the computer program.

Images