Video monitoring method and device

Abstract

The invention discloses a video monitoring method and device. The method comprises the following steps: at each acquisition moment, acquiring a video frame image of a pan-tilt camera in a current visual angle range; analyzing the video frame image by using a multi-target tracking algorithm to obtain a first position detection frame of each to-be-tracked object in the video frame image at the current acquisition moment and a first position prediction frame of each to-be-tracked object at the next acquisition moment; determining a target visual angle range comprising each first position detection frame and each first position prediction frame; and adjusting the current visual angle range of the PTZ camera to the target visual angle range by adjusting the rotation angle and / or the focal length of the PTZ camera. According to the invention, the technical problems of lack of flexibility in video acquisition by a camera and poor target tracking effect in a related video monitoring scheme are solved.

Description

Technical Field

[0001] The present application relates to the field of video surveillance technology, and in particular to a video surveillance method and device. Background Art

[0002] The video intercom PTZ camera combines video recording and real-time intercom technology, providing not only high-definition video surveillance, but also remote intercom through a built-in two-way audio system. Users can easily view real-time images through a mobile app or computer, and communicate instantly with people at the camera location. In the home, this type of camera can be used for remote care, home security monitoring, and communication with family members; in commercial places, it is often used to monitor the safety of employees, customers, or materials to improve management efficiency and safety. Some advanced models are also equipped with intelligent recognition functions, such as face recognition or motion detection, to further enhance security protection capabilities.

[0003] However, traditional video intercom PTZ cameras have some limitations in multi-target tracking: for simultaneous tracking of multiple targets, it is easy to have problems such as tracking loss and mistracking, especially when there are occlusions, intersections or similar appearances between targets, traditional tracking algorithms are difficult to accurately identify and track each target; during the tracking process, how to automatically adjust the camera's field of view according to the position and movement of the target to ensure that the image within a certain range is presented in the screen is still a challenge. The performance of existing systems in this regard is not satisfactory, which may cause some targets to exceed the range of the screen, or there are a large number of invalid areas in the screen, reducing the monitoring effect and information integrity; when the other party needs to pay attention to multiple targets, it is impossible to automatically enlarge or reduce the display ratio according to the distribution and importance of the targets, so that users cannot clearly see the details of all targets during the audio and video call, or the screen is too large or too small to affect the understanding of the overall situation.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a video monitoring method and device to at least solve the technical problems of the lack of flexibility in video acquisition by cameras and poor target tracking effect in related video monitoring solutions.

[0006] According to one aspect of an embodiment of the present application, a video surveillance method is provided, including: at each acquisition moment, acquiring a video frame image of a pan-tilt camera within a current viewing angle; using a multi-target tracking algorithm to analyze the video frame image to obtain a first position detection frame of each object to be tracked in the video frame image at the current acquisition moment, and a first position prediction frame of each object to be tracked at the next acquisition moment; determining a target viewing angle range including each first position detection frame and each first position prediction frame; and adjusting the current viewing angle range of the pan-tilt camera to the target viewing angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera.

[0007] Optionally, the multi-target tracking algorithm is a simple online real-time tracking SORT algorithm including a target detection algorithm, a Kalman filter algorithm and a Hungarian algorithm. The multi-target tracking algorithm is used to analyze the video frame image to obtain a position detection frame of each object to be tracked in the video frame image at the current acquisition moment, and a position prediction frame of each object to be tracked at the next acquisition moment, including: using the target detection algorithm to analyze the video frame image to obtain a second position detection frame of multiple objects; obtaining the second position prediction frame of each object determined by the Kalman filter algorithm at the previous acquisition moment; using the Hungarian algorithm to match each second position detection frame with each second position prediction frame, and determining the first position detection frame of each object to be tracked at the current acquisition moment based on the matching result; using the Kalman filter algorithm to predict the first position prediction frame of each object to be tracked at the next acquisition moment based on the first position detection frame of each object to be tracked.

[0008] Optionally, the Hungarian algorithm is used to match each second position detection frame with each second position prediction frame, and the first position detection frame of each object to be tracked at the current acquisition moment is determined based on the matching result, including: respectively calculating the intersection-and-union ratio between each second position detection frame and each second position prediction frame, and constructing a cost matrix based on the obtained multiple intersection-and-union ratios; using the Hungarian algorithm to solve a target matching scheme that minimizes the cost of the cost matrix, wherein the target matching scheme includes multiple groups of matched second position detection frames and second position prediction frames, and the intersection-and-union ratio between each group of matched second position detection frames and second position prediction frames is greater than a preset threshold; for each group of matched second position detection frames and second position prediction frames, it is determined that the object corresponding to the second position detection frame and the second position prediction frame is an existing object to be tracked, and the second position detection frame is determined to be the first position detection frame of the object to be tracked; for the second position detection frame that is not successfully matched, it is determined that the object corresponding to the second position detection frame is a new object to be tracked, and the second position detection frame is determined to be the first position detection frame of the object to be tracked; if the position prediction frames corresponding to any object at multiple acquisition moments are not successfully matched, the object is no longer tracked.

[0009] Optionally, a Kalman filter algorithm is used to predict the first position prediction frame of each object to be tracked at the next acquisition moment based on the first position detection frame of each object to be tracked, including: for a new object to be tracked, generating a tracker for the object to be tracked based on the first position detection frame of the object to be tracked; for an existing object to be tracked, updating the tracker of the object to be tracked based on the first position detection frame of the object to be tracked; for each object to be tracked, predicting the state of the tracker of the object to be tracked at the next acquisition moment based on the state of the tracker of the object to be tracked at the current acquisition moment, to obtain the first position prediction frame of the object to be tracked at the next acquisition moment.

[0010] Optionally, determining a target viewing angle range including each first position detection frame and each first position prediction frame includes: determining a minimum bounding box including each first position detection frame and each first position prediction box; and enlarging the minimum bounding box by a preset ratio to obtain the target viewing angle range.

[0011] Optionally, when the current viewing angle range of the pan-tilt camera cannot be adjusted to the target viewing angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera, a first rotation angle is determined at which the viewing angle range of the pan-tilt camera includes the most objects to be tracked when the focal length of the pan-tilt camera is maximum, wherein, for any object to be tracked, the object to be tracked is determined to be included in the viewing angle range of the pan-tilt camera only when the first position detection frame and the first position prediction frame of the object to be tracked are simultaneously included in the viewing angle range of the pan-tilt camera; or, a multi-target tracking algorithm is used to determine the type of each object to be tracked at the current acquisition moment, and a priority level of each object to be tracked is determined according to a preset mapping relationship table, and a second rotation angle is determined at which the total priority levels of all objects to be tracked included in the viewing angle range of the pan-tilt camera are the highest when the focal length of the pan-tilt camera is maximum, wherein the mapping relationship table stores priority levels corresponding to different types of objects; the focal length of the pan-tilt camera is adjusted to the maximum, and the rotation angle of the pan-tilt camera is adjusted to the first rotation angle or the second rotation angle.

[0012] Optionally, in response to a tracking instruction for a target tracking object, a minimum bounding box including a first position detection box and a first position prediction box of the target tracking object is determined; the minimum bounding box is enlarged by a preset ratio to obtain a tracking viewing angle range; and the current viewing angle range of the gimbal camera is adjusted to the tracking viewing angle range by adjusting the rotation angle and / or focal length of the gimbal camera.

[0013] According to another aspect of an embodiment of the present application, a video surveillance device is also provided, including: an acquisition module, used to acquire video frame images of a pan-tilt camera within a current viewing angle at each acquisition moment; an analysis module, used to analyze the video frame images using a multi-target tracking algorithm to obtain a first position detection frame of each object to be tracked in the video frame image at the current acquisition moment, and a first position prediction frame of each object to be tracked at the next acquisition moment; a viewing angle determination module, used to determine a target viewing angle range including each first position detection frame and each first position prediction frame; and an adjustment module, used to adjust the current viewing angle range of the pan-tilt camera to the target viewing angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera.

[0014] According to another aspect of an embodiment of the present application, a computer program product is further provided, the computer program product comprising: a computer program, wherein the computer program implements the above-mentioned video monitoring method when executed by a processor.

[0015] According to another aspect of an embodiment of the present application, an electronic device is further provided. The electronic device includes: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned video monitoring method through the computer program.

[0016] In an embodiment of the present application, at each acquisition moment, a video frame image of the pan-tilt camera within the current viewing angle is acquired; the video frame image is analyzed using a multi-target tracking algorithm to obtain a first position detection frame of each object to be tracked in the video frame image at the current acquisition moment, and a first position prediction frame of each object to be tracked at the next acquisition moment; a target viewing angle range including each first position detection frame and each first position prediction frame is determined; and the current viewing angle range of the pan-tilt camera is adjusted to the target viewing angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera. In this way, a multi-target tracking algorithm (such as the SORT algorithm) is used to analyze the video frame image, obtain the target position detection frame at the current acquisition moment, and predict the target position prediction frame at the next acquisition moment, thereby improving the accuracy of multi-target tracking and reducing the problems of tracking loss and mistracking. The current detection frame and the prediction frame are combined to determine the target viewing angle range, ensuring that all objects to be tracked are within the field of view, ensuring the integrity and clarity of the monitoring picture, and adjusting the rotation angle and / or focal length of the pan-tilt camera to adjust the current viewing angle range to the target viewing angle range, ensuring that all targets are always within the field of view and preventing some targets from exceeding the picture range, thereby solving the technical problems of lack of flexibility in camera video acquisition and poor target tracking effect in related video surveillance solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 is a flow chart of an optional video monitoring method according to an embodiment of the present application;

[0019] Figure 2 is a schematic structural diagram of an optional video surveillance device according to an embodiment of the present application;

[0020] Figure 3 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first", "second", etc. in the specification, claims and drawings of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] In order to better understand the embodiments of the present application, some nouns or terms that appear in the description of the embodiments of the present application are first translated and explained as follows:

[0024] SORT (Simple Online and Realtime Tracking) algorithm: is an online real-time multi-target tracking algorithm. It mainly uses Kalman filtering to predict the position of the target object in the future frame, and uses IOU (intersection over union) as a metric to establish the relationship between targets and achieve multi-target tracking. The core of the SORT algorithm includes the following modules: Kalman filtering, which is used to predict and update the target state; Hungarian algorithm, which is used for data association, that is, to establish the association relationship between the current frame and the previous frame; detection module, the SORT algorithm uses a target detector (such as Faster R-CNN) to detect the position and bounding box of the target object in each frame.

[0025] Example 1

[0026] According to an embodiment of the present application, a video surveillance method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] Figure 1 is a flow chart of a video monitoring method provided according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0028] Step S102, at each acquisition moment, acquiring a video frame image of the PTZ camera within the current viewing angle;

[0029] Step S104, analyzing the video frame image using a multi-target tracking algorithm to obtain a first position detection frame of each object to be tracked in the video frame image at the current acquisition time, and a first position prediction frame of each object to be tracked at the next acquisition time;

[0030] Step S106, determining a target viewing angle range including each first position detection frame and each first position prediction frame;

[0031] Step S108, adjusting the current viewing angle range of the pan / tilt camera to the target viewing angle range by adjusting the rotation angle and / or focal length of the pan / tilt camera.

[0032] The following describes each step of the video monitoring method in conjunction with a specific implementation process.

[0033] In order to solve the technical problems of the lack of flexibility in video acquisition and poor target tracking effect in related video surveillance solutions, this application proposes to improve the problem by taking the following measures:

[0034] Firstly, at each acquisition moment, the video frame images within the current viewing angle of the pan-tilt camera are acquired; the video frame images are analyzed using a multi-target tracking algorithm to obtain the first position detection frame of each object to be tracked in the video frame image at the current acquisition moment, and the first position prediction frame of each object to be tracked at the next acquisition moment.

[0035] Optionally, the multi-target tracking algorithm is a simple online real-time tracking SORT algorithm including a target detection algorithm, a Kalman filter algorithm and a Hungarian algorithm.

[0036] The multi-target tracking algorithm is used to analyze the video frame image to obtain the position detection frame of each object to be tracked in the video frame image at the current acquisition time, and the position prediction frame of each object to be tracked at the next acquisition time, which mainly includes the following steps:

[0037] S1, analyzing the video frame image using a target detection algorithm to obtain second position detection frames of multiple objects;

[0038] S2, obtaining a second position prediction frame of each object determined by using a Kalman filter algorithm at the last acquisition moment;

[0039] S3, using the Hungarian algorithm to match each second position detection frame with each second position prediction frame, and determining the first position detection frame of each object to be tracked at the current acquisition moment according to the matching result;

[0040] S4, using a Kalman filter algorithm to predict the first position prediction frame of each object to be tracked at the next acquisition moment according to the first position detection frame of each object to be tracked.

[0041] Among them, the target detection algorithm can apply Faster R-CNN or YOLO algorithm to obtain the detection result of the current video frame. Assuming that YOLO is used for target detection, the output of YOLO is usually a list containing multiple detection results, and each detection result contains the following information: Class Label, which indicates the detected target category, such as "car", "person", etc.; Confidence Score, which indicates the credibility of the detected result, usually a floating point number between 0 and 1; Bounding box, which indicates the position of the detected target in the image, usually expressed by four coordinate values, namely (x1, y1) in the upper left corner and (x2, y2) in the lower right corner, or the center point (x, y) and width (w), height (h), that is, the position detection box of each object to be tracked in the video frame image at the current acquisition time, and the position prediction box of each object to be tracked at the next acquisition time are obtained.

[0042] Among them, the Hungarian algorithm is used to match each second position detection frame with each second position prediction frame, and the first position detection frame of each object to be tracked at the current acquisition moment is determined according to the matching result. The following steps can be used: the intersection and union ratio between each second position detection frame and each second position prediction frame is calculated respectively, and a cost matrix is ​​constructed according to the obtained multiple intersection and union ratios; the Hungarian algorithm is used to solve the target matching scheme that minimizes the cost of the cost matrix, wherein the target matching scheme includes multiple groups of matched second position detection frames and second position prediction frames, and the intersection and union ratio between each group of matched second position detection frames and second position prediction frames is greater than a preset threshold; for each group of matched second position detection frames and second position prediction frames, it is determined that the object corresponding to the second position detection frame and the second position prediction frame is an existing object to be tracked, and the second position detection frame is determined to be the first position detection frame of the object to be tracked; for the second position detection frame that is not successfully matched, it is determined that the object corresponding to the second position detection frame is a new object to be tracked, and the second position detection frame is determined to be the first position detection frame of the object to be tracked; the general formula of the intersection and union ratio is as follows:

[0043]

[0044] Where IoU represents the intersection-over-union ratio, ∩ represents the intersection area of ​​two boxes, ∪ represents the union area of ​​two boxes, and the value of IoU ranges from 0 to 1. The larger the value, the higher the overlap of the two boxes. For each second position detection box D i And each second position prediction box T j , calculate their IoU value IoU ij .

[0045] The Hungarian algorithm is used to solve the optimal matching problem between the detection box and the tracker. It constructs a cost matrix based on the IoU between the detection box and the prediction box, and finds the matching solution with the minimum cost.

[0046] Using 1-IoU ij As an element of the cost matrix C, this is because the Hungarian algorithm is usually used to find the minimum cost match, and a high IoU value (i.e., two regions highly overlap) should correspond to a low cost, so C[i][j] = 1-IoU ij , the cost matrix C is input into the Hungarian algorithm.

[0047] The Hungarian algorithm searches for the minimum cost match through a series of row and column subtraction operations (called row reduction and column reduction), as well as possible marking and search steps. The output of the algorithm is a set of matching pairs, with each second position detection box D i Assigned to a second position prediction box T j , so that the total cost (that is, the sum of the costs of all matching pairs) is minimized.

[0048] Among them, the Kalman filter algorithm is used to predict the first position prediction frame of each object to be tracked at the next acquisition moment based on the first position detection frame of each object to be tracked. The following steps can be adopted: for a new object to be tracked, a tracker for the object to be tracked is generated based on the first position detection frame of the object to be tracked; for an existing object to be tracked, the tracker of the object to be tracked is updated based on the first position detection frame of the object to be tracked; for each object to be tracked, the state of the tracker of the object to be tracked at the next acquisition moment is predicted based on the state of the tracker of the object to be tracked at the current acquisition moment, so as to obtain the first position prediction frame of the object to be tracked at the next acquisition moment.

[0049] For example, for each set of matched detection boxes D i and the predicted box P j , determine D i and P j If the corresponding object is an existing object to be tracked, D i Determine the first position detection frame of the object to be tracked, and for the detection frame D that is not successfully matched i , then determine D i The corresponding object is the new object to be tracked. i A first position detection frame is determined as a new object to be tracked.

[0050] Determine a target viewing angle range including each first position detection frame and each first position prediction frame; determine a minimum bounding box including each first position detection frame and each first position prediction frame; and enlarge the minimum bounding box by a preset ratio to obtain a target viewing angle range.

[0051] By using the above method, after determining the target viewing angle range, the current viewing angle range of the pan-tilt camera is adjusted to the target viewing angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera.

[0052] When the current viewing angle cannot be adjusted to the target viewing angle by adjusting the rotation angle and / or focal length of the PTZ camera, the following strategy is used to determine the optimal rotation angle:

[0053] When the current viewing angle range of the pan-tilt camera cannot be adjusted to the target viewing angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera, determine a first rotation angle that includes the most objects to be tracked within the viewing angle range of the pan-tilt camera when the focal length of the pan-tilt camera is maximum, wherein, for any object to be tracked, the object to be tracked is determined to be included in the viewing angle range of the pan-tilt camera only when the first position detection frame and the first position prediction frame of the object to be tracked are simultaneously included in the viewing angle range of the pan-tilt camera; alternatively, determine the type of each object to be tracked at the current acquisition moment using a multi-target tracking algorithm, determine the priority level of each object to be tracked according to a preset mapping relationship table, and determine a second rotation angle that has the highest total priority level of all objects to be tracked within the viewing angle range of the pan-tilt camera when the focal length of the pan-tilt camera is maximum, wherein the mapping relationship table stores priority levels corresponding to different types of objects; adjust the focal length of the pan-tilt camera to the maximum, and adjust the rotation angle of the pan-tilt camera to the first rotation angle or the second rotation angle.

[0054] Through the above adjustment strategy, even when the viewing angle range cannot be adjusted by conventional methods, it can be ensured that the viewing angle range of the PTZ camera includes as many objects to be tracked as possible, or that important objects are preferentially included.

[0055] In response to a tracking instruction for a target tracking object, a minimum bounding box including a first position detection box and a first position prediction box of the target tracking object is determined; the minimum bounding box is enlarged by a preset ratio to obtain a tracking viewing angle range; and the current viewing angle range of the pan-tilt camera is adjusted to the tracking viewing angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera.

[0056] Specifically, it can be explained as follows: in response to the tracking instruction of the target tracking object, the minimum bounding box including the first position detection box and the first position prediction box of the target tracking object is first determined. This minimum bounding box contains the predicted positions of the target object at the current and next moments, ensuring that the target object is always within the field of view. The minimum bounding box is enlarged by a preset ratio to obtain the tracking viewing angle range. The enlargement ratio can be set according to actual needs. The purpose is to ensure that there is a certain buffer area around the target object to prevent the target object from suddenly exceeding the field of view during movement. By adjusting the rotation angle and / or focal length of the pan-tilt camera, the current viewing angle range of the pan-tilt camera is adjusted to the tracking viewing angle range. This step ensures that the camera can focus on the target object and maintain the stability and clarity of the target object in the field of view.

[0057] In an embodiment of the present application, at each acquisition moment, a video frame image of the pan-tilt camera within the current viewing angle is acquired; the video frame image is analyzed using a multi-target tracking algorithm to obtain a first position detection frame of each object to be tracked in the video frame image at the current acquisition moment, and a first position prediction frame of each object to be tracked at the next acquisition moment; a target viewing angle range including each first position detection frame and each first position prediction frame is determined; and the current viewing angle range of the pan-tilt camera is adjusted to the target viewing angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera. In this way, a multi-target tracking algorithm (such as the SORT algorithm) is used to analyze the video frame image, obtain the target position detection frame at the current acquisition moment, and predict the target position prediction frame at the next acquisition moment, thereby improving the accuracy of multi-target tracking and reducing the problems of tracking loss and mistracking. The current detection frame and the prediction frame are combined to determine the target viewing angle range, ensuring that all objects to be tracked are within the field of view, ensuring the integrity and clarity of the monitoring picture, and adjusting the rotation angle and / or focal length of the pan-tilt camera to adjust the current viewing angle range to the target viewing angle range, ensuring that all targets are always within the field of view and preventing some targets from exceeding the picture range, thereby solving the technical problems of lack of flexibility in camera video acquisition and poor target tracking effect in related video surveillance solutions.

[0058] Example 2

[0059] According to an embodiment of the present application, a video monitoring device for implementing the video monitoring method in Embodiment 1 is also provided. Figure 2 As shown, the video monitoring device at least includes: a collection module 21, an analysis module 22, a viewing angle determination module 23 and an adjustment module 24, wherein:

[0060] The acquisition module 21 is used to acquire the video frame images of the PTZ camera within the current viewing angle at each acquisition moment;

[0061] The analysis module 22 is used to analyze the video frame image using a multi-target tracking algorithm to obtain a first position detection frame of each object to be tracked in the video frame image at the current acquisition time, and a first position prediction frame of each object to be tracked at the next acquisition time;

[0062] A viewing angle determination module 23, used to determine a target viewing angle range including each first position detection frame and each first position prediction frame;

[0063] The adjustment module 24 is used to adjust the current viewing angle range of the pan / tilt camera to the target viewing angle range by adjusting the rotation angle and / or focal length of the pan / tilt camera.

[0064] The functions of each module of the video surveillance device are described below in conjunction with a specific implementation process.

[0065] The acquisition module acquires the video frame images within the current viewing angle of the PTZ camera at each acquisition moment; the analysis module analyzes the video frame images using a multi-target tracking algorithm to obtain the first position detection frame of each object to be tracked in the video frame image at the current acquisition moment, and the first position prediction frame of each object to be tracked at the next acquisition moment.

[0066] Optionally, the multi-target tracking algorithm is a simple online real-time tracking SORT algorithm including a target detection algorithm, a Kalman filter algorithm and a Hungarian algorithm.

[0067] The multi-target tracking algorithm is used to analyze the video frame image to obtain the position detection frame of each object to be tracked in the video frame image at the current acquisition time, and the position prediction frame of each object to be tracked at the next acquisition time, which mainly includes the following steps:

[0068] S1, analyzing the video frame image using a target detection algorithm to obtain second position detection frames of multiple objects;

[0069] S2, obtaining a second position prediction frame of each object determined by using a Kalman filter algorithm at the last acquisition moment;

[0070] S3, using the Hungarian algorithm to match each second position detection frame with each second position prediction frame, and determining the first position detection frame of each object to be tracked at the current acquisition moment according to the matching result;

[0071] S4, using a Kalman filter algorithm to predict the first position prediction frame of each object to be tracked at the next acquisition moment according to the first position detection frame of each object to be tracked.

[0072] Among them, the Hungarian algorithm is used to match each second position detection frame with each second position prediction frame, and the first position detection frame of each object to be tracked at the current acquisition moment is determined according to the matching result. The following steps can be used: the intersection and union ratio between each second position detection frame and each second position prediction frame is calculated respectively, and a cost matrix is ​​constructed according to the obtained multiple intersection and union ratios; the Hungarian algorithm is used to solve the target matching scheme that minimizes the cost of the cost matrix, wherein the target matching scheme includes multiple groups of matched second position detection frames and second position prediction frames, and the intersection and union ratio between each group of matched second position detection frames and second position prediction frames is greater than a preset threshold; for each group of matched second position detection frames and second position prediction frames, it is determined that the object corresponding to the second position detection frame and the second position prediction frame is an existing object to be tracked, and the second position detection frame is determined to be the first position detection frame of the object to be tracked; for the second position detection frame that is not successfully matched, it is determined that the object corresponding to the second position detection frame is a new object to be tracked, and the second position detection frame is determined to be the first position detection frame of the object to be tracked; the general formula of the intersection and union ratio is expressed as follows:

[0073]

[0074] Where IoU represents the intersection-over-union ratio, ∩ represents the intersection area of ​​two boxes, ∪ represents the union area of ​​two boxes, and the value of IoU ranges from 0 to 1. The larger the value, the higher the overlap of the two boxes. For each second position detection box D i And each second position prediction box P j , calculate their IoU value IoU (D i ,P j ).

[0075] According to the calculated multiple intersection ratios, a cost matrix C is constructed. The elements of the cost matrix C ij Denotes each second position detection box D i And each second position prediction box P j The cost between them is usually defined as 1-IoU(D i ,P j ), the higher the intersection-union ratio, the lower the cost. The Hungarian algorithm is used to solve the target matching solution that minimizes the cost matrix. The Hungarian algorithm can find a matching method that minimizes the sum of the costs of all matches.

[0076] Among them, the Kalman filter algorithm is used to predict the first position prediction frame of each object to be tracked at the next acquisition moment based on the first position detection frame of each object to be tracked. The following steps can be adopted: for a new object to be tracked, a tracker for the object to be tracked is generated based on the first position detection frame of the object to be tracked; for an existing object to be tracked, the tracker of the object to be tracked is updated based on the first position detection frame of the object to be tracked; for each object to be tracked, the state of the tracker of the object to be tracked at the next acquisition moment is predicted based on the state of the tracker of the object to be tracked at the current acquisition moment, so as to obtain the first position prediction frame of the object to be tracked at the next acquisition moment.

[0077] For example, for each set of matched detection boxes D i and the predicted box P j , determine D i and P j If the corresponding object is an existing object to be tracked, D i Determine the first position detection frame of the object to be tracked, and for the detection frame D that is not successfully matched i , then determine D i The corresponding object is the new object to be tracked. i A first position detection frame is determined as a new object to be tracked.

[0078] As an optional implementation, the perspective determination module determines the target perspective range including each first position detection frame and each first position prediction frame, which may include: determining the minimum bounding box of each first position detection frame and each first position prediction frame; and enlarging the minimum bounding box by a preset proportion to obtain the target perspective range.

[0079] By using the above method, after determining the target viewing angle range, the current viewing angle range of the pan-tilt camera is adjusted to the target viewing angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera.

[0080] Optionally, when the current viewing angle range cannot be adjusted to the target viewing angle range by adjusting the rotation angle and / or focal length of the PTZ camera, the following strategy is used to determine the optimal rotation angle:

[0081] When the current viewing angle range of the pan-tilt camera cannot be adjusted to the target viewing angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera, determine a first rotation angle that includes the most objects to be tracked within the viewing angle range of the pan-tilt camera when the focal length of the pan-tilt camera is maximum, wherein, for any object to be tracked, the object to be tracked is determined to be included in the viewing angle range of the pan-tilt camera only when the first position detection frame and the first position prediction frame of the object to be tracked are simultaneously included in the viewing angle range of the pan-tilt camera; alternatively, determine the type of each object to be tracked at the current acquisition moment using a multi-target tracking algorithm, determine the priority level of each object to be tracked according to a preset mapping relationship table, and determine a second rotation angle that has the highest total priority level of all objects to be tracked within the viewing angle range of the pan-tilt camera when the focal length of the pan-tilt camera is maximum, wherein the mapping relationship table stores priority levels corresponding to different types of objects; adjust the focal length of the pan-tilt camera to the maximum, and adjust the rotation angle of the pan-tilt camera to the first rotation angle or the second rotation angle.

[0082] Through the above adjustment strategy, even when the viewing angle range cannot be adjusted by conventional methods, it can be ensured that the viewing angle range of the PTZ camera includes as many objects to be tracked as possible, or that important objects are preferentially included.

[0083] In response to a tracking instruction for a target tracking object, a minimum bounding box including a first position detection box and a first position prediction box of the target tracking object is determined; the minimum bounding box is enlarged by a preset ratio to obtain a tracking viewing angle range; and the current viewing angle range of the pan-tilt camera is adjusted to the tracking viewing angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera.

[0084] Specifically, it can be explained as follows: in response to the tracking instruction of the target tracking object, the minimum bounding box including the first position detection box and the first position prediction box of the target tracking object is first determined. This minimum bounding box contains the predicted positions of the target object at the current and next moments, ensuring that the target object is always within the field of view. The minimum bounding box is enlarged by a preset ratio to obtain the tracking viewing angle range. The enlargement ratio can be set according to actual needs. The purpose is to ensure that there is a certain buffer area around the target object to prevent the target object from suddenly exceeding the field of view during movement. By adjusting the rotation angle and / or focal length of the pan-tilt camera, the current viewing angle range of the pan-tilt camera is adjusted to the tracking viewing angle range. This step ensures that the camera can focus on the target object and maintain the stability and clarity of the target object in the field of view.

[0085] It should be noted that each module in the video surveillance device in the embodiment of the present application corresponds one-to-one to each implementation step of the video surveillance method in Example 1. Since a detailed description has been given in Example 1, some details not reflected in this embodiment can be referred to Example 1 and will not be repeated here.

[0086] Example 3

[0087] According to an embodiment of the present application, a computer program product is also provided. The computer program product includes a computer program, wherein the video monitoring method in Embodiment 1 is implemented when the computer program is executed by a processor.

[0088] According to an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the video surveillance method in Example 1 by running the computer program.

[0089] According to an embodiment of the present application, a processor is further provided, which is used to run a computer program, wherein the video monitoring method in Example 1 is executed when the computer program is running.

[0090] According to an embodiment of the present application, an electronic device is also provided, which includes: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the video monitoring method in Example 1 through the computer program.

[0091] Specifically, the following steps are executed when the computer program is running: at each acquisition moment, a video frame image of the pan-tilt camera within the current viewing angle is acquired; the video frame image is analyzed using a multi-target tracking algorithm to obtain a first position detection frame of each object to be tracked in the video frame image at the current acquisition moment, and a first position prediction frame of each object to be tracked at the next acquisition moment; a target viewing angle range including each first position detection frame and each first position prediction frame is determined; and the current viewing angle range of the pan-tilt camera is adjusted to the target viewing angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera.

[0092] As an optional implementation, the electronic device may be in the form of a mobile terminal, a computer terminal or a similar computing device. Figure 3 FIG. 1 shows a hardware structure block diagram of an electronic device for implementing a video monitoring method. Figure 3As shown, the electronic device 30 may include one or more (302a, 302b, ..., 302n are used to illustrate) processors 302 (the processor 302 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 304 for storing data, and a transmission device 306 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 3 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 3 More or fewer components as shown, or with Figure 3 Different configurations are shown.

[0093] It should be noted that the one or more processors 302 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the electronic device 30. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0094] The memory 304 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the video monitoring method in the embodiment of the present application. The processor 302 executes various functional applications and data processing by running the software programs and modules stored in the memory 304, that is, to implement the vulnerability detection method of the above-mentioned application. The memory 304 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 304 may further include a memory remotely arranged relative to the processor 302, and these remote memories may be connected to the electronic device 30 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0095] The transmission device 306 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the electronic device 30. In one example, the transmission device 306 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 306 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0096] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the electronic device 30 .

[0097] The serial numbers of the above embodiments are only for description and do not represent the advantages or disadvantages of the embodiments.

[0098] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0101] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program code.

[0103] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A video monitoring method, characterized in that: include: At each acquisition moment, the video frame image of the PTZ camera within the current viewing angle is acquired; Analyzing the video frame image using a multi-target tracking algorithm to obtain a first position detection frame of each object to be tracked in the video frame image at a current acquisition time, and a first position prediction frame of each object to be tracked at a next acquisition time; Determine a target viewing angle range including each of the first position detection frames and each of the first position prediction frames; By adjusting the rotation angle and / or focal length of the pan / tilt camera, the current viewing angle range of the pan / tilt camera is adjusted to the target viewing angle range.

2. The method according to claim 1, characterized in that The multi-target tracking algorithm is a simple online real-time tracking SORT algorithm including a target detection algorithm, a Kalman filter algorithm and a Hungarian algorithm. The multi-target tracking algorithm is used to analyze the video frame image to obtain the position detection frame of each object to be tracked in the video frame image at the current acquisition time, and the position prediction frame of each object to be tracked at the next acquisition time, including: Analyzing the video frame image using a target detection algorithm to obtain second position detection frames of multiple objects; Obtaining a second position prediction frame of each object determined by using a Kalman filter algorithm at the last acquisition moment; Using the Hungarian algorithm to match each of the second position detection frames with each of the second position prediction frames, and determining the first position detection frame of each of the objects to be tracked at the current acquisition moment according to the matching result; A Kalman filter algorithm is used to predict the first position prediction frame of each of the objects to be tracked at the next acquisition moment based on the first position detection frame of each of the objects to be tracked.

3. The method according to claim 2, characterized in that Using the Hungarian algorithm to match each of the second position detection frames and each of the second position prediction frames, and determining the first position detection frame of each object to be tracked at the current acquisition moment according to the matching result, including: Respectively calculating the intersection-and-union ratio between each second position detection frame and each second position prediction frame, and constructing a cost matrix according to the obtained multiple intersection-and-union ratios; A target matching scheme is solved by using the Hungarian algorithm to minimize the cost of the cost matrix, wherein the target matching scheme includes multiple groups of matched second position detection frames and second position prediction frames, and an intersection-over-union ratio between each group of matched second position detection frames and second position prediction frames is greater than a preset threshold; For each group of the matched second position detection frames and second position prediction frames, determining that the objects corresponding to the second position detection frames and the second position prediction frames are the existing objects to be tracked, and determining that the second position detection frame is the first position detection frame of the object to be tracked; For the second position detection frame that is not successfully matched, determining that the object corresponding to the second position detection frame is the new object to be tracked, and determining that the second position detection frame is the first position detection frame of the object to be tracked; If the position prediction boxes corresponding to any object at multiple acquisition times are not matched successfully, the object will no longer be tracked.

4. The method according to claim 3, characterized in that Predicting the first position prediction frame of each of the objects to be tracked at the next acquisition moment according to the first position detection frame of each of the objects to be tracked using a Kalman filter algorithm includes: For the new object to be tracked, generating a tracker for the object to be tracked according to the first position detection frame of the object to be tracked; For the existing object to be tracked, updating the tracker of the object to be tracked according to the first position detection frame of the object to be tracked; For each of the objects to be tracked, the state of the tracker of the object to be tracked at the next acquisition time is predicted according to the state of the tracker of the object to be tracked at the current acquisition time, so as to obtain the first position prediction frame of the object to be tracked at the next acquisition time.

5. The method according to claim 1, characterized in that Determining a target viewing angle range including each of the first position detection frames and each of the first position prediction frames includes: Determine a minimum bounding box including each of the first position detection boxes and each of the first position prediction boxes; The minimum bounding box is enlarged by a preset ratio to obtain the target viewing angle range.

6. The method according to claim 1, characterized in that The method further comprises: In the case where the current viewing angle range of the pan / tilt camera cannot be adjusted to the target viewing angle range by adjusting the rotation angle and / or focal length of the pan / tilt camera, Determine a first rotation angle that includes the most objects to be tracked within the viewing angle of the pan-tilt camera when the focal length of the pan-tilt camera is at a maximum, wherein, for any object to be tracked, it is determined that the object to be tracked is included within the viewing angle of the pan-tilt camera only when the first position detection frame and the first position prediction frame of the object to be tracked are both included within the viewing angle of the pan-tilt camera; or, Determine the type of each of the objects to be tracked at the current acquisition moment by using a multi-target tracking algorithm, determine the priority level of each of the objects to be tracked according to a preset mapping relationship table, and determine a second rotation angle at which the sum of the priority levels of all the objects to be tracked within the viewing angle of the pan-tilt camera is the highest when the focal length of the pan-tilt camera is the largest, wherein the mapping relationship table stores the priority levels corresponding to different types of objects; The focal length of the pan / tilt camera is adjusted to the maximum, and the rotation angle of the pan / tilt camera is adjusted to the first rotation angle or the second rotation angle.

7. The method according to claim 1, characterized in that The method further comprises: In response to a tracking instruction for a target tracking object, determining a minimum bounding box including the first position detection box and the first position prediction box of the target tracking object; Enlarging the minimum bounding box by a preset ratio to obtain a tracking viewing angle range; The current viewing angle range of the pan / tilt camera is adjusted to the tracking viewing angle range by adjusting the rotation angle and / or focal length of the pan / tilt camera.

8. A video monitoring device, characterized in that: include: The acquisition module is used to acquire the video frame images of the PTZ camera within the current viewing angle at each acquisition moment; An analysis module is used to analyze the video frame image using a multi-target tracking algorithm to obtain a first position detection frame of each object to be tracked in the video frame image at a current acquisition time, and a first position prediction frame of each object to be tracked at a next acquisition time; A viewing angle determination module, used to determine a target viewing angle range including each of the first position detection frames and each of the first position prediction frames; The adjustment module is used to adjust the current viewing angle range of the pan-tilt camera to the target viewing angle range by adjusting the rotation angle and / or focal length of the pan-tilt camera.

9. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, the video surveillance method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the video surveillance method according to any one of claims 1 to 7 through the computer program.

Images