Large-scene video command tracking method and system and medium

Abstract

The invention provides a large-scene video command tracking method and system and a medium, and the method comprises the steps: marking an erected array camera and a high-power dome camera image, obtaining a calibration point group, storing the calibration point group in a pre-marking library, marking all camera resources in a vision field to a panoramic video under the array camera, and carrying out the marking of all camera resources in the vision field; a user can directly look up and control the corresponding camera through different camera resources; a linkage target area is selected through panoramic video automatic identification or manual frame selection of a user, the center point of the target area is adopted as a target point B, and normalized coordinates of Q in a rotary table coordinate system are calculated based on scene geometry by using a homography transformation algorithm in combination with eight calibration point groups closest to the target point B; a specific PT parameter of Q is obtained in combination with a specific turntable hardware parameter, and a PTZ parameter corresponding to the dome camera is further obtained; and issuing the PTZ parameters corresponding to the dome cameras to the corresponding dome cameras for real-time tracking. According to the invention, the problems of small picture visual range, low pixel and poor target tracking effect in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the field of video surveillance technology, and in particular to a large-scene video command tracking method, system and medium. Background Art

[0002] In the existing video command and tracking system, gun-shooting cameras and photoelectric dome cameras are two common devices. Gun-shooting cameras cannot rotate and have a relatively fixed viewing angle, and are used to monitor key areas of a fixed viewing angle. Photoelectric dome cameras have functions such as rotation, tilt, and zoom, and can observe and magnify specific areas in the monitoring scene with high precision.

[0003] In order to improve the intelligence and automation level of the monitoring system, some monitoring systems based on the linkage of gunshot and photoelectric dome cameras have gradually emerged in recent years. That is, when an abnormality is detected in the key area monitored by the gunshot camera, the nearest dome camera will be called to zoom in and point to the abnormal area for key observation, hereinafter referred to as gun-ball linkage. The gun-ball linkage system realizes all-round monitoring of the monitoring scene through the collaborative work of two cameras. The main process is as follows:

[0004] 1. Target detection and identification: Fixed-barrel cameras monitor a large area in real time. When a suspicious target (such as a person or vehicle) is detected, the system automatically identifies the target and determines its location in the monitoring scene;

[0005] 2. Target positioning and tracking: The system calculates the target's position coordinates and transmits this information to the photoelectric ball camera through a preset linkage protocol;

[0006] 3. Photoelectric ball camera linkage: After receiving the information of the target location, the photoelectric ball camera quickly rotates, tilts and zooms to the area where the target is located to monitor and track the target in detail. The photoelectric ball camera can continuously adjust the angle and focal length of the camera according to the movement of the target to ensure that the target is always within the monitoring field of view;

[0007] 4. Alarm linkage: In some cases, the system can also be linked with the alarm device. When the target's behavior meets certain preset abnormal conditions, the alarm is triggered to notify the monitoring personnel or initiate other safety measures.

[0008] Some gun-ball linkage systems also rely on specific hardware devices or sensors, especially in large-scale monitoring scenarios. However, in the entire command and tracking system, how to quickly and accurately convert the target point in the panoramic coordinate system into the coordinate point in the (ball camera) turntable coordinate system, and then operate the turntable to the corresponding position is still the core of the entire system. That is, how to achieve efficient linkage of multiple perspectives and multiple cameras is still a major problem faced by existing technologies.

[0009] In the prior art, there are two main ways to control the linkage of cameras: one is to achieve it through grid annotation, that is, calibration is performed every certain pixel value in the horizontal and vertical directions, and then the position of the target in the turntable coordinate system is calculated based on one or two marking points near the target point. This requires prior grid annotation of the panoramic field of view, but for large scenes, for example, when the scene is a sea area with a resolution of 23040×6480, most of the picture is the sea surface, and it is difficult to match part of the panoramic picture with the ball camera picture because of the lack of target objects and the inability to achieve uniform and detailed annotation, resulting in poor actual linkage effect and inability to accurately link to the corresponding position. The other is to calibrate the internal and external parameters of the camera for linkage. This method requires converting the internal and external parameters of multiple devices into a unified coordinate system (geod coordinates), which leads to a complicated process and an exponential increase in errors. It can only be tracked by the tracking algorithm of the ball camera itself. Once the target moves too fast and exceeds the range of the ball camera, the target is lost and difficult to find. Moreover, this algorithm is only applicable to monocular cameras. For other panoramic cameras such as array cameras, it is necessary to calculate each of its sub-lens lenses one by one. The calculation is very complicated, which seriously affects timeliness.

[0010] In addition, the existing gun-type or gun-ball linkage systems have small FOV range and insufficient video imaging, and cannot achieve real-time monitoring and target recognition of large-scale monitoring scenes (such as large sea areas, border defense, etc.). Although the video pixels can meet the needs of target recognition through optical zoom and electronic amplification of the photoelectric ball camera, it is impossible to achieve real-time monitoring and recognition of the entire area because only the area within the current field of view can be taken into account during polling or scanning. In terms of user use, whether it is through digital twins or fisheye cameras as the main use page, there are defects such as lack of intuitiveness, insufficient pixels, and inability to view global information in real time. The main reason for digital twins is that the underlying data cannot support the real-time update requirements of global digital twins, resulting in insufficient real-time data coverage and timeliness. However, due to the excessive distortion of fisheye cameras, the overall viewing experience will be greatly reduced. The global video of simple panoramic cameras cannot meet the global target recognition and tracking under a large field of view due to insufficient pixels.

[0011] In view of this, it is urgent to study an efficient video command and tracking method that can be applied to large scenarios such as sea areas. Summary of the invention

[0012] The technical problem to be solved by the present invention is to provide a large-scene video command tracking method, system and medium to solve the problems of small visible range, low pixel and poor target tracking effect in the existing picture.

[0013] In a first aspect, the present invention provides a method for tracking large-scale video commands, comprising the following steps:

[0014] Step S1: annotate the array camera and high-magnification ball camera images that have been set up, obtain the calibration point group and store it in the pre-annotation library, annotate all camera resources in the field of view on the panoramic video under the array camera, so that users can directly view and control the corresponding camera through different camera resources;

[0015] Step S2: select the linkage target area through automatic recognition of the panoramic video or manual selection by the user, and use the center point of the target area as the target point B. Based on the scene geometry, the homography transformation algorithm is used to combine the eight nearest calibration point groups of the target point B in the panoramic video screen coordinate system to calculate the normalized coordinates of Q in the turntable coordinate system, and the specific PT parameters of Q are obtained in combination with the specific turntable hardware parameters, and the PTZ parameters corresponding to the ball camera are further obtained;

[0016] Step S3: Send the PTZ parameters corresponding to the ball camera to the corresponding ball camera for real-time tracking.

[0017] Furthermore, the method for obtaining the calibration point group in step S1 is specifically as follows:

[0018] Obtain a landmark A (such as a fixed building, object, tree, etc.) in the panoramic video, select the target in the panoramic video, and record the normalized coordinates P (X p ,Y p ), where 1≥X p ≥0, 1≥Y p ≥0;

[0019] At the same time, manually point the high-power ball camera video center to be calibrated to the marker and record its actual PT parameter Q(P Q ,T Q ), where P Q and T Q Depends on the horizontal and vertical rotation values ​​of the actual turntable;

[0020] Combined with the actual limit value, the parameter Q is normalized to M(X M ,Y M ), where 1≥X M ≥0, 1≥Y M ≥0;

[0021] Record and save the corresponding relationship between P and M, and create calibration points;

[0022] Repeat the above steps to obtain 10-20 scattered calibration points and save their data to the pre-annotated library.

[0023] Furthermore, the step S2 specifically includes:

[0024] Step S21: select the linkage target area through automatic recognition of the panoramic video or manual selection by the user, and use the center point of the target area as the target point B (X b ,Y b );

[0025] Step S22: Select the 8 points with the closest Euclidean distance to the target point B from the calibration point group stored in step 1, and record them as P i (X i ,Y i )(0≤i≤7) and M i (X i ,Y i )(0≤i≤7);

[0026] Step S23: i and M i The array is used through the findHomography method in OpenCV to calculate the homography matrix H based on the RANSAC (Random Sample Consensus) algorithm;

[0027] Step S24: transform the target point B(X b ,Y b ) is transformed into the normalized coordinate M(X m ,Y m );

[0028] Step S25, converting and restoring the actual PT parameters according to the actual PT limit value of the ball camera;

[0029] Step S26, calculate the Zoom value of the ball camera to obtain the PTZ parameters of the ball camera.

[0030] Furthermore, first, in the panoramic video screen, the FOV of the target is approximately calculated by the panoramic field of view FOV and the percentage of the target area occupying the panoramic width. 2 , and set the FOV 2 As the field of view angle value that the ball camera should have after zooming, FOV 2 The calculation formula is as follows:

[0031]

[0032] Then according to the target field of view FOV after zooming 2 Calculate the focal length F that the ball camera should zoom to 2 , the focal length F is calculated according to the following formula 2 :

[0033]

[0034] Where S is the size of the photosensitive element;

[0035] Finally adjust the dome camera's Zoom value to focal length F 2 You can get the PTZ parameters of the ball camera.

[0036] Furthermore, when the user selects a target area in the panoramic video, the nearest dome camera is pointed to the area by default and supports manual designation of the corresponding dome camera.

[0037] Furthermore, AI is used to identify targets in panoramic videos, and models in different application scenarios are trained to perform real-time detection and tracking. When an early warning is detected, the nearest dome camera is automatically called to point to and track the target.

[0038] In a second aspect, the present invention provides a large-scene video command tracking system, the system comprising:

[0039] The annotation module is used to annotate the array cameras and high-magnification ball camera images that have been set up, obtain the calibration point group and store it in the pre-annotation library, and annotate all camera resources in the field of view to the panoramic video under the array camera, so that users can directly view and control the corresponding camera through different camera resources;

[0040] The parameter calculation module is used to select the linkage target area through automatic recognition of the panoramic video or manual selection by the user, and use the center point of the target area as the target point B (X b ,Y b ), based on the scene geometry, the homography transformation algorithm is used to combine the eight nearest calibration point groups of the target point B in the panoramic video screen coordinate system to calculate the normalized coordinates of Q in the turntable coordinate system, and the specific PT parameters of Q are obtained in combination with the specific turntable hardware parameters, and the PTZ parameters corresponding to the ball camera are further obtained;

[0041] The target tracking module is used to send the PTZ parameters corresponding to the ball camera to the corresponding ball camera for real-time tracking.

[0042] Furthermore, the method for obtaining the calibrated point group in the annotation module is as follows:

[0043] Obtain a landmark A (such as a fixed building, object, tree, etc.) in the panoramic video, select the target in the panoramic video, and record the normalized coordinates P (X p ,Y p ), where 1≥X p ≥0, 1≥Y p ≥0;

[0044] At the same time, manually point the high-power ball camera video center to be calibrated to the marker and record its actual PT parameter Q(P Q ,T Q), where P Q and T Q Depends on the horizontal and vertical rotation values ​​of the actual turntable;

[0045] Combined with the actual limit value, the parameter Q is normalized to M(X M ,Y M ), where 1≥X M ≥0, 1≥Y M ≥0;

[0046] Record and save the corresponding relationship between P and M, and create calibration points;

[0047] Repeat the above steps to obtain 10-20 scattered calibration points and save their data to the pre-annotated library.

[0048] Furthermore, the parameter calculation module specifically includes:

[0049] The linkage target area is selected through automatic recognition of the panoramic video or manual selection by the user, and the center point of the target area is used as the target point B (X b ,Y b );

[0050] Select the 8 points with the closest Euclidean distance to the target point B from the calibration point group stored in the annotation module, denoted as P i (X i ,Y i )(0≤i≤7) and M i (X i ,Y i )(0≤i≤7);

[0051] P i and M i The array is used through the findHomography method in OpenCV to calculate the homography matrix H based on the RANSAC (RandomSample Consensus) algorithm;

[0052] Through the homography matrix H, the target point B(X b ,Y b ) is transformed into the normalized coordinate M(X m ,Y m );

[0053] The actual PT parameters are restored based on the actual PT limit value of the ball camera;

[0054] Calculate the Zoom value of the ball camera and obtain the PTZ parameters of the ball camera.

[0055] Furthermore, the calculation process of the dome camera Zoom value is as follows: First, in the panoramic video screen, the FOV of the target is approximately calculated by the panoramic field of view FOV and the percentage of the target area occupying the panoramic width. 2 , and set the FOV 2 As the field of view angle value that the ball camera should have after zooming, FOV 2 The calculation formula is as follows:

[0056]

[0057] Then according to the target field of view FOV after zooming 2 Calculate the focal length F that the ball camera should zoom to 2 , the focal length F is calculated according to the following formula 2 :

[0058]

[0059] Where S is the size of the photosensitive element;

[0060] Finally adjust the dome camera's Zoom value to focal length F 2 You can get the PTZ parameters of the ball camera.

[0061] Furthermore, when the user selects a target area in the panoramic video, the nearest dome camera is pointed to the area by default and supports manual designation of the corresponding dome camera.

[0062] Furthermore, AI is used to identify targets in panoramic videos, and models in different application scenarios are trained to perform real-time detection and tracking. When an early warning is detected, the nearest dome camera is automatically called to point to and track the target.

[0063] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0064] The advantages of the present invention are:

[0065] 1. The present invention uses an array camera and a ball camera for linkage control. By utilizing the characteristics of the array camera that can see far and wide (the video pixels reach billions, and the horizontal FOV reaches 75°), it can realize a panoramic video command and tracking system in a large sea area. It is more intuitive for users and can quickly locate the specific location of camera resources in the field of view. In addition, combined with the early warning function, all events of interest in the field of view can be presented at the corresponding spatial position, which is more conducive to the overall coordination and control of managers than directly using GIS or digital twins.

[0066] 2. By combining spatial geometry (homography transformation model) with the video annotation method of the present invention, the target can be quickly and accurately located and the PTZ and other parameters required by the ball camera can be obtained by simply annotating the installed array camera and high-magnification ball camera. There is no need to annotate the internal and external parameters of each camera lens in turn. The PTZ parameters of the target area can be directly calculated and sent to the ball camera. The entire control algorithm is simple and the process is fast and accurate.

[0067] 3. All data of the present invention are obtained through calculation and normalization, and have nothing to do with the specific hardware support functions. Therefore, no matter what brand of ball machine or turntable, no matter whether it supports PelcoD, OnVif, GB28181 or private protocol, it can be compatible with the control method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0069] Figure 1 This is a flowchart of a large-scene video command tracking method in Embodiment 1 of the present invention;

[0070] Figure 2 This is a structural diagram of a large-scene video command and tracking system in Embodiment 2 of the present invention;

[0071] Figure 3 Schematic diagram of the structure of the medium in the third embodiment of the present invention.

[0072] Figure 4 It is a schematic diagram of the equipment installation of the present invention.

[0073] Figure 5 A flowchart of the process for marking the object of the present invention.

[0074] Figure 6 It is a schematic diagram of the coordinate system of the geometric model of the present invention.

[0075] Figure 7 This is a flow chart of the execution of the parameter calculation algorithm of the present invention.

[0076] Figure 8 The figure is a schematic diagram of the process of obtaining the Zoom parameters in the PTZ parameters of the ball camera of the present invention.

[0077] Fig. 9 It is a schematic diagram of the target tracking process of the present invention.

[0078] Fig.10 It is a schematic diagram of the interface effect in one embodiment of the present invention. DETAILED DESCRIPTION

[0079] The embodiments of the present application provide a large-scene video command and tracking method, system and medium based on a scene geometric model, so as to solve the problems of insufficient target tracking control accuracy and poor timeliness, as well as poor panoramic display effect, existing in the existing large-scene video command and tracking system.

[0080] The technical solution in the embodiments of the present application has the following general idea: the present invention combines the characteristics of array cameras that can see wide and far (panoramic video pixels are 150 million, resolution is 23040×6480, and horizontal FOV is 75°), and can perform multi-target recognition and tracking on the whole-domain video, and the panoramic picture spliced ​​by the array camera calculation camera has the characteristic of no distortion. At the same time, because the array camera itself is heavy after installation, it has good wind and rain resistance. Its field of view is relatively fixed in the outdoor environment for a long time, and it is not affected by the environment, and the fused video does not have the distortion phenomenon of fisheye lens. The present invention proposes a method for linkage control of array cameras and photoelectric ball cameras based on scene geometry. By applying the homography transformation model, it can more accurately calculate the spatial relationship between cameras, and realize faster and more accurate linkage operation.

[0081] In order to make the present invention more clearly understood, preferred embodiments are now described in detail with reference to the accompanying drawings.

[0082] Embodiment 1

[0083] This embodiment provides a large scene video command tracking method based on scene geometric model. Figure 1 As shown, the method includes:

[0084] Step S1: annotate the array camera and high-power ball camera images that have been set up, obtain the calibration point group and store it in the pre-annotation library, annotate all camera resources in the field of view on the panoramic video under the array camera, so that users can directly view and control the corresponding camera through different camera resources, such as Fig.10 As shown;

[0085] Step S2: select the linkage target area through automatic recognition of the panoramic video or manual selection by the user, and use the center point of the target area as the target point B (X b ,Y b ), based on the scene geometry, the homography transformation algorithm is used to combine the eight nearest calibration point groups of the target point B in the panoramic video screen coordinate system to calculate the normalized coordinates of Q in the turntable coordinate system, and the specific PT parameters of Q are obtained in combination with the specific turntable hardware parameters, and the PTZ parameters corresponding to the ball camera are further obtained;

[0086] Step S3: Send the PTZ parameters corresponding to the ball camera to the corresponding ball camera for real-time tracking.

[0087] Better, such as Figure 5As shown, the method for obtaining the calibration point group in step S1 is specifically as follows:

[0088] Obtain a landmark A (such as a fixed building, object, tree, etc.) in the panoramic video, select the target in the panoramic video, and record the normalized coordinates P (X p ,Y p ), where 1≥X p ≥0, 1≥Y p ≥0;

[0089] At the same time, manually point the high-power ball camera video center to be calibrated to the marker and record its actual PT parameter Q(P Q ,T Q ), where P Q and T Q Depends on the horizontal and vertical rotation values ​​of the actual turntable;

[0090] Combined with the actual limit value, the parameter Q is normalized to M(X M ,Y M ), where 1≥X M ≥0, 1≥Y M ≥0; For example, in the control of OnVif / PelcoD and other protocols, the values ​​of P and T will be sent. The range of these values ​​is determined by the hardware manufacturer. Some are continuous intervals, and some may be discontinuous intervals. For example, the actual vertical rotation range of a turntable is ±30°, and the horizontal rotation range is ±180°. When actually issuing a command, the value range of P may be [-180, +180], or [0, 360], etc.; and the value range of T may be [0, 60], or (0, 30] ∪ [330, 360], etc. At this time, the normalization processing can be set as follows: when the value P range is [-180, +180], the normalized value is (P + 180) ÷ 360; when the value P range is [0, 360], the normalized value is P ÷ 360; when the value T range is [0, 60], the normalized value is T ÷ 60; when the value T range is (0, 30] ∪ [330, 360], the normalized value is:

[0091]

[0092] In this way, the specifications of different hardware manufacturers are normalized to make their values ​​uniform within the range of [0,1]. After this processing, the overall algorithm will not be related to the hardware, and the robustness is high, so it can be compatible with devices of different manufacturers, types, and protocols.

[0093] Record and save the corresponding relationship between P and M, and create calibration points;

[0094] Repeat the above steps to obtain 10-20 scattered calibration points (the calibration points are taken from the key areas that can be calibrated in the scene. For example, for scene videos containing sea areas, the annotation points should be taken from points on the coastline because the sea surface and the sky are not suitable for calibration) and save their data to the pre-annotated library.

[0095] Better, such as Figure 6 and Figure 7 As shown, the step S2 specifically includes:

[0096] Step S21: select the linkage target area through automatic recognition of the panoramic video or manual selection by the user, and use the center point of the target area as the target point B (X b ,Y b );

[0097] Step S22: Select the 8 points with the closest Euclidean distance to the target point B from the calibration point group stored in step 1, and record them as P i (X i ,Y i )(0≤i≤7) and M i (X i ,Y i )(0≤i≤7);

[0098] Step S23: i and M i The array is used through the findHomography method in OpenCV to calculate the homography matrix H based on the RANSAC (Random Sample Consensus) algorithm;

[0099] Step S24: transform the target point B(X b ,Y b ) is transformed into the normalized coordinate M(X m ,Y m );

[0100] Step S25, converting and restoring the actual PT parameters according to the actual PT limit value of the ball camera, that is, restoring the normalized data to the actual value;

[0101] Step S26, calculate the Zoom value of the ball camera to obtain the PTZ parameters of the ball camera.

[0102] The homography transformation refers to the existence of homography between images of the same planar object taken from different positions by a camera without lens distortion, which can be represented by a projection transformation (homography transformation). Homography (also known as perspective transformation) allows us to switch from one view of the same scene to another by multiplying the homography matrix with the points in one view to find their corresponding positions in another view. Here, we use a virtual PT parameter coordinate system that does not actually exist, and take the scene taken by the panoramic camera as the first plane, and the plane obtained by scanning the ball camera in the panoramic camera field of view as the second plane, as shown in Figure 4 As shown in the figure, the panoramic video images are obtained by setting up array cameras, and the dome camera is set up to observe the target and rotate and stitch it into an overall field of view. The coordinates (x, y) of the first plane are defined as the normalized coordinates of the target in the panoramic video, and the coordinates (x, y) of the second plane are defined as the PT parameters of the dome camera and the normalized coordinates converted according to the actual limit of the dome camera, as shown in Figure 6 As shown, two corresponding coordinate system models are constructed. Then, the homography transformation matrix model is calculated based on the 8 annotation points closest to the target point in the panoramic field of view, and the PT parameter coordinates of the target point in the scanning field of view of the ball camera are deduced through the matrix, and the actual PT parameters of the ball camera are further deduced, and the Z parameters are deduced based on the maximum magnification, field of view angle, and field of view angle of the panoramic camera, so as to command the ball camera to point to the corresponding PTZ area.

[0103] The step S23 specifically includes:

[0104] P i (x i ',y i ') and M i (x i ,y i )'s array is expanded as follows:

[0105]

[0106] From the correspondence between plane coordinates and homogeneous coordinates You can get:

[0107]

[0108] Written in matrix form as AH = 0, then:

[0109]

[0110] When solving, add the constraint h33=1, and use the RANSAC algorithm to calculate the homography matrix H based on the 8 points passed in. Since the homography matrix has 8 degrees of freedom, the constraint h33=1 is generally added when solving, so there are a total of 8 unknowns, and at least 4 non-collinear points are required to solve the matrix parameter H. Since the sampling points obtained by annotation may have noise (machine errors and manual errors will cause deviations), taking only 4 points will often lead to inaccurate results due to noise. Therefore, the RANSAC algorithm is used here to calculate the homography matrix H, and passing in 8 points can effectively reduce the impact of noise.

[0111] Better, such as Figure 8 As shown, the step S26 specifically includes: first, in the panoramic video screen, the FOV of the target is approximately calculated by the panoramic field of view FOV and the percentage of the panoramic width occupied by the target area (the percentage is the normalized width W of the area) 2 , and set the FOV 2 As the field of view angle value that the ball camera should have after zooming, FOV 2 The calculation formula is as follows:

[0112]

[0113] Then according to the target field of view FOV after zooming 2 Calculate the focal length F that the ball camera should zoom to 2 , the focal length F is calculated according to the following formula 2 :

[0114]

[0115] Where S is the size of the photosensitive element, usually in millimeters (mm), which can be either width or height, depending on whether the horizontal or vertical field of view is calculated, and F 2 is the focal length, expressed in millimeters (mm), and FOV is the field of view, expressed in degrees. 2 is the target field of view after zooming;

[0116] Finally adjust the dome camera's Zoom value to focal length F 2 You can get the PTZ parameters of the ball camera.

[0117] After the above steps, the target can be identified in real time in the panorama, and the target position can be calculated into the PTZ parameters of the ball camera and sent to the ball camera for real-time tracking. Fig. 9 It has nothing to do with whether the ball camera itself supports the target tracking function, as long as the ball camera supports the turntable protocol, it can realize this function.

[0118] Preferably, when the user selects a target area in the panoramic video, the nearest dome camera is pointed to the area by default and manual designation of the corresponding dome camera is supported. If the nearest dome camera is not desired to be pointed to, manual designation can be performed.

[0119] Preferably, AI is used to identify targets in panoramic videos, and models in different application scenarios are trained for real-time detection and tracking. When an early warning is detected, the nearest ball camera is automatically called to point to and track the target.

[0120] Based on the same inventive concept, the present application also provides a system corresponding to the method in Example 1, see Example 2 for details.

[0121] Embodiment 2

[0122] In this embodiment, a large scene video command tracking method based on scene geometric model is provided. Figure 2 As shown, the system comprises:

[0123] The annotation module is used to annotate the array cameras and high-magnification ball cameras that have been set up, obtain the calibration point group and store it in the pre-annotation library, and annotate all camera resources in the field of view to the panoramic video under the array camera, so that users can directly view and control the corresponding camera through different camera resources, such as Fig.10 As shown;

[0124] The parameter calculation module is used to select the linkage target area through automatic recognition of the panoramic video or manual selection by the user, and use the center point of the target area as the target point B (X b ,Y b ), based on the scene geometry, the homography transformation algorithm is used to combine the eight nearest calibration point groups of the target point B in the panoramic video screen coordinate system to calculate the normalized coordinates of Q in the turntable coordinate system, and the specific PT parameters of Q are obtained in combination with the specific turntable hardware parameters, and the PTZ parameters corresponding to the ball camera are further obtained;

[0125] The target tracking module is used to send the PTZ parameters corresponding to the ball camera to the corresponding ball camera for real-time tracking.

[0126] Better, such as Figure 5 As shown, the method for obtaining the calibrated point group in the marking module is specifically as follows:

[0127] Obtain a landmark A (such as a fixed building, object, tree, etc.) in the panoramic video, select the target in the panoramic video, and record the normalized coordinates P (X p ,Y p ), where 1≥X p ≥0, 1≥Y p ≥0;

[0128] At the same time, manually point the high-power ball camera video center to be calibrated to the marker and record its actual PT parameter Q(P Q ,T Q ), where P Q and T Q Depends on the horizontal and vertical rotation values ​​of the actual turntable;

[0129] Combined with the actual limit value, the parameter Q is normalized to M(X M ,Y M ), where 1≥X M ≥0, 1≥Y M ≥0; For example, in the control of OnVif / PelcoD and other protocols, the values ​​of P and T will be sent. The range of these values ​​is determined by the hardware manufacturer. Some are continuous intervals, and some may be discontinuous intervals. For example, the actual vertical rotation range of a turntable is ±30°, and the horizontal rotation range is ±180°. When actually issuing a command, the value range of P may be [-180, +180], or [0, 360], etc.; and the value range of T may be [0, 60], or (0, 30] ∪ [330, 360], etc. At this time, the normalization processing can be set as follows: when the value P range is [-180, +180], the normalized value is (P + 180) ÷ 360; when the value P range is [0, 360], the normalized value is P ÷ 360; when the value T range is [0, 60], the normalized value is T ÷ 60; when the value T range is (0, 30] ∪ [330, 360], the normalized value is:

[0130]

[0131] In this way, the specifications of different hardware manufacturers are normalized to make their values ​​uniform within the range of [0,1]. After this processing, the overall algorithm will not be related to the hardware, and the robustness is high, so it can be compatible with devices of different manufacturers, types, and protocols.

[0132] Record and save the corresponding relationship between P and M, and create calibration points;

[0133] Repeat the above steps to obtain 10-20 scattered calibration points (the calibration points are taken from the key areas that can be calibrated in the scene. For example, for scene videos containing sea areas, the annotation points should be taken from points on the coastline because the sea surface and the sky are not suitable for calibration) and save their data to the pre-annotated library.

[0134] Better, such as Figure 6 and Figure 7 As shown, the parameter calculation module specifically includes:

[0135] The linkage target area is selected through automatic recognition of the panoramic video or manual selection by the user, and the center point of the target area is used as the target point B (X b ,Y b );

[0136] Select the 8 points with the closest Euclidean distance to the target point B from the calibration point group stored in the annotation module, denoted as P i (X i ,Y i )(0≤i≤7) and M i (X i ,Y i )(0≤i≤7);

[0137] P i and M i The array is used through the findHomography method in OpenCV to calculate the homography matrix H based on the RANSAC (RandomSample Consensus) algorithm;

[0138] Through the homography matrix H, the target point B(X b ,Y b ) is transformed into the normalized coordinate M(X m ,Y m );

[0139] The actual PT parameters are converted and restored according to the actual PT limit value of the ball camera, that is, the normalized data is restored to the actual value;

[0140] Calculate the Zoom value of the ball camera and obtain the PTZ parameters of the ball camera.

[0141] The homography transformation refers to the existence of homography between images of the same planar object taken from different positions by a camera without lens distortion, which can be represented by a projection transformation (homography transformation). Homography (also known as perspective transformation) allows us to switch from one view of the same scene to another by multiplying the homography matrix with the points in one view to find their corresponding positions in another view. Here, we use a virtual PT parameter coordinate system that does not actually exist, and take the scene taken by the panoramic camera as the first plane, and the plane obtained by scanning the ball camera in the panoramic camera field of view as the second plane, as shown in Figure 4 As shown in the figure, the panoramic video images are obtained by setting up array cameras, and the dome camera is set up to observe the target and rotate and stitch it into an overall field of view. The coordinates (x, y) of the first plane are defined as the normalized coordinates of the target in the panoramic video, and the coordinates (x, y) of the second plane are defined as the PT parameters of the dome camera and the normalized coordinates converted according to the actual limit of the dome camera, as shown in Figure 6As shown, two corresponding coordinate system models are constructed. Then, the homography transformation matrix model is calculated based on the 8 annotation points closest to the target point in the panoramic field of view, and the PT parameter coordinates of the target point in the scanning field of view of the ball camera are deduced through the matrix, and the actual PT parameters of the ball camera are further deduced, and the Z parameters are deduced based on the maximum magnification, field of view angle, and field of view angle of the panoramic camera, so as to command the ball camera to point to the corresponding PTZ area.

[0142] The homography matrix calculation process is as follows:

[0143] P i (x i ',y i ') and M i (x i ,y i )'s array is expanded as follows:

[0144]

[0145] From the correspondence between plane coordinates and homogeneous coordinates You can get:

[0146]

[0147] Written in matrix form as AH = 0, then:

[0148]

[0149] When solving, add the constraint h33=1, and use the RANSAC algorithm to calculate the homography matrix H based on the 8 points passed in. Since the homography matrix has 8 degrees of freedom, the constraint h33=1 is generally added when solving, so there are a total of 8 unknowns, and at least 4 non-collinear points are required to solve the matrix parameter H. Since the sampling points obtained by annotation may have noise (machine errors and manual errors will cause deviations), taking only 4 points will often lead to inaccurate results due to noise. Therefore, the RANSAC algorithm is used here to calculate the homography matrix H, and passing in 8 points can effectively reduce the impact of noise.

[0150] Better, such as Figure 8 As shown in the figure, the calculation process of the dome camera Zoom value is as follows: First, in the panoramic video screen, the FOV of the target is approximately calculated by the panoramic field of view FOV and the percentage of the target area occupying the panoramic width (the percentage is the normalized width W of the area). 2 , and set the FOV 2 As the field of view angle value that the ball camera should have after zooming, FOV 2 The calculation formula is as follows:

[0151]

[0152] Then according to the target field of view FOV after zooming 2 Calculate the focal length F that the ball camera should zoom to 2 , the focal length F is calculated according to the following formula 2 :

[0153]

[0154] Where S is the size of the photosensitive element, usually in millimeters (mm), which can be either width or height, depending on whether the horizontal or vertical field of view is calculated, and F 2 is the focal length, expressed in millimeters (mm), and FOV is the field of view, expressed in degrees. 2 is the target field of view after zooming;

[0155] Finally adjust the dome camera's Zoom value to focal length F 2 You can get the PTZ parameters of the ball camera.

[0156] After the above steps, the target can be identified in real time in the panorama, and the target position can be calculated into the PTZ parameters of the ball camera and sent to the ball camera for real-time tracking. Fig. 9 It has nothing to do with whether the ball camera itself supports the target tracking function, as long as the ball camera supports the turntable protocol, it can realize this function.

[0157] Preferably, when the user selects a target area in the panoramic video, the nearest dome camera is pointed to the area by default and manual designation of the corresponding dome camera is supported. If the nearest dome camera is not desired to be pointed to, manual designation can be performed.

[0158] Preferably, AI is used to identify targets in panoramic videos, and models in different application scenarios are trained for real-time detection and tracking. When an early warning is detected, the nearest ball camera is automatically called to point to and track the target.

[0159] Since the device introduced in the second embodiment of the present invention is a device used to implement the method of the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0160] Based on the same inventive concept, the present application provides a storage medium corresponding to Example 1, see Example 3 for details.

[0161] Embodiment 3

[0162] This embodiment provides a computer-readable storage medium, such as Figure 3As shown, a computer program is stored thereon, and when the computer program is executed by a processor, any implementation method in Example 1 can be implemented.

[0163] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] In summary, the advantages of the present invention are as follows: The present invention combines the characteristics of the array camera that can see wide and far (the video pixels reach the billion level, and the horizontal FOV reaches 75°), and can perform multi-target recognition and tracking on the global video, solving the real-time monitoring of large scene sea areas that cannot be solved by using multiple shooting videos, fisheye lenses or ball camera scanning solutions in the existing system, and can realize global and full-time target recognition and AI empowerment, such as multi-target detection and tracking of ships by array cameras in offshore wind farms. By adopting the method of spatial geometry (homography transformation model), the complexity of the camera internal and external parameter calibration method that requires calibration of more than a dozen sub-lenses in the array camera is solved, and the problem of slow linkage speed of the ball camera and high complexity of the control algorithm is solved. The method of the present invention only needs to mark the video of the array camera and the high-magnification ball camera, and then the target can be quickly and accurately located, and the PTZ and other parameters required by the ball camera can be obtained. There is no need to mark the internal and external parameters of each camera lens in turn, and the PTZ parameters of the target area are directly calculated and sent to the ball camera. The whole process is fast and accurate. This solution supports labeling at any position within the field of view. Key areas can be densely labeled, and non-key areas can be sparsely labeled. At the same time, in practical applications, the algorithm using spatial geometry is more robust, more accurate and stable than the gridding method.

[0165] Since all data are calculated and normalized, they are not related to the specific hardware support functions. Therefore, no matter what brand of ball camera or turntable, no matter whether it supports PelcoD, OnVif, GB28181 or private protocol, it can be compatible with this solution.

[0166] Since panoramic video can be AI-enabled, key targets can be tracked and detected by the panoramic view all the time, so that the ball camera can perform full-time tracking or multi-target polling through calculation, regardless of whether the ball camera supports this function. This reduces the hardware dependence of the tracking and detection function on the ball camera.

[0167] The present invention can realize a panoramic-based video command and tracking system, which is more intuitive for users and can quickly locate the specific location of camera resources in the field of view. In addition, combined with the early warning function, all events of interest in the field of view can be presented at the corresponding spatial location, which is more conducive to the overall coordination and control of managers than directly using GIS or digital twins.

[0168] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A large-scene video command tracking method, characterized by: The steps include: Step S1: annotate the array camera and high-magnification ball camera images that have been set up, obtain a calibration point group and store it in a pre-annotation library, annotate all camera resources in the field of view on the panoramic video under the array camera, so that users can directly check and control the corresponding camera through different camera resources. The calibration point group includes a plurality of normalized coordinates P of the panoramic video calibration points A and the normalized coordinates M of the ball camera PT parameters Q that correspond one to one; Step S2: select the linkage target area through automatic recognition of the panoramic video or manual selection by the user, and use the center point of the target area as the target point B. Based on the scene geometry, the homography transformation algorithm is used to combine the eight nearest calibration point groups of the target point B in the panoramic video screen coordinate system to calculate the normalized coordinates of Q in the turntable coordinate system, and the specific PT parameters of Q are obtained in combination with the specific turntable hardware parameters, and the PTZ parameters corresponding to the ball camera are further obtained; Step S3: Send the PTZ parameters corresponding to the ball camera to the corresponding ball camera for real-time tracking.

2. A large-scene video command tracking method as claimed in claim 1, characterized in that: The specific method of obtaining the calibration point group in step S1 is as follows: Get the landmark A in the panoramic video, select the target in the panoramic video, and record the normalized coordinates P (X p ,Y p ), where 1≥X p ≥0, 1≥Y p ≥0; At the same time, manually point the high-power ball camera video center to be calibrated to the marker and record its actual PT parameter Q(P Q ,T Q ), where P Q and T Q Depends on the horizontal and vertical rotation values ​​of the actual turntable; Combined with the actual limit value, the parameter Q is normalized to M(X M ,Y M ), where 1≥X M ≥0, 1≥Y M ≥0; Record and save the corresponding relationship between P and M, and create calibration points; Repeat the above steps to obtain 10-20 scattered calibration points and save their data to the pre-annotated library.

3. A large-scene video command tracking method as claimed in claim 1, characterized in that: The step S2 specifically includes: Step S21: select the linkage target area through automatic recognition of the panoramic video or manual selection by the user, and use the center point of the target area as the target point B (X b ,Y b ); Step S22: Select the 8 points with the closest Euclidean distance to the target point B from the calibration point group stored in step 1, and record them as P i (X i ,Y i )(0≤i≤7) and M i (X i ,Y i )(0≤i≤7); Step S23: i and M i The array is used through the findHomography method in OpenCV to calculate the homography matrix H based on the RANSAC algorithm; Step S24: transform the target point B(X b ,Y b ) is transformed into the normalized coordinate M(X m ,Y m ); Step S25, converting and restoring the actual PT parameters according to the actual PT limit value of the ball camera; Step S26, calculate the Zoom value of the ball camera to obtain the PTZ parameters of the ball camera.

4. A large-scene video command tracking method as claimed in claim 3, characterized in that: The step S26 specifically includes: first, in the panoramic video screen, the FOV2 of the target is approximately calculated by the panoramic field of view FOV and the percentage of the target area occupying the panoramic width, and FOV2 is used as the field of view value that the ball camera should have after the zoom is changed. The calculation formula of FOV2 is as follows: Then calculate the focal length F2 that the ball camera should zoom to based on the target field of view FOV2 after zooming. The focal length F2 is calculated according to the following formula: Where S is the size of the photosensitive element; Finally, adjust the Zoom value of the ball camera to the focal length F2 to obtain the PTZ parameters of the ball camera.

5. A large-scene video command and tracking system, characterized by: The system comprises: The annotation module is used to annotate the array cameras and high-magnification ball camera images that have been set up, obtain the calibration point group and store it in the pre-annotation library, and annotate all camera resources in the field of view to the panoramic video under the array camera, so that users can directly view and control the corresponding camera through different camera resources; The parameter calculation module is used to select the linkage target area through automatic recognition of the panoramic video or manual selection by the user, and use the center point of the target area as the target point B. Based on the scene geometry, the homography transformation algorithm is used to combine the eight nearest calibration point groups of the target point B in the panoramic video screen coordinate system to calculate the normalized coordinates of Q in the turntable coordinate system, and the specific PT parameters of Q are obtained in combination with the specific turntable hardware parameters, and the PTZ parameters corresponding to the ball camera are further obtained; The target tracking module is used to send the PTZ parameters corresponding to the ball camera to the corresponding ball camera for real-time tracking.

6. A large-scene video command and tracking system as claimed in claim 5, characterized in that: The specific method of obtaining the calibrated point group in the annotation module is as follows: Get the landmark A in the panoramic video, select the target in the panoramic video, and record the normalized coordinates P (X p ,Y p ), where 1≥X p ≥0, 1≥Y p ≥0; At the same time, manually point the high-power ball camera video center to be calibrated to the marker and record its actual PT parameter Q(P Q ,T Q ), where P Q and T Q Depends on the horizontal and vertical rotation values ​​of the actual turntable; Combined with the actual limit value, the parameter Q is normalized to M(X M ,Y M ), where 1≥X M ≥0, 1≥Y M ≥0; Record and save the corresponding relationship between P and M, and create calibration points; Repeat the above steps to obtain 10-20 scattered calibration points and save their data to the pre-annotated library.

7. A large-scene video command and tracking system as claimed in claim 5, characterized in that: The parameter calculation module specifically includes: The linkage target area is selected through automatic recognition of the panoramic video or manual selection by the user, and the center point of the target area is used as the target point B (X b ,Y b ); Select the 8 points with the closest Euclidean distance to the target point B from the calibration point group stored in the annotation module, denoted as P i (X i ,Y i )(0≤i≤7) and M i (X i ,Y i )(0≤i≤7); P i and M i The array is used through the findHomography method in OpenCV to calculate the homography matrix H based on the RANSAC algorithm; Through the homography matrix H, the target point B(X b ,Y b ) is transformed into the normalized coordinate M(X m ,Y m ); The actual PT parameters are restored based on the actual PT limit value of the ball camera; Calculate the Zoom value of the ball camera and obtain the PTZ parameters of the ball camera.

8. A large-scene video command and tracking system as claimed in claim 7, characterized in that: The calculation process of the ball camera Zoom value is as follows: First, in the panoramic video screen, the target FOV2 is approximately calculated by the panoramic field of view FOV and the percentage of the target area occupying the panoramic width, and FOV2 is used as the field of view value that the ball camera should have after the Zoom is changed. The calculation formula of FOV2 is as follows: Then calculate the focal length F2 that the ball camera should zoom to based on the target field of view FOV2 after zooming. The focal length F2 is calculated according to the following formula: Where S is the size of the photosensitive element; Finally, adjust the Zoom value of the ball camera to the focal length F2 to obtain the PTZ parameters of the ball camera.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images