Cooperative target positioning system and method under condition of unknown dual-camera center position

Abstract

The application discloses a cooperative target positioning system and method under the condition that the center positions of double cameras are unknown, and comprises the following steps: S1: the first camera is at the origin of the coordinate system, and the center position of the PTZ camera is unknown; S2: when the target appears in the first camera and the PTZ simultaneously, the first camera records the target coordinate, records the horizontal and vertical angles that the PTZ needs to rotate at this time, and establishes a contradictory equation set by using the coordinate geometric relation of the first camera and the PTZ; S3: the unknown position is obtained by solving the contradictory equation set by using the least square equation method, and the unknown position is optimized by using the gradient optimization; S4: when the first camera detects the target, the 3D coordinate of the target is obtained, and the horizontal and vertical rotation angles of the PTZ camera are calculated by using the contradictory equation set. The application realizes the cooperative target positioning of the two cameras under the condition that the coordinate relation between the center of the 3D camera and the center of the PTZ camera is unknown, and provides a new cooperative thought for the cooperative target positioning of the eagle-eye system.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a collaborative target localization system and method for situations where the center position of two cameras is unknown. Background Technology

[0002] Reference (Sun Zhuojin, 2012) describes the successful construction of a dual-camera automatic collaborative multi-target eagle-eye observation system in a PC environment. The main method used in this dual-camera collaborative system is the lookup table method. After fixing the relative positions of the wide-angle camera and the PTZ camera, the key monitoring area, the type of observed target, and the corresponding region of interest for the wide-angle camera are set. The wide-angle camera image is divided into a grid, and the PTZ camera is controlled to rotate and align with the grid intersections. The position M(x, y) of the grid intersection and the corresponding rotation angle (a, B) of the PTZ camera are recorded. Interpolation is performed at non-grid intersections to establish a lookup table containing the relationship between any position M(x, y) in the wide-angle camera image and the corresponding rotation angle (a, B) of the PTZ camera. This establishes a mapping relationship between any position M(x, y) in the wide-angle camera image area and the rotation angle (a, B). Through calibration, the key monitoring area in the wide-angle camera image is obtained by aligning the PTZ camera with the wide-angle camera image.

[0003] Ling Jiayao et al. proposed a panoramic camera and PTZ camera linkage monitoring system. This system achieves large-scale monitoring and detailed observation and analysis of designated areas by establishing a unified position association model. In this system, the panoramic camera captures the scene with its wide field of view, while the PTZ camera utilizes its automatic zoom and long-distance observation capabilities to conduct detailed monitoring of specific areas. To achieve effective linkage between the two cameras, a position association model is proposed. This model can accurately transform the target position in the panoramic image to the PTZ camera image, thus providing a foundation for accurate tracking and analysis of monitored targets. The model defines two coordinate systems: the panoramic camera coordinate system and the PTZ camera coordinate system. The distance between the camera centers of the two cameras is projected onto the X, Y, and Z coordinate axes as Lx, Ly, and Lz, respectively. Assuming there is a point P in the panoramic camera coordinate system, the point in the PTZ coordinate system is obtained through camera translation and rotation. The corresponding horizontal and pitch coordinates are calculated using specific formulas.

[0004] Reference (Bai Yang, 2020) describes the design of a target detection and recognition system based on dual-camera collaborative operation. For dual-camera calibration, the system uses Zhang's calibration method to calculate the internal parameters of the cameras. After the calibration of a single camera, dual-camera localization is performed on the images acquired by both cameras, thus completing the system calibration. The main focus is on constructing the system's world coordinate system. A certain coordinate transformation relationship is established between the coordinates of the two cameras themselves, the camera image coordinates, and the world coordinate system. This allows the camera image coordinates to be transformed into the world coordinate system. Based on the detected target's position in the world coordinate system, the telephoto camera can be rotated to the corresponding position to align with the target. The system's dual-camera collaborative strategy employs a combination of telephoto and short-focus cameras. The short-focus camera remains stationary, acquiring image data from the entire large field of view for target detection, while the telephoto camera, after detecting a target, moves under the control of a turntable to align with the target area and acquire high-resolution target image data for target recognition.

[0005] The system proposed by Sun Zhuojin uses a wide-angle camera and a PTZ camera for collaborative localization, employing a lookup table method for calibration. During calibration, the dual-camera calibration relies on a pre-made calibration table containing camera parameters calculated under specific distances and conditions. When the target distance exceeds the range covered by the calibration table, the system may not provide accurate parameters and may default to using the parameters of the nearest point in the table, even if the actual distance of this point does not match the target distance. Calibration tables often only cover a limited range of distances. When the target exceeds this range, the lookup table method can only use the parameters already present in the table, even if these parameters are not optimal.

[0006] The position association model proposed by Ling Jiayao et al. includes translation and rotation models between cameras. However, in practice, since both cameras have outer shells, it is difficult to accurately obtain the camera coordinate center and the rotation angle between the cameras. Therefore, this algorithm will inevitably have errors in practical applications, and these errors increase as the distance between the target and the camera increases.

[0007] The drawback of algorithms such as Bai Yang's is that after the telephoto and short-focus cameras are calibrated using Zhang Zhengyou's algorithm, the telephoto camera undergoes a horizontal and vertical rotation. This rotation can cause the calibration to fail, thus affecting the subsequent dual-camera collaboration to some extent. Summary of the Invention

[0008] In practical applications, due to factors such as camera housing encapsulation and mounting structure obstruction, the optical center positions of the two cameras are often difficult to measure directly or obtain accurately, making it impossible to directly use the aforementioned geometric relationships for accurate angle calculations. If the relative positions of the two cameras can be accurately obtained, then by combining the 3D coordinates of the target of interest obtained by the 3D camera with the precise relative positional relationship between the two cameras, the required horizontal and vertical rotation angles for PTZ can be calculated.

[0009] The main problem this invention addresses is the collaborative target localization of two cameras when the positions of the 3D camera and the PTZ sensor are unknown.

[0010] The cooperative target localization method for situations where the center position of two cameras is unknown, proposed in this invention, includes the following steps:

[0011] S1: Establish a coordinate system. The first camera is located at the origin of the coordinate system. The center position of the PTZ camera, which has horizontal and vertical rotation and target scaling functions, is unknown. Define the standard position of the PTZ camera;

[0012] It should be noted that the first camera can be a 3D camera with a wide field of view (or a combination of multiple 3D cameras with a narrow field of view), and the PTZ camera has horizontal and vertical rotation and target scaling functions.

[0013] S2: When a target appears simultaneously in both the first camera and the PTZ camera, the first camera records the target's 3D coordinates. And record the horizontal angle that the PTZ camera needs to rotate relative to the standard position when it rotates from the standard position to the point where the target is in the center of the PTZ camera's field of view. and vertical angle And establish the following contradictory system of equations:

[0014] ;

[0015] S3: Move the target and repeat step S2. Second-rate, For positive integers greater than 1, we can construct A system of contradictory equations with three unknowns is solved using the least squares method to obtain the calculated unknown positions. The calculated unknown position is used as the initial value for gradient optimization, and the coordinates of the PTZ camera are optimized using gradient optimization. ;

[0016] S4: When the target moves to position j, utilize... Combined with the target's 3D coordinates detected by the first camera The required horizontal and vertical rotation angles of the PTZ camera relative to the standard position can be calculated using the equations in step S2. and .

[0017] It should be noted that the standard position of the PTZ camera is when the shooting center of the PTZ camera is parallel to the positive Z-axis of the coordinate system, and the horizontal rotation angle α=0° and the vertical rotation angle β=0° of the PTZ camera are used as the reference for subsequent angle measurements.

[0018] Preferably, optimizing the coordinates of the PTZ camera using the gradient optimization method in S3 includes:

[0019] Collect the theoretically calculated angles of the PTZ camera based on the equations in step S2, as well as the observation angles rotated by the PTZ camera in step S2, and optimize the coordinate parameters of the PTZ camera using the gradient descent method. The optimization objective is to minimize the mixed loss function, which is the angle matching loss. :

[0020] Angle error: Calculate the difference between the theoretical angle and the actual observed angle under the current PTZ coordinate parameters. For each sample point, calculate the difference between the theoretically expected camera angle and the actual recorded angle under the current PTZ position parameters.

[0021] Angle matching loss The calculation is as follows:

[0022]

[0023] In the formula: The theoretically calculated horizontal angle for the i-th group, in degrees.

[0024] The actual observed horizontal angle of the i-th group, in degrees;

[0025] The vertical angle calculated theoretically for the i-th group is expressed in degrees.

[0026] The actual observed vertical angle of the i-th group, in degrees.

[0027] Minimize the hybrid loss function = angle matching loss + distance constraint loss;

[0028] The gradient calculation uses a numerical differentiation method to minimize the mixed loss function with respect to three location parameters. The gradient of the parameter x is calculated by perturbing the parameter x by a small amount Δx and observing the change in loss ΔLoss. The gradient component is calculated as ΔLoss / Δx. Similarly, the gradient components of the parameters y and z are calculated.

[0029] If the parameter change between consecutive iterations is less than the threshold (the threshold is 1mm), then the condition is considered to have converged and the optimization is complete. After the optimization is completed, the calibration parameters are updated, and the original coordinate parameters of the PTZ camera are replaced with the optimized new position parameters.

[0030] Specifically, in step S2: the first camera deploys a YOLOv8 model for face detection and combines depth information to obtain the three-dimensional spatial coordinates of the target face. Simultaneously, it uses the ArcFace model to extract the feature vector of the face. The PTZ camera also uses the YOLOv8 model for face detection and drives the pan-tilt unit to rotate via closed-loop control, aligning the target with the image center. Then, its ArcFace features are extracted. The similarity between the features extracted by the first camera and the features extracted by the PTZ camera after alignment is calculated. If the similarity exceeds a preset threshold, the targets in the two fields of view are determined to be the same object. At this time, the three-dimensional coordinates of the target and the horizontal and vertical rotation angles of the PTZ camera relative to the reference position are recorded simultaneously. Specifically, the PTZ camera loads an ArcFace face detection model to identify and locate facial targets in the video stream. It calculates the Euclidean distance from the center of the face bounding box to the image center. When the target deviates from the center by more than a threshold, the motion time is dynamically calculated based on the offset; the larger the offset, the longer the motion time. Relative movement is used to control the PTZ camera, moving it horizontally and vertically according to the offset to gradually bring the target closer to the image center. When the target stabilizes in the center region of the image, the PTZ camera is considered to be aligned with the target face. The current horizontal pan and vertical tilt states of the PTZ camera are read and converted into angle values. = pan × 180° = 45°×(1 - tilt), thus recording the horizontal angle that the PTZ camera needs to rotate from the standard position to when the target is at the center of the PTZ field of view. and vertical angle .

[0031] Preferably, the threshold for the target's deviation from the center is: ±20 pixels horizontally and ±10 pixels vertically.

[0032] Specifically, the first camera is a 3D camera.

[0033] A collaborative target localization system for situations where the center position of two cameras is unknown, used to implement the above method, includes a first camera, a PTZ camera, a processor, and a memory. The processor is connected to the first camera and the PTZ camera and receives data detected by the first camera and the PTZ camera. The processor is also used to control the rotation of the PTZ camera.

[0034] The proposed collaborative target localization system and method for situations where the center position of two cameras is unknown solves the error problem existing in the current technology when PTZ cameras and 3D cameras are used for collaborative localization. When directly using 3D cameras and PTZ cameras in collaboration, it is impossible to accurately obtain the positional relationship between the two due to the existence of shells and other reasons for both cameras. However, this invention calculates the center coordinates of the PTZ camera based on the 3D camera, thereby improving the detection accuracy.

[0035] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0036] Figure 1 This is a schematic diagram illustrating the relationship between a 3D camera and PTZ coordinates according to an embodiment of the present invention;

[0037] Figure 2 This is a flowchart of the method of the present invention;

[0038] Figure 3 Infographics captured by a 3D camera;

[0039] Figure 4 Information captured by a PTZ camera. Detailed Implementation

[0040] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0041] This method uses a 3D camera for pedestrian detection and 3D localization, then converts the relevant location information into coordinates for the PTZ camera, controls the PTZ camera's pan-tilt-zoom (PTZ) unit to rotate and zoom using relevant algorithms, and finally magnifies and identifies the target of interest.

[0042] The Orbbec Astra Pro depth camera was used as the 3D camera, enabling real-time monitoring of the experimental scene and acquisition of the target's 3D coordinates and depth information. The Astra Pro offers a wide field of view of 58.4° horizontally and 45.5° vertically, allowing the camera to cover a large scene. The Astra Pro's working range is 0.6-8 meters, providing a suitable focal length for medium to long-distance applications. However, it may be less flexible for extremely long or short-distance applications. Furthermore, the Astra Pro uses a monocular structured light design, making it more suitable for indoor use.

[0043] The PTZ camera in this embodiment is a Hikvision PTZ camera. This patent uses a Hikvision 4MP wireless DS-2DC2402IW-D3 / W network camera, which can provide a wide field of view and track and capture targets of interest through zoom function. The horizontal monitoring range is 0-350° and the vertical monitoring range is 0-90°, achieving all-round monitoring coverage.

[0044] Since the 3D camera's ranging capability is limited to within 8 meters indoors, both the 3D camera and the PTZ sensor use face detection algorithms (YOLOv8 face detection; if there are multiple people in the scene, it is recommended to use the ArcFace algorithm for face comparison and recognition between the 3D camera and the PTZ sensor to ensure that they are co-locating the same target). Because both cameras use face detection algorithms, adaptive collaborative target localization is possible.

[0045] like Figure 1 - Figure 2 The cooperative target localization method shown in the case of unknown dual-camera center positions includes the following steps:

[0046] like Figure 1 As shown, S1: Establish a coordinate system. The first camera is located at point 0 of the coordinate system. The first camera is a 3D camera. It also includes a PTZ camera with horizontal and vertical rotation and target scaling functions. The center position of the PTZ camera is unknown. Define the standard position of the PTZ camera. The standard position of the PTZ camera is the orientation of the shooting center of the PTZ camera to be parallel to the positive Z-axis of the coordinate system. The horizontal rotation angle α of the PTZ camera is 0° and the vertical rotation angle β is 0°, which serve as the reference for subsequent angle measurements.

[0047] S2: When the target appears simultaneously on both the first camera and the PTZ camera, as shown in... Figure 3 As shown, the first camera records the 3D coordinates of the target. And record the horizontal angle that the PTZ camera needs to rotate relative to the standard position when it rotates from the standard position to the point where the target is in the center of the PTZ camera's field of view. and vertical angle And establish the following system of equations:

[0048] ;

[0049] Specifically: The first camera loads a YOLOv8 face detection model to obtain target coordinates, and the PTZ camera loads an ArcFace face detection model to identify and locate face targets in the video stream. It calculates the Euclidean distance from the center of the face bounding box to the image center. When the target deviates from the center by more than a threshold, the motion time is dynamically calculated based on the offset; the larger the offset, the longer the motion time. Relative movement is used to control the PTZ camera, controlling its relative movement in the horizontal and vertical directions according to the offset, gradually bringing the target closer to the image center. When the target stabilizes in the image center area, it is considered that the PTZ camera has aligned with the target face. The current horizontal and vertical rotation angles (pan and tilt) of the PTZ camera are read, and the rotation data is converted into angle values. = pan × 180° = 45° × (1 - tilt), thus recording the horizontal angle that the PTZ camera needs to rotate from the standard position to when the target is at the center of the PTZ field of view. and vertical angle ;

[0050] Preferably, the threshold for the target's deviation from the center is: ±20 pixels horizontally and ±10 pixels vertically.

[0051] Specifically, the face detection model can be the YOLOv8 face detection model or the ArcFace model.

[0052] S3: Move the target and repeat step S2. Second-rate, M is a positive integer greater than 1; in this embodiment, M is 8, which can be used to construct... A system of contradictory equations with three unknowns is solved using the least squares method to obtain the calculated unknown positions. The unknown position of the PTZ camera is optimized using the gradient optimization method. ;

[0053] Collect the theoretically calculated angles of the PTZ camera based on the equations in step S2, as well as the observation angles rotated by the PTZ camera in step S2, and optimize the coordinate parameters of the PTZ camera using the gradient descent method. The optimization objective is to minimize the mixed loss function:

[0054] Minimize the mixture loss function = angle matching loss;

[0055] Angle matching loss The calculation is as follows:

[0056]

[0057] In the formula: Calculate the horizontal angle for the i-th theoretical group, in degrees.

[0058] The actual observed horizontal angle of the i-th group, in degrees;

[0059] Calculate the vertical angle for the i-th theoretical group, in degrees.

[0060] The actual observed vertical angle for the i-th group is expressed in degrees.

[0061] Specifically, calculation based on the contradictory function set and ;

[0062]

[0063] The loss for a single sample of the i-th sample is: ;

[0064] The angle matching loss is the average of all samples: ;

[0065] Then, gradient calculation is performed: a small perturbation Δx is applied to the parameter x;

[0066] Calculate the loss L before the disturbance old ;calculate Keeping y and z constant, recalculate. and L new ;

[0067] Gradient components: ;

[0068] Similarly, calculate: ;

[0069] Step 5: Parameter Update

[0070]

[0071] Where η is the learning rate, which determines the step size of the parameters along the gradient direction in each iteration. It is a positive number that needs to be preset or dynamically adjusted (e.g., 0.001 or 0.01) to control the step size of the PTZ camera coordinates in each iteration. The correction range, this threshold can be adjusted manually, and can be flexibly adjusted according to actual application scenarios and other conditions.

[0072] Then, convergence iteration is performed. If the parameter change between consecutive iterations is less than the threshold of 1mm, the condition is considered to have converged and the optimization is complete. After optimization, the calibration parameters are updated, and the original coordinate parameters of the PTZ camera are replaced with the optimized new position parameters.

[0073] S4: When the target moves to position j, use the optimized coordinates of the PTZ camera combined with the target's 3D coordinates detected by the first camera. The required horizontal and vertical rotation angles of the PTZ camera relative to the standard position can be calculated using the equations in step S2. and The display of the PTZ camera is as follows Figure 4 As shown;

[0074] Specifically:

[0075] ;

[0076] ;

[0077] Then calculate the required horizontal and vertical rotation angles of the PTZ camera. and .

[0078] A collaborative target localization system for situations where the center position of two cameras is unknown, used to implement the above method, includes a first camera, a PTZ camera, a processor, and a memory. The processor is connected to the first camera and the PTZ camera and receives data detected by the first camera and the PTZ camera. The processor is also used to control the rotation of the PTZ camera.

[0079] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A cooperative target localization method for situations where the center positions of two cameras are unknown, characterized in that, Includes the following steps: S1: Establish a coordinate system. The first camera is located at the origin of the coordinate system. The center position of the PTZ camera, which has horizontal and vertical rotation and target scaling functions, is unknown. Define the standard position of the PTZ camera; S2: When a target appears simultaneously in both the first camera and the PTZ camera, the first camera records the target's 3D coordinates. And record the horizontal angle that the PTZ camera needs to rotate relative to the standard position when it rotates from the standard position to the point where the target is in the center of the PTZ camera's field of view. and vertical angle And establish the following contradictory system of equations: ； S3: Move the target and repeat step S2. Second-rate, For positive integers greater than 1, we can construct A system of contradictory equations with three unknowns is solved using the least squares method to obtain the calculated unknown positions. The calculated unknown position is used as the initial value for gradient descent, which allows for a more accurate estimation of the unknown position of the PTZ camera. ; S4: When the target moves to At that time, the unknown position of the PTZ camera is combined with the target's 3D coordinates detected by the first camera. The required horizontal and vertical rotation angles of the PTZ camera relative to the standard position can be calculated using the equations in step S2. and .

2. The cooperative target localization method for situations where the center position of two cameras is unknown, as described in claim 1, is characterized in that... Gradient descent includes the following steps: Collect the theoretically calculated angles of the PTZ camera based on the equations in step S2, as well as the observation angles rotated by the PTZ camera in step S2, and optimize the unknown position of the PTZ camera using the gradient descent method. The optimization objective is to minimize the mixed loss function: Minimize the mixture loss function = angle matching loss; Angle matching loss The calculation is as follows: In the formula: The theoretically calculated horizontal angle for the i-th group, in degrees. The actual observed horizontal angle of the i-th group, in degrees. The vertical angle calculated theoretically for the i-th group is expressed in degrees. The actual observed vertical angle of the i-th group, in degrees. N is a positive integer; The gradient calculation uses a numerical differentiation method to minimize the mixed loss function with respect to three location parameters. The gradient of the parameter x is calculated by perturbing the parameter x by a small amount Δx and observing the change in loss ΔLoss. The gradient component is calculated as ΔLoss / Δx. Similarly, the gradient components of the parameters y and z are calculated. If the parameter change between consecutive iterations is less than the threshold during the iteration process, the condition is considered to have converged and the optimization is complete. After the optimization is completed, the calibration parameters are updated, and the original coordinate parameters of the PTZ camera are replaced with the optimized new position parameters.

3. The cooperative target localization method for situations where the center position of two cameras is unknown, as described in claim 2, is characterized in that... Calculate based on the contradictory equations in step S2 and ; The loss for a single sample of the i-th sample is: ; The angle matching loss is the average of all samples: .

4. The cooperative target localization method for situations where the center position of two cameras is unknown, as described in claim 1, is characterized in that... The standard position of the PTZ camera is when the shooting center of the PTZ camera is aligned with the first camera in the positive Z-axis direction of the coordinate system.

5. The cooperative target localization method for situations where the center position of two cameras is unknown, as described in claim 1, is characterized in that... In step S2: The first camera deploys a YOLOv8 model to perform face detection and combines depth information to obtain the three-dimensional spatial coordinates of the target face. At the same time, it uses the ArcFace model to extract the feature vector of the face. The PTZ camera also uses the YOLOv8 model to perform face detection and drives the pan-tilt unit to rotate through closed-loop control so that the PTZ camera is aligned with the center of the image. Then, its ArcFace features are extracted. The similarity between the features extracted by the first camera and the features extracted by the PTZ camera after alignment is calculated. If the similarity exceeds a preset threshold, it is determined that the target in the two fields of view is the same object. At this time, the three-dimensional coordinates of the target and the horizontal and vertical rotation angles of the PTZ camera relative to the reference position are recorded simultaneously.

6. The cooperative target localization method for situations where the center position of two cameras is unknown, as described in claim 5, is characterized in that... The threshold for the target's deviation from the center is: ±20 pixels horizontally and ±10 pixels vertically.

7. The cooperative target localization method for situations where the center position of two cameras is unknown, as described in claim 1, is characterized in that... The first camera is a 3D camera.

8. A cooperative target localization system for situations where the center position of two cameras is unknown, characterized in that, It includes a first camera, a PTZ camera, a processor, and a memory. The processor is connected to the first camera and the PTZ camera and receives data detected by the first camera and the PTZ camera. The processor is also used to control the rotation of the PTZ camera.

Images