Target tracking system

Abstract

The invention belongs to the field of dynamic target tracking, and particularly relates to a target tracking system. The device comprises a processing module which is used for matching a tracking object selected from a panoramic video stream image with each target identified in a close-up video stream image, and selecting the target matched with the tracking object as the target of the same object corresponding to the tracking object, target tracking is carried out on the tracking object and the matching target in the panoramic video stream and the close-up video stream respectively; when the target is tracked, according to the tracking result and the position of the acquisition device corresponding to the close-up video stream, obtaining positioning information of a tracked object in the current frame of the panoramic video, and adjusting the view of the acquisition device according to the positioning information; therefore, detailed and visual details can be provided through synchronous tracking of the close-up video; and the view of the close-up video can be adjusted according to the wide-range positioning of the tracking object by the panoramic video, so that the detailed and intuitive detail features and the wide tracking range of the tracking object are considered.

Description

Technical Field

[0001] The invention belongs to the field of dynamic target tracking, and in particular relates to a target tracking system. Background Art

[0002] At present, with the development of artificial intelligence technology and imaging technology, target detection technology based on artificial intelligence has been more and more widely used. Existing transmission technology and image fusion processing technology can already support drones to receive videos and analyze them frame by frame to achieve target tracking.

[0003] For example, the Chinese invention patent application with application publication number CN106485736A discloses a method for panoramic visual tracking of unmanned aerial vehicles, an unmanned aerial vehicle, and a control terminal, which obtains images taken by multiple cameras at the same time point; stitches the images taken by the multiple cameras at the same time point to form a panoramic image; and transmits the panoramic image stitched each time to a control terminal wirelessly connected to the unmanned aerial vehicle. The method for panoramic visual tracking of unmanned aerial vehicles, an unmanned aerial vehicle, and a control terminal provided by the present application use multiple cameras to obtain multiple camera images and then stitch them to form a panoramic image and transmit them to the tracking terminal for use. Not only can a 360-degree panoramic image be obtained, but also a panoramic imaging of multiple cameras and the fusion of multiple camera image data can be realized, and based on the panoramic image, full-view target tracking of the unmanned aerial vehicle can be realized.

[0004] However, when tracking a target with a panoramic image, although the viewing angle is wide and the field of view is large, it is convenient to observe the tracked target in a large range, but for the tracked target, more detailed features will be lost visually, and the image of the target has the problem of image distortion that is difficult to avoid, which makes it difficult to accurately obtain the detailed features of the tracked target, which is not conducive to intuitive observation; but the target tracking method based on close-up image shooting with an ordinary lens can observe a small range. When tracking a target with a fast moving speed or a large moving range, it is easy for the tracked object to exceed the shooting range of the ordinary lens, resulting in the disappearance of the target, and once the target exceeds the shooting range of the ordinary lens, it is more difficult to track the target again. It can be seen that the target tracking method in the prior art is difficult to take into account both the more detailed and intuitive detailed features of the tracked target and the larger tracking range. Summary of the invention

[0005] The object of the present invention is to provide a target tracking system for solving the problem that the target tracking schemes in the prior art are difficult to take into account both the more detailed and intuitive detail features of the tracked target and the larger tracking range.

[0006] In order to achieve the above object, the present invention provides a target tracking system, including a processing module, for:

[0007] Matching the tracked object selected in the panoramic video stream image with each target identified in the close-up video stream image, selecting the target matched with the tracked object as the target corresponding to the same object as the tracked object, and performing target tracking on the tracked object and the matched target in the panoramic video stream and the close-up video stream respectively;

[0008] When tracking a target, the positioning information of the tracked object is obtained in the current frame of the panoramic video based on the tracking result and the position of the acquisition device corresponding to the close-up video stream, and the field of view of the acquisition device is adjusted using the positioning information so that the close-up video stream acquired by the acquisition device contains the matching target.

[0009] Further, according to the tracking result and the position of the acquisition device corresponding to the close-up video stream, the method of obtaining the positioning information of the tracked object in the current frame of the panoramic video includes:

[0010] According to the tracking result of the target tracking of the tracked object in the panoramic video stream, the image pixel coordinates corresponding to the tracked object in the current frame of the panoramic video are determined as the positioning information of the tracked object.

[0011] Furthermore, the acquisition device is also used to be carried on a drone; the method of adjusting the field of view of the acquisition device corresponding to the close-up video stream using the positioning information includes:

[0012] Calculating the rotation angle of the lens of the acquisition device mounted on the drone using the positioning information of the tracked object, and sending a corresponding instruction to control the lens of the acquisition device to adjust according to the rotation angle;

[0013] The method of calculating the rotation angle of the lens of the acquisition device by using the positioning information of the tracked object includes:

[0014] 1) According to the attitude information of the UAV, determine the rotation matrix corresponding to the attitude of the UAV; the rotation matrix includes a yaw matrix, a pitch matrix and a roll matrix;

[0015] 2) according to the internal parameters of the lens of the acquisition device, the image pixel coordinates corresponding to the tracked object are converted into three-dimensional space coordinates in the lens coordinate system of the acquisition device; and then according to the rotation matrix obtained in step 1), the three-dimensional space coordinates are converted into three-dimensional coordinates of the image pixel in the local coordinate system of the drone;

[0016] 3) Convert the current GPS position of the drone into the ECEF coordinates of the current position of the drone; rotate the three-dimensional coordinates of the image pixel in the local coordinate system of the drone according to the current posture of the drone so as to align it with the earth reference coordinate system to obtain the rotated coordinates; then add the rotated coordinates to the ECEF coordinates of the current position of the drone to obtain the ECEF coordinates of the image pixel as the ECEF coordinates of the object corresponding to the tracked object;

[0017] 4) Obtain a relative position vector according to the ECEF coordinates of the lens of the acquisition device determined by the ECEF coordinates of the current position of the drone and the ECEF coordinates of the object corresponding to the tracked object; transform the relative position vector according to the rotation matrix in step 1) to obtain a vector representation of the relative position vector in the local coordinate system of the drone; obtain the azimuth and pitch angles of the object corresponding to the tracked object relative to the drone according to the vector representation, and obtain the rotation angle of the lens of the acquisition device according to the azimuth and pitch angle.

[0018] Furthermore, the method of selecting a target matching the tracked object in the close-up video stream includes:

[0019] Determine the target features corresponding to the tracked object and each target identified in the close-up video stream image; compare the target features corresponding to the tracked object with the target features corresponding to each target identified in the close-up video stream image, and select the target whose target features among the targets identified in the close-up video stream image have the greatest similarity with the target features corresponding to the tracked object as the target that matches the tracked object;

[0020] The target features corresponding to the tracked object / targets are obtained based on the pixel features, motion features and position features of the tracked object / targets in the corresponding video stream images; during the target tracking process, the pixel features of the tracked object / targets include color, texture and shape information, the motion features of the tracked object / targets include their movement trajectories in continuous video frames, which are determined by the optical flow method; the position features of the tracked object / targets in the corresponding video stream images include their specific positions and sizes in the video frames, which are determined by the corresponding deep learning model.

[0021] Furthermore, the method of performing target tracking on the tracking object and the matching target in the panoramic video stream and the close-up video stream respectively includes:

[0022] Specially marking the tracked object in the panoramic video stream, and performing the same special marking on the matching target in the close-up video stream for synchronous display;

[0023] The DeepSORT algorithm is used to perform data association and feature extraction on the tracked object and the matching target in the panoramic video stream and the close-up video stream respectively, and the position and motion trajectory of the tracked object and the matching target are obtained in real time; the position includes the corresponding image pixel coordinates.

[0024] Furthermore, it also includes a collection device; the collection device includes collection devices corresponding to the panoramic video stream and the close-up video stream, which are respectively used to collect the corresponding panoramic video stream and the close-up video stream, and transmit them to the processing module.

[0025] Further, the acquisition device corresponding to the close-up video stream includes a visible light camera and an infrared camera, and the close-up video stream includes a visible light video stream and an infrared video stream;

[0026] The visible light camera and the infrared camera are mounted at the same position of the same UAV via the same optoelectronic pod;

[0027] The processing module sends corresponding instructions to control the lens of the acquisition device to adjust according to the rotation angle, including: sending corresponding control instructions to the optoelectronic pod according to the rotation angle, controlling the optoelectronic pod to rotate the corresponding angle, so that the lenses of the visible light camera and the infrared camera carried by the optoelectronic pod are pointed to the area where the object corresponding to the tracked object is currently located.

[0028] Furthermore, it also includes a human-computer interaction module for displaying the panoramic video stream and the close-up video stream;

[0029] The processing module is also used to perform target detection in the panoramic video stream, and assign corresponding different identifiers to each target detected and send them to the human-computer interaction module; the human-computer interaction module is also used to display the different identifiers for relevant personnel to select the corresponding identifier, and return the selection result to the processing module; the processing module is also used to select the corresponding tracking object in the panoramic video stream image through the corresponding identifier selected by the relevant personnel.

[0030] Furthermore, the processing module is also used to convert the projection mode of the panoramic video stream image displayed by the human-computer interaction module between an equirectangular projection mode and a cubic projection mode;

[0031] The human-computer interaction module is also used to display options of different projection modes for relevant personnel to choose from, and return the selection results to the processing module;

[0032] The processing module is also used to convert the panoramic video stream image displayed by the human-computer interaction module into views in six directions when the received selection result is a cubic projection mode; and to convert the panoramic video stream image displayed by the human-computer interaction module into an overall view when the received selection result is an equirectangular projection mode.

[0033] Furthermore, the processing module is also used to track the GPS position of the object; the GPS position is calculated in the following ways:

[0034] According to the conversion formula from ECEF coordinates to LLA coordinates, the ECEF coordinates of the object corresponding to the tracked object are converted into the LLA coordinates of the object corresponding to the tracked object, and the GPS position of the tracked object is determined according to the LLA coordinates of the object corresponding to the tracked object.

[0035] The above technical solution of the present invention provides an improved target tracking system based on the existing panoramic video target tracking and close-up video target tracking technologies, and its beneficial effects include:

[0036] The panoramic video stream and the close-up video stream are linked together through the processing module, which not only utilizes the larger field of view of the panoramic video to facilitate the selection and tracking of the corresponding tracking object from the large range, but also can synchronously track the tracking object in the close-up video stream to compensate for the defects of image distortion and lack of details in the panoramic video, which is convenient for intuitive observation; and, the tracking object can be located within the large field of view of the panoramic video, which serves as a basis for adjusting the field of view of the close-up video. When the field of view adjustment is completed, the problem that the position of the tracking object may exceed the shooting range of the close-up video can be solved, thereby realizing continuous tracking of the close-up video within a large range, compensating for the defects of the close-up video having a small field of view and difficulty in continuously tracking targets with a fast moving speed or a large moving range, thereby taking into account both the more detailed and intuitive detail features of the tracking target and the larger tracking range. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flowchart of the processing module in the target tracking system embodiment of the present invention;

[0038] Figure 2 It is a schematic diagram of the principle of the target tracking system in the embodiment of the target tracking system of the present invention;

[0039] Figure 3 This is an example diagram of the display effect of the human-computer interaction module in the target tracking system embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0041] Target Tracking System Embodiment

[0042] This embodiment provides a technical solution for a target tracking system. Figure 1 , including processing modules; see Figure 2 , the processing module is used to:

[0043] The tracking object selected in the panoramic video stream image is matched with each target identified in the close-up video stream image, and the target matching the tracking object is selected in the close-up video stream as the target corresponding to the same object as the tracking object, and the tracking object and the target matching the tracking object are tracked in the panoramic video stream and the close-up video stream respectively. Specifically, taking the matching of a single tracking object as an example, the tracking object in the panoramic video stream and the target matching the tracking object selected in the close-up video stream correspond to the same object in the actual scene, and the method of selecting the target matching the tracking object in the close-up video stream is the process of linking the panoramic video stream with the target to be tracked in the close-up video stream through the actual characteristics of the object; tracking the tracking object and the target matching the tracking object in the panoramic video stream and the close-up video stream respectively is equivalent to tracking the object in the panoramic video stream and the close-up video stream respectively.

[0044] When tracking a target, according to the tracking result and the position of the acquisition device corresponding to the close-up video stream, the positioning information of the tracked object is obtained in the current frame of the panoramic video, and the acquisition device corresponding to the close-up video stream is adjusted according to the positioning information so that the close-up video stream acquired by the acquisition device corresponding to the close-up video stream contains the target that matches the tracked object. The close-up video stream refers to the corresponding video stream obtained by shooting with a single camera, which has a smaller field of view and clearer details.

[0045] Therefore, the panoramic video stream and the close-up video stream are linked together through the processing module, which not only utilizes the larger field of view of the panoramic video to facilitate the selection and tracking of the corresponding tracking object from the large range, but also can synchronously track the tracking object in the close-up video stream to make up for the defects of image distortion and lack of details in the panoramic video, which is convenient for intuitive observation; and, the location of the tracking object can be found within the large field of view of the panoramic video, which serves as a basis for adjusting the field of view of the close-up video. When the field of view adjustment is completed, the problem that the position of the tracking object may exceed the shooting range of the close-up video can be solved, thereby realizing continuous tracking of the close-up video in a large range, making up for the defects of the close-up video with a small field of view and difficulty in continuously tracking targets with a fast moving speed or a large moving range, thereby taking into account both the more detailed and intuitive detail features of the tracking target and the larger tracking range.

[0046] To obtain panoramic video streams and close-up video streams in a timely manner, refer to Figure 2The target tracking system also includes a collection device; the collection device includes collection devices corresponding to the panoramic video stream and the close-up video stream, which are respectively used to collect the corresponding panoramic video stream and the close-up video stream, and transmit them to the processing module. In order to facilitate accurate and effective target recognition both during the day and at night, the collection device corresponding to the close-up video stream includes a visible light camera and an infrared camera; therefore, the close-up video stream includes a visible light video stream and an infrared video stream.

[0047] In order to make the position of the acquisition device more flexible and controllable and obtain a larger acquisition range, the acquisition device is also used to be mounted on a drone; wherein, each acquisition device (such as a fisheye camera) corresponding to the panoramic video stream is respectively mounted on each drone located at different positions to obtain images of various perspectives at the same time point, and the images are stitched together by the airborne computing module on the drone to obtain a foreground image with a 360° all-round perspective; specifically, the acquisition device corresponding to the panoramic video stream is a group of panoramic cameras, and a group of panoramic cameras is composed of 8 fisheye cameras. The airborne computing module on the drone obtains the videos of the 8 fisheye cameras at the same time point. The image data of these videos first undergo a preprocessing stage, including de-distortion and color correction, to ensure the quality of subsequent stitching; then, using an image stitching algorithm based on feature matching, the images taken by the 8 fisheye cameras at the same time point are stitched to form a panoramic image, and image transformation is supported; in this embodiment, the image stitching applies an image stitching algorithm based on feature point detection and matching, SIFT (scale-invariant feature transform), to identify and match the overlapping areas between the images; the image stitching algorithm can process and fuse images from multiple perspectives to generate a seamless and continuous panoramic image. In other embodiments, the splicing operation of the airborne computing modules on the above-mentioned drone may also be performed by the processing module.

[0048] In order to achieve synchronous adjustment of the field of view of the visible light camera and the infrared camera included in the acquisition device corresponding to the close-up video stream, and to achieve control of the field of view adjustment of the ordinary visible light camera and the infrared camera, the visible light camera and the infrared camera are mounted at the same position of the same drone through the same optoelectronic pod; the processing module can adjust the field of view of the acquisition device corresponding to the close-up video stream by controlling the selected angle of the optoelectronic pod. Therefore, the synchronous adjustment of the field of view of the visible light camera and the infrared camera can be achieved directly by controlling the angle of the optoelectronic pod, and there is no need to use a special camera with an automatic rotation angle function, and only an optoelectronic pod that can simultaneously install the visible light camera and the infrared camera is required.

[0049] In combination with the characteristics of panoramic video and close-up video, the above-mentioned matching of the tracked object selected in the panoramic video stream image with each target identified in the close-up video stream image, and the method of selecting the target matching the tracked object in the close-up video stream specifically includes:

[0050] Determine the target features corresponding to the tracked object selected in the panoramic video stream image and the targets identified in the close-up video stream image; compare the target features corresponding to the tracked object with the target features corresponding to the targets identified in the close-up video stream image, and take the target with the greatest similarity between the target features of the targets identified in the close-up video stream image and the target features corresponding to the tracked object as the target that matches the tracked object;

[0051] The target features corresponding to the tracked object / targets are obtained based on the pixel features, motion features and position features of the tracked object / targets in the corresponding video stream image. In the target tracking process, the pixel features of the target are extracted by image processing technology, including color, texture and shape information. The optical flow method is usually used to analyze the motion features of the target to estimate its movement trajectory in continuous video frames. In addition, it is necessary to locate the position features of the target and use a deep learning model to determine its specific position and size in the video frame. Finally, these extracted features are matched with known targets or targets in previous frames, and feature similarity scoring is used to achieve continuous tracking of the target. In this way, the accuracy of matching can be improved, ensuring that the tracked object in the panoramic video stream and the target selected in the close-up video stream that matches the tracked object correspond to the same object in the actual scene, thereby improving the accuracy of linking the panoramic video stream with the target to be tracked in the close-up video stream through the actual characteristics of the object.

[0052] In this embodiment, the tracking object can be one or more. The processing module first synchronously receives and preprocesses the video streams from various cameras, and then applies a target detection algorithm such as YOLO to detect and locate multiple targets in the video stream respectively. For multiple tracking objects, the target matching the tracking object is selected in the close-up video stream in accordance with the above-mentioned method, and each target matching each tracking object is selected, and a multi-target tracking algorithm is used to track them separately. Then, the method of tracking the tracking object and the target matching the tracking object in the panoramic video stream and the close-up video stream includes:

[0053] The tracked object is specially marked in the panoramic video stream, and the target matching the tracked object is specially marked in the close-up video stream for synchronous display; specifically, once the matching target is determined, the position of the target can be identified by the special mark, and the corresponding control command is issued to synchronously display the same special mark in the display windows of the panoramic video and the visible light video; the special mark can be a target frame of different colors, a graphic or text mark on / in the target frame, etc., which can be flexibly set according to specific needs.

[0054] The DeepSORT algorithm is used to perform data association and feature extraction on the tracked object and the target that matches the tracked object in the panoramic video stream and the close-up video stream, respectively, to obtain the position and motion trajectory of the tracked object and the target that matches the tracked object in real time; the positions of the tracked object and the target that matches the tracked object here include the image pixel coordinates corresponding to the tracked object and the target that matches the tracked object. Among them, the DeepSORT algorithm includes tracking using the Kalman filter and the Hungarian algorithm, and optimizing the appearance features using the CNN model. The algorithm continuously updates the state of each target, effectively handles the appearance and disappearance of the target, and optimizes the tracking results to reduce identity switching and omissions. Finally, the DeepSORT algorithm can output the position and motion trajectory of each target in real time.

[0055] On the basis of being able to implement target tracking of the tracked object and the target matching the tracked object in the panoramic video stream and the close-up video stream, respectively, in this embodiment, the method of obtaining the positioning information of the tracked object in the current frame of the panoramic video according to the tracking result and the position of the acquisition device corresponding to the close-up video stream includes:

[0056] According to the tracking result of the target tracking of the tracked object in the panoramic video stream, the image pixel coordinates corresponding to the tracked object in the current frame of the panoramic video are determined as the positioning information of the tracked object. Specifically, this embodiment extracts the pixel coordinates of the target object (i.e., the image pixel coordinates corresponding to the tracked object) from the image captured by the drone camera through image processing technology.

[0057] The acquisition device is also used to be carried on a drone; the method of using the positioning information to adjust the field of view of the acquisition device corresponding to the close-up video stream includes:

[0058] Calculate the rotation angle of the lens of the acquisition device mounted on the drone using the positioning information of the tracked object, and send corresponding instructions to control the lens of the acquisition device to adjust according to the rotation angle;

[0059] In this embodiment, the method of calculating the rotation angle of the lens of the acquisition device by using the positioning information of the tracked object includes:

[0060] 1) According to the attitude information of the UAV, determine the rotation matrix corresponding to the attitude of the UAV; the rotation matrix corresponding to the attitude of the UAV includes the yaw matrix, the pitch matrix and the roll matrix;

[0061] 2) According to the internal parameters of the lens of the acquisition device, the image pixel coordinates corresponding to the tracked object are converted into three-dimensional space coordinates in the lens coordinate system of the acquisition device; then according to the rotation matrix obtained in step 1), the three-dimensional space coordinates are converted into three-dimensional coordinates of the image pixels corresponding to the tracked object in the local coordinate system of the drone; this step is equivalent to mapping the image pixel coordinates to the physical coordinates, that is, by establishing a mapping relationship between the image pixel coordinates and the actual physical position, combined with the current position and posture information of the drone, the three-dimensional space position of the target object relative to the drone is calculated.

[0062] 3) Convert the current GPS position of the drone into the ECEF coordinates of the current position of the drone; rotate the three-dimensional coordinates of the image pixels corresponding to the tracked object in the local coordinate system of the drone according to the current posture of the drone, so that they are aligned with the earth reference coordinate system to obtain the rotated coordinates; then add the rotated coordinates to the ECEF coordinates of the current position of the drone to obtain the ECEF coordinates of the image pixels corresponding to the tracked object, which are used as the ECEF coordinates of the object corresponding to the tracked object; this step is equivalent to position coordinate conversion, that is, combining the current GPS position information of the drone, converting the obtained three-dimensional space position coordinates into the position in the ECEF coordinate system.

[0063] 4) Obtain a relative position vector according to the ECEF coordinates of the lens of the acquisition device determined by the ECEF coordinates of the current position of the UAV and the ECEF coordinates of the object corresponding to the tracked object; transform the relative position vector according to the rotation matrix in step 1) to obtain a vector representation of the relative position vector in the local coordinate system of the UAV; obtain the azimuth and pitch angles of the object corresponding to the tracked object relative to the UAV according to the vector representation, and obtain the rotation angle of the lens of the acquisition device according to the azimuth and pitch angle.

[0064] The method of using the positioning information to adjust the field of view of the acquisition device corresponding to the close-up video stream includes:

[0065] The positioning information of the tracked object is used to calculate the rotation angle of the lens of the acquisition device corresponding to the close-up video stream carried on the drone, and a corresponding instruction is sent to control the lens of the acquisition device corresponding to the close-up video stream to adjust according to the calculated rotation angle.

[0066] The processing module is also used to track the GPS position of the object; the GPS position is calculated in the following ways:

[0067] According to the conversion formula from ECEF coordinates to LLA coordinates, the ECEF coordinates of the object corresponding to the tracked object are converted into the LLA coordinates of the object corresponding to the tracked object, and the GPS position of the tracked object is determined according to the LLA coordinates of the object corresponding to the tracked object.

[0068] The following is a process for calculating the rotation angle of the lens of the acquisition device (which also includes the relative direction vector calculation for indicating the relative direction from the drone to the target point in the local coordinate system of the drone, and the GPS position calculation of the tracked object. The three parameters of the rotation angle of the lens of the acquisition device, the relative direction vector, and the GPS position of the tracked object can all be calculated and output by the processing module. In other embodiments, the calculation of at least one of the two parameters of the relative direction vector and the GPS position of the tracked object can be removed as needed):

[0069] Assumptions: The current GPS position of the drone is: longitude λ uav = -117.234°, latitude φ uav =32.123°, height h uav = 200 meters. The attitude information of the drone is: pitch angle θ = 5°, yaw angle ψ = 10°, roll angle φ = 2°. The pixel coordinates of the target object in the image (i.e. the image pixel coordinates corresponding to the tracked object) are: (x, y) = (1024, 768), with the upper left corner of the image as the origin, the right as the positive direction of the X axis, and the downward as the positive direction of the Y axis. The field of view (FOV) of the drone camera is: 90° horizontally and 60° vertically. Image resolution: 1920×1080 pixels.

[0070] ① Conversion from image pixel coordinates to sight angle: First, calculate the sight angle of the target object relative to the center line of the drone camera based on the image pixel coordinates of the target object and the field of view of the camera. Assume that the horizontal and vertical sight angles are α and α respectively. x and α y .

[0071] The method of determining the center pixel coordinates includes: the resolution of the image is W pixels in width and H pixels in height, and the pixel coordinates of the center of the image are (W / 2, H / 2). Calculate pixel offset: for a target point in the image, its pixel coordinates are (x, y), calculate the pixel offset of the point relative to the center of the image (Δx, Δy) = (xW / 2, yH / 2).

[0072] The method of calculating the sight angle includes: using the horizontal and vertical FOV of the camera, and calculating the sight angle based on the pixel offset. Horizontal sight angle α x It can be calculated by Δx and horizontal FOV. Assuming that the angle corresponding to each pixel in the horizontal direction is FOVx / W, then α x =Δx×(FOVx / W). Similarly, the vertical line of sight angle α x It can be calculated by Δy and vertical FOV. Assuming that the angle corresponding to each pixel in the vertical direction is FOVy / H, then α y =Δy×(FOVy / H).

[0073] The content of step ① belongs to the content of solving the relative direction vector. The content of this step is not required for solving the two parameters, namely, the rotation angle of the lens of the acquisition device and the GPS position of the tracked object.

[0074] ② From the sight angle to the target vector in the local coordinate system of the drone: Using the drone attitude information and the sight angle, the direction vector of the target object relative to the local coordinate system of the drone can be calculated. This requires combining the sight angle and the drone attitude angle (i.e., the drone attitude information) through a three-dimensional rotation transformation.

[0075] The sight angle obtained from image processing is the horizontal angle relative to the center line of the camera (α x ) and the vertical angle (α y ), converting these angles into a three-dimensional direction vector in the local coordinate system of the drone. This vector represents the direction from the drone to the target point. The details are as follows:

[0076] Initialization vector: First, initialize a direction vector based on the line of sight. In the camera coordinate system, it can be assumed that this vector points to α x and α y Defined direction, assuming the initial vector is where v x and v y can be calculated by the sight angle, and v z Typically set to 1.

[0077] Apply rotation: Next, apply a three-dimensional rotation based on the drone’s attitude angle. This involves constructing three rotation matrices: yaw (R yaw )、Pitch(R pitch ) and Roll (R roll ). Yaw rotation matrix R yaw Rotation around the Z axis, pitch rotation matrix R pitch Rotate around the Y axis, roll the rotation matrix R roll Rotate around the X axis.

[0078] Calculate the final direction vector: The final direction vector By setting the initial vector Multiply by the rotation matrix. The order of this multiplication depends on the order in which the rotations are applied, usually This final direction vector It represents the relative direction from the UAV to the target point in the UAV local coordinate system, that is, the relative direction vector.

[0079] The content of step ② also belongs to the content of solving the relative direction vector. The content of this step is not required for solving the two parameters, namely, the rotation angle of the lens of the acquisition device and the GPS position of the tracked object.

[0080] ③ Mapping of image pixel coordinates to physical coordinates: This process involves converting the two-dimensional pixel position in the image into a three-dimensional spatial position in the real world, which requires the use of the drone’s attitude information and the camera’s internal parameters.

[0081] Camera model: mainly includes the internal parameter matrix. The internal parameter matrix of the camera includes the focal length (f x ,f y ) and the optical center (c x ,c y ) that describe how the camera maps 3D world coordinates to pixel coordinates on the 2D image plane.

[0082] Conversion from pixel coordinates to normalized image coordinates: First, convert the pixel coordinates of the image (x p ,y p ) is converted to normalized image coordinates (x i ,y i ). This can be achieved using the internal parameter matrix and the following formula:

[0083]

[0084] Mapping of normalized image coordinates to three-dimensional space coordinates:

[0085] (a) Depth information: In order to map from 2D image coordinates to 3D space coordinates, it is necessary to know the depth information (Z) of the point in the scene, that is, the distance from the camera to the target. In this application, it is obtained through the flight control system.

[0086] (b) Calculation of three-dimensional space coordinates: After obtaining the depth information, the normalized image coordinates (x i ,y i ) is converted to the three-dimensional space coordinates (X, Y, Z) in the camera coordinate system:

[0087] X=x i ·Z,Y=y i ·Z,Z=Z

[0088] Here, Z is the distance from the camera to the target, while X and Y are the position of the target in the camera coordinate system.

[0089] Conversion from camera coordinate system to drone local coordinate system: Finally, the target point needs to be converted from the camera coordinate system to the drone local coordinate system. The conversion between coordinate systems is performed using the drone's attitude information (yaw, pitch, roll). The conversion involves applying a rotation matrix R constructed from the drone's attitude angles to the three-dimensional coordinates (X, Y, Z) calculated in the previous step:

[0090]

[0091] Among them, (X u ,Y u ,Z u ) is the three-dimensional coordinate in the local coordinate system of the drone, and R is the rotation matrix constructed according to the pitch, yaw, and roll angles of the drone.

[0092] The content of step ③ belongs to the content of solving the rotation angle of the lens of the acquisition device (equivalent to the specific content of the above-mentioned step 2). The solution of the relative direction vector does not require the content of this step. The solution of the GPS position of the tracked object can directly skip this step ③ and directly use the result obtained from the solution of the rotation angle of the lens of the acquisition device (the ECEF coordinates of the object corresponding to the tracked object). Alternatively, the content of step ③ can be used to perform a corresponding calculation in the solution of the GPS position of the tracked object.

[0093] ④ Position coordinate conversion: First, convert the current GPS position of the drone into the ECEF coordinates of the current position of the drone; if the three-dimensional spatial position coordinates (X u ,Y u ,Z u ) in the local coordinate system of the drone, these coordinates describe the direction and distance of a target relative to the current position of the drone. The current GPS position of the drone provides the position of the drone relative to the earth, usually in longitude (λ), latitude and altitude (h), where altitude is relative to the Earth's surface.

[0094] Based on the current GPS position of the drone, a conversion is performed from the local coordinate system (drone local coordinate system) to the ECEF coordinate system; the ECEF coordinate system is an earth-fixed coordinate system with its origin at the center of mass of the earth, the Z axis pointing to the North Pole, and the X and Y axes pointing to the equator on the prime meridian and the 90 degrees east longitude point on the equator, respectively. Converting a point in the drone's local coordinate system to the ECEF coordinate system involves the following steps:

[0095] a. ECEF coordinates of the drone’s location

[0096] First, you need to get the current GPS position of the drone (longitude λ, latitude The height h) is converted to ECEF coordinates. This can be done using the following formula:

[0097] X ecef =(N+h)·cos(φ)·cos(λ)

[0098] Y ecef =(N+h)·cos(φ)·sin(λ)

[0099] Z ecef =(N·(1-e 2 )+h)·sin(φ)

[0100] Where N is a function of the Earth's radius and Earth's flattening, calculated as a is the equatorial radius of the Earth and e is the eccentricity of the Earth.

[0101] b. Conversion from local coordinate system to ECEF

[0102] Then, you need to convert the point (X u ,Y u ,Z u ) is converted to the ECEF coordinate system. This conversion needs to take into account the attitude of the drone and the curvature of the earth's surface. It is usually assumed that the earth is a perfect sphere and is approximated using a series of rotation and translation operations. The specific conversion method is to rotate the three-dimensional coordinates of the image pixels in the drone's local coordinate system according to the drone's current attitude so that they are aligned with the earth's reference coordinate system to obtain the rotated coordinates, and then add the rotated coordinates to the ECEF coordinates of the drone's current position to obtain the ECEF coordinates of the image pixels as the ECEF coordinates of the object corresponding to the tracked object. The example is as follows:

[0103] The points in the local coordinate system are rotated according to the current attitude of the drone (yaw, pitch, roll) to align them with the Earth reference coordinate system. Translation: The rotated coordinates are then added to the ECEF coordinates of the drone's current position to obtain the ECEF coordinates of the target point, which are the ECEF coordinates of the image pixels corresponding to the tracked object. Ultimately, this process produces the ECEF coordinates of the target point (X target ,Y target ,Z target ), which is the ECEF coordinate of the object corresponding to the tracked object.

[0104] The content of step ④ belongs to the content of solving the rotation angle of the lens of the acquisition device (equivalent to the specific content of the above-mentioned step 3). The solution of the relative direction vector does not require the content of this step. The solution of the GPS position of the tracked object can directly skip the process of step ④ and directly use the result obtained in step ④ (the ECEF coordinates of the object corresponding to the tracked object). It can also use the content of step ④ in the solution of the GPS position of the tracked object to perform a corresponding calculation to obtain the ECEF coordinates of the object corresponding to the tracked object.

[0105] ⑤Conversion from ECEF coordinates to LLA coordinates:

[0106] The point (X target ,Y target ,Z target ) is converted to the LLA coordinate system in is the latitude, λ is the longitude, and h is the altitude, which can be achieved by the following steps:

[0107]

[0108] in For WGS84 the parameters are as follows: a = 6378137, b = a(1-f) = 6356752.31424518, f = 1 / 298.257223563, The LLA coordinates of the object corresponding to the tracked object include the information required to determine the GPS position of the tracked object.

[0109] The content of step ⑤ belongs to the solution of the GPS position of the tracking object (equivalent to the specific content of the GPS position solution method of the tracking object mentioned above). The solution of the rotation angle and relative direction vector of the lens of the acquisition device does not require the content of this step.

[0110] ⑥Calculate the rotation angle of the camera pod:

[0111] According to the ECEF coordinates of the lens of the acquisition device determined by the ECEF coordinates of the current position of the drone and the ECEF coordinates of the object corresponding to the tracking object, a relative position vector is obtained; specifically, a method of calculating the relative position vector between the target position (in the ECEF coordinate system) and the current position of the pod (also in the ECEF coordinate system) includes: assuming that the ECEF coordinates of the target are (X t ,Y t ,Z t ), the ECEF coordinates of the pod are (X c ,Y c ,Z c ), then the relative position vector for:

[0112]

[0113] Then, the relative position vector is transformed according to the rotation matrix in step 1) to obtain the vector representation of the relative position vector in the local coordinate system of the drone; specifically, the relative position vector is transformed using the rotation matrix Transform to obtain the vector representation in the local coordinate system.

[0114]

[0115] Among them, R is the rotation matrix constructed according to the pitch, yaw, and roll angles of the drone.

[0116] After that, the azimuth and pitch angles of the object corresponding to the tracking object relative to the drone are obtained according to the vector representation, and the rotation angle is calculated: In the local coordinate system, the rotation angle can be directly calculated from the relative position vector. Let the relative position vector in the local coordinate system be The parameters that can be determined based on the relative position vector include:

[0117] Azimuth: It is the angle from the north direction of the pod to the direction of the vector projection on the ground in a clockwise direction. It can be obtained by calculating the angle of the vector projection on the east-north plane:

[0118] Azimuth = atan2(v e ,v n )

[0119] Elevation: It is the angle from the horizontal plane of the acquisition device to the vector direction. It can be calculated by the angle between the three-dimensional vector and the ground:

[0120]

[0121] Finally, according to the calculated azimuth and pitch angles, the command for controlling the rotation of the camera pod is generated. The above is the specific process of calculating the position of the target object and using the position calculation result to calculate the rotation angle of the pod.

[0122] The content of step ⑥ belongs to the solution of the rotation angle of the lens of the acquisition device (equivalent to the specific content of the above-mentioned step 4)). The solution of the GPS position of the tracking object and the solution of the relative direction vector do not require the content of this step.

[0123] On the basis of the setting that the visible light camera and the infrared camera are carried at the same position of the same unmanned aerial vehicle through the same optoelectronic pod, the processing module sends corresponding instructions to control the lens of the acquisition device to adjust according to the calculated rotation angle, including: sending corresponding control instructions to the optoelectronic pod according to the calculated rotation angle, controlling the optoelectronic pod to rotate the corresponding angle, so that the lenses of the visible light camera and the infrared camera carried by the optoelectronic pod are pointed to the area where the object corresponding to the tracked object is currently located.

[0124] Specifically, the precise position of the tracked object in the panoramic video is determined by using the above-mentioned method of target tracking of the tracked object in the panoramic video stream, and then the linkage mode is entered to convert the position information of the tracked object into an optoelectronic pod control instruction and send it to the optoelectronic pod to control the optoelectronic pod accordingly; this process includes target GPS solution, calculating the direction and distance of the object corresponding to the tracked object relative to the current pod position, and generating a series of control signals to guide the pod to adjust its angle so as to point the lenses of the visible light camera and infrared camera installed on the optoelectronic pod to the specified target (i.e., the object corresponding to the tracked object).

[0125] To facilitate users to view panoramic videos and close-up videos and interact with data, refer to Figure 1 ,The target tracking system also includes a human-computer interaction module for displaying the panoramic video stream and the close-up video stream;

[0126] The processing module is also used to perform target detection in the panoramic video stream, and assign corresponding different identifiers to each target detected and send them to the human-computer interaction module; the human-computer interaction module is also used to display these different identifiers for relevant personnel to select the corresponding identifier, and return the selection result to the processing module; the processing module is also used to select the corresponding tracking object in the panoramic video stream image through the corresponding identifier selected by the relevant personnel. Therefore, after performing target detection, the user can select one or more targets of interest from the detected targets as tracking objects for in-depth tracking, which can facilitate the user to freely select the tracking object to be tracked on the basis of being able to intuitively observe the images of each detected target.

[0127] In this embodiment, the human-computer interaction module realizes interaction through a human-computer interface, such as Figure 3As shown in the figure, the left side of the interface shows the real-time panoramic video and close-up video (including close-up videos collected by visible light and infrared cameras respectively), and the right side of the interface shows the ID number, pixel coordinates, GPS location information and timestamp of each target after target recognition and detection in different videos (panoramic, visible light, infrared). The processing module detects the targets in the scene through the panoramic video stream and assigns a unique ID. When the operator enters a specific target ID in the human-computer interface for query, the human-computer interaction module sends the queried target ID to the processing module, and the processing module uses the above-mentioned positioning and tracking algorithm to determine the precise position of the target in the panoramic video.

[0128] In addition, the processing module is also used to convert the projection mode of the panoramic video stream image displayed by the human-computer interaction module between the equidistant cylindrical projection mode and the cubic projection mode; the human-computer interaction module is also used to display options of different projection modes for relevant personnel to choose, and return the selection result to the processing module; the processing module is also used to convert the panoramic video stream image displayed by the human-computer interaction module into a view in six directions when the received selection result is the cubic projection mode; when the received selection result is the equidistant cylindrical projection mode, the panoramic video stream image displayed by the human-computer interaction module is converted into an overall view. Thus, the user can flexibly choose to convert the panoramic image displayed by the human-computer interface in the human-computer interaction module between the equidistant cylindrical projection and the cubic projection according to needs; when the cubic projection is selected, the processing module converts the panoramic image into a view in six directions and displays it through the human-computer interaction module, with less distortion, which is convenient for detailed observation; when the equidistant cylindrical projection is selected, the processing module uses the panoramic image as an overall view and displays it through the human-computer interaction module, and the field of view can cover a large range, which is convenient for large-scale observation.

[0129] The present invention has the following characteristics:

[0130] 1) The panoramic video stream and the close-up video stream are linked by the processing module, which not only utilizes the larger field of view of the panoramic video to facilitate the selection and tracking of the corresponding tracking object from the large range, but also can synchronously track the tracking object in the close-up video stream to make up for the defects of image distortion and lack of details in the panoramic video, which is convenient for intuitive observation; and, the location of the tracking object can be found within the large field of view of the panoramic video, which serves as the basis for adjusting the field of view of the close-up video. When the field of view adjustment is completed, the problem that the position of the tracking object may exceed the shooting range of the close-up video can be solved, thereby realizing the continuous tracking of the close-up video in a large range, making up for the defects of the close-up video with a small field of view and difficulty in continuously tracking the target with a fast moving speed or a large moving range, thereby taking into account both the more detailed and intuitive detail features of the tracking target and the larger tracking range.

[0131] 2) The visible light camera and the infrared camera are mounted at the same position on the same UAV via the same optoelectronic pod; the processing module can adjust the field of view of the acquisition device corresponding to the close-up video stream by controlling the selected angle of the optoelectronic pod; thereby, the synchronous adjustment of the fields of view of the visible light camera and the infrared camera can be achieved directly by controlling the angle of the optoelectronic pod, and there is no need to use a special camera with an automatic rotation angle function, and only an optoelectronic pod that can simultaneously install the visible light camera and the infrared camera is required.

[0132] 3) By obtaining the target features corresponding to the tracked object / targets for matching through the pixel features, motion features and position features of the tracked object / targets in the corresponding video stream images, the matching accuracy can be improved, ensuring that the tracked object in the panoramic video stream and the target selected in the close-up video stream to match the tracked object correspond to the same object in the actual scene, thereby improving the accuracy of linking the panoramic video stream with the target to be tracked in the close-up video stream through the actual characteristics of the object.

[0133] 4) The acquisition device corresponding to the close-up video stream includes a visible light camera and an infrared camera. The close-up video stream includes a visible light video stream and an infrared video stream, which can facilitate accurate and effective target recognition both during the day and at night.

[0134] 5) The different identifiers corresponding to each target detected in the panoramic video stream are displayed through the human-computer interaction module for selection by relevant personnel, so that the processing module can select the corresponding tracking object in the panoramic video stream image according to the selection result of the relevant personnel in the human-computer interaction module; thereby, the user can select one or more targets of interest as tracking objects from the various targets detected in the panoramic video stream with a large field of view for in-depth tracking, which can facilitate the user to freely select the tracking object to be tracked on the basis of being able to intuitively observe the images of each detected target.

[0135] It should be understood that the above specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention, but do not constitute a limitation to the present invention.

Claims

1. A target tracking system, characterized in that: Includes processing modules for: Matching the tracked object selected in the panoramic video stream image with each target identified in the close-up video stream image, selecting the target matched with the tracked object as the target corresponding to the same object as the tracked object, and performing target tracking on the tracked object and the matched target in the panoramic video stream and the close-up video stream respectively; When tracking a target, the positioning information of the tracked object is obtained in the current frame of the panoramic video based on the tracking result and the position of the acquisition device corresponding to the close-up video stream, and the field of view of the acquisition device is adjusted using the positioning information so that the close-up video stream acquired by the acquisition device contains the matching target.

2. The target tracking system according to claim 1, characterized in that: According to the tracking result and the position of the acquisition device corresponding to the close-up video stream, the method of obtaining the positioning information of the tracked object in the current frame of the panoramic video includes: According to the tracking result of the target tracking of the tracked object in the panoramic video stream, the image pixel coordinates corresponding to the tracked object in the current frame of the panoramic video are determined as the positioning information of the tracked object.

3. The target tracking system according to claim 2, characterized in that: The acquisition device is also used to be carried on a drone; the method of using the positioning information to adjust the field of view of the acquisition device corresponding to the close-up video stream includes: Calculating the rotation angle of the lens of the acquisition device mounted on the drone using the positioning information of the tracked object, and sending a corresponding instruction to control the lens of the acquisition device to adjust according to the rotation angle; The method of calculating the rotation angle of the lens of the acquisition device by using the positioning information of the tracked object includes: 1) According to the attitude information of the UAV, determine the rotation matrix corresponding to the attitude of the UAV; the rotation matrix includes a yaw matrix, a pitch matrix and a roll matrix; 2) according to the internal parameters of the lens of the acquisition device, the image pixel coordinates corresponding to the tracked object are converted into three-dimensional space coordinates in the lens coordinate system of the acquisition device; and then according to the rotation matrix obtained in step 1), the three-dimensional space coordinates are converted into three-dimensional coordinates of the image pixel in the local coordinate system of the drone; 3) Convert the current GPS position of the drone into the ECEF coordinates of the current position of the drone; rotate the three-dimensional coordinates of the image pixel in the local coordinate system of the drone according to the current posture of the drone so as to align it with the earth reference coordinate system to obtain the rotated coordinates; then add the rotated coordinates to the ECEF coordinates of the current position of the drone to obtain the ECEF coordinates of the image pixel as the ECEF coordinates of the object corresponding to the tracked object; 4) Obtain a relative position vector according to the ECEF coordinates of the lens of the acquisition device determined by the ECEF coordinates of the current position of the drone and the ECEF coordinates of the object corresponding to the tracked object; transform the relative position vector according to the rotation matrix in step 1) to obtain a vector representation of the relative position vector in the local coordinate system of the drone; obtain the azimuth and pitch angles of the object corresponding to the tracked object relative to the drone according to the vector representation, and obtain the rotation angle of the lens of the acquisition device according to the azimuth and pitch angle.

4. The target tracking system according to any one of claims 1 to 3, characterized in that: Methods for selecting targets in the close-up video stream that match the tracked object include: Determine the target features corresponding to the tracked object and each target identified in the close-up video stream image; compare the target features corresponding to the tracked object with the target features corresponding to each target identified in the close-up video stream image, and select the target whose target features among the targets identified in the close-up video stream image have the greatest similarity with the target features corresponding to the tracked object as the target that matches the tracked object; The target features corresponding to the tracked object / targets are obtained based on the pixel features, motion features and position features of the tracked object / targets in the corresponding video stream images; during the target tracking process, the pixel features of the tracked object / targets include color, texture and shape information, the motion features of the tracked object / targets include their movement trajectories in continuous video frames, which are determined by the optical flow method; the position features of the tracked object / targets in the corresponding video stream images include their specific positions and sizes in the video frames, which are determined by the corresponding deep learning model.

5. The target tracking system according to any one of claims 1 to 3, characterized in that: The method of tracking the tracking object and the matching target in the panoramic video stream and the close-up video stream respectively includes: Specially marking the tracked object in the panoramic video stream, and performing the same special marking on the matching target in the close-up video stream for synchronous display; The DeepSORT algorithm is used to perform data association and feature extraction on the tracked object and the matching target in the panoramic video stream and the close-up video stream respectively, and the position and motion trajectory of the tracked object and the matching target are obtained in real time; the position includes the corresponding image pixel coordinates.

6. The target tracking system according to claim 3, characterized in that: It also includes a collection device; the collection device includes collection devices corresponding to the panoramic video stream and the close-up video stream, which are respectively used to collect the corresponding panoramic video stream and the close-up video stream, and transmit them to the processing module.

7. The target tracking system according to claim 6, characterized in that: The acquisition device corresponding to the close-up video stream includes a visible light camera and an infrared camera, and the close-up video stream includes a visible light video stream and an infrared video stream; The visible light camera and the infrared camera are mounted at the same position of the same UAV via the same optoelectronic pod; The processing module sends corresponding instructions to control the lens of the acquisition device to adjust according to the rotation angle, including: sending corresponding control instructions to the optoelectronic pod according to the rotation angle, controlling the optoelectronic pod to rotate the corresponding angle, so that the lenses of the visible light camera and the infrared camera carried by the optoelectronic pod are pointed to the area where the object corresponding to the tracked object is currently located.

8. The target tracking system according to any one of claims 1 to 3, characterized in that: It also includes a human-computer interaction module for displaying the panoramic video stream and the close-up video stream; The processing module is also used to perform target detection in the panoramic video stream, and assign corresponding different identifiers to each target detected and send them to the human-computer interaction module; the human-computer interaction module is also used to display the different identifiers for relevant personnel to select the corresponding identifier, and return the selection result to the processing module; the processing module is also used to select the corresponding tracking object in the panoramic video stream image through the corresponding identifier selected by the relevant personnel.

9. The target tracking system according to claim 8, characterized in that: The processing module is also used to convert the projection mode of the panoramic video stream image displayed by the human-computer interaction module between an equirectangular projection mode and a cubic projection mode; The human-computer interaction module is also used to display options of different projection modes for relevant personnel to choose from, and return the selection results to the processing module; The processing module is also used to convert the panoramic video stream image displayed by the human-computer interaction module into views in six directions when the received selection result is a cubic projection mode; and to convert the panoramic video stream image displayed by the human-computer interaction module into an overall view when the received selection result is an equirectangular projection mode.

10. The target tracking system according to claim 3, characterized in that: The processing module is also used to track the GPS position of the object; the GPS position is calculated in the following ways: According to the conversion formula from ECEF coordinates to LLA coordinates, the ECEF coordinates of the object corresponding to the tracked object are converted into the LLA coordinates of the object corresponding to the tracked object, and the GPS position of the tracked object is determined according to the LLA coordinates of the object corresponding to the tracked object.

Images