Visual tracking system based on a modularized gimbal

Abstract

The application discloses a visual tracking system based on a modularized holder, relates to the technical field of visual dynamic tracking, and comprises the following: a modularized holder, which is used for erecting a visual collection module; the visual collection module, which is used for acquiring image data in real time; a data processing module, which is used for processing the obtained image data and determining an image key frame according to a processing result; a visual analysis module, which is used for determining a target feature to be tracked in the image key frame; and a visual tracking module, which is used for generating a cooperative intelligent control strategy according to the moving track of the target feature; through feature priority division and dynamic key frame demarcation range adjustment, full-range demarcation is adopted to focus on a core target when there is a high-priority feature, and a default range is adopted when there is no highest-priority feature, so that redundant data processing is effectively reduced, the computing power is inclined to high-value core features, and the accuracy and real-time performance of tracking are improved.

Description

Technical Field

[0001] This invention relates to the field of visual dynamic tracking technology, specifically a visual tracking system based on a modular gimbal. Background Technology

[0002] Existing visual tracking systems mostly use a single camera or a standalone gimbal, which suffers from limited field of view and easy target loss in complex scenarios. To address this, we are now providing a visual tracking system based on a modular gimbal. Summary of the Invention

[0003] The purpose of this invention is to provide a visual tracking system based on a modular gimbal.

[0004] The objective of this invention can be achieved through the following technical solution: a visual tracking system based on a modular gimbal, comprising:

[0005] Modular gimbal, used to set up visual acquisition modules;

[0006] The visual acquisition module is used to acquire image data in real time;

[0007] The data processing module is used to process the acquired image data and determine the image keyframes based on the processing results.

[0008] The visual analysis module is used to determine the features of the target to be tracked based on keyframes of the image;

[0009] The visual tracking module is used to generate collaborative intelligent control strategies based on the movement trajectory of the target features.

[0010] Furthermore, the modular gimbal includes a pitch unit and a steering unit. The pitch unit is used to adjust the pitch of the constructed visual acquisition terminal, and the steering unit is used to adjust the horizontal position of the constructed visual acquisition module.

[0011] Furthermore, the process by which the visual acquisition module acquires image data in real time includes:

[0012] Several modular gimbals equipped with visual acquisition modules were deployed in the target area, and the initial state of the modular gimbals was reset to their initial positions.

[0013] Image data of the target area is acquired in real time through the visual acquisition module mounted on each modular gimbal. Each visual acquisition module is set with a shooting range.

[0014] Construct a timeline associated with each visual acquisition module, and map the image data obtained by the visual acquisition module to the corresponding timeline according to the acquisition time;

[0015] The video data obtained by each visual acquisition module mapped to the timeline are time-aligned according to the acquisition time.

[0016] Furthermore, the data processing module processes the acquired image data, and the process of determining image keyframes based on the processing results includes:

[0017] The image data within each time axis is converted into image frames, and each image frame is marked with a timestamp;

[0018] Set a time window, the time span of which is [t1, t2];

[0019] The system acquires image frames from the video data obtained by each visual acquisition module at time t2, extracts features from each image frame, and filters the extracted features to obtain key features.

[0020] Based on the obtained key features, the key frame calibration range is determined, and image frames within the time window are selected as image key frames according to the determined key frame calibration range.

[0021] Furthermore, feature extraction is performed on each image frame, and the extracted features are filtered to obtain key features. This process includes:

[0022] The image frames are processed by grayscale and rasterization, and features are extracted from the processed image frames.

[0023] The extracted features are filtered through a pre-defined list of basic reference features, and the corresponding features within the image frame are marked as key features based on the filtering results.

[0024] Furthermore, the process of determining the keyframe calibration range based on the obtained key features, and selecting image frames within the time window as image keyframes according to the determined keyframe calibration range, includes:

[0025] Obtain the key features present in the image frame corresponding to each visual acquisition module, obtain the priority level of each key feature, and determine whether there is a key feature with the highest priority level. If not, set the key frame calibration range of the image frame to the default range.

[0026] If it exists, the corresponding key feature is marked as the core feature, and the key frame calibration range of the image frame is set to the full range;

[0027] Based on the set keyframe calibration range, select image frames within the time window as image keyframes.

[0028] Furthermore, the process by which the visual analysis module determines the target features to be tracked based on image keyframes includes:

[0029] Mark the image frames containing core features in each image frame corresponding to the current moment, as well as the selected corresponding image keyframes;

[0030] The motion trajectory of the core feature is generated based on the marked image keyframes, and the motion vector is generated at the location of the core feature within the image frame at the current moment.

[0031] The simulated motion trajectory of the core features is generated based on the motion vector of the core features;

[0032] Obtain the motion trajectory of other key features and generate the corresponding motion vector within the image frame at the current moment, and generate simulated motion trajectories corresponding to each key feature;

[0033] The motion vector includes the motion speed and the motion direction;

[0034] Then the correlation influence values ​​between each key feature and the core feature are obtained;

[0035] Set an influence threshold, compare the correlation influence values ​​of each key feature and core feature with the influence threshold, and determine the target feature based on the comparison results.

[0036] Furthermore, the process by which the visual tracking module generates a collaborative intelligent control strategy based on the target's movement trajectory includes:

[0037] The visual acquisition module that is closest to the core feature and within the shooting range is marked as the main tracking device;

[0038] Similarly, the visual acquisition module that is closest to each target feature and is within the shooting range is identified and denoted as the secondary tracking device.

[0039] Based on the motion vector of the core feature, a corresponding time series is generated on the simulated motion trajectory of the core feature;

[0040] Align the center of the main tracking device's shooting range with the location of the core feature, and obtain the motion vectors of the core feature in the horizontal and vertical directions based on the time series on the simulated motion trajectory, thereby setting the rotation direction and speed of the pitch unit and steering unit of the main tracking device.

[0041] Similarly, based on the motion vectors of each target feature, a corresponding time series is generated on the simulated motion trajectory;

[0042] Align the center of the shooting range of the secondary tracking device with the center point between the target feature and the core feature. Based on the time series on the simulated motion trajectory, obtain the motion vectors of the target feature in the horizontal and vertical directions, and thus set the rotation direction and speed of the pitch unit and steering unit of the secondary tracking device.

[0043] Compared with the prior art, the beneficial effects of the present invention are:

[0044] 1. By prioritizing features and adjusting the dynamic keyframe calibration range, full-range calibration is used to focus on the core target when high-priority features exist, and the default range is used when there is no highest priority feature. This effectively reduces redundant data processing, ensures that computing power is tilted towards high-value core features, and improves the accuracy and real-time performance of tracking.

[0045] 2. Based on motion vectors, simulated motion trajectories and time series are generated to drive the gimbal to perform predictive pre-rotation, eliminating the response lag of traditional passive tracking. At the same time, by dividing the tracking into primary and secondary devices, the primary device accurately locks the core features, and the secondary device aligns the target with the center point of the core features, realizing intelligent division of labor and collaborative cooperation among multiple gimbals in complex multi-target scenarios. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0047] Figure 1 This is a schematic diagram of the principle of the present invention. Detailed Implementation

[0048] like Figure 1 As shown, the visual tracking system based on a modular gimbal includes:

[0049] Modular gimbal, used to set up visual acquisition modules;

[0050] The visual acquisition module is used to acquire image data in real time;

[0051] The data processing module is used to process the acquired image data and determine the image keyframes based on the processing results.

[0052] The visual analysis module is used to determine the features of the target to be tracked based on keyframes of the image;

[0053] The visual tracking module is used to generate collaborative intelligent control strategies based on the movement trajectory of the target features.

[0054] It should be further explained that, in the specific implementation process, the modular gimbal includes a pitch unit and a steering unit. The pitch unit is used to adjust the pitch of the constructed visual acquisition terminal, and the steering unit is used to adjust the horizontal position of the constructed visual acquisition module.

[0055] Record the initial positions of the pitch unit and the steering unit, and based on the initial positions, denote the pitch adjustment angle range as [w1, w2] and the horizontal adjustment angle range as [s1, s2], where w1 and s1 are both less than 0, and w2 and s2 are both greater than 0. It should be noted that the pitch angle and horizontal angle corresponding to the initial positions of the pitch unit and the steering unit are both 0.

[0056] It should be further explained that, in the specific implementation process, the process of the visual acquisition module acquiring image data in real time includes:

[0057] Several modular gimbals equipped with visual acquisition modules were deployed in the target area, and the initial state of the modular gimbals was reset to their initial positions.

[0058] Image data of the target area is acquired in real time through the visual acquisition module mounted on each modular gimbal;

[0059] It should be noted that each visual acquisition module has a set shooting range, and the combination of the shooting ranges of all visual acquisition modules can cover the entire target area.

[0060] Each visual acquisition module is labeled as i, where i = 1, 2, ..., n;

[0061] Construct a timeline associated with each visual acquisition module, and map the image data obtained by the visual acquisition module labeled i to the corresponding timeline according to the acquisition time;

[0062] The video data obtained by each visual acquisition module mapped to the timeline are time-aligned according to the acquisition time.

[0063] It should be further explained that, in the specific implementation process, the data processing module processes the acquired image data, and the process of determining the image keyframes based on the processing results includes:

[0064] The image data within each time axis is converted into image frames, and each image frame is marked with a timestamp;

[0065] Set a time window with a time span of [t1, t2], where t2 corresponds to the current time and t1 is the previous time.

[0066] The system acquires image frames from the video data obtained by each visual acquisition module at time t2, extracts features from each image frame, and filters the extracted features to obtain key features.

[0067] Based on the obtained key features, the key frame calibration range is determined, and image frames within the time window are selected as image key frames according to the determined key frame calibration range.

[0068] It should be further explained that the process of extracting features from each image frame and filtering the extracted features to obtain key features includes:

[0069] The image frames are processed by grayscale and rasterization, and features are extracted from the processed image frames.

[0070] The extracted features are filtered through a preset basic reference feature list. Based on the filtering results, the corresponding features within the image frame are marked as key features. It should be noted that the basic reference feature list consists of several reference features, and different reference features are assigned corresponding priority coefficients, grayscale value ranges, and priority levels. The higher the priority coefficient, the higher the importance of the corresponding reference feature. By matching the grayscale value corresponding to the extracted feature with the grayscale value of each reference feature, if a match is found with the grayscale value of any reference feature, the corresponding feature is marked as a key feature.

[0071] It should be further explained that, in the specific implementation process, the process of determining the keyframe calibration range based on the obtained key features, and selecting image frames within the time window as image keyframes according to the determined keyframe calibration range, includes:

[0072] Obtain the key features present in the image frame corresponding to the visual acquisition module labeled i, and obtain the priority level of each key feature. Determine whether there is a key feature with the highest priority level. If not, set the key frame calibration range of the image frame to the default range.

[0073] If it exists, the corresponding key feature is marked as the core feature, and the key frame calibration range of the image frame is set to the full range; it should be further noted that there is only one core feature, the full range refers to the entire time window, and the default range refers to a part of the time window;

[0074] Based on the set keyframe calibration range, select image frames within the time window as image keyframes.

[0075] It should be further explained that, in the specific implementation process, the visual analysis module determines the target features to be tracked based on image keyframes, including:

[0076] Mark the image frames containing core features in each image frame corresponding to the current moment, as well as the selected corresponding image keyframes;

[0077] The motion trajectory of the core feature is generated based on the marked image keyframes, and the motion vector is generated at the location of the core feature within the image frame at the current moment.

[0078] The simulated motion trajectory of the core features is generated based on the motion vector of the core features;

[0079] Obtain the motion trajectory of other key features and generate the corresponding motion vector within the image frame at the current moment, and generate simulated motion trajectories corresponding to each key feature;

[0080] The motion vector includes the motion speed and the motion direction;

[0081] Then the correlation influence values ​​between each key feature and the core feature are obtained;

[0082] Set an influence threshold, compare the correlation influence values ​​of each key feature and core feature with the influence threshold, and determine the target feature based on the comparison results;

[0083] Specifically:

[0084] Each key feature is labeled and denoted as j, where j = 1, 2, ..., m;

[0085] The positions where the simulated motion trajectory corresponding to the key feature labeled j is closest to the simulated motion trajectory of the core feature are marked as key boundary points and core boundary points, respectively.

[0086] Let the straight-line distance between the critical boundary and the core boundary of the critical feature labeled j be denoted as . ;

[0087] Furthermore, based on the motion vectors, the arrival times of key features and core features at key and core boundaries are obtained, respectively, and the corresponding time differences are then obtained, denoted as . ;

[0088] Then the correlation influence value between the key feature and the core feature labeled j is obtained, denoted as . ,in:

[0089] ;

[0090] Where L0 is a distance constant and t0 is a time constant;

[0091] Set the impact threshold as G0;

[0092] like If ≥G0, it means that the corresponding key feature is strongly related to the core feature, and the key feature is recorded as the target feature. Conversely, if the corresponding key feature is weakly related to the core feature, the key feature is ignored.

[0093] It should be further explained that, in the specific implementation process, the process by which the visual tracking module generates a collaborative intelligent control strategy based on the target's movement trajectory includes:

[0094] The visual acquisition module that is closest to the core feature and within the shooting range is marked as the main tracking device;

[0095] Similarly, the visual acquisition module that is closest to each target feature and within the shooting range is selected and denoted as the secondary tracking device. It should be noted that if the visual acquisition module is both the primary tracking device and the secondary tracking device, it will automatically be overridden as the primary tracking device.

[0096] Based on the motion vector of the core feature, a corresponding time series is generated on the simulated motion trajectory of the core feature;

[0097] Align the center of the main tracking device's shooting range with the location of the core feature, and obtain the motion vectors of the core feature in the horizontal and vertical directions based on the time series on the simulated motion trajectory, thereby setting the rotation direction and speed of the pitch unit and steering unit of the main tracking device.

[0098] Similarly, based on the motion vectors of each target feature, a corresponding time series is generated on the simulated motion trajectory;

[0099] Align the center of the shooting range of the secondary tracking device with the center point between the target feature and the core feature. Based on the time series on the simulated motion trajectory, obtain the motion vectors of the target feature in the horizontal and vertical directions, and thus set the rotation direction and speed of the pitch unit and steering unit of the secondary tracking device.

[0100] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications or equivalent substitutions made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A visual tracking system based on a modular gimbal, characterized in that, include: Modular gimbal, used to set up visual acquisition modules; The visual acquisition module is used to acquire image data in real time; The data processing module is used to process the acquired image data and determine the image keyframes based on the processing results. The visual analysis module is used to determine the features of the target to be tracked based on keyframes of the image; The visual tracking module is used to generate collaborative intelligent control strategies based on the movement trajectory of the target features.

2. The visual tracking system based on a modular gimbal according to claim 1, characterized in that, The modular gimbal includes a pitch unit and a steering unit. The pitch unit is used to adjust the pitch of the constructed visual acquisition terminal, and the steering unit is used to adjust the horizontal position of the constructed visual acquisition module.

3. The visual tracking system based on a modular gimbal according to claim 2, characterized in that, The process of the visual acquisition module acquiring image data in real time includes: Several modular gimbals equipped with visual acquisition modules were deployed in the target area, and the initial state of the modular gimbals was reset to their initial positions. Image data of the target area is acquired in real time through the visual acquisition module mounted on each modular gimbal. Each visual acquisition module is set with a shooting range. Construct a timeline associated with each visual acquisition module, and map the image data obtained by the visual acquisition module to the corresponding timeline according to the acquisition time; The video data obtained by each visual acquisition module mapped to the timeline are time-aligned according to the acquisition time.

4. The visual tracking system based on a modular gimbal according to claim 3, characterized in that, The data processing module processes the acquired image data, and the process of determining image keyframes based on the processing results includes: The image data within each time axis is converted into image frames, and each image frame is marked with a timestamp; Set a time window, the time span of which is [t1, t2]; The system acquires image frames from the video data obtained by each visual acquisition module at time t2, extracts features from each image frame, and filters the extracted features to obtain key features. Based on the obtained key features, the key frame calibration range is determined, and image frames within the time window are selected as image key frames according to the determined key frame calibration range.

5. The visual tracking system based on a modular gimbal according to claim 4, characterized in that, The process of extracting features from each image frame and filtering the extracted features to obtain key features includes: The image frames are processed by grayscale and rasterization, and features are extracted from the processed image frames. The extracted features are filtered through a pre-defined list of basic reference features, and the corresponding features within the image frame are marked as key features based on the filtering results.

6. The visual tracking system based on a modular gimbal according to claim 5, characterized in that, The process of determining the keyframe calibration range based on the obtained key features, and selecting image frames within the time window as image keyframes according to the determined keyframe calibration range, includes: Obtain the key features present in the image frame corresponding to each visual acquisition module, obtain the priority level of each key feature, and determine whether there is a key feature with the highest priority level. If not, set the key frame calibration range of the image frame to the default range. If it exists, the corresponding key feature is marked as the core feature, and the key frame calibration range of the image frame is set to the full range; Based on the set keyframe calibration range, select image frames within the time window as image keyframes.

7. The visual tracking system based on a modular gimbal according to claim 6, characterized in that, The process by which the visual analysis module determines the features of the target to be tracked based on keyframes of an image includes: Mark the image frames containing core features in each image frame corresponding to the current moment, as well as the selected corresponding image keyframes; The motion trajectory of the core feature is generated based on the marked image keyframes, and the motion vector is generated at the location of the core feature within the image frame at the current moment. The simulated motion trajectory of the core features is generated based on the motion vector of the core features; Obtain the motion trajectory of other key features and generate the corresponding motion vector within the image frame at the current moment, and generate simulated motion trajectories corresponding to each key feature; The motion vector includes the motion speed and the motion direction; Then the correlation influence values ​​between each key feature and the core feature are obtained; Set an influence threshold, compare the correlation influence values ​​of each key feature and core feature with the influence threshold, and determine the target feature based on the comparison results.

8. The visual tracking system based on a modular gimbal according to claim 7, characterized in that, The process by which the visual tracking module generates a collaborative intelligent control strategy based on the target's movement trajectory includes: The visual acquisition module that is closest to the core feature and within the shooting range is marked as the main tracking device; Similarly, the visual acquisition module that is closest to each target feature and is within the shooting range is identified and denoted as the secondary tracking device. Based on the motion vector of the core feature, a corresponding time series is generated on the simulated motion trajectory of the core feature; Align the center of the main tracking device's shooting range with the location of the core feature, and obtain the motion vectors of the core feature in the horizontal and vertical directions based on the time series on the simulated motion trajectory, thereby setting the rotation direction and speed of the pitch unit and steering unit of the main tracking device. Similarly, based on the motion vectors of each target feature, a corresponding time series is generated on the simulated motion trajectory; Align the center of the shooting range of the secondary tracking device with the center point between the target feature and the core feature. Based on the time series on the simulated motion trajectory, obtain the motion vectors of the target feature in the horizontal and vertical directions, and thus set the rotation direction and speed of the pitch unit and steering unit of the secondary tracking device.

Images