Target tracking shooting method, device, equipment and storage medium
Through the combination of the binocular ranging module and the three-axis tracking module, the target is identified and tracked, and the problem of low target tracking accuracy in dynamic background is solved, achieving a more stable and efficient shooting effect.
Patent Information
- Application Number
- CN202410993075.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-24
AI Technical Summary
Existing target tracking photography methods have low tracking accuracy for shooting targets in dynamic backgrounds, especially when targets are deformed or occluded, and are sensitive to noise and light changes.
The binocular ranging module is used to collect multiple frames of original binocular images for multiple image enhancement, identify the initial moving target and still target, generate shooting depth information through parallax conversion, determine the target and background using the background separation model, and extract the edge of the target image to calculate the geometric distance. , generate target features, predict target offsets, and generate tracking and shooting parameters through the three-axis tracking module to achieve accurate tracking of the target.
Improves the accuracy of target tracking in dynamic contexts, enhances stability and continuity to shooting targets, and reduces sensitivity to noise and light changes.
Smart Images

Figure CN118524289B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of target tracking technology, and in particular to a target tracking shooting method, device, equipment and storage medium. Background Art
[0002] In the field of photography and videography, the introduction of automation technology has greatly improved shooting efficiency and quality. Traditional shooting methods mainly rely on manual operation, and the photographer manually adjusts the angle, focal length and position of the camera to capture the target. This method is not only time-consuming and laborious, but also difficult to ensure the stability and continuity of the shooting effect when facing fast-moving targets. In response to this, in order to improve shooting efficiency and quality, automatic target tracking shooting technology came into being.
[0003] At present, target tracking shooting technologies mainly include target recognition and tracking based on image processing, target positioning and tracking based on sensors, etc. However, these methods often have problems such as performance degradation when the target is deformed or occluded, sensitivity to noise and lighting changes, and difficulty in adapting to dynamic changes in the scene and the similarity between the target and the background. In other words, the existing target tracking shooting methods have low tracking accuracy for the shooting target under dynamic background. Summary of the invention
[0004] The main purpose of the present invention is to solve the problem that the existing target tracking and shooting methods have low tracking accuracy for shooting targets under dynamic backgrounds.
[0005] A first aspect of the present invention provides a target tracking shooting method, which is applied to a pan-tilt shooting and tracking system, wherein the pan-tilt shooting and tracking system includes a binocular ranging module and a three-axis tracking module, and the target tracking shooting method includes: collecting multiple frames of original binocular images shot by the pan-tilt through the binocular ranging module, and performing multiple image enhancement on the original binocular images to obtain enhanced original binocular images; identifying initial moving targets and initial stationary targets of different frame rates in the enhanced original binocular images, and performing parallax conversion on the enhanced original binocular images to generate shooting depth information, and based on the initial moving targets and the initial stationary targets, using a preset background separation model to determine the initial shooting targets and the initial stationary targets in the original binocular images. The invention relates to an initial shooting background; extracting the target image edge corresponding to the initial shooting target, and based on the target image edge, calculating the geometric distance of the initial shooting target to generate the target object feature; determining the target offset of the initial shooting target based on the initial shooting background, the target object feature and the shooting depth information, and predicting the target prediction state of the next frame of the initial shooting target based on the target offset; generating the tracking shooting parameters of the initial shooting target through the three-axis tracking module based on the target prediction state and the target object feature, and centering the shooting picture of the initial shooting target and tracking and shooting the initial shooting target based on the tracking shooting parameters to obtain the target tracking shooting result.
[0006] Optionally, in a first implementation method of the first aspect of the present invention, performing multiple image enhancements on the original binocular image to obtain an enhanced original binocular image includes: performing image grayscale and median filtering on the original binocular image, and adjusting image parameters of the filtered original binocular image to obtain an adjusted original binocular image; performing histogram equalization on the adjusted original binocular image, and based on the calibration parameters of the binocular camera in the binocular ranging module, performing geometric correction of image distortion on the equalized original binocular image to obtain an enhanced original binocular image.
[0007] Optionally, in a second implementation manner of the first aspect of the present invention, the identifying of initial moving targets and initial stationary targets at different frame rates in the enhanced original binocular image, and performing disparity conversion on the enhanced original binocular image to generate shooting depth information include: based on a preset initial target area, identifying and marking the change areas in the enhanced original binocular image at different frame rates to generate potential moving targets and potential stationary targets; matching multiple image features of the potential moving targets and the potential stationary targets, and based on the matching results, performing multi-region attribute marking on the potential moving targets and the potential stationary targets to obtain the initial moving targets and the initial stationary targets; selecting the enhanced original binocular image of one purpose to perform pixel matching cost on the enhanced original binocular image of another purpose, and performing cost aggregation and disparity optimization on each pixel after matching cost to obtain an original disparity map; based on the internal and external parameters of the camera in the binocular ranging module, performing disparity value conversion on the original disparity map to obtain shooting depth information.
[0008] Optionally, in a third implementation manner of the first aspect of the present invention, the initial shooting target and the initial shooting background in the original binocular image are determined based on the initial moving target and the initial stationary target using a preset background separation model, including: determining the target shooting scene shot by the gimbal based on the initial moving target and the initial stationary target, and matching a background separation model of a corresponding scene type from a preset model library based on the target shooting scene; separating the image foreground mask in the initial moving target and the initial stationary target using the background separation model, and performing morphological operations and target area extraction on the image foreground mask based on the target shooting scene to obtain the initial shooting target; separating the image frame of the initial shooting target from the image foreground mask using the background separation model, and performing noise smoothing and motion compensation on the separated background image to obtain the initial shooting background.
[0009] Optionally, in a fourth implementation manner of the first aspect of the present invention, the extracting the target image edge corresponding to the initial photographed target, and calculating the geometric distance of the initial photographed target based on the target image edge to generate the target object feature includes: calculating a plurality of edge attribute parameters corresponding to the initial photographed target, and generating an initial target edge corresponding to the initial photographed target based on the edge attribute parameters; performing pixel tracking of a closed contour on the initial target edge to obtain at least one target image edge with a closed contour, and calculating the target geometric distance of the target image edge;
[0010] Based on the target geometric distance, multiple target attribute parameters of the initial shooting target are determined, and based on the target attribute parameters, a geometric feature vector corresponding to the target shape is constructed; the edge feature vector corresponding to the initial target edge and the geometric feature vector are weightedly fused to generate target object features.
[0011] Optionally, in a fifth implementation manner of the first aspect of the present invention, determining the target offset of the initial shooting target based on the initial shooting background, the target object features and the shooting depth information, and predicting the target prediction state of the next frame of the initial shooting target based on the target offset, includes: calculating the target trajectory parameters of the initial shooting target in each frame image based on the initial shooting background, the target object features and the shooting depth information; comparing the center of mass difference of the initial shooting target in each frame image based on the target center of mass coordinates in the target attribute parameters and the target trajectory parameters to obtain the target offset. , and determine whether the target offset is greater than a preset offset threshold; if the target offset is not greater than the preset offset threshold, then based on the target centroid coordinates and the target trajectory parameters, determine the target motion trajectory of the initial shooting target, and based on the target motion trajectory, predict the target prediction state of the next frame of the initial shooting target; if the target offset is greater than the preset offset threshold, then based on the target centroid coordinates and the target trajectory parameters, determine the target shape features and target posture of the initial shooting target, and based on the target shape features, predict the target prediction state of the next frame of the initial shooting target at the target posture.
[0012] Optionally, in a sixth implementation manner of the first aspect of the present invention, the tracking and shooting parameters of the initial shooting target are generated by the three-axis tracking module based on the target prediction state and the target object characteristics, including: based on the target prediction state and the target object characteristics, the three-axis tracking module calculates the background motion vector of the pixel points in the continuous frame images, and uses the background motion vector to perform background compensation on the target prediction state to obtain a compensated target prediction state; based on the compensated target prediction state and the background motion vector, calculates the motion displacement parameters of the binocular ranging module, and based on the motion displacement parameters, generates the tracking and shooting parameters of the initial shooting target.
[0013] The second aspect of the present invention provides a target tracking shooting device, which is applied to a pan-tilt shooting and tracking system. The pan-tilt shooting and tracking system includes a binocular ranging module and a three-axis tracking module. The target tracking shooting device includes: an image enhancement module, which is used to collect multiple frames of original binocular images shot by the pan-tilt through the binocular ranging module, and perform multiple image enhancements on the original binocular images to obtain enhanced original binocular images; a target recognition module, which is used to identify initial moving targets and initial stationary targets at different frame rates in the enhanced original binocular images, and perform parallax conversion on the enhanced original binocular images to generate shooting depth information, and based on the initial moving targets and the initial stationary targets, use a preset background separation model to determine the initial shooting targets and initial shooting targets in the original binocular images. The invention relates to a method for obtaining a target object of the invention and a background; a feature extraction module, which is used to extract the edge of the target image corresponding to the initial shooting target, and based on the edge of the target image, calculate the geometric distance of the initial shooting target to generate the target object feature; a state prediction module, which is used to determine the target offset of the initial shooting target based on the initial shooting background, the target object feature and the shooting depth information, and predict the target prediction state of the next frame of the initial shooting target based on the target offset; a tracking shooting module, which is used to generate the tracking shooting parameters of the initial shooting target through the three-axis tracking module based on the target prediction state and the target object feature, and based on the tracking shooting parameters, center the shooting picture of the initial shooting target and track and shoot the initial shooting target to obtain the target tracking shooting result.
[0014] The third aspect of the present invention provides a target tracking and shooting device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the target tracking and shooting device executes each step of the above-mentioned target tracking and shooting method.
[0015] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute each step of the above-mentioned target tracking and shooting method.
[0016] In an embodiment of the present invention, a plurality of frames of original binocular images captured by a pan / tilt are collected by a binocular ranging module, and multiple image enhancements are performed on the original binocular images to obtain enhanced original binocular images; initial moving targets and initial stationary targets of different frame rates in the enhanced original binocular images are identified, and parallax conversion is performed on the enhanced original binocular images to generate shooting depth information, and based on the initial moving target and the initial stationary target, an initial shooting target and an initial shooting background in the original binocular image are determined by using a preset background separation model; a target image edge corresponding to the initial shooting target is extracted, and based on the target image edge, a geometric distance of the initial shooting target is calculated to generate target object features; based on the initial shooting background, the target object features and the shooting depth information, a target offset of the initial shooting target is determined, and based on the target offset, a target prediction state of the next frame of the initial shooting target is predicted; based on the target prediction state and the target object features, a tracking shooting parameter of the initial shooting target is generated by a three-axis tracking module, and based on the tracking shooting parameter, a shooting picture of the initial shooting target is centered and the initial shooting target is tracked and shot to obtain a target tracking shooting result. Compared with the prior art, the present application performs parallax conversion and shooting background separation on multiple frames of original binocular images collected by the binocular ranging module, and performs image edge detection and geometric clustering calculation on the extracted initial shooting target to obtain the target object features, and then determines the target offset of the initial shooting target based on the initial shooting background, target object features and shooting depth information, and generates the target prediction state at the next moment based on the offset of the initial shooting target, thereby generating corresponding tracking shooting parameters based on the target prediction state, and controlling the gimbal to perform centering shooting and tracking shooting of the shooting target through the three-axis tracking module, thereby improving the tracking accuracy of the shooting target under a dynamic background.
[0017] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.
[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 Schematic diagram of a first embodiment of a target tracking and shooting method according to an embodiment of the present invention;
[0020] Figure 2 Schematic diagram of a second embodiment of the target tracking and shooting method in an embodiment of the present invention;
[0021] Figure 3A schematic diagram of an embodiment of a target tracking and shooting device in an embodiment of the present invention;
[0022] Figure 4 Schematic diagram of an embodiment of a target tracking and shooting device in an embodiment of the present invention. DETAILED DESCRIPTION
[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0024] The terms "including" and "having" and any variations thereof mentioned in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device end including a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or device ends.
[0025] To facilitate understanding of this embodiment, the specific process of the embodiment of the present invention is described below. Figure 1 The first embodiment of the target tracking and shooting method in the embodiment of the present invention includes:
[0026] 101. Collect multiple frames of original binocular images taken by the gimbal through the binocular ranging module, and perform multiple image enhancement on the original binocular images to obtain enhanced original binocular images;
[0027] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0028] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0029] In this embodiment, the pan-tilt shooting and tracking system includes a binocular ranging module and a three-axis tracking module, wherein the binocular ranging module is an image acquisition module composed of a binocular camera and a galvanometer, and the three-axis tracking module is an adjustment control module that adjusts the speed and direction of the pan-tilt based on relevant tracking parameters so that the center of the field of view of the binocular camera is consistent with the center of the target object. The original binocular image is grayed and median filtered, and the image parameters of the filtered original binocular image are adjusted to obtain the adjusted original binocular image; the adjusted original binocular image is histogram-equalized, and based on the calibration parameters of the binocular camera in the binocular ranging module, the equalized original binocular image is geometrically corrected for image distortion to obtain an enhanced original binocular image.
[0030] In practical applications, multiple frames of original binocular images taken by the current gimbal are collected through the binocular ranging module, and then the original binocular image of the color image is converted into a grayscale image, and the median of the neighborhood pixels of each pixel is calculated, and the median is used to replace the original pixel value to obtain the original binocular image after median filtering, and then the original binocular image after median filtering is adjusted, including optimizing the parameters such as image brightness, contrast, and saturation to enhance the visual effect of the image (wherein, this step is based on the scene mode shot by the current gimbal, and the corresponding parameters need to be adjusted based on the corresponding shooting scene to ensure that the adjusted image can provide the best visual information and detail presentation); and then by adjusting the grayscale distribution of the image, the brightness of the image is The values are more evenly distributed, thereby enhancing the details and contrast of the image. After completing the histogram equalization, based on the calibration parameters of the binocular camera in the binocular ranging module (wherein the calibration parameters of the binocular camera include the intrinsic parameters and extrinsic parameters of the camera, the intrinsic parameters describe the focal length, optical center and other parameters of the camera, and the extrinsic parameters describe the relative position and posture between the two cameras), the equalized original binocular image is geometrically corrected for image distortion, that is, each pixel is reversely mapped, and the pixels in the distorted image are mapped to the undistorted image plane. By using the calibration parameters of the binocular camera, the corrected position of each pixel is accurately calculated, and the distortion caused by the camera lens and the shooting angle is eliminated, thereby obtaining the enhanced original binocular image.
[0031] 102. Identify initial moving targets and initial stationary targets at different frame rates in the enhanced original binocular image, perform parallax conversion on the enhanced original binocular image, generate shooting depth information, and determine the initial shooting target and the initial shooting background in the original binocular image based on the initial moving target and the initial stationary target using a preset background separation model;
[0032] In this embodiment, the initial moving target here refers to an object that moves relatively in the shooting picture, including the shooting target and background objects, etc.; the initial stationary target here refers to an object that remains relatively prohibited in the shooting picture; the shooting depth information here refers to the three-dimensional depth information of each object in the shooting picture; the initial shooting target here refers to the target to be tracked and shot in the current shooting picture; the initial shooting background here refers to the screen objects other than the initial shooting target in the current shooting picture as the shooting background. Based on the preset initial target area, the change area in the enhanced original binocular image with different frame rates is identified and marked to generate potential moving targets and potential stationary targets; the potential moving targets and potential stationary targets are matched with multiple image features, and based on the matching results, the potential moving targets and potential stationary targets are marked with attributes in multiple regions to obtain the initial moving targets and initial stationary targets; the original binocular image enhanced by one purpose is selected to match the original binocular image enhanced by another purpose with pixel matching costs, and each pixel after matching costs is subjected to cost aggregation and disparity optimization to obtain the original disparity map; based on the internal and external parameters of the camera in the binocular ranging module, the original The disparity map is converted into disparity values to obtain shooting depth information; based on the initial moving target and the initial stationary target, the target shooting scene shot by the gimbal is determined, and based on the target shooting scene, a background separation model of the corresponding scene type is matched from a preset model library; the image foreground masks in the initial moving target and the initial stationary target are separated by the background separation model, and based on the target shooting scene, morphological operations and target area extraction are performed on the image foreground masks to obtain the initial shooting target; the image foreground mask is separated from the image frame of the initial shooting target by the background separation model, and noise smoothing and motion compensation are performed on the separated background image to obtain the initial shooting background.
[0033] In practical applications, based on the preset initial target area, the changed areas in the enhanced original binocular image with different frame rates are identified and marked, that is, by comparing the differences between frames, the moving areas and static areas in the image are identified, wherein the detection of the changed areas is achieved by a simple frame difference method, the pixel-level differences between consecutive frames are calculated, and the areas with significant differences are marked as potential moving targets, while the areas with small or no changes are regarded as potential static targets; then, a variety of feature information in the image is extracted, such as color features, texture features, shape features, etc., and matched with the known target template, and then, based on the matching results, the potential moving targets and potential static targets are marked with multi-region attributes, wherein these attribute tags include the target's motion state, shape, size, color and other information, through these attribute tags, the characteristics of the initial moving target and the initial static target are more accurately described, thereby obtaining the initial moving target and the initial static target; then, a frame of the enhanced original binocular image is selected. The image performs pixel matching cost calculation on the original binocular image enhanced by another frame, that is, matching is performed by comparing the similarity of corresponding pixels in the two images, and after the pixel matching cost calculation, cost aggregation and disparity optimization are performed on each pixel, that is, local costs are aggregated into global costs and a shooting target optimization problem is solved to determine the best disparity map, and the original disparity map is obtained (wherein the original disparity map is a grayscale image, and the grayscale value of each pixel represents the disparity value of the pixel, that is, the displacement difference between the corresponding pixels in the left and right frames of the image, the larger the disparity value, the closer the target is to the camera, and the smaller the disparity value, the farther the target is from the camera); then based on the internal and external parameters of the binocular camera in the binocular ranging module (such as the focal length and optical center of the two cameras), the original disparity map is converted into a disparity value, and through these parameters, the disparity value is converted into an actual depth value through the triangulation formula, that is, the real distance of the target in the scene shot by the camera, thereby obtaining the shooting depth information, and obtaining the three-dimensional structure information of the entire scene.
[0034] Then, by analyzing the marked moving and stationary targets, the main shooting object and its environment are identified to determine the target shooting scene of the gimbal, and the background separation model of the corresponding scene type is matched from the preset model library, wherein the preset model library stores background separation models of various scene types, each model is optimized for specific scene features, such as indoor, outdoor, urban environment, natural landscape, etc.; then, the selected background separation model is used to process the initial moving target and the initial stationary target to separate the image foreground mask (which is a binary image, in which white pixels represent foreground targets, i.e., moving and stationary targets, and black pixels represent background), which is obtained by comparing the difference between the current frame and the background model; and based on the target shooting scene, morphological operations are performed on the separated image foreground mask, including expansion Dilation, erosion, opening and closing operations are performed to remove noise, fill small holes inside the target object, and smooth the target boundary, thereby improving the accuracy and completeness of the target area. Connected domain analysis is used to identify continuous areas in the mask, and the characteristics of the regions (such as area, compactness, center point, etc.) are used to determine whether they are the shooting targets to be found, thereby obtaining the initial shooting target; then the foreground mask is subtracted from the current frame using the background separation model to obtain an image containing only the background, and the noise of the separated background image is smoothed by applying Gaussian filtering or median filtering to remove the noise introduced by the target movement or image acquisition process, and then the background image is adjusted according to the motion information of the target object to reduce the blur or ghosting caused by rapid motion, and the initial shooting background is extracted from the original binocular image.
[0035] 103. Extracting the target image edge corresponding to the initial shooting target, and calculating the geometric distance of the initial shooting target based on the target image edge, and generating the target object feature;
[0036] In this embodiment, the target image edge here refers to the image edge feature of the photographed target; the geometric distance here refers to the original moment, central moment and normalized moment of the target object, which includes the shape, position, direction and scale information of the target object.
[0037] In practical applications, multiple edge attribute parameters corresponding to the initial photographed target are calculated, and based on the edge attribute parameters, an initial target edge corresponding to the initial photographed target is generated; then, pixel tracking of a closed contour is performed on the initial target edge to obtain at least one target image edge with a closed contour, and a target geometric distance of the target image edge is calculated; then, based on the target geometric distance, multiple target attribute parameters of the initial photographed target are determined, and based on the target attribute parameters, a geometric feature vector corresponding to the target shape is constructed; thus, the edge feature vector corresponding to the initial target edge and the geometric feature vector are weightedly fused to generate target object features.
[0038] 104. Determine a target offset of the initial shooting target based on the initial shooting background, target object features and shooting depth information, and predict a target prediction state of the initial shooting target in the next frame based on the target offset;
[0039] In this embodiment, the target offset here refers to the offset of the center of mass of the photographed target relative to the previous frame.
[0040] In practical applications, the target trajectory parameters of the initial shooting target in each frame are calculated based on the initial shooting background, target object features and shooting depth information, that is, the state of the target in the next frame is predicted through the Kalman filter. The prediction steps are expressed as: in, Is the current frame The state prediction, is the state transition matrix, is the state estimate of the previous frame, is the control matrix, is the control input, is process noise, where the trajectory parameters usually include the position, velocity, acceleration, etc. of the target; based on the target centroid coordinates and target trajectory parameters in the target attribute parameters, the centroid difference of the target in each frame is compared by calculating the position difference of the target centroid in two consecutive frames of images. The centroid difference calculation formula is: in, and The current frame and previous frame The target centroid coordinates are obtained to obtain the target offset, and it is determined whether the target offset is greater than the preset offset threshold. if , then the target offset is large. Here, The Euclidean norm of the center of mass difference (i.e., the preset offset threshold) is represented. If the target offset is not greater than the preset offset threshold, the target is considered to be moving smoothly. Based on the target center of mass coordinates and target trajectory parameters, the Kalman filter update step is used to fuse the observed data and the predicted data. The update formula is: ,in, is the observation value of the current frame, H is the observation matrix, K is the Kalman gain, and based on the determined target motion trajectory, the target prediction state of the next frame of the initial shooting target is predicted: ; If the target offset is greater than the preset offset threshold, it means that the target's movement is beyond the expected range, and it is necessary to re-evaluate the state of the target. Based on the target center of mass coordinates and target trajectory parameters, determine the target shape features and target posture of the initial shooting target, where the target shape features are a description of the target's appearance, and the target posture is a description of the target's direction and angle in three-dimensional space. By analyzing the shape features and posture of the target in the current frame, combined with the shape changes and posture changes of the target, such as a deep learning model, by learning the motion patterns in the historical data of the corresponding shooting scene in the database, the target prediction state of the initial shooting target in the next frame when the target posture is predicted.
[0041] 105. Based on the target prediction state and the target object characteristics, a tracking shooting parameter of the initial shooting target is generated through a three-axis tracking module, and based on the tracking shooting parameter, a shooting picture of the initial shooting target is centered and the initial shooting target is tracked and shot to obtain a target tracking shooting result.
[0042] In this embodiment, based on the target prediction state and the target object features, a three-axis tracking module is used to adopt a feature point matching rule to obtain a background motion vector by identifying feature points in the image and matching the position changes of these feature points in consecutive frames; and after obtaining the background motion vector, the background motion vector is applied to the target prediction state to adjust the position and posture of the target. For example, if the background motion vector indicates that the background moves to the right, then the position of the target also needs to be adjusted to the right accordingly to compensate for the movement of the background, so as to improve the accuracy of the target position prediction, so that the system can track the target more stably in the dynamic shooting environment; then based on the compensated target prediction state and background motion vector, by analyzing the pixel parallax in the left and right images, the depth information of the target in the three-dimensional space is calculated, and the motion displacement parameters of the relative position change of the shooting target at different time points are obtained, and the left and right images are aligned through image registration technology, and then the parallax of each pixel is calculated, and the depth information of the target is obtained through the principle of triangulation, and the motion displacement parameters are calculated by combining the background motion vector and the target prediction state; then based on the motion displacement parameters, the tracking shooting parameters of the initial shooting target are generated (where the tracking shooting parameters include the position, angle and camera focal length of the gimbal, etc.), and these parameters are used to adjust the posture and position of the camera so that the target is always in the center of the shooting picture. By adjusting the tracking shooting parameters in real time, it is ensured that the target always maintains a clear and stable picture throughout the shooting process. For example, if the target moves to the left, the system will adjust the camera angle and position so that the target is centered again. For example, the motor in the three-axis tracking module drives the movement of the gimbal, and the position and angle of the camera are continuously adjusted through a feedback loop to ensure that the target is always in the center of the shooting image.
[0043] In the embodiment of the present invention, parallax conversion and shooting background separation are performed on multiple frames of original binocular images collected by a binocular ranging module, and image edge detection and geometric aggregation calculation are performed on the extracted initial shooting target to obtain target object features, and then based on the initial shooting background, target object features and shooting depth information, the target offset of the initial shooting target is determined, and the target prediction state at the next moment is generated based on the offset of the initial shooting target, so as to generate corresponding tracking shooting parameters based on the target prediction state, and the three-axis tracking module is used to control the centering shooting and tracking shooting of the shooting target by the gimbal, thereby improving the tracking accuracy of the shooting target under a dynamic background.
[0044] See also Figure 2 The second embodiment of the target tracking and shooting method in the embodiment of the present invention includes:
[0045] 201. Collect multiple frames of original binocular images taken by the PTZ through a binocular ranging module, and perform multiple image enhancement on the original binocular images to obtain enhanced original binocular images;
[0046] 202. Identify initial moving targets and initial stationary targets at different frame rates in the enhanced original binocular image, perform parallax conversion on the enhanced original binocular image, generate shooting depth information, and determine the initial shooting target and the initial shooting background in the original binocular image based on the initial moving target and the initial stationary target using a preset background separation model;
[0047] 203. Calculate a plurality of edge attribute parameters corresponding to the initial shooting target, and generate an initial target edge corresponding to the initial shooting target based on the edge attribute parameters;
[0048] In this embodiment, edge detection is performed on the initial photographed target, that is, by calculating the gradient change of the image brightness, the part of the image with drastic grayscale changes is identified, thereby determining the edge of the target. The generated edge image contains the edge information of the initial photographed target, that is, the outline of the object, and then based on the edge image obtained by edge detection, a variety of edge attribute parameters corresponding to the initial photographed target are calculated. These attribute parameters include edge length, direction, curvature, gradient amplitude, etc. For example, the edge length is obtained by calculating the number of edge pixels; the edge direction is obtained by calculating the gradient direction; the edge curvature is calculated by analyzing the shape change of the edge; and the gradient amplitude represents the strength of the edge. By combining these edge attribute parameters, the edge features (that is, the initial target edge) that fully describe the initial photographed target are generated.
[0049] 204. Perform pixel tracking of a closed contour on the initial target edge to obtain at least one target image edge with a closed contour, and calculate a target geometric distance of the target image edge;
[0050] In this embodiment, the pixel tracking of the closed contour of the initial target edge is performed by image scanning and pattern recognition technology. For example, an active contour model (snakes) is used to track the edge and make it closed. The iterative update formula of the active contour model is: in, Indicates at time The outline of is the regularization parameter, It is the gradient of the energy function relative to the contour, so as to obtain at least one target image edge with a closed contour, and then use the closed target image edge to calculate the target geometric distance, such as area , Centroid , moment of inertia, etc. For example, the perimeter is obtained by calculating the total length of edge pixels; the area is obtained by calculating the number of pixels surrounded by the contour; the centroid is the center point of the target contour, which is obtained by calculating the average position of all edge pixels. The centroid calculation formula is as follows: in, are the coordinates of edge pixels, and n is the total number of pixels. The principal axis direction is calculated by the principal component analysis (PCA) method, which represents the main direction of the target contour. The compactness is the ratio of the contour area to the square of the perimeter, which is used to describe the shape complexity of the target.
[0051] 205. Based on the target geometric distance, determine multiple target attribute parameters of the initial shooting target, and construct a geometric feature vector corresponding to the target shape based on the target attribute parameters;
[0052] In this embodiment, based on the geometric features of the target, multiple target attribute parameters (such as shape parameters, position parameters, direction parameters, etc.) of the initial photographed target are determined. For example, shape parameters include the aspect ratio, roundness, rectangularity, etc. of the target; position parameters include the centroid coordinates, bounding box, etc. of the target; direction parameters include the main axis direction, rotation angle, etc. of the target. By combining these target attribute parameters, the geometric features of the target are fully described; and after obtaining the target attribute parameters, these parameters are combined into a vector to construct a geometric feature vector corresponding to the multi-dimensional vector of the target shape, where each dimension represents a geometric feature parameter.
[0053] 206. Perform weighted fusion on the edge feature vector and the geometric feature vector corresponding to the initial target edge to generate target object features;
[0054] In this embodiment, the edge feature vector (including the detailed information of the target edge) and the geometric feature vector (including the overall shape information of the target) corresponding to the initial target edge are subjected to nonlinear weighting and other methods to generate the target object features. These features not only include the edge information of the target, but also the geometric shape information of the target, providing a comprehensive feature description for the recognition and classification of photographic targets.
[0055] 207. Determine a target offset of the initial shooting target based on the initial shooting background, target object features and shooting depth information, and predict a target prediction state of the initial shooting target in the next frame based on the target offset;
[0056] 208. Based on the target prediction state and the target object features, a tracking shooting parameter of the initial shooting target is generated through a three-axis tracking module, and based on the tracking shooting parameter, a shooting picture of the initial shooting target is centered and the initial shooting target is tracked and shot to obtain a target tracking shooting result.
[0057] In the embodiment of the present invention, parallax conversion and shooting background separation are performed on multiple frames of original binocular images collected by a binocular ranging module, and image edge detection and geometric aggregation calculation are performed on the extracted initial shooting target to obtain target object features, and then based on the initial shooting background, target object features and shooting depth information, the target offset of the initial shooting target is determined, and the target prediction state at the next moment is generated based on the offset of the initial shooting target, so as to generate corresponding tracking shooting parameters based on the target prediction state, and the three-axis tracking module is used to control the centering shooting and tracking shooting of the shooting target by the gimbal, thereby improving the tracking accuracy of the shooting target under a dynamic background.
[0058] The above describes the target tracking and shooting method in the embodiment of the present invention. The following describes the target tracking and shooting device in the embodiment of the present invention. Figure 3 , an embodiment of the target tracking and shooting device in the embodiment of the present invention includes:
[0059] The image enhancement module 301 is used to collect multiple frames of original binocular images taken by the pan / tilt via the binocular ranging module, and perform multiple image enhancement on the original binocular images to obtain enhanced original binocular images;
[0060] The target recognition module 302 is used to recognize the initial moving target and the initial stationary target of different frame rates in the enhanced original binocular image, and perform parallax conversion on the enhanced original binocular image to generate shooting depth information, and determine the initial shooting target and the initial shooting background in the original binocular image based on the initial moving target and the initial stationary target using a preset background separation model;
[0061] The feature extraction module 303 is used to extract the target image edge corresponding to the initial shooting target, and calculate the geometric distance of the initial shooting target based on the target image edge to generate the target object feature;
[0062] A state prediction module 304 is used to determine a target offset of the initial shooting target based on the initial shooting background, the target object features and the shooting depth information, and predict a target prediction state of the initial shooting target in the next frame based on the target offset;
[0063] The tracking and shooting module 305 is used to generate tracking and shooting parameters of the initial shooting target through the three-axis tracking module based on the target prediction state and the target object characteristics, and based on the tracking and shooting parameters, center the shooting screen of the initial shooting target and track and shoot the initial shooting target to obtain the target tracking and shooting result.
[0064] In the embodiment of the present invention, the present application performs parallax conversion and shooting background separation on multiple frames of original binocular images collected by the binocular ranging module, and performs image edge detection and geometric aggregation calculation on the extracted initial shooting target to obtain target object features, and then determines the target offset of the initial shooting target based on the initial shooting background, target object features and shooting depth information, and generates a target prediction state at the next moment based on the offset of the initial shooting target, thereby generating corresponding tracking shooting parameters based on the target prediction state, and controlling the gimbal to perform centering shooting and tracking shooting of the shooting target through a three-axis tracking module, thereby improving the tracking accuracy of the shooting target under a dynamic background.
[0065] above Figure 3 The target tracking and shooting device in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The target tracking and shooting device in the embodiment of the present invention is described in detail from the perspective of hardware processing.
[0066] Figure 4 4 is a schematic diagram of the structure of a target tracking and shooting device provided by an embodiment of the present invention. The target tracking and shooting device 400 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 410 (for example, one or more processors) and a memory 420, and one or more storage media 430 (for example, one or more mass storage devices) storing application programs 433 or data 432. Among them, the memory 420 and the storage medium 430 can be short-term storage or permanent storage. The program stored in the storage medium 430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the target tracking and shooting device 400. Furthermore, the processor 410 may be configured to communicate with the storage medium 430 to execute a series of instruction operations in the storage medium 430 on the target tracking and shooting device 400.
[0067] The target tracking and shooting device 400 may also include one or more power supplies 440, one or more wired or wireless network interfaces 450, one or more input and output interfaces 460, and / or one or more operating systems 431, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 4 The target tracking and shooting device structure shown does not constitute a limitation on the target tracking and shooting device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.
[0068] The present invention also provides a target tracking and shooting device, wherein the computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes each step of the target tracking and shooting method in the above-mentioned embodiments.
[0069] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes each step of the target tracking and shooting method.
[0070] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0071] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.
[0072] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0073] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A target tracking shooting method, applied to a pan-tilt shooting tracking system, It is characterized in that The pan-tilt shooting and tracking system includes a binocular ranging module and a three-axis tracking module, and the target tracking shooting method includes: The binocular ranging module collects multiple frames of original binocular images taken by the pan / tilt head, and performs multiple image enhancement on the original binocular images to obtain enhanced original binocular images; Identifying initial moving targets and initial stationary targets at different frame rates in the enhanced original binocular image, performing parallax conversion on the enhanced original binocular image, generating shooting depth information, and determining an initial shooting target and an initial shooting background in the enhanced original binocular image based on the initial moving target and the initial stationary target using a preset background separation model; Extracting the target image edge corresponding to the initial photographed target, and calculating the geometric distance of the initial photographed target based on the target image edge to generate target object features; Determine a target offset of the initial shooting target based on the initial shooting background, the target object feature and the shooting depth information, and predict a target prediction state of the initial shooting target in a next frame based on the target offset; Based on the target prediction state and the target object features, the tracking shooting parameters of the initial shooting target are generated by the three-axis tracking module, and based on the tracking shooting parameters, the shooting screen of the initial shooting target is centered and the initial shooting target is tracked and shot to obtain the target tracking shooting result.
2. The target tracking shooting method according to claim 1, It is characterized in that The performing multiple image enhancement on the original binocular image to obtain an enhanced original binocular image includes: Performing image graying and median filtering on the original binocular image, and adjusting image parameters of the filtered original binocular image to obtain an adjusted original binocular image; The adjusted original binocular image is subjected to histogram equalization, and based on the calibration parameters of the binocular camera in the binocular ranging module, the equalized original binocular image is subjected to geometric correction of image distortion to obtain an enhanced original binocular image.
3. The target tracking shooting method according to claim 1, It is characterized in that The identifying of initial moving targets and initial stationary targets at different frame rates in the enhanced original binocular image, and performing parallax conversion on the enhanced original binocular image to generate shooting depth information includes: Based on a preset initial target area, identifying and marking the changed areas in the enhanced original binocular image at different frame rates, and generating potential moving targets and potential stationary targets; Matching multiple image features of the potential moving target and the potential stationary target, and based on the matching results, marking attributes of multiple regions of the potential moving target and the potential stationary target to obtain an initial moving target and an initial stationary target; Select the original binocular image enhanced by one purpose to perform pixel matching cost on the original binocular image enhanced by another purpose, and perform cost aggregation and disparity optimization on each pixel after matching cost to obtain the original disparity map; Based on the internal and external parameters of the camera in the binocular ranging module, the original disparity map is converted into a disparity value to obtain shooting depth information.
4. The target tracking shooting method according to claim 3, It is characterized in that The method of determining the initial shooting target and the initial shooting background in the enhanced original binocular image based on the initial moving target and the initial stationary target by using a preset background separation model includes: Based on the initial moving target and the initial stationary target, determining the target shooting scene shot by the gimbal, and matching a background separation model of a corresponding scene type from a preset model library based on the target shooting scene; Separating the image foreground masks in the initial moving target and the initial stationary target by using the background separation model, and performing morphological operations and target region extraction on the image foreground masks based on the target shooting scene to obtain an initial shooting target; The background separation model is used to separate the image frame of the initial shooting target from the image foreground mask, and noise smoothing and motion compensation are performed on the separated background image to obtain the initial shooting background.
5. The target tracking shooting method according to claim 1, It is characterized in that The step of extracting the target image edge corresponding to the initial photographed target, and calculating the geometric distance of the initial photographed target based on the target image edge to generate the target object feature includes: Calculating a plurality of edge attribute parameters corresponding to the initial shooting target, and generating an initial target edge corresponding to the initial shooting target based on the edge attribute parameters; Perform pixel tracking of a closed contour on the initial target edge to obtain at least one target image edge with a closed contour, and calculate a target geometric distance of the target image edge; Based on the target geometric distance, determining a plurality of target attribute parameters of the initial photographed target, and constructing a geometric feature vector corresponding to the target shape based on the target attribute parameters; The edge feature vector corresponding to the initial target edge and the geometric feature vector are weightedly fused to generate target object features.
6. The target tracking and shooting method according to claim 5, It is characterized in that The step of determining a target offset of the initial shooting target based on the initial shooting background, the target object feature and the shooting depth information, and predicting a target prediction state of the initial shooting target in a next frame based on the target offset includes: Calculating target trajectory parameters of the initial shooting target in each frame of the image based on the initial shooting background, the target object features and the shooting depth information; Based on the target centroid coordinates in the target attribute parameters and the target trajectory parameters, the centroid difference of the initial captured target in each frame of the image is compared to obtain a target offset, and it is determined whether the target offset is greater than a preset offset threshold; If the target offset is not greater than a preset offset threshold, determining the target motion trajectory of the initial shooting target based on the target centroid coordinates and the target trajectory parameters, and predicting the target prediction state of the next frame of the initial shooting target based on the target motion trajectory; If the target offset is greater than a preset offset threshold, the target shape features and target posture of the initial photographed target are determined based on the target center of mass coordinates and the target trajectory parameters, and based on the target shape features, the target prediction state of the initial photographed target in the next frame when the target posture is predicted.
7. The target tracking shooting method according to claim 1, It is characterized in that The step of generating tracking shooting parameters of the initial shooting target by the three-axis tracking module based on the target prediction state and the target object features includes: Based on the target prediction state and the target object features, the three-axis tracking module calculates the background motion vector of the pixel points in the continuous frame images, and uses the background motion vector to perform background compensation on the target prediction state to obtain the compensated target prediction state; Based on the compensated target prediction state and the background motion vector, the motion displacement parameters of the binocular ranging module are calculated, and based on the motion displacement parameters, the tracking shooting parameters of the initial shooting target are generated.
8. A target tracking shooting device, applied to a pan-tilt shooting tracking system, It is characterized in that The pan-tilt shooting and tracking system includes a binocular ranging module and a three-axis tracking module, and the target tracking shooting device includes: An image enhancement module is used to collect multiple frames of original binocular images taken by the pan / tilt platform through the binocular ranging module, and perform multiple image enhancement on the original binocular images to obtain enhanced original binocular images; a target recognition module, configured to recognize initial moving targets and initial stationary targets at different frame rates in the enhanced original binocular image, and to perform parallax conversion on the enhanced original binocular image to generate shooting depth information, and to determine an initial shooting target and an initial shooting background in the enhanced original binocular image based on the initial moving target and the initial stationary target using a preset background separation model; A feature extraction module, used to extract the edge of the target image corresponding to the initial shooting target, and based on the edge of the target image, calculate the geometric distance of the initial shooting target to generate the target object feature; A state prediction module, used to determine a target offset of the initial shooting target based on the initial shooting background, the target object features and the shooting depth information, and predict a target prediction state of the initial shooting target in a next frame based on the target offset; The tracking and shooting module is used to generate tracking and shooting parameters of the initial shooting target through the three-axis tracking module based on the target prediction state and the target object characteristics, and based on the tracking and shooting parameters, center the shooting picture of the initial shooting target and track and shoot the initial shooting target to obtain the target tracking and shooting result.
9. A target tracking and shooting device, It is characterized in that The target tracking and shooting device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the target tracking and shooting device executes each step of the target tracking and shooting method as described in any one of claims 1-7.
10. A computer-readable storage medium having instructions stored thereon, It is characterized in that When the instructions are executed by the processor, the steps of the target tracking and shooting method as described in any one of claims 1 to 7 are implemented.
