Dynamic target tracking system and method based on image detection and pan / tilt

Abstract

The present application relates to a dynamic target tracking system and method based on image detection and pan-tilt. The system includes a data processing terminal, a pan-tilt, a camera, a single-chip microcomputer and a motion unit. When performing target detection, the system collects and annotates target images and corresponding negative sample data under diverse environmental conditions for model training, and performs target detection through sub-graph partitioning processing, thereby improving the robustness, speed and detection accuracy of target detection. When performing dynamic target tracking, the deviation angle between the center point of the target annotation frame and the center point of the image screen is converted into a PWM signal for real-time and accurate control of the pan-tilt servo, and by introducing the pan-tilt speed intelligent coefficient and decision boundary, the pan-tilt movement is made smoother and more stable, and the target is continuously kept in the center of the image screen. The tracking distance between the pan-tilt and the target is monitored and maintained by a laser radar, so that the pan-tilt can stably track the moving target.

Description

Technical Field

[0001] The present application relates to the technical field of target detection and tracking, and in particular to a dynamic target tracking system and method based on image detection and a pan / tilt platform. Background Art

[0002] With the improvement of computing power and the development of deep learning algorithms, target detection and gimbal tracking technologies based on deep learning have made significant progress, and can quickly and accurately identify and locate objects. These technological advances have made gimbal tracking systems increasingly important in monitoring and tracking specific targets.

[0003] However, in terms of detection, target detection models are usually only effective in specific scenarios and cannot support all real-world targets. Therefore, it is necessary to fine-tune the existing models and train models suitable for specific applications to solve problems such as inaccurate target detection, low recognition rate, and poor robustness. In addition, most gimbal tracking systems also have some limitations. For example, traditional tracking methods usually fix the gimbal in one position, affecting the continuous monitoring of the target. In addition, during the movement of the gimbal, the rotation angle and accuracy of the servo and various environmental factors often cause the target to not be kept in the center of the picture. This non-smoothness of the movement may lead to the loss of the target, and after losing the target, the system fails to automatically find the target in time, further affecting the effectiveness of tracking.

[0004] Therefore, improving the motion control capability of the gimbal to ensure that it can flexibly and smoothly follow the target, while improving the adaptability of the target detection model in a dynamic environment, and achieving more efficient and reliable dynamic target tracking have become technical problems that need to be urgently solved by technical personnel in this field. Summary of the invention

[0005] Based on this, it is necessary to provide a dynamic target tracking system and method based on image detection and gimbal to address the above technical problems.

[0006] A dynamic target tracking system based on image detection and pan-tilt, the system comprising: a data processing terminal, a pan-tilt, a camera, a single-chip microcomputer and a motion unit;

[0007] Among them, the data processing terminal is connected to the camera and the single-chip microcomputer respectively, including a display, a control module and a target detection module; the target detection module is used to form a training data set with the pre-annotated collected images under diverse environmental conditions and the corresponding negative sample data and input them into the target detection model based on deep learning for training, and based on the trained target detection model, target detection is performed through sub-image division; the display is connected to the target detection module to display and output the target annotation box in the image; the control module is connected to the display to calculate and obtain the deviation angle between the center point of the target annotation box and the center point of the image screen, and by introducing the pan-tilt speed intelligent coefficient, the deviation angle is converted into a PWM signal and imported into the single-chip microcomputer;

[0008] The gimbal is fixedly connected to the camera. Through the movement of the servo in the gimbal, the camera is controlled to adjust the shooting angle, realize dynamic tracking and image acquisition of moving targets, and upload the acquired images to the target detection module in the data processing terminal;

[0009] The two ends of the single-chip microcomputer are respectively connected to the steering gear and the data processing terminal, and are used to forward the PWM signal input by the control module in the data processing terminal to the steering gear, and control the movement of the steering gear in combination with the PID algorithm deployed in the steering gear;

[0010] The motion unit with laser radar and automatic obstacle avoidance function is installed at the bottom of the gimbal. It is used to monitor and keep the tracking distance between the gimbal and the target within the set threshold range through the laser radar during the dynamic tracking of the moving target.

[0011] Furthermore, the camera is connected to the data processing terminal via a USB cable; the single-chip microcomputer is connected to the data processing terminal via a serial port and communicates with the data processing terminal.

[0012] Furthermore, the target detection module is used to form a training data set with the pre-annotated collected images under diverse environmental conditions and the corresponding negative sample data and input them into a target detection model based on deep learning for training, and perform target detection by sub-image division based on the trained target detection model, including:

[0013] The target detection module converts the pre-annotated collected images under diverse environmental conditions from JSON format to TXT format, obtains the TXT annotation file, and integrates the annotated collected images, the corresponding TXT annotation file and the corresponding negative sample data into the image and label directory of the COCO dataset to obtain a training dataset, and inputs the training dataset into a deep learning-based target detection model for training until a trained target detection model is obtained; wherein, the target detection model is based on the YOLO model;

[0014] The image captured by the camera in real time is input into the trained target detection model for target detection. The model first divides the input image into sub-images, and each obtained sub-image has overlapping parts; then, each sub-image is input into the YOLO model for detection in sequence. When the target is detected in the central area of ​​the sub-image, the target detection is ended and the next image is detected; when the target is detected in the edge area of ​​a sub-image, the adjacent sub-image of the edge area where the target is located is merged with the sub-image where the target is located into one sub-image, and target detection is performed again until the target is detected in the central area of ​​the sub-image, at which point the target detection is ended and the next image is detected; finally, after target detection is performed on all sub-images, the target coordinates obtained by detection are mapped to the initial input image according to the position coordinates of the sub-image, and the final target detection result is output to the display.

[0015] Furthermore, the control module is connected to the display to calculate the deviation angle between the center point of the target annotation frame and the center point of the image screen, and by introducing the intelligent coefficient of the pan / tilt speed, the deviation angle is converted into a PWM signal and imported into the single chip microcomputer, including:

[0016] The control module is connected to the display to calculate the horizontal position deviation and vertical position deviation between the center point of the target annotation frame and the center point of the image screen, and calculates according to the horizontal position deviation, vertical position deviation and the field of view angle in the corresponding direction to obtain the horizontal deviation angle and vertical deviation angle between the center point of the target annotation frame and the center point of the image screen, which are expressed as

[0017] ;

[0018] in, and Respectively represent the horizontal deviation angle and vertical deviation angle between the center point of the target annotation box and the center point of the image screen; and Respectively represent the horizontal position deviation and vertical position deviation between the center point of the target annotation box and the center point of the image screen; and Respectively represent the horizontal and vertical field of view of the camera; the resolution of the target image is , is the width, is the height; Indicates the field of view angle corresponding to each pixel in the horizontal direction, Indicates the field of view angle corresponding to each pixel in the vertical direction;

[0019] On the basis of obtaining the deviation angle between the center point of the target annotation frame and the center point of the image screen, the gimbal speed intelligent coefficient K is introduced, and the tracking speed of the gimbal is controlled by intelligently adjusting the K value to adapt to the dynamic tracking of the moving target; specifically, when the distance between the center point of the target annotation frame and the center point of the image screen increases, the deviation angle between the two increases synchronously, indicating that the target is deviating from the current line of sight center of the gimbal, and the K value is automatically increased to reduce the deviation angle, and after the deviation angle is converted into a PWM signal and imported into the single-chip microcomputer, the tracking speed of the gimbal is increased through the single-chip microcomputer control to ensure that the target remains in the center of the image screen; on the contrary, when the distance between the center point of the target annotation frame and the center point of the image screen decreases, the deviation angle between the two decreases synchronously, indicating that the target is approaching the center of the screen, and at this time, it is determined whether the K value is the initial value, if so, the initial value is maintained; if not, the K value is automatically reduced to increase the deviation angle, and after the deviation angle is converted into a PWM signal and imported into the single-chip microcomputer, the tracking speed of the gimbal is reduced through the single-chip microcomputer control until it is reduced to the initial value of the K value;

[0020] Among them, the relationship between the gimbal speed intelligent coefficient K and the gimbal tracking speed is expressed as: ,in, Indicates the gimbal tracking speed. It indicates the distance of a single rotation of the PTZ servo, and t indicates the time of a single rotation of the PTZ servo. When the t value remains unchanged, the distance of a single rotation of the servo is controlled by introducing the K value. , and then control the gimbal tracking speed.

[0021] Furthermore, the horizontal deviation angle between the center point of the target annotation box and the center point of the image screen is and vertical deviation angle The conversion relationship between the PWM signal value is expressed as

[0022] ;

[0023] Among them, the initial value of the PWM signal is 1500, that is, the zero position is 1500, and the range of the PWM signal is 500-2500.

[0024] Furthermore, the control module also efficiently locates the center point of the target annotation box by setting a decision boundary; specifically, when the gimbal dynamically tracks the moving target, it continuously monitors the horizontal deviation angle between the center point of the target annotation box and the center point of the image screen. and vertical deviation angle When the two deviation angles are within the preset threshold range respectively, it is determined that the center of the target annotation box is successfully positioned at the center of the image screen.

[0025] Further, the gimbal is composed of a first steering gear, a second steering gear, a first U-shaped beam, a second U-shaped beam, a first metal steering disc, a second metal steering disc, a long U bracket, a friction ring and two multifunctional brackets;

[0026] The two multifunctional brackets are fixed by eight M3 bolts and eight M3 nuts and placed horizontally. On the horizontal plane formed by the tops of the two multifunctional brackets, a second U-shaped beam is fixed by four M3 bolts and four M3 nuts. The second servo for controlling the horizontal movement of the gimbal is vertically loaded in the second U-shaped beam and connected and fixed to the second U-shaped beam by two M5 bolts and two M5 nuts on each side of the second servo;

[0027] The second metal steering disc is clamped directly above the second steering gear, and the second metal steering disc is connected and fixed to the first U-shaped beam through four M3 bolts. When the second steering gear is started, the first U-shaped beam moves in the horizontal direction;

[0028] The first servo for controlling the vertical movement of the gimbal is horizontally loaded inside the first U-shaped beam and is connected and fixed to the first U-shaped beam by two M5 bolts and two M5 nuts on each side of the first servo;

[0029] The first metal steering disc is clamped on the side of the first steering gear, and the first metal steering disc is connected and fixed to one side of the long U bracket by four M3 bolts, and the other side of the long U bracket is connected and fixed to the first U-shaped beam by an M3 bolt and an M3 nut, and a friction ring is installed at the connection between the long U bracket and the first U-shaped beam to increase friction. When the first steering gear is started, the long U bracket rotates in the vertical direction.

[0030] Furthermore, the camera is fixedly mounted on the top of the long U-bracket of the gimbal. When the camera encounters an unexpected situation and fails to detect the target in a certain frame of the captured image, it starts counting. When the target is not detected for thirty consecutive frames, it is determined that the target is lost, and a control instruction is sent through the control module in the data processing terminal to control the gimbal to continuously rotate back and forth within the extreme angle range in the horizontal direction until the camera re-locks the target; wherein, the extreme angle range of the gimbal in the horizontal direction is [-135°, 135°].

[0031] Furthermore, when the system is restarted, the gimbal automatically returns to the initial position, and the PWM signal output by the control module returns to the initial value.

[0032] A method for dynamic target tracking based on image detection and pan-tilt, the method is implemented based on the above-mentioned dynamic target tracking system based on image detection and pan-tilt, and comprises the following steps:

[0033] In the target detection module of the data processing terminal, the pre-annotated collected images under diverse environmental conditions and the corresponding negative sample data are combined into a training data set and input into a target detection model based on deep learning for training until a trained target detection model is obtained; wherein the target detection model performs target detection by sub-graph division;

[0034] The gimbal and camera are fixedly connected to perform dynamic tracking and image acquisition of moving targets. The motion unit with laser radar and automatic obstacle avoidance function installed at the bottom of the gimbal monitors the tracking distance between the gimbal and the target. If the tracking distance is outside the set threshold range, the motion unit moves the gimbal toward the target and automatically avoids obstacles during the movement.

[0035] If the tracking distance is within the set threshold range, the image currently captured by the camera is uploaded to the target detection module for target detection, and it is determined whether the display in the data processing terminal can output the target annotation frame. If not, it means that the target is not detected. Counting starts from the current frame. When the target is not detected for thirty consecutive frames, it is determined that the target is lost, and a control instruction is issued through the control module in the data processing terminal to control the gimbal to continue to rotate back and forth within the limit angle range in the horizontal direction until the camera re-locks the target and the display successfully outputs the target annotation frame in the captured image;

[0036] If yes, the control module calculates the deviation angle between the center point of the target annotation box and the center point of the image screen, and converts the deviation angle into a PWM signal by introducing the intelligent coefficient of the gimbal speed and imports it into the single-chip microcomputer. The single-chip microcomputer further connects to the gimbal servo, forwards the PWM signal to the servo, and controls the movement of the servo in combination with the PID algorithm deployed in the servo;

[0037] It is further determined whether the deviation angle between the center point of the target annotation frame and the center point of the image screen is within a preset threshold range. If so, it is determined that the center of the target annotation frame is successfully positioned at the center of the image screen; if not, the tracking speed of the gimbal is controlled by intelligently adjusting the value of the gimbal speed intelligent coefficient until the center of the target annotation frame is successfully positioned at the center of the image screen, and the position of the gimbal itself is further updated, the tracking distance between the gimbal and the target is continuously monitored, and the moving target is continuously tracked.

[0038] Compared with the prior art, the above-mentioned dynamic target tracking system and method based on image detection and pan / tilt has the following technical effects:

[0039] 1. When performing target detection, by collecting target images under diverse environmental conditions and the corresponding negative sample data and inputting them into the target detection model for training, the model can be made more robust and learn more features, thereby improving the robustness of target detection. Through sub-graph division processing, the detection speed and accuracy can be further improved.

[0040] 2. When the target is dynamically tracked based on the fixed connection between the gimbal and the camera, the control module calculates the deviation angle between the center point of the target annotation frame and the center point of the image screen, and by introducing the gimbal speed intelligent coefficient, the deviation angle is converted into a PWM signal and imported into the single-chip microcomputer. Based on the PWM signal forwarded by the single-chip microcomputer and the PID algorithm deployed in the servo, the real-time and precise control of the gimbal servo can be achieved. In addition, through the intelligent adjustment of the gimbal speed intelligent coefficient value, the acceleration and deceleration of the gimbal can be automatically controlled, overcoming the problem of uneven gimbal movement under certain circumstances, and continuously keeping the target in the center of the image screen.

[0041] 3. The control module also sets a decision boundary. When the deviation angle between the center point of the target annotation box and the center point of the image screen enters the preset threshold range, it determines that the center of the target annotation box is successfully positioned at the center of the image screen. This can prevent the gimbal from twitching continuously due to tiny errors, thereby ensuring the stability of the gimbal.

[0042] 4. Install the motion unit with laser radar and automatic obstacle avoidance function at the bottom of the gimbal. Use the laser radar to monitor and keep the tracking distance between the gimbal and the target within the set threshold range to avoid the impact of the target being too far or too close to the gimbal on the tracking, so that the gimbal can stably track the moving target.

[0043] 5. The system also has the function of automatically searching for the target after losing the target and automatically returning to zero after restarting, ensuring the convenience and consistency of system operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a structural schematic diagram of a dynamic target tracking system based on image detection and a pan / tilt system in one embodiment;

[0045] Figure 2 A schematic diagram of a process for intelligently adjusting the pan / tilt speed intelligent coefficient value in one embodiment;

[0046] Figure 3 A schematic diagram of a fixed connection between a pan / tilt head and a camera in one embodiment;

[0047] Figure 4 is a top view of a pan / tilt platform in one embodiment;

[0048] Figure 5 is a side view of a pan / tilt platform in one embodiment;

[0049] Figure numerals: camera 1; first servo 2; second servo 3; M3 bolt 4; second metal steering disc 5; first U-shaped beam 6; friction ring 7; long U bracket 8; first metal steering disc 9; second U-shaped beam 10; M3 nut 11; multi-function bracket 12; M5 nut 13; M5 bolt 14. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] In one embodiment, Figure 1 As shown, a dynamic target tracking system based on image detection and a pan-tilt system is provided, the system comprising: a data processing terminal, a pan-tilt system, a camera, a single-chip microcomputer and a motion unit.

[0052] Among them, the data processing terminal is connected to the camera and the single-chip microcomputer respectively, including a display, a control module and a target detection module; the target detection module is used to form a training data set with the pre-annotated collected images under diverse environmental conditions and the corresponding negative sample data and input them into the target detection model based on deep learning for training, and based on the trained target detection model, target detection is performed through sub-image division; the display is connected to the target detection module to display and output the target annotation box in the image; the control module is connected to the display to calculate and obtain the deviation angle between the center point of the target annotation box and the center point of the image screen, and by introducing the gimbal speed intelligent coefficient, the deviation angle is converted into a PWM (PulseWidth Modulation) signal and imported into the single-chip microcomputer.

[0053] The gimbal is fixedly connected to the camera. Through the movement of the servo in the gimbal, the camera is controlled to adjust the shooting azimuth angle, realize dynamic tracking and image acquisition of moving targets, and upload the acquired images to the target detection module in the data processing terminal.

[0054] The two ends of the single-chip microcomputer are respectively connected to the steering gear and the data processing terminal, and are used to forward the PWM signal input by the control module in the data processing terminal to the steering gear, and control the movement of the steering gear in combination with the PID algorithm deployed in the steering gear. Specifically, this application uses an STM32 single-chip microcomputer. Among them, the PID algorithm is a control algorithm that combines three links of proportion, integration and differentiation.

[0055] A motion unit (such as a car) with a laser radar and automatic obstacle avoidance function is installed at the bottom of the gimbal. It is used to monitor and keep the tracking distance between the gimbal and the target within the set threshold range through the laser radar during the dynamic tracking of the moving target. Specifically, by setting a minimum distance D1 and a maximum distance D2 to control the tracking distance between the gimbal and the target, it is avoided that when the target is too close, the target will occupy too large a proportion in the field of view, resulting in a situation where even a small movement may cause tracking failure. At the same time, it can also avoid the situation where the target is out of the detection range and cannot be detected when the target is too far away. By monitoring and keeping the tracking distance between the gimbal and the target within the set threshold range through the laser radar, it can ensure that the gimbal can continue to effectively track the moving target regardless of whether the target is too close or too far away.

[0056] Furthermore, the camera is connected to the data processing terminal via a USB cable; the single-chip microcomputer is connected to the data processing terminal via a serial port and communicates with the data processing terminal.

[0057] Furthermore, the target detection module converts the pre-annotated collected images under diverse environmental conditions from JSON format (JavaScript Object Notation, a lightweight data exchange format) to TXT format (text format), obtains the TXT annotation file, and integrates the annotated collected images with the corresponding TXT annotation file and the corresponding negative sample data into the image and label directory of the COCO dataset to obtain the training dataset. The training dataset is further input into the deep learning-based target detection model for training until a trained target detection model is obtained; wherein, the target detection model is based on the YOLO model.

[0058] It can be understood that the target detection module obtains target images under various environmental conditions captured by the camera, including indoor, outdoor and scenes under different lighting conditions, and accurately annotates the captured images to ensure that the size of the annotation box is accurate and not excessive, so as to facilitate subsequent target tracking. In addition, the annotation results are converted from JSON format to TXT format to adapt to model training requirements. The annotated images and their corresponding TXT annotation files are integrated into the image and label directories of the COCO dataset to ensure the integrity of the dataset, so that other images in the dataset can be used to fine-tune the model, configure the dataset path, and integrate it into the model training process. At the same time, during the model training process, in order to make the model more robust and learn more features, a large amount of related image negative sample data is also put in to improve the robustness of target detection.

[0059] The image captured by the camera in real time is input into the trained target detection model for target detection. The model first performs sub-image division processing on the input image. Each sub-image obtained has an overlapping part. Specifically, the 4k image can be divided into 16 sub-images. Then, each sub-image is input into the YOLO model for detection in sequence. When the target is detected in the central area of ​​the sub-image, the target detection is ended and the next image is detected. When the target is detected in the edge area of ​​a sub-image, the adjacent sub-image of the edge area where the target is located is merged with the sub-image where the target is located into a sub-image, and the target detection is performed again until the target is detected in the central area of ​​the sub-image. The target detection is ended and the next image is detected. Finally, after all sub-images are detected, the target coordinates obtained by detection are mapped to the initial input image according to the position coordinates of the sub-image, and the final target detection results are output to the display. The speed and accuracy of target detection can be improved through sub-image division processing.

[0060] Furthermore, the control module is connected to the display to calculate the horizontal position deviation and vertical position deviation between the center point of the target annotation frame and the center point of the image screen, and calculates according to the horizontal position deviation, vertical position deviation and the field of view angle in the corresponding direction to obtain the horizontal deviation angle and vertical deviation angle between the center point of the target annotation frame and the center point of the image screen, which are respectively expressed as

[0061] ;

[0062] in, and Respectively represent the horizontal deviation angle and vertical deviation angle between the center point of the target annotation box and the center point of the image screen; and Respectively represent the horizontal position deviation and vertical position deviation between the center point of the target annotation box and the center point of the image screen. If If it is greater than 0, it means the target is on the right side of the camera's field of view. is a positive number, if If it is less than 0, it means the target is on the left side of the camera's field of view. Is a negative number, if If it is greater than 0, it means the target is below the camera's field of view. is a positive number, if If it is less than 0, it means the target is above the camera's field of view. is a negative number; and Respectively represent the horizontal and vertical field of view of the camera; the resolution of the target image is , is the width, is the height; Indicates the field of view angle corresponding to each pixel in the horizontal direction, Indicates the field of view angle corresponding to each pixel in the vertical direction.

[0063] On the basis of obtaining the deviation angle between the center point of the target annotation frame and the center point of the image screen, the gimbal speed intelligent coefficient K is introduced, and the gimbal tracking speed is controlled by intelligently adjusting the K value to adapt to the dynamic tracking of the moving target; specifically, the intelligent adjustment process of the gimbal speed intelligent coefficient value is as follows: Figure 2 As shown in the figure, when the distance between the center point of the target annotation frame and the center point of the image screen increases, the deviation angle between the two increases synchronously, indicating that the target is deviating from the current line of sight of the gimbal. At this time, the K value is automatically increased to reduce the deviation angle, and after the deviation angle is converted into a PWM signal and imported into the single-chip microcomputer, the gimbal tracking speed is increased through the single-chip microcomputer control to ensure that the target remains in the center of the image screen; on the contrary, when the distance between the center point of the target annotation frame and the center point of the image screen decreases, the deviation angle between the two decreases synchronously, indicating that the target is approaching the center of the screen. At this time, it is determined whether the K value is the initial value. If so, the initial value is maintained; if not, the K value is automatically reduced to increase the deviation angle, and after the deviation angle is converted into a PWM signal and imported into the single-chip microcomputer, the gimbal tracking speed is reduced through the single-chip microcomputer control until it is reduced to the initial value of the K value. Among them, the relationship between the size of the gimbal speed intelligent coefficient K and the gimbal tracking speed is expressed as: ,in, Indicates the gimbal tracking speed. It indicates the distance of a single rotation of the PTZ servo, and t indicates the time of a single rotation of the PTZ servo. When the t value remains unchanged, the distance of a single rotation of the servo is controlled by introducing the K value. , and then control the gimbal tracking speed.

[0064] Introducing the K value can automatically control the acceleration and deceleration of the gimbal, making the gimbal movement smoother and keeping the target in the center of the image. Taking the horizontal direction as an example, when the deviation angle is H, because the rotation of the servo is controlled by the PWM signal, the angle is rounded up and only the angle of ⌈H×K⌉ is rotated. This angle is converted into the corresponding PWM value and imported, and then compared with the horizontal angle H in the next frame. 1 For comparison, if H 1 >H-⌈H×K⌉, when the angle between the target and the center of the picture increases, it indicates that the target is deviating from the current center of the gimbal's vision. The system will automatically increase the K value to increase the gimbal's tracking speed and ensure that the target remains in the center of the field of view. Continue to read the next frame. If H 2 <H 1 -⌈H 1×K⌉, where K is the increased K, that is, when the target approaches the center of the screen, the system gradually reduces the K value until it returns to the initial coefficient K to achieve fine tracking control; if the horizontal angle H is still decreasing under the initial coefficient K, the K value is kept unchanged. The same applies to the vertical direction.

[0065] Furthermore, the horizontal deviation angle between the center point of the target annotation box and the center point of the image screen is and vertical deviation angle The conversion relationship between the PWM signal value is expressed as

[0066] ;

[0067] The initial value of the PWM signal is 1500, that is, the zero position is 1500, and the range of the PWM signal is 500-2500. and vertical deviation angle The converted PWM signal can control the movement of the servo. At the same time, combined with the PID algorithm deployed inside the servo, the precise control of the servo rotation can be achieved through a feedback closed loop.

[0068] Furthermore, due to the influence of mechanical errors, control signal delays, environmental interference and other factors, it is often difficult for the servo to achieve completely accurate center point alignment, that is, the horizontal deviation angle and vertical deviation angle To address this challenge, the control module also sets a decision boundary to efficiently locate the center point of the target annotation box; specifically, when the gimbal dynamically tracks a moving target, it continuously monitors the horizontal deviation angle between the center point of the target annotation box and the center point of the image screen. and vertical deviation angle When the two deviation angles are within the preset threshold range, the center of the target annotation box is determined to be successfully positioned at the center of the image screen, preventing the gimbal from twitching due to tiny errors, thereby ensuring the stability of the gimbal.

[0069] Furthermore, if Figures 3 to 5 As shown, the gimbal is composed of a first steering gear 2, a second steering gear 3, a first U-shaped beam 6, a second U-shaped beam 10, a first metal steering plate 9, a second metal steering plate 5, a long U bracket 8, a friction ring 7 and two multifunctional brackets 12;

[0070] The two multifunctional brackets 12 are fixed and placed horizontally by eight M3 bolts 4 and eight M3 nuts 11. On the horizontal plane formed by the tops of the two multifunctional brackets 12, a second U-shaped beam 10 is fixed by four M3 bolts 4 and four M3 nuts 11. The second steering gear 3 for controlling the horizontal movement of the pan / tilt head is vertically loaded in the second U-shaped beam 10, and is connected and fixed to the second U-shaped beam 10 by two M5 bolts 14 and two M5 nuts 13 on each side of the second steering gear 3.

[0071] The second metal steering plate 5 is clamped on the top of the second steering gear 3, and the second metal steering plate 5 is connected and fixed to the first U-shaped beam 6 by four M3 bolts 4. When the second steering gear 3 is started, the first U-shaped beam 6 moves in the horizontal direction.

[0072] The first servo 2 for controlling the vertical movement of the gimbal is horizontally loaded inside the first U-shaped beam 6, and is connected and fixed to the first U-shaped beam 6 by two M5 bolts 14 and two M5 nuts 13 on each side of the first servo 2;

[0073] The first metal steering plate 9 is clamped on the side of the first steering gear 2, and the first metal steering plate 9 is connected and fixed to one side of the long U-bracket 8 by four M3 bolts 4, and the other side of the long U-bracket 8 is connected and fixed to the first U-shaped beam 6 by an M3 bolt 4 and an M3 nut 11, and a friction ring 7 is installed at the connection between the long U-bracket 8 and the first U-shaped beam 6 to increase friction. When the first steering gear 2 is started, the long U-bracket 8 rotates in the vertical direction.

[0074] Furthermore, the camera 1 is fixedly mounted on the top of the long U-bracket 8 of the gimbal. When the camera 1 encounters an unexpected situation and fails to detect the target in a certain frame of the captured image, it starts counting. When the target is not detected for thirty consecutive frames, it is determined that the target is lost, and a control instruction is issued through the control module in the data processing terminal to control the gimbal to continuously rotate back and forth within the extreme angle range in the horizontal direction until the camera 1 re-locks the target; wherein, the extreme angle range of the gimbal in the horizontal direction is [-135°, 135°].

[0075] The purpose of this is to prevent the twitching problem caused by the target not being detected in a certain frame due to various reasons, but being detected in the next frame. At the same time, the gimbal uses variable speed search, because through feedback, we can know the current position of the gimbal, that is, the PWM signal value, so in the process of re-searching, when the PWM signal value is 1400-1600, the gimbal moves in steps of 5, that is, the PWM signal changes by 5 in each frame, when the PWM signal value is 1300-1400 and 1600-1700, the gimbal moves in steps of 7, that is, the PWM signal changes by 7 in each frame, when the PWM signal value is 1000-1300 and 1700-2000, the gimbal moves in steps of 8, that is, the PWM signal changes by 8 in each frame, when the PWM signal value is 500-1000 and 2000-2500, the gimbal moves in steps of 10, that is, the PWM signal changes by 10 in each frame. According to the above operation, let the gimbal search slowly near the initial position. According to historical data, the most likely location of the target can be analyzed, and slow search can be performed in these areas first. This application uses the initial region. Normally, the frame rate per second FPS is set to 15, that is, 15 frames per second.

[0076] Furthermore, when the system restarts, the gimbal automatically returns to its initial position, and the PWM signal output by the control module returns to its initial value, ensuring the convenience and consistency of system operation.

[0077] In one embodiment, a dynamic target tracking method based on image detection and pan-tilt is provided. The method is implemented based on the above-mentioned dynamic target tracking system based on image detection and pan-tilt. The method comprises the following steps:

[0078] In the target detection module of the data processing terminal, the pre-annotated collected images under diverse environmental conditions and the corresponding negative sample data are combined into a training data set and input into a target detection model based on deep learning for training until a trained target detection model is obtained; wherein the target detection model performs target detection by sub-graph division;

[0079] The gimbal and camera are fixedly connected to perform dynamic tracking and image acquisition of moving targets. The motion unit with laser radar and automatic obstacle avoidance function installed at the bottom of the gimbal monitors the tracking distance between the gimbal and the target. If the tracking distance is outside the set threshold range, the motion unit moves the gimbal toward the target and automatically avoids obstacles during the movement.

[0080] If the tracking distance is within the set threshold range, the image currently captured by the camera is uploaded to the target detection module for target detection, and it is determined whether the display in the data processing terminal can output the target annotation frame. If not, it means that the target is not detected. Counting starts from the current frame. When the target is not detected for thirty consecutive frames, it is determined that the target is lost, and a control instruction is issued through the control module in the data processing terminal to control the gimbal to continue to rotate back and forth within the limit angle range in the horizontal direction until the camera re-locks the target and the display successfully outputs the target annotation frame in the captured image;

[0081] If yes, the control module calculates the deviation angle between the center point of the target annotation box and the center point of the image screen, and converts the deviation angle into a PWM signal by introducing the intelligent coefficient of the gimbal speed and imports it into the single-chip microcomputer. The single-chip microcomputer further connects to the gimbal servo, forwards the PWM signal to the servo, and controls the movement of the servo in combination with the PID algorithm deployed in the servo;

[0082] It is further determined whether the deviation angle between the center point of the target annotation frame and the center point of the image screen is within a preset threshold range. If so, it is determined that the center of the target annotation frame is successfully positioned at the center of the image screen; if not, the tracking speed of the gimbal is controlled by intelligently adjusting the value of the gimbal speed intelligent coefficient until the center of the target annotation frame is successfully positioned at the center of the image screen, and the position of the gimbal itself is further updated, the tracking distance between the gimbal and the target is continuously monitored, and the moving target is continuously tracked.

[0083] The above method can improve the motion control capability of the gimbal, ensuring that it can flexibly and smoothly follow the target. At the same time, it can improve the adaptability of the target detection model in a dynamic environment and achieve more efficient and reliable dynamic target tracking.

[0084] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A dynamic target tracking system based on image detection and pan / tilt, characterized in that: The system includes: a data processing terminal, a pan / tilt head, a camera, a single chip microcomputer and a motion unit; The data processing terminal is connected to the camera and the single-chip microcomputer respectively, and includes a display, a control module and a target detection module; the target detection module is used to form a training data set with the pre-annotated collected images under diversified environmental conditions and the corresponding negative sample data and input them into a target detection model based on deep learning for training, and perform target detection through sub-graph division based on the trained target detection model; the display is connected to the target detection module to display and output the target annotation box in the image; the control module is connected to the display to calculate and obtain the deviation angle between the center point of the target annotation box and the center point of the image screen, and by introducing the pan-tilt speed intelligent coefficient, the deviation angle is converted into a PWM signal and imported into the single-chip microcomputer; The pan-tilt is fixedly connected to the camera, and the camera is controlled to adjust the shooting azimuth angle through the movement of the servo in the pan-tilt, so as to realize dynamic tracking and image acquisition of the moving target, and upload the acquired image to the target detection module in the data processing terminal; The two ends of the single chip microcomputer are respectively connected to the steering gear and the data processing terminal, and are used to forward the PWM signal input by the control module in the data processing terminal to the steering gear, and control the movement of the steering gear in combination with the PID algorithm deployed in the steering gear; The motion unit with laser radar and automatic obstacle avoidance function is installed at the bottom of the gimbal, and is used to monitor and keep the tracking distance between the gimbal and the target within a set threshold range through the laser radar during the dynamic tracking of the moving target; The control module is connected to the display to calculate the deviation angle between the center point of the target annotation frame and the center point of the image screen, and converts the deviation angle into a PWM signal and imports it into the single-chip microcomputer by introducing the pan / tilt speed intelligent coefficient, including: The control module is connected to the display to calculate the horizontal position deviation and vertical position deviation between the center point of the target annotation frame and the center point of the image screen, and calculates according to the horizontal position deviation, vertical position deviation and the field of view in the corresponding direction to obtain the horizontal deviation angle and vertical deviation angle between the center point of the target annotation frame and the center point of the image screen, which are respectively expressed as ； in, and Respectively represent the horizontal deviation angle and vertical deviation angle between the center point of the target annotation box and the center point of the image screen; and Respectively represent the horizontal position deviation and vertical position deviation between the center point of the target annotation box and the center point of the image screen; and Respectively represent the horizontal and vertical field of view of the camera; the resolution of the target image is , is the width, is the height; Indicates the field of view angle corresponding to each pixel in the horizontal direction, Indicates the field of view angle corresponding to each pixel in the vertical direction; On the basis of obtaining the deviation angle between the center point of the target annotation frame and the center point of the image screen, the gimbal speed intelligent coefficient K is introduced, and the tracking speed of the gimbal is controlled by intelligently adjusting the K value to adapt to the dynamic tracking of the moving target; specifically, when the distance between the center point of the target annotation frame and the center point of the image screen increases, the deviation angle between the two increases synchronously, indicating that the target is deviating from the current line of sight center of the gimbal, and the K value is automatically increased to reduce the deviation angle, and after the deviation angle is converted into a PWM signal and imported into the single-chip microcomputer, the tracking speed of the gimbal is increased through the single-chip microcomputer control to ensure that the target remains in the center of the image screen; on the contrary, when the distance between the center point of the target annotation frame and the center point of the image screen decreases, the deviation angle between the two decreases synchronously, indicating that the target is approaching the center of the screen, and at this time, it is determined whether the K value is the initial value, if so, the initial value is maintained; if not, the K value is automatically reduced to increase the deviation angle, and after the deviation angle is converted into a PWM signal and imported into the single-chip microcomputer, the tracking speed of the gimbal is reduced through the single-chip microcomputer control until it is reduced to the initial value of the K value; Among them, the relationship between the gimbal speed intelligent coefficient K and the gimbal tracking speed is expressed as: ,in, Indicates the gimbal tracking speed. It indicates the distance of a single rotation of the PTZ servo, and t indicates the time of a single rotation of the PTZ servo. When the t value remains unchanged, the distance of a single rotation of the servo is controlled by introducing the K value. , and then control the gimbal tracking speed.

2. The dynamic target tracking system based on image detection and pan / tilt according to claim 1, characterized in that: The camera is connected to the data processing terminal via a USB cable; the single chip microcomputer is connected to the data processing terminal via a serial port and communicates with the data processing terminal.

3. The dynamic target tracking system based on image detection and pan / tilt according to claim 1, characterized in that: The target detection module is used to form a training data set by combining the pre-annotated collected images under diverse environmental conditions and the corresponding negative sample data and inputting them into a target detection model based on deep learning for training, and based on the trained target detection model, performing target detection by sub-image division, including: The target detection module converts the pre-annotated collected images under diverse environmental conditions from JSON format to TXT format, obtains TXT annotation files, and integrates the annotated collected images, the corresponding TXT annotation files, and the corresponding negative sample data into the image and label directory of the COCO dataset to obtain a training dataset, and inputs the training dataset into a deep learning-based target detection model for training until a trained target detection model is obtained; wherein the target detection model is based on the YOLO model; The image captured by the camera in real time is input into the trained target detection model for target detection. The model first performs sub-image processing on the input image, and each obtained sub-image has an overlapping part; then, each sub-image is sequentially input into the YOLO model for detection. When the target is detected in the central area of ​​the sub-image, the target detection is ended and the target detection is performed on the next image; when the target is detected in the edge area of ​​a certain sub-image, the adjacent sub-image of the edge area where the target is located and the sub-image where the target is located are merged into one sub-image, and the target detection is performed again until the target is detected in the central area of ​​the sub-image, and the target detection is ended and the target detection is performed on the next image; finally, after the target detection is performed on all sub-images, the target coordinates obtained by the detection are mapped to the initial input image according to the position coordinates of the sub-image, and the final target detection result is output to the display.

4. The dynamic target tracking system based on image detection and pan / tilt according to claim 1, characterized in that: The horizontal deviation angle between the center point of the target annotation box and the center point of the image screen and vertical deviation angle The conversion relationship between the PWM signal value is expressed as ； Among them, the initial value of the PWM signal is 1500, that is, the zero position is 1500, and the range of the PWM signal is 500-2500.

5. The dynamic target tracking system based on image detection and pan / tilt according to claim 1, characterized in that: The control module also efficiently locates the center point of the target annotation frame by setting a decision boundary; specifically, when the gimbal dynamically tracks the moving target, the horizontal deviation angle between the center point of the target annotation frame and the center point of the image screen is continuously monitored. and vertical deviation angle When the two deviation angles are within the preset threshold range respectively, it is determined that the center of the target annotation box is successfully positioned at the center of the image screen.

6. The dynamic target tracking system based on image detection and pan / tilt according to claim 1, characterized in that: The gimbal is composed of a first steering gear, a second steering gear, a first U-shaped beam, a second U-shaped beam, a first metal steering disc, a second metal steering disc, a long U bracket, a friction ring and two multifunctional brackets; The two multifunctional brackets are fixed by eight M3 bolts and eight M3 nuts and placed horizontally. On the horizontal plane formed by the tops of the two multifunctional brackets, a second U-shaped beam is fixed by four M3 bolts and four M3 nuts. The second steering gear that controls the horizontal movement of the gimbal is vertically loaded in the second U-shaped beam and is connected and fixed to the second U-shaped beam by two M5 bolts and two M5 nuts on each side of the second steering gear. The second metal steering disc is clamped directly above the second steering gear, and the second metal steering disc is connected and fixed to the first U-shaped beam through four M3 bolts. When the second steering gear is started, the first U-shaped beam moves in the horizontal direction; The first servo for controlling the vertical movement of the gimbal is horizontally loaded inside the first U-shaped beam, and is connected and fixed to the first U-shaped beam by two M5 bolts and two M5 nuts on both sides of the first servo; The first metal steering disc is clamped on the side of the first steering gear, and the first metal steering disc is connected and fixed to one side of the long U bracket through four M3 bolts, and the other side of the long U bracket is connected and fixed to the first U-shaped beam through an M3 bolt and an M3 nut, and a friction ring is installed at the connection between the long U bracket and the first U-shaped beam to increase friction. When the first steering gear is started, the long U bracket rotates in the vertical direction.

7. The dynamic target tracking system based on image detection and pan / tilt according to claim 6, characterized in that: The camera is fixedly mounted on the top of the long U bracket of the gimbal. When the camera encounters an unexpected situation and fails to detect the target in a certain frame of the captured image, it starts counting. When the target is not detected for thirty consecutive frames, it is determined that the target is lost, and a control instruction is issued through the control module in the data processing terminal to control the gimbal to continuously rotate back and forth within the extreme angle range in the horizontal direction until the camera re-locks the target; wherein, the extreme angle range of the gimbal in the horizontal direction is [-135°, 135°].

8. The dynamic target tracking system based on image detection and pan / tilt according to claim 7, characterized in that: When the system restarts, the pan / tilt automatically returns to the initial position, and the PWM signal output by the control module returns to the initial value.

9. A method for dynamic target tracking based on image detection and pan-tilt, the method being implemented based on the dynamic target tracking system based on image detection and pan-tilt according to any one of claims 1 to 8, characterized in that: The method comprises the following steps: In the target detection module of the data processing terminal, the pre-annotated collected images under diverse environmental conditions and the corresponding negative sample data are combined into a training data set and input into a target detection model based on deep learning for training until a trained target detection model is obtained; wherein the target detection model performs target detection by sub-graph division; Dynamic tracking and image acquisition of moving targets are performed through a fixedly connected gimbal and camera, and a motion unit with a laser radar and automatic obstacle avoidance function installed at the bottom of the gimbal monitors the tracking distance between the gimbal and the target. If the tracking distance is outside the set threshold range, the motion unit carries the gimbal toward the target and automatically avoids obstacles during the movement. If the tracking distance is within the set threshold range, the image currently captured by the camera is uploaded to the target detection module for target detection, and it is determined whether the display in the data processing terminal can output the target annotation box. If not, it means that the target is not detected. Counting starts from the current frame. When the target is not detected for thirty consecutive frames, it is determined that the target is lost, and a control instruction is issued through the control module in the data processing terminal to control the pan / tilt to continue to rotate back and forth within the limit angle range in the horizontal direction until the camera re-locks the target and the display successfully outputs the target annotation box in the captured image; If yes, the control module calculates the deviation angle between the center point of the target annotation box and the center point of the image screen, and converts the deviation angle into a PWM signal by introducing the intelligent coefficient of the gimbal speed and imports it into the single-chip microcomputer. The single-chip microcomputer is further connected to the gimbal steering gear, forwards the PWM signal to the steering gear, and controls the movement of the steering gear in combination with the PID algorithm deployed in the steering gear; It is further determined whether the deviation angle between the center point of the target annotation frame and the center point of the image screen is within a preset threshold range. If so, it is determined that the center of the target annotation frame is successfully positioned at the center of the image screen; if not, the tracking speed of the gimbal is controlled by intelligently adjusting the value of the gimbal speed intelligent coefficient until the center of the target annotation frame is successfully positioned at the center of the image screen, and the position of the gimbal itself is further updated, and the tracking distance between the gimbal and the target is continuously monitored to continuously track the moving target; The control module calculates and obtains the deviation angle between the center point of the target annotation frame and the center point of the image screen, and converts the deviation angle into a PWM signal and imports it into the single-chip microcomputer by introducing the intelligent coefficient of the pan / tilt speed, including: The control module is connected to the display to calculate the horizontal position deviation and vertical position deviation between the center point of the target annotation frame and the center point of the image screen, and calculates according to the horizontal position deviation, vertical position deviation and the field of view in the corresponding direction to obtain the horizontal deviation angle and vertical deviation angle between the center point of the target annotation frame and the center point of the image screen, which are respectively expressed as ； in, and Respectively represent the horizontal deviation angle and vertical deviation angle between the center point of the target annotation box and the center point of the image screen; and Respectively represent the horizontal position deviation and vertical position deviation between the center point of the target annotation box and the center point of the image screen; and Respectively represent the horizontal and vertical field of view of the camera; the resolution of the target image is , is the width, is the height; Indicates the field of view angle corresponding to each pixel in the horizontal direction, Indicates the field of view angle corresponding to each pixel in the vertical direction; On the basis of obtaining the deviation angle between the center point of the target annotation frame and the center point of the image screen, the gimbal speed intelligent coefficient K is introduced, and the tracking speed of the gimbal is controlled by intelligently adjusting the K value to adapt to the dynamic tracking of the moving target; specifically, when the distance between the center point of the target annotation frame and the center point of the image screen increases, the deviation angle between the two increases synchronously, indicating that the target is deviating from the current line of sight center of the gimbal, and the K value is automatically increased to reduce the deviation angle, and after the deviation angle is converted into a PWM signal and imported into the single-chip microcomputer, the tracking speed of the gimbal is increased through the single-chip microcomputer control to ensure that the target remains in the center of the image screen; on the contrary, when the distance between the center point of the target annotation frame and the center point of the image screen decreases, the deviation angle between the two decreases synchronously, indicating that the target is approaching the center of the screen, and at this time, it is determined whether the K value is the initial value, if so, the initial value is maintained; if not, the K value is automatically reduced to increase the deviation angle, and after the deviation angle is converted into a PWM signal and imported into the single-chip microcomputer, the tracking speed of the gimbal is reduced through the single-chip microcomputer control until it is reduced to the initial value of the K value; Among them, the relationship between the gimbal speed intelligent coefficient K and the gimbal tracking speed is expressed as: ,in, Indicates the gimbal tracking speed. It indicates the distance of a single rotation of the PTZ servo, and t indicates the time of a single rotation of the PTZ servo. When the t value remains unchanged, the distance of a single rotation of the servo is controlled by introducing the K value. , and then control the gimbal tracking speed.

Images