A method for solving the target loss of a live follow-up holder

Abstract

The application provides a method for solving the problem of live tracking gimbal target loss: S1 acquires video image data; S2, tracking condition a: if the determination result is TRUE, directly enter the KCF tracking module, otherwise enter S3 judgment; S3 motor stop state: if TRUE, enter S4, otherwise continue to execute S1; S4 detection link; S5 tracking condition b: if the tracking condition b is TRUE, enter S6, otherwise continue S1; S6 tracking link: including KCF tracking module, tracking verification module, verification frequency setting module and updating tracking condition a; wherein the tracking verification mechanism composed of the tracking verification module and the verification frequency setting module adopts a local CNN human body detection algorithm, verifies the actual size of the human body frame and synchronously updates the KCF parameters, and only performs human body detection in the region of interest; S7, gimbal control link: including calculating gimbal control signal module, motor execution module, acquiring motor state module; and updating motor stop state. Solve the problem of tracking target loss.

Description

Technical Field

[0001] This invention relates to the field of intelligent video processing technology, and in particular to a method for solving the problem of target loss in live streaming follow-up shooting with a gimbal. Background Technology

[0002] With the development of computer technology and the widespread application of computer vision principles, the use of computer image processing technology for real-time target detection and tracking has become increasingly popular. Dynamic real-time tracking and localization of targets for human detection in intelligent AI image recognition systems and intelligent monitoring systems has broad application value. A common system application is the use of the OpenCV HOG algorithm to detect human figures, and then controlling the gimbal rotation based on the deviation of the human figure detection results to achieve real-time tracking.

[0003] However, existing methods for detecting humanoid figures have some drawbacks, as follows:

[0004] 1. The human tracking method based on OpenCV HOG algorithm has several drawbacks. First, the algorithm is relatively simple and cannot adapt to human figures in various lighting conditions and poses, which can easily lead to missed detections and false detections. Second, it does not consider how to select the best tracking target when multiple human figures appear in the camera's field of view. Third, when human detection fails or there is a human figure in the frame but the human detection has not provided a result, simply waiting for the next correct result can easily result in losing the moving human figure and causing tracking failure.

[0005] 2. In the KCF algorithm tracking, the target scale does not change with the size of the target. This can easily lead to a large scale change in the actual target area when the tracked target moves significantly relative to the camera. However, since KCF cannot change the scale of the detection result in real time, two phenomena occur: First, the detection box is smaller than the actual target area, resulting in the loss of a lot of effective data and the target is easily lost. Second, the detection box is larger than the actual target area, which can easily introduce a lot of noise data and cause the tracked target to be lost.

[0006] In addition, the following are commonly used terms in the prior art:

[0007] 1. Motion Detection: Based on continuous video image data, determine whether there is a moving target in the image. If so, the coordinate position of the rectangle will be returned in the form of a rectangle.

[0008] 2. CNN Human Detection: Based on YOLOv3-tiny and other CNN convolutional neural networks, human figures are detected in video images, and the bounding boxes of the human figures are calculated.

[0009] 3. Two-axis gimbal: A device equipped with a camera that can achieve two degrees of freedom of movement.

[0010] 4. KCF Tracking Algorithm: Kernel Correlation Filter (KCF) is a fast tracking algorithm that can adjust the bounding box of a target object based on a rectangular region in an image. Summary of the Invention

[0011] To address the problems in the prior art, the purpose of this application is to propose a new control flow:

[0012] 1. Using Yolov3-tiny for human detection can perform human detection efficiently and accurately.

[0013] 2. Using the KCF tracking algorithm, the humanoid results can be updated in real time, and the algorithm can be run on low-power chips.

[0014] 3. A tracking verification mechanism is used to solve the problem of target loss caused by the KCF tracking algorithm's inability to change the target bounding box scale.

[0015] 4. We propose using motion detection results as target optimization and false human detection filtering, which can enable human tracking in complex scenarios.

[0016] 5. Use motion detection to effectively improve target tracking when human detection fails, and improve the situation of losing human tracking.

[0017] Specifically, the present invention provides a method for solving the problem of target loss in live streaming follow-up shooting, the method comprising the following steps:

[0018] S1. Acquire video image data: Obtain real-time bitstream data through the camera installed on the PTZ to obtain image data for each frame;

[0019] S2, Tracking condition a: Determine whether to directly enter the KCF tracking module. If the determination result is TRUE, then directly enter the KCF tracking module; otherwise, enter the motor stop state determination.

[0020] S3, Motor Stopped Status: If TRUE, proceed to the S4 detection step; otherwise, continue with step S1.

[0021] S4. Detection process: including motion detection module, human detection module, and target selection module;

[0022] S5. Tracking condition b: If the tracking condition b is TRUE, proceed to the S6 tracking step; otherwise, continue with step S1 to acquire image data.

[0023] S6. Tracking stage: including: KCF tracking module, tracking verification module, verification frequency setting module; and updating tracking condition a; among which, the tracking verification mechanism composed of the tracking verification module and the verification frequency setting module adopts the local CNN human detection algorithm to verify the actual size of the human frame and synchronously update the KCF parameters so as to improve the KCF tracking effect. The local CNN human detection algorithm only performs human detection within the region of interest, meeting the speed requirement of tracking.

[0024] S7. PTZ control stage: including calculating the PTZ control signal module, motor execution module, obtaining the motor status module; and updating the motor stop status.

[0025] The step S4 further includes:

[0026] S4.1. Moving detection module: Using the image frame difference method with a time interval of T frames to implement moving target detection, determining whether there is a moving target in the image. If so, retain the result, i.e., the position of the rectangular box, for use in the optimization process.

[0027] S4.2. Human detection module: Using the Yolov3-tiny CNN convolutional neural network to perform human detection on the current image. If a human is detected, retain all the detection results, i.e., the rectangular boxes, for use in the optimization process.

[0028] S4.3. Target optimization module: Comprehensive optimization of the results of moving detection and human detection to obtain the final tracking target.

[0029] In the target optimization module in S4.3, the specific optimization logic is as follows:

[0030] Case 1: When only the moving detection has a result, first compare the width and height b_wh of the rectangular box of the detection result with the preset values min_wh < b_wh < max_wh. Retain the rectangular boxes within the condition range, and select the one with the largest area among the retained rectangular boxes as the final rectangular box.

[0031] Case 2: When only the human detection result exists, since no moving detection result is detected, it means there is no moving target in the picture, and there is no need to readjust the position of the PTZ. Set the tracking condition b to FALSE.

[0032] Scenario 3: When both human detection results and motion detection results are available, perform a bitwise AND operation on the motion detection bounding box and the human detection bounding box. After the AND operation, retain the human detection bounding box that intersects and the intersection reaches a preset value, and record it as B1. Otherwise, set the tracking condition b to FALSE. If the current preset tracking mode is to prioritize tracking historical targets, perform a bitwise AND operation on B1 and the historical human detection bounding box B2. The target with the largest intersection is taken as the final target detection box, and the tracking condition b is set to TRUE. If there is no intersection, the tracking condition b is set to FALSE. If the tracking mode is to prioritize tracking moving human targets, select the largest area in B1 as the final human result and set the tracking condition b to TRUE.

[0033] In case 1, min_wh is 50 and max_wh is frame_h / 2; frame_h is the height of the entire frame image; in case 3, the preset value is 0.5.

[0034] Step S6 further includes:

[0035] S6.1 Perform KCF tracking module: take the input rectangle and the current image together as input data for KCF algorithm; and obtain the result of KCF algorithm, which includes the tracking status S, indicating whether the tracking failed, 0 indicates failure, 1 indicates success, and the adjusted output rectangle B3;

[0036] S6.2 Update tracking condition a: Set tracking condition a to FALSE if tracking status S=0, and set tracking condition to TRUE if S=1;

[0037] S6.3, Perform tracking verification module: Input the target box box from the KCF tracking module, expand the target box width horizontally by kw times and expand the target box height vertically by kh times to obtain a new target box newbox. Use newbox to extract the data of interest from the image data, and use it as the input data for the CNN human detection algorithm. Obtain all human target boxes pboxes in the region of interest through forward inference. Use the IOU method to calculate the intersection-union ratio between pboxes and box, take the pbox with the largest intersection-union ratio score as the verified target box, and re-perform KCF tracking with pbox as the target box;

[0038] S6.4 Verification Frequency Setting Module: This module determines the execution frequency of the tracking verification module, including the frame count parameter `check_count` and the verification frequency parameter `check_freq`. Each time the KCF tracking module is entered, `check_count` is incremented by 1, and it is checked whether `check_count` is greater than or equal to `check_freq` (`check_freq check_count>=check_freq`). If the condition is met, the tracking verification module is executed, and `check_count` is set to 0; otherwise, the KCF tracking module continues. In step S6.3, the empirical value of `kw` is set to 2.0, and `kh` is set to 1.2 times the empirical value. These `kw` and `kh` can be adjusted according to different resolutions and actual conditions.

[0039] Step S7 further includes:

[0040] S7.1 Calculate the gimbal control signal module: Take the rectangle B3 obtained in step S6.1, and take the deviation dx and dy between the center point of B3 and the center point of the image. The center point is the origin of the coordinate system. dx represents the deviation of the center point of B3 from the center point of the image on the horizontal axis in pixels, and dy represents the deviation of the center point of B3 from the center point of the image on the vertical axis in pixels.

[0041] S7.2, Motor Actuation Module: Based on the dx and dy calculated in S7.1, the motor control signal step size or rotation speed is converted, and the motor actuator performs the action.

[0042] S7.3, Motor Status Acquisition Module: Mainly acquires the motor actuator stop signal STOP. STOP=1 indicates that the motor is in a stopped state, and STOP=0 indicates that the motor is in motion.

[0043] S7.4 Update motor stop status: Based on the status obtained in S7.2, if STOP=0, set the motor stop status to FALSE; if STOP=2, set the motor stop status to TRUE.

[0044] Therefore, the advantage of this application is:

[0045] 1. This application proposes a method to solve the problem of target tracking loss in live streaming gimbals.

[0046] 2. This application uses a fast and accurate Yolov3-tiny CNN convolutional neural network for human detection, which can greatly improve the detection effect and effectively improve the human detection results in various lighting scenes, various human poses, and situations where human figures are not fully displayed.

[0047] 3. This application adopts the KCF algorithm. When the human detection result is obtained, the result is fed into the KCF algorithm with the human result as the center to track the result. The execution efficiency of the KCF algorithm is about ten times faster than that of the YOLOv3-tiny algorithm. It can achieve real-time updates of human results and can run on low-power embedded chips such as INGENIC T-series chips and achieve real-time effect.

[0048] 4. This application incorporates a self-developed motion detection algorithm, which can effectively solve the problem of false detection of some stationary human-like figures. Furthermore, based on the motion detection algorithm results, it can prioritize tracking moving targets, allowing the focus to be placed on dynamic human figures.

[0049] 5. This application uses a motion detection algorithm to compensate for the failure of human detection, i.e., when there is a moving human figure in the frame but no feedback is received. The motion detection result can replace the human detection result, and the motor control of the gimbal can ensure that the moving target is always in the video field of view, so as to ensure that the target person is still in the image when the next human detection is performed.

[0050] 6. This application incorporates a tracking verification mechanism, which effectively solves the problem of target loss caused by the KCF algorithm's inability to update the target bounding box in real time during the tracking process. This tracking mechanism mainly adopts a local CNN human detection algorithm, which can verify the actual size of the human bounding box and update the KCF parameters synchronously, thereby effectively improving the KCF tracking effect. By using a local CNN human detection algorithm, human detection is only performed within the region of interest, which can improve detection efficiency and meet the speed requirements of tracking. Attached Figure Description

[0051] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0052] Figure 1 This is a flowchart of the method of the present invention.

[0053] Figure 2 This is a schematic diagram of the electric vehicle detection system in an embodiment of the present invention. Detailed Implementation

[0054] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0055] like Figure 1 As shown, this application relates to a method for solving the problem of target loss in live streaming with a gimbal, the method comprising the following steps:

[0056] S1. Acquire video image data: Obtain real-time bitstream data through the camera installed on the PTZ to obtain image data for each frame;

[0057] S2, Tracking condition a: Determine whether to directly enter the KCF tracking module. If the determination result is TRUE, then directly enter the KCF tracking module; otherwise, enter the motor stop state determination.

[0058] S3, Motor Stopped Status: If TRUE, proceed to the S4 detection step; otherwise, continue with step S1.

[0059] S4. Detection process: including motion detection module, human detection module, and target selection module;

[0060] S5. Tracking condition b: If the tracking condition b is TRUE, proceed to the S6 tracking step; otherwise, continue with step S1 to acquire image data.

[0061] S6. Tracking process: includes: KCF tracking module, tracking verification module, verification frequency setting module; and updating tracking condition a; among which, the tracking verification mechanism consisting of the tracking verification module and the verification frequency setting module adopts a local CNN human detection algorithm to verify the actual size of the human bounding box and update the KCF parameters synchronously to improve the KCF tracking effect. The local CNN human detection algorithm only performs human detection in the region of interest to meet the speed requirements of tracking.

[0062] Among them, the local CNN human detection algorithm: The CNN human detection algorithm is based on CNN convolutional neural networks such as Yolov3-tiny to detect human figures in video images and calculate the rectangular box of the human figure. The local human detection algorithm does not use the whole image for detection input image, but extracts the area to be detected for CNN human detection, which will greatly improve efficiency.

[0063] Among them, the region of interest (ROI) is defined in machine vision and image processing as a region that needs to be processed from the image, using shapes such as rectangles, circles, ellipses, and irregular polygons. In machine vision software such as Halcon, OpenCV, and Matlab, various operators and functions are often used to find the ROI and perform further image processing.

[0064] S7. Gimbal Control Module: This includes the module for calculating gimbal control signals, the motor execution module, the module for acquiring motor status, and the module for updating the motor stop status.

[0065] For ease of understanding, such as Figure 2 As shown, the method of this application can also be described as follows:

[0066] 1. Obtain video image data: Obtain real-time stream data through the camera installed on the pan-tilt, and obtain each frame of image data.

[0067] 2. Tracking condition a: Determine whether to directly enter the KCF tracking module. If the determination result is TRUE, directly enter the KCF tracking module; otherwise, enter the motor stop state determination.

[0068] Tracking condition a: Judge the situation of the state machine. The states that the state machine can represent are: detection state, tracking state, and waiting state. If it is judged that the current state machine is in the tracking state, the result is TRUE.

[0069] 3. Motor stop state: If it is TRUE, enter the detection link; otherwise, continue to execute 1.

[0070] 4. Motion detection module: Use the method of frame difference of images separated by T frames to implement moving target detection. Judge whether there is a moving target in the image. If there is, retain the result (the position of the rectangular box) for use in optimization.

[0071] Among them, the frame difference method of images: Select the current frame and record it as F n , select another frame after an interval of T frames and record it as F n+T , for F n , F n+T , take the absolute value of the difference of the gray values of the corresponding pixel points to obtain the difference image Fd, then perform thresholding processing to obtain a binary image, and then perform morphological processing to obtain the final binary image F. Perform contour search on the binary image F to obtain n (n is an integer greater than or equal to 0) contour data. Calculate the minimum bounding rectangle for the n contours to obtain n rectangular boxes. When the number of rectangular boxes is not zero, it is considered that there is a moving target. The rectangular box represents the position information of the moving target on the image; otherwise, no moving target appears.

[0072] 5. Human detection module: Use the Yolov3-tiny CNN convolutional neural network to perform human detection on the current picture. If a human is detected, retain all the detected human results (rectangular boxes) for use in optimization.

[0073] 6. Optimization module: Optimize the results of motion detection and human detection comprehensively to obtain the final tracking target. The specific optimization logic is as follows:

[0074] Case 1: When only motion detection has a result, first compare the width and height b_wh of the detection result rectangular box with the preset values (empirical values min_wh: 50, max_wh: frame_h / 2; frame_h is the height of the entire frame image). min_wh < b_wh < max_wh. Retain the rectangular boxes within the condition range, and select the one with the largest area among the retained rectangular boxes as the final rectangular box.

[0075] Scenario 2: When only human detection results are available, since no motion detection results are detected, it means that no moving target appears in the picture. There is no need to readjust the position of the gimbal. Set the tracking condition b to FALSE.

[0076] Scenario 3: When both human detection and motion detection results are available, perform a bitwise AND operation on the motion detection bounding box and the human detection bounding box. After the AND operation, retain the human detection bounding box (B1) with an intersection that reaches a preset value (empirical value 0.5). Otherwise, set tracking condition b to FALSE. If the current preset tracking mode prioritizes tracking historical targets, perform a bitwise AND operation on B1 and the historical human detection bounding box B2. The target with the largest intersection is selected as the final target detection box, and tracking condition b is set to TRUE. If there is no intersection, tracking condition b is set to FALSE. If the tracking mode prioritizes tracking moving human targets, select the largest area in B1 as the final human result and set tracking condition b to TRUE.

[0077] 7. Tracking condition b: If tracking condition b is TRUE, proceed to the KCF tracking module; otherwise, continue with step 1 to acquire image data.

[0078] 8. KCF Tracking Module: This module takes the input bounding box and the current image as input data for the KCF algorithm. It then obtains the KCF algorithm result, which includes the tracking status S (whether tracking failed, 0 for failure, 1 for success) and the adjusted output bounding box B3.

[0079] 9. Update tracking condition a: Set tracking condition a to FALSE if tracking status S=0, and set tracking condition to TRUE if S=1.

[0080] 10. Tracking and Verification Module: The module takes the target bounding box (box) from the KCF tracking module as input. It expands the target bounding box horizontally by a factor of kw and vertically by a factor of kh to obtain a new target bounding box (newbox). The empirical values ​​for kw and kh are set to 2.0 and 1.2 respectively. These kw and kh can be adjusted based on different resolutions and actual conditions. The newbox is used to extract the data of interest from the image data and serves as the input data for the CNN human detection algorithm. Through forward inference, all human target bounding boxes (pboxes) within the region of interest are obtained. The Intersection over Union (IOU) method is used to calculate the intersection-union ratio (IoU) between the pboxes and the boxes. The pbox with the highest IoU score is selected as the verified target bounding box, and KCF tracking is performed again using this pbox as the target bounding box. IoU is the ratio of the intersection to the union of the predicted bounding box and the ground truth bounding box.

[0081] 11. Verification Frequency Setting Module: This module determines the execution frequency of the tracking verification module. It includes the frame count parameter `check_count` and the verification frequency parameter `check_freq`. Each time the KCF tracking module is entered, `check_count` is incremented by 1, and it is checked whether `check_count` is greater than or equal to `check_freq` (`check_freq check_count>=check_freq`). If the condition is met, the tracking verification module is executed, and `check_count` is set to 0. Otherwise, the KCF tracking module continues.

[0082] 12. Calculate the gimbal control signal module: Take the rectangle B3 obtained in step 8, and take the deviation dx and dy between the center point of B3 and the center point of the image (the center point is the origin of the coordinate system). Here, dx represents the deviation of the center point of B3 from the center point of the image on the horizontal axis in pixels, and dy represents the deviation of the center point of B3 from the center point of the image on the vertical axis in pixels.

[0083] 13. Motor Actuation Module: Based on the dx and dy calculated in section 12, this module converts them into control signal step size or rotational speed for the motor. The motor actuator performs the actions.

[0084] 14. Motor Status Acquisition Module: This module mainly acquires the STOP signal of the motor actuator. STOP=1 indicates that the motor is in a stopped state, and STOP=0 indicates that the motor is in motion.

[0085] 15. Update motor stop status: Based on the status obtained in step 14, set the motor stop status to FALSE if STOP=0, and set the motor stop status to TRUE if STOP=2.

[0086] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for solving the problem of target loss in live streaming with a gimbal, characterized in that, The method includes the following steps: S1. Obtain video image data: Obtain real-time stream data through a camera installed on a pan-tilt, and obtain image data for each frame. S2. Tracking condition a: Determine whether to directly enter the KCF tracking module. If the determination result is TRUE, directly enter the KCF tracking module in step S6; otherwise, enter step S3 to judge the motor stop state. The tracking condition a: Judge the situation of the state machine. The states represented by the state machine are: detection state, tracking state, and waiting state. If it is judged that the current state machine is in the tracking state, the result is TRUE. S3. Motor stop state: If it is TRUE, enter step S4 for the detection link; otherwise, continue to execute step S1. S4. Detection link: Includes a motion detection module, a human detection module, and a target optimization module. Among them, the specific optimization logic of the target optimization module is: Case 1: When only the motion detection has a result, first compare the width and height b_wh of the rectangle frame of the detection result with a preset value min_wh < b_wh < max_wh. The rectangle frames within the condition range are retained, and the rectangle frame with the largest area among the retained ones is selected as the final rectangle frame. Case 2: When only the human detection result is available, since no motion detection result is detected, it means that there are no moving targets in the picture, and there is no need to re-adjust the position of the pan-tilt. Set the tracking condition b to FALSE. Case 3: When there are both human detection results and motion detection results, perform an AND operation on the rectangle frame of the motion detection result and the human detection rectangle frame. After the AND operation, retain the human detection frames with intersections and the intersections reaching the preset value, denoted as B1; otherwise, set the tracking condition b to FALSE. If the preset current tracking mode is to preferentially track historical targets, perform an AND operation on B1 and the historical human detection frame B2. The target with the largest intersection is used as the final target detection frame, and the tracking condition b is set to TRUE; if there is no intersection, the tracking condition b is set to FALSE. If the tracking mode is to preferentially track moving human targets, select the one with the largest area in B1 as the final human result and set the tracking condition b to TRUE. S5. Tracking condition b: If it is determined that the tracking condition b is TRUE, enter step S6 for the tracking link; otherwise, continue with step S1 to obtain image data. S6. Tracking link: Includes: a KCF tracking module, a tracking verification module, a verification frequency setting module; and update the tracking condition a. Among them, the tracking verification mechanism composed of the tracking verification module and the verification frequency setting module uses a local CNN human detection algorithm to verify the actual size of the human frame and synchronously update the KCF parameters, and only performs human detection within the region of interest. S7. Pan-tilt control link: Includes a module for calculating pan-tilt control signals, a motor execution module, a module for obtaining motor status; and update the motor stop state.

2. The method for solving the problem of target loss in live streaming with a gimbal according to claim 1, characterized in that, Step S4 further includes: S4.

1. Perform the motion detection module: Use the method of frame difference of images separated by T frames to implement the detection of moving targets, and judge whether there are moving targets in the image. If so, retain the result, that is, the position of the rectangle frame for use in optimization. S4.2 Human detection module: Use Yolov3-tiny CNN convolutional neural network to detect human figures in the current image. If a human figure is detected, all detected human figure results, i.e., bounding boxes, are retained for selection and use. S4.3 Target Selection Module: Combines the motion detection results and human detection results to select the best target and obtain the final tracking target.

3. The method for solving the problem of target loss in live streaming with a gimbal according to claim 2, characterized in that, The image frame difference method in step S4.1: Select the current frame and record F. n Select one frame after an interval of T frames and record F. n+T , will F n F n+T The difference image F is obtained by taking the absolute value of the difference between the gray values ​​of the corresponding pixels. d The thresholding process is then performed to obtain a binary image, followed by morphological processing to obtain the final binary image F. Contour search is performed on the binary image F to obtain n contour data, where n is an integer >= 0. The minimum bounding rectangle is calculated on the n contours to obtain n bounding boxes. When the number of bounding boxes is not zero, it is considered that there is a moving target. The bounding boxes represent the position information of the moving target on the image; otherwise, no moving target appears.

4. The method for solving the problem of target loss in live streaming with a gimbal according to claim 1, characterized in that, In case 1, min_wh is 50 and max_wh is frame_h / 2; frame_h is the height of the entire frame image; in case 3, the preset value is 0.

5.

5. The method for solving the problem of target loss in live streaming with a gimbal according to claim 1, characterized in that, Step S6 further includes: S6.1 Perform KCF tracking module: take the input rectangle and the current image together as input data for KCF algorithm; and obtain the result of KCF algorithm, which includes the tracking status S, indicating whether the tracking failed, 0 indicates failure, 1 indicates success, and the adjusted output rectangle B3; S6.2 Update tracking condition a: Set tracking condition a to FALSE if tracking status S=0, and set tracking condition a to TRUE if S=1; S6.3, Perform tracking verification module: Input the target box box from the KCF tracking module, expand the target box width horizontally by kw times and expand the target box height vertically by kh times to obtain a new target box newbox. Use newbox to extract the data of interest from the image data, and use it as the input data for the CNN human detection algorithm. Obtain all human target boxes pboxes in the region of interest through forward inference. Use the IOU method to calculate the intersection-union ratio between pboxes and box, take the pbox with the largest intersection-union ratio score as the verified target box, and re-perform KCF tracking with pbox as the target box; S6.4, Verification Frequency Setting Module: Determines the execution frequency of the tracking verification module, which includes the frame count parameter check_count and the verification frequency parameter check_freq. Each time the KCF tracking module is entered, check_count is incremented by 1, and it is determined whether check_count is greater than or equal to check_freq. If the condition is met, the tracking verification module is executed and check_count is set to 0. Otherwise, the KCF tracking module continues.

6. The method for solving the problem of target loss in live streaming with a gimbal according to claim 5, characterized in that, In step S6.3, the empirical value of kw is set to 2.0, and the empirical value of kh is set to 1.2; kw and kh are adjusted according to different resolutions and actual conditions.

7. The method for solving the problem of tracking loss by a live-streaming gimbal according to claim 5, characterized in that, Step S7 further includes: S7.1 Calculate the gimbal control signal module: Take the rectangle B3 obtained in step S6.1, and calculate the deviations dx and dy between the center point of B3 and the center point of the image. Here, the center point of B3 is the origin of the coordinate system, and dx represents the deviation of the center point of B3 from the center point of the image on the horizontal axis, in pixels. dy represents the deviation of the center point of B3 from the center point of the image on the vertical axis, in pixels; S7.2 Execute the motor actuation module: Based on the dx and dy calculated in step S7.1, convert them into the motor control signal step size or rotation speed, and the motor actuation mechanism performs the action; S7.3 Perform the motor status acquisition module: acquire the motor actuator stop signal STOP, STOP=1 indicates that the motor is in a stopped state, STOP=0 indicates that the motor is in a moving state; S7.4 Update the motor stop status: Based on the status obtained in step S7.2, set the motor stop status to FALSE if STOP=0, and set the motor stop status to TRUE if STOP=2.

8. The method for solving the problem of target loss in live streaming with a gimbal according to claim 1, characterized in that, In step S6, the local CNN human detection algorithm detects human figures in video images based on the Yolov3-tiny CNN convolutional neural network and calculates the bounding box of the human figure. The local human detection algorithm does not use the entire image for detection input images, but instead extracts the region to be detected for CNN human detection.

Images