A method for solving the target loss of low-power live tracking cloud platform

Abstract

The application provides a method for solving the target loss of low-power live tracking cloud platform, which comprises the following steps: S1, acquiring video image data: acquiring real-time code stream data through a camera installed on the cloud platform to obtain each frame of image data; S2, tracking condition a: judging whether to directly enter S6, if the judgment result is TRUE, directly entering S6, otherwise entering S3; S3, motor stop state: if TRUE, entering S4, otherwise continuing to execute S1; S4, detection link: comprising a moving detection module, a human body detection module and a target optimization module; S5, tracking condition b: judging whether to directly enter the KCF tracking module of the tracking link, if the judgment result is TRUE, directly entering S6.2, otherwise continuing to execute S1; S6, tracking link: comprising a KCF pre-tracking module, a KCF tracking module and an updated tracking condition a; S7, cloud platform control link: comprising a cloud platform control signal calculation module, a motor execution module, a motor state acquisition module and an updated motor stop state.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent video processing, in particular to a method for solving the target loss of low-power live tracking gimbal. BACKGROUND

[0002] With the development of computer technology and the wide application of computer vision principles, it has become increasingly popular to use computer image processing technology to detect and track targets in real time. In intelligent AI image recognition systems, intelligent monitoring systems, etc., dynamic real-time tracking and positioning of targets are used to detect human forms, which have wide application value. Among the common system applications, for example, human forms are detected based on the OpenCV HOG algorithm, and the gimbal is controlled by the deviation of the human form result to achieve the purpose of real-time tracking.

[0003] However, the existing human form detection method has some defects, as follows:

[0004] 1. The tracking method based on the OpenCV HOG algorithm for detecting human forms cannot adapt to various light and various postures of human forms due to the simplicity of the detection algorithm, which can easily lead to missed detection and false detection, etc. Secondly, it does not consider how to optimize the tracking target when multiple human form targets appear in the camera view. When human form detection fails and there is a human form in the picture but the human form detection does not give a detection result, if only waiting for the correct result of the human form next time, it is easy to lose the moving human form target and cause tracking failure.

[0005] 2. The CNN deep learning algorithm is used for human form detection, which greatly improves the detection effect. Considering that low-power chip platforms (generally referring to CPU frequency lower than 1GHz, ddr frequency lower than 500MHz), generally do not have deep learning inference engines, which can cause long CNN forward inference time and cannot meet the actual use situation. When a moving target is detected, the time length of the next frame is generally greater than 200-300ms, which can cause a large deviation between the initial tracking target box and the actual target in the subsequent real-time tracking process, resulting in tracking process failure.

[0006] In addition, the common terms in the prior art are as follows:

[0007] 1. Moving detection: based on continuous video image data, it is determined whether there is a moving target in the image. If there is, the coordinate position of the rectangular box is fed back in the form of a rectangular box.

[0008] 2. CNN human form detection: based on CNN convolutional neural network such as Yolov3-tiny, human forms in video images are detected, and the rectangular box of the human form is calculated.

[0009] 3、Two-axis gimbal: a device with a camera device that can achieve two degrees of freedom motion, 4、KCF tracking algorithm: full name 'Kernel Correlation Filter' kernel correlation filter algorithm, which can quickly track and adjust the target frame according to a given rectangular area on the image. SUMMARY

[0010] In order to solve the above problems in the prior art, the purpose of the present application is to propose a new control process:

[0011] 1. Use Yolov3-tiny for human detection, which can efficiently and accurately detect human shapes.

[0012] 2. A pre-tracking module is proposed, which can effectively reduce the tracking loss problem caused by the large difference between the tracking initialization result and the actual result caused by the low efficiency of CNN human detection on low-power platforms.

[0013] 3. Use KCF tracking algorithm to update human shape results in real time, and run the algorithm on a low-power chip.

[0014] 4. Use the moving detection result as the target optimization and false detection human filtering, which can realize human tracking in complex scenes.

[0015] 5. Use moving detection to effectively improve target tracking when human detection fails, and improve human tracking loss.

[0016] Specifically, the present application provides a method for solving the target loss of low-power live broadcast tracking gimbal, which comprises the following steps:

[0017] S1, obtaining video image data: obtaining real-time code stream data through the camera installed on the gimbal to obtain each frame of image data;

[0018] S2, tracking condition a: determining whether to directly enter the tracking link, if the determination result is TRUE, directly enter step S6 tracking link, otherwise enter step S3 motor stop state determination;

[0019] S3, motor stop state: if TRUE, enter step S4 detection link, otherwise continue step S1;

[0020] S4, detection link: including moving detection module, human detection module, target optimization module;

[0021] S5, tracking condition b: determining whether to directly enter the KCF tracking module of the tracking link, if the determination is TRUE, directly entering S6.2 of S6 tracking link; otherwise, continue step S1 to obtain image data;

[0022] S6, tracking link: including KCF pre-tracking module, KCF tracking module; and update tracking condition a; wherein, using the pre-tracking module of local CNN human detection algorithm: after detecting the human form result in the full frame, enter the KCF tracking process, first use the local CNN human detection algorithm to perform secondary human detection positioning around the initial target area, update the target frame position, and use it as the initial tracking parameter of KCF for subsequent KCF tracking;

[0023] S7, pan-tilt control link: including calculating pan-tilt control signal module, motor execution module, and obtaining motor state module; and updating motor stop state.

[0024] In step S2, tracking condition a: used to judge the state machine, the state machine can represent three states: detection state, tracking state and waiting state, if the current state machine is judged to be in tracking state, the result is TRUE.

[0025] In step S5, tracking condition b is a state identifier, and the specific identifier setting condition of the state is: see below, wherein, the optimization module: the results of moving detection and human detection are combined to optimize the final tracking target.

[0026] S4.1, moving detection module: using image frame difference method with T frames apart to realize moving target detection, judging whether there is a moving target in the image, if there is, the result (rectangular frame position) is retained for optimization; wherein, in the image frame difference method with T frames apart, T: the value is used as a configurable parameter for user configuration, ensuring that it is a number greater than 1, including 3, 4, 5, 6, and the default configuration is 4; the image frame difference method: selecting the current frame as F n , selecting another frame T frames apart as F n+T , taking the absolute value of the difference between the gray values of the corresponding pixel points to obtain the difference image F n， , F n+T , thresholding the difference image F d to obtain a binary image, and then performing morphological processing to obtain the final binary image F, and performing contour searching on the binary image F to obtain n (n >= 0 integer) contour data, and performing minimum circumscribed rectangle calculation on the n contours to obtain n rectangular frames, when the number of rectangular frames is not zero, it is considered that there is a moving target, and the rectangular frame represents the position information of the moving target in the image, otherwise no moving target appears.

[0027] S4.2, human detection module: using Yolov3-tiny CNN convolutional neural network to detect human in the current image, if human is detected, all human detection results (rectangular frames) are retained for optimization.

[0028] S4.3, performing a target optimization module: the results of the movement detection and the human detection results are integrated and optimized to obtain the final tracking target.

[0029] In the step S4.3 target optimization module, the specific optimization logic is as follows:

[0030] Case 1: when only the movement detection has results, first compare the width and height b_wh of the detection result rectangular frame with the preset value min_wh < b_wh < max_wh, the rectangular frame within the condition range is retained, and the retained rectangular frame with the largest area is selected as the final rectangular frame;

[0031] Case 2: when only the human detection result is obtained, because no movement detection result is detected, it indicates that there is no moving target in the picture, and there is no need to re-adjust the position of the holder, and the tracking condition b is set to FALSE;

[0032] Case 3: when there are both human detection results and movement detection results, the movement detection structure rectangular frame and the human detection rectangular frame are ANDed, and the human detection frame with intersection and intersection reaching the preset value is retained as B1, otherwise the tracking condition b is set to FALSE; if the preset current tracking mode is to track the historical target preferentially, then B1 and the historical human detection frame B2 are ANDed, and the target with intersection and the largest intersection is taken as the final target detection frame, and the tracking condition b is set to TRUE, and if there is no intersection, the tracking condition b is set to FALSE; if the tracking mode is to track the moving human target preferentially, then the largest area in B1 is selected as the final human result and the tracking condition b is set to TRUE.

[0033] In the case 1, the experience values min_wh and max_wh are 50 and frame_h / 2 respectively; frame_h is the height of the whole frame image; in the case 3, the intersection after the AND operation reaches the preset value of 0.5.

[0034] The step S6 further comprises:

[0035] S6.1, performing a KCF pre-tracking module: the input rectangular frame and the current picture are taken as the input data of the module, and the module is triggered only when the tracking condition b is TRUE for the first time; the module takes the input rectangular frame box as the reference, performs the expansion operation in the width direction and the height direction to obtain new_box, expands a frame centering on a circle point, and expands the box width and height by kw and kh times, wherein the experience value of kw is 2, and the experience value of kh is 1.2, then the interested region in the picture is deducted according to new_box and human detection is performed, the human result person_box obtained by the detection is IOU operated with box, and the one with the largest score is taken as the output result of the module;

[0036] S6.2, a KCF tracking module: the incoming rectangular frame and the current image are taken as the input data of the KCF algorithm. The result of the KCF algorithm is obtained, which contains the state S of this tracking, indicating whether the tracking fails, 0 indicating failure, 1 indicating success, and the adjusted output rectangular frame B3;

[0037] S6.3, update tracking condition a: set tracking condition a to FALSE according to tracking state S = 0, and set tracking condition to TRUE according to S = 1.

[0038] The step S7, the control link of the holder, further comprises:

[0039] S7.1, a holder control signal calculation module: the rectangular frame B3 obtained in step S7 is taken, and the deviation dx, dy of the center point of B3 from the center point of the image (the center point is the coordinate origin) is calculated, wherein dx represents the deviation amount of the center point of B3 from the center point of the image on the horizontal axis, and dy represents the deviation amount of the center point of B3 from the center point of the image on the vertical axis, both in pixels;

[0040] S7.2, a motor execution module: according to the dx and dy calculated in 9, the control signal step or rotation speed of the motor is converted, and the motor execution mechanism is actuated;

[0041] S7.3, a motor state acquisition module: mainly acquires the motor stop signal STOP, STOP = 1 indicating that the motor is in a stop state, and STOP = 0 indicating that the motor is in a motion state;

[0042] S7.4, update motor stop state: according to the state acquired in 11, set the motor stop state to FALSE when STOP = 0, and set the motor stop state to TRUE when STOP = 2.

[0043] Therefore, the advantages of the present application are:

[0044] 1. The present application uses the Yolov3-tiny CNN convolutional neural network with fast speed, high detection accuracy and human body detection, which can greatly improve the detection effect and effectively improve the human body detection result in various light scenes, various human body postures and incomplete human body display.

[0045] 2. The present application adopts KCF algorithm, and when the human body detection result is obtained, the result is input into KCF algorithm with the human body result as the center to track the result. The KCF algorithm has an execution efficiency about ten times faster than the Yolov3-tiny algorithm, can achieve real-time updating of the human body result, and can run on a low-power embedded chip such as INGENIC T-series chip of Beijing Junzheng Integrated Circuit Co., Ltd. (hereinafter referred to as Beijing Junzheng) and achieve real-time effect.

[0046] 3、The application joins the self-research mobile detection algorithm, can effectively solve the false detection of some static human-like objects. At the same time, the moving target can be selected according to the result of the mobile detection algorithm to track, and the focus can be focused on the dynamic human form.

[0047] 4、The application uses the mobile detection algorithm to make up for the case when the human detection fails, that is, there is a human form moving in the picture but no human form result is fed back. The result of the mobile detection can replace the human detection result, and the motor control of the holder can ensure that the moving target is always in the video field of view. To ensure that the target person remains in the image when the human detection is performed.

[0048] 5、The application proposes a pre-tracking module using a local CNN human detection algorithm. After the human form result is detected in the full frame, the KCF tracking process is started. First, the local CNN human detection algorithm is used to perform secondary human detection positioning around the initial target area. The target frame position can be updated in a relatively fast manner, and the KCF initial tracking parameters are used for subsequent KCF tracking. The KCF effect can be effectively guaranteed. BRIEF DESCRIPTION OF DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application.

[0050] Figure 1 is a flowchart of the method of the present application.

[0051] Figure 2 is a flowchart of the electric vehicle detection system in the embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to more clearly understand the technical content and advantages of the present application, the present application will be further described in detail in conjunction with the drawings.

[0053] As shown in Figure 1 , the method of the present application relates to a method for solving the loss of a live tracking holder, the method comprising the following steps:

[0054] S1, acquiring video image data: acquiring real-time code stream data through a camera installed on the holder, and obtaining each frame of image data;

[0055] S2, tracking condition a: determining whether to directly enter the tracking link, if the determination result is TRUE, directly entering step S6 tracking link, otherwise entering step S3 motor stop state determination;

[0056] S3, motor stop state: if TRUE, enter step S4 detection link, otherwise continue to execute step S1;

[0057] S4, detection link: including mobile detection module, human detection module, target optimization module;

[0058] S5, tracking condition b: judge whether to directly enter the KCF tracking module of the tracking link, if the judgment is TRUE, directly enter S6.2 of the tracking link S6; otherwise continue step S1 operation to obtain image data;

[0059] S6, tracking link: including KCF pre-tracking module, KCF tracking module; and update tracking condition a;

[0060] S7, head control link: including calculating head control signal module, motor execution module, obtaining motor state module; and updating motor stop state.

[0061] For easy understanding, as shown in Figure 2 The embodiments of the application can be further described as:

[0062] 1, obtain video image data: obtain real-time code stream data through the camera installed on the head, and obtain each frame of image data.

[0063] 2, tracking condition a: judge whether to directly enter the KCF tracking module, if the judgment result is TRUE, directly enter the KCF tracking module, otherwise enter the motor stop state judgment.

[0064] 3, motor stop state: if TRUE, enter the detection link, otherwise continue to execute 1.

[0065] 4, mobile detection module: use the image frame difference method with T frames apart to realize mobile target detection, judge whether there is a mobile target in the image, if there is, reserve the result (rectangular frame position) for optimization.

[0066] 5, human detection module: use Yolov3-tiny CNN convolutional neural network to detect human in the current picture, if human is detected, reserve all human detection results (rectangular frame) for optimization.

[0067] 6, optimization module: comprehensively optimize the results of mobile detection and human detection to obtain the final tracking target. The specific optimization logic is as follows:

[0068] Case 1: When there is only mobile detection result, first compare the width and height of the detection result rectangle frame b_wh with the preset value (empirical value min_wh: 50, max_wh: frame_h / 2; frame_h is the height of the whole frame image) min_wh < b_wh < max_wh, the rectangle frame within the condition range is retained, and the largest area of the retained rectangle frame is selected as the final rectangle frame.

[0069] Case 2: When there is only human detection result, because there is no mobile detection result, it means that there is no moving target in the picture, and there is no need to re-adjust the position of the gimbal, and the tracking condition b is set to FALSE.

[0070] Case 3: When there is both human detection result and mobile detection result, the mobile detection structure rectangle frame and the human detection rectangle frame are operated by 'and', and the human detection frame with intersection and intersection reaching the preset value (empirical value 0.5) is retained as B1, otherwise the tracking condition b is set to FALSE. If the preset current tracking mode is to track the historical target preferentially, then B1 and the historical human detection frame B2 are operated by 'and', and the target with intersection and the largest intersection is taken as the final target detection frame, and the tracking condition b is set to TRUE, and if there is no intersection, the tracking condition b is set to FALSE. If the tracking mode is to track the moving human target preferentially, then the largest area in B1 is selected as the final human result, and the tracking condition b is set to TRUE.

[0071] 7. Tracking condition b: if the tracking condition b is determined to be TRUE, enter the KCF tracking module, otherwise continue to operate 1 to obtain image data.

[0072] 8. KCF pre-tracking module: the input rectangle frame and the current picture are taken as the input data of the module, and the module is triggered only when the tracking condition b is first satisfied as TURE. The module takes the input rectangle frame box as the reference, performs expansion operation in the width direction and the height direction to obtain new_box, expands a frame center as a circle point, expands the box width and height by kw and kh times, wherein kw is an empirical value of 2, and kh is an empirical value of 1.2, then the interested region in the picture is deducted according to new_box and human detection is performed, the human result person_box obtained by detection is operated by IOU with box, and the maximum is taken as the output result of the module.

[0073] 9. KCF tracking module: the input rectangle frame and the current image are taken as the input data of KCF algorithm, and the result of KCF algorithm is obtained, which includes the state S of this tracking (whether the tracking fails, 0 represents failure, and 1 represents success) and the adjusted output rectangle frame B3.

[0074] 10. Update tracking condition a: set tracking condition a to FALSE when tracking state S=0, set tracking condition to TRUE when S=1;

[0075] 11. Calculate the pan-tilt control signal module: take the rectangular frame B3 obtained in 7, and take the deviation dx, dy of the center point of B3 from the center point of the image (the center point is the coordinate origin), wherein dx represents the deviation of the center point of B3 from the center point of the image on the horizontal axis, and dy represents the deviation of the center point of B3 from the center point of the image on the vertical axis, both in units of pixels.

[0076] 12. Motor execution module: convert dx, dy obtained in 9 into motor control signal step or rotation speed, etc. The motor execution mechanism is actuated.

[0077] 13. Get motor state module: mainly obtain the motor execution mechanism stop signal STOP, STOP=1 indicates that the motor is in a stop state, and STOP=0 indicates that the motor is in a motion state.

[0078] 14. Update motor stop state: set the motor stop state to FALSE when the state obtained in 11 is STOP=0, and set the motor stop state to TRUE when STOP=2.

[0079] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application can be variously changed and modified. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for solving the problem of target loss of low-power live tracking gimbal, characterized in that, The method comprises the following steps: S1, acquiring video image data: acquiring real-time code stream data through a camera installed on a pan-tilt head to obtain each frame of image data; S2, tracking condition a: judging whether to directly enter a tracking link, if the judgment result is TRUE, directly entering step S6 tracking link, otherwise entering step S3 motor stop state judgment; the tracking condition a: used for judging the state machine, the state machine indicates that there are three states: detection state, tracking state and waiting state, if the current state machine is the tracking state, the result is TRUE; S3, motor stop state: if TRUE, entering step S4 detection link, otherwise continuing to execute step S1; S4, detection link: including a moving detection module, a human body detection module and a target optimization module; the specific optimization logic of the target optimization module is as follows: Case 1: when there is only a moving detection result, first comparing the width and height b_wh of the detection result rectangular frame with a preset value min_wh < b_wh < max_wh, retaining the rectangular frame within the condition range, and selecting the largest area of the retained rectangular frame as the final rectangular frame; Case 2: when there is only a human body detection result, because no moving detection result is detected, it is indicated that there is no moving target in the picture, and the position of the pan-tilt head does not need to be adjusted, and the tracking condition b is set to FALSE; Case 3: when there are both human body detection results and moving detection results, the moving detection structure rectangular frame and the human body detection rectangular frame are operated, and after the operation, the human body detection frame with intersection and the intersection reaching a preset value is retained as B1, otherwise the tracking condition b is set to FALSE; if the preset tracking mode is to track the historical target preferentially, the B1 and the historical human body detection frame B2 are operated, the target with intersection and the largest intersection is selected as the final target detection frame, and the tracking condition b is set to TRUE, and if there is no intersection, the tracking condition b is set to FALSE; if the tracking mode is to track the moving human body target preferentially, the largest area in B1 is selected as the final human body result, and the tracking condition b is set to TRUE; S5, tracking condition b: judging whether to directly enter the KCF tracking module of the tracking link, if the judgment result is TRUE, directly entering S6.2 of the tracking link S6, otherwise continuing to execute step S1 to acquire image data; S6, tracking link: including a KCF pre-tracking module and a KCF tracking module, and updating the tracking condition a; wherein the pre-tracking module using the local CNN human body detection algorithm: after detecting the human body result in the whole frame, entering the KCF tracking process, first using the local CNN human body detection algorithm to perform secondary human body detection positioning around the initial target area, updating the target frame position, and taking the same as the initial tracking parameter of KCF to perform subsequent KCF tracking; S7, pan-tilt head control link: including a calculation pan-tilt head control signal module, a motor execution module and an acquisition motor state module, and updating the motor stop state.

2. The method of claim 1, wherein, The step S4 further comprises: S4.1, a moving detection module is performed: a moving target detection is realized by using an image frame difference method with T frames apart, whether a moving target exists in the image is judged, if yes, the result, i.e. a rectangular frame position, is reserved for selection and optimization; S4.2, a human body detection module is performed: a Yolov3-tiny CNN convolutional neural network is used to detect human bodies in the current picture, if a human body is detected, all the human body detection results, i.e. rectangular frames, are reserved for selection and optimization; S4.3, a target selection and optimization module is performed: the results of the moving detection in step S4.1 and the human body detection results in step S4.2 are comprehensively selected and optimized to obtain the final tracking target.

3. The method of claim 2, wherein, In the image frame difference method with T frames apart, T: the value is configured by a user as a configurable parameter, which is ensured to be greater than 1, including 3, 4, 5 and 6, and the default configuration is 4; the image frame difference method: a current frame is selected as Fn, a frame apart by T frames is selected as Fn+T, the gray values of the corresponding pixels of Fn and Fn+T are subtracted to obtain a differential image Fd, then threshold processing is performed to obtain a binary image, then morphological processing is performed to obtain a final binary image F, contour searching is performed on the binary image F to obtain n contour data, where n is an integer greater than or equal to 0, and the minimum circumscribed rectangle of the n contours is calculated to obtain n rectangular frames, when the number of the rectangular frames is not zero, it is considered that a moving target exists, the rectangular frame represents the position information of the moving target on the image, otherwise, no moving target exists.

4. The method of claim 1, wherein, In the case 1, the experience value min_wh is 50, and the experience value max_wh is frame_h / 2; frame_h is the height of the whole frame image; in the case 3, the preset value is 0.

5.

5. The method of claim 1, wherein, The step S6 further includes: S6.1, a KCF pre-tracking module is performed: the input rectangular frame and the current picture are used as the input data of the module, the module is triggered only when the tracking condition b is first satisfied as TRUE; the module takes the input rectangular frame box as a reference, performs expansion operation in the width direction and the height direction to obtain new_box, the center of the expansion frame is a circle point, the expansion box width and height are multiplied by kw and kh, respectively, where the experience value of kw is 2, and the experience value of kh is 1.2, then the region of interest in the picture is cut according to new_box, and human body detection is performed, the human body detection result person_box is subjected to IOU operation with box, and the result with the maximum score is taken as the output result of the module; S6.2, a KCF tracking module is performed: the input rectangular frame and the current image are used as the input data of the KCF algorithm; and the result of the KCF algorithm is obtained, which includes the state S of the current tracking, indicating whether the tracking fails, 0 indicating failure, and 1 indicating success, and the output rectangular frame B3 after adjustment; S6.3, an update tracking condition a is performed: the tracking condition a is set as FALSE when the tracking state S is 0, and the tracking condition a is set as TRUE when the tracking state S is 1.

6. The method of claim 1, wherein, The step S7 is a control link of the pan-tilt. S7.1, gimbal control signal calculation module: take the deviation dx, dy of the center point of the rectangle B3 obtained in step S7 from the center point of the image, wherein the center point of B3 is the coordinate origin, dx represents the deviation of the center point of B3 from the center point of the image on the horizontal axis, and dy represents the deviation of the center point of B3 from the center point of the image on the vertical axis, both in units of pixels; S7.2, motor execution module: convert dx, dy calculated in step S7.1 into motor control signal step or rotation speed, and actuate the motor execution mechanism; S7.3, motor state acquisition module: acquire the motor execution mechanism stop signal STOP, STOP=1 indicates that the motor is in a stopped state, and STOP=0 indicates that the motor is in a moving state; S7.4, update motor stop state: according to the state acquired in step S7.3, set the motor stop state to FALSE when STOP=0, and set the motor stop state to TRUE when STOP=2.

Images