A solution to a low-power live follow-up cloud platform multi-target complex scene

Abstract

The application provides a solution to a low-power live tracking cloud platform in a multi-target complex scene, comprising a local fusion separation module: a local fusion separation algorithm is used to input a rectangular frame box, the number of the input rectangular frame box is determined, if the number is less than 1, the single-target update module (T) is entered, if the number is greater than 1, the multi-target update module (T) is entered, wherein (T) represents the Tth frame; further comprising an update motion state: when the input box meets the candidate condition, the state count variable S is updated according to the situation; an update separation start signal (T) is provided: when the separation start signal is true, the separation signal is set to true, and the separation start signal is modified to false at the same time, so as to prevent the separation signal from being repeatedly updated. The problem of tracking target switching in the case of multi-target occlusion and overlap is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent video processing, in particular to a solution method for solving the multi-target complex scene of low-power live tracking and following the holder. BACKGROUND

[0002] In the prior art, with the development of science and technology, the dynamic real-time tracking and positioning of the target used in intelligent AI image recognition system, intelligent monitoring system and the like are increasingly widely used for detecting human forms. Common system applications, such as detecting human forms based on OpenCV HOG algorithm, and controlling the holder to rotate through the deviation of the human form result, achieve the purpose of real-time tracking.

[0003] However, the existing method for detecting human forms has some defects, as follows:

[0004] 1. The tracking method based on OpenCV HOG algorithm for detecting human forms, firstly, the detection algorithm is relatively simple and cannot adapt to various light and various postures of human forms, which is prone to miss detection and false detection, etc. Secondly, it does not consider how to optimize the tracking target when multiple human form targets appear in the camera view. When human form detection fails and there is a human form in the picture but the human form detection does not give a detection result, if only waiting for the correct result of the human form next time, it is easy to lose the moving human form target and cause tracking failure.

[0005] 2. The CNN deep learning algorithm is used for human form detection, and the detection effect is greatly improved. Considering that the low-power chip platform (generally referring to CPU frequency lower than 1GHz, ddr frequency lower than 500MHz), generally does not have a deep learning inference engine, which will cause the CNN forward inference time to be too long to meet the actual use, and when a moving target is detected, the time length of the next frame will generally be greater than 200-300ms, which will cause a large deviation between the initial tracking target box and the actual target in the subsequent real-time tracking process, resulting in tracking process failure.

[0006] 3. In particular, when multiple targets appear in the picture and the tracking target overlaps with other targets, it is easy to cause the tracking target to switch and fail to continuously track the original target.

[0007] In addition, the common terms in the prior art are as follows:

[0008] 1. Moving detection: based on continuous video image data, it is judged whether there is a moving target in the image, and if there is, the coordinate position of the rectangular box is fed back in the form of a rectangular box.

[0009] 2. CNN human form detection: based on Yolov3-tiny and other CNN convolutional neural network, the human form in the video image is detected, and the rectangular box of the human form is calculated.

[0010] 3、Two-axis gimbal: a device with a camera device that can achieve two degrees of freedom movement,

[0011] 4、KCF tracking algorithm: full name 'Kernel Correlation Filter' kernel correlation filter algorithm, which can quickly track and adjust the target frame according to a given rectangular area on the image.

[0012] 5、Image frame difference method: select the current frame as Fn, select another frame as Fn+T after interval T, take the absolute value of the difference between the gray values of the corresponding pixels of Fn and Fn+T to get the difference image Fd, then do thresholding to get the binary image, and then do morphological processing to get the final binary image F, where the highlighted part is the moving target area.

[0013] 6、IOU(Intersection-over-Union), Intersection-over-Union, a concept used in target detection, is the intersection rate of the generated candidate frame (candidate bound) and the original labeled frame (ground truth bound), that is, the ratio of their intersection to union. The most ideal case is complete overlap, that is, the ratio is 1. SUMMARY

[0014] In order to solve the problems in the prior art, the purpose of the present application is:

[0015] 1、The present application uses the Yolov3-tiny CNN convolutional neural network with fast speed and high detection accuracy for human detection, greatly improves the detection effect, and effectively improves the human detection results of various light scenes, various human postures and incomplete human display.

[0016] 2、The present application adopts KCF algorithm, and when the human detection result is obtained, the result is transmitted into the KCF algorithm for tracking based on the human result as the center. The execution efficiency of the KCF algorithm is about ten times faster than that of the Yolov3-tiny algorithm, which can achieve real-time updating of the human result, and can run on a low-power embedded chip such as the INGENIC T-series chip of Beijing Junzheng Co., Ltd. and achieve real-time effect.

[0017] 3、The present application adds a self-developed moving detection algorithm, which can effectively solve the false detection of human detection results on some static-like objects. At the same time, the moving target can be preferentially selected for tracking according to the moving detection algorithm result, and the focus can be focused on the dynamic human.

[0018] 4、The mobile detection algorithm used in the present application can compensate for the situation when the human shape detection fails, i.e. when there is a human shape moving in the picture but no human shape result is fed back, the mobile detection result can replace the human shape detection result, and the motor control of the holder can ensure that the moving target is always in the video field of view, so as to ensure that the target person remains in the image when the human shape detection is performed next time.

[0019] 5、The present application proposes a pre-tracking module using a local CNN human shape detection algorithm. After the human shape result is detected in the full frame, the KCF tracking process is entered, and the local CNN human shape detection algorithm is used first to perform secondary human shape detection positioning around the initial target area, so that the target frame position can be updated in a relatively fast manner, and the KCF tracking is performed subsequently using the updated target frame position as the initial tracking parameter of KCF, so that the KCF effect can be effectively ensured.

[0020] 6、In particular, the present application proposes a processing method of a local fusion segmentation module, which solves the problems that the human shape detection algorithm cannot detect the target when the target is blocked, and the KCF tracking algorithm cannot distinguish the tracking target from other targets when similar targets are blocked and overlapped. The algorithm in the local fusion segmentation module can identify the multi-target overlapping event and record the motion state before and after the target overlapping, and the output of the human shape detection algorithm and the KCF tracking algorithm is controlled and adjusted according to the motion state, so as to solve the problem of tracking target mis-switching when multiple targets are blocked and overlapped.

[0021] Specifically, the present application provides a solution to a low-power live broadcast and tracking holder in a multi-target complex scene, which comprises a local fusion separation module step: the module is used to implement a local fusion separation algorithm, and a rectangular frame box is input,

[0022] Firstly, it is judged whether the number of input rectangular frames box is greater than 1,

[0023] When it is not greater than 1, a single target updating module (T) is entered;

[0024] When it is greater than 1, a multi-target updating module (T) is entered, wherein (T) represents the T-th frame, and when it is greater than 1, the candidate condition needs to be met at the same time, and the motion state S(T) is updated, but only the state S of the (T-1)-th frame, i.e. the state S of the previous frame, enters the single target updating module (T), and not the state of the current frame;

[0025] The single target updating module: the box entering the single target module is obtained, and the following processing is performed according to the motion state S(T-1) of the previous frame, wherein (T-1) represents the motion state S(T-1) of the (T-1)-th frame, i.e. the motion state S of the previous frame of the current frame, and the motion state S value has S=0, S<0, and S>0:

[0026] When S>0, adjust the box, the center point is translated by pixels Δx in the positive direction of X, Δx is an empirical value, here taking 0.2 of the width w of the box, that is, Δx=w*0.2, at the same time, the width w of the box is changed to 0.7 times of the original, that is, the new width new_w=0.7*w, and the separation start signal (T) is updated to true;

[0027] When S<0, adjust the box, the center point is translated by pixels Δx in the negative direction of X, Δx is an empirical value, here taking 0.2 of the width w of the box, that is, Δx=w*0.2, at the same time, the width w of the box is changed to 0.7 times of the original, that is, the new width new_w=0.7*w, and the separation start signal (T) is updated to true;

[0028] When S=0, the box is not adjusted, and the separation start signal (T) is not updated;

[0029] The multi-target updating module: record the current tracking target box as TB, calculate the intersection over union IOU of TB and all boxes, when there are more than 0 boxes with IOU>thr, wherein thr is an empirical value of 0.5 by default, take the largest area in the box as the output of the multi-target updating module; when the IOU of all boxes is not greater than thr, select the largest area target box in the box as MB, update the center point coordinates of TB to the center point coordinates of MB, keep the width and height of TB unchanged, and take TB as the output of the multi-target updating module;

[0030] When the single-target updating module (T) is completed, update the separation start signal (T), when the separation start signal is true, set the separation signal to true, and modify the separation start signal to false to prevent repeated updating of the separation signal.

[0031] The method comprises the following steps:

[0032] S1, acquiring video image data: acquiring real-time code stream data through a camera installed on a pan-tilt head to obtain each frame of image data;

[0033] S2, tracking condition a: used to judge the state machine, the state machine indicates that there are three states: detection state, tracking state and waiting state, if it is judged that the current state machine is in the tracking state, the result is TRUE, if the judgment result is TRUE, directly enter step S6 of the tracking link, otherwise enter step S3 of the motor stop state judgment;

[0034] S3, motor stop state: if TRUE, enter step S4 of the detection link, otherwise continue to execute step S1;

[0035] S4, detection link: including a movement detection module, a human body detection module and an optimization module;

[0036] S5, tracking condition b: for a state identifier, the specific identifier setting condition is updated by the optimization module in step S4, if it is judged that the tracking condition b is TRUE, enter the KCF tracking module of step S6, otherwise continue to operate step S1 to obtain image data;

[0037] S6, tracking link: including KCF pre-tracking module, local fusion separation module, KCF tracking module; and update tracking condition a; wherein the local fusion separation module takes the output result of the KCF pre-tracking module in the last step as input;

[0038] S7, head control link: including calculating head control signal module, motor execution module, obtaining motor state module; and updating motor stop state.

[0039] The step S6 further comprises:

[0040] S6.1, KCF pre-tracking module: the incoming rectangular box and the current picture are taken as the input data of the module, and the module is triggered only when the tracking condition b is TRUE for the first time. The module takes the incoming rectangular box box as the reference, performs expansion operation in the width direction and the height direction to obtain new_box, expands a box center to a circle point, expands the box width w and height h by kw and kh times, wherein kw is an empirical value of 2, kh is an empirical value of 1.2, then the interested region in the picture is deducted according to new_box and human body detection is performed, and the human body detection result person_box is taken as the output result of the module;

[0041] S6.2, local fusion separation module: taking the output result of the last step S6.2 as input; judging whether the number of input rectangular boxes box is greater than 1,

[0042] If it is not greater than 1, enter the single target update module (T), the single target update module: simultaneously according to the motion state S (T-1), wherein (T-1) represents the T-1th frame, that is, the motion state S (T-1) of the last frame of the current frame; the motion state S value has S=0, S<0, S>0; when S>0, adjust the box, the center point is translated by pixels Δx in the X positive direction, Δx is an empirical value, which is 0.2 of the box width here, that is, Δx=w*0.2, at the same time, the width of the box becomes 0.7 times of the original, that is, new_w=0.7*w, and the separation start signal (T) is updated to true; when S<0, adjust the box, the center point is translated by pixels Δx in the X negative direction, Δx is an empirical value, which is 0.2 of the box width here, that is, Δx=w*0.2, at the same time, the width of the box becomes 0.7 times of the original, that is, new_w=0.7*w, and the separation start signal (T) is updated to true; when S=0, do not adjust the box, and do not update the separation start signal (T);

[0043] If greater than 1, enter the multi-target update module (T), where (T) represents the Tth frame;

[0044] The multi-target update module: record the current tracking target box as TB, calculate the IOU of TB and all boxes, when there are greater than 0 boxes with IOU>thr, thr takes the experience value 0.5 by default, take the largest area in the box as the output of the multi-target update module. When the IOU of all boxes is not greater than thr, select the largest area target box in the box as MB, update the center point coordinates of TB to the center point coordinates of MB, keep the width and height of TB unchanged, and take TB as the output of the multi-target update module;

[0045] If greater than 1, update the motion state when the input box meets the following candidate conditions:

[0046] Condition a: the height of the box is less than 1.2 times the height of the tracking target box TB and greater than 0.8 times the height of TB;

[0047] Condition b: the number of boxes meeting condition a is greater than 1;

[0048] For all boxes that meet the candidate box, record the box center point coordinates x as box.x, the input image center point coordinates x as F.x, and the state count variable as S. When box.x>F.x+7, S is increased by 1, when box.x<F.x-7, S is decreased by 1, and in other cases, S=0, and the traversal is ended in advance;

[0049] Further comprising: separate signal update, the specific operation is that if the last frame is a separate start signal (T-1), the current frame is updated to a waiting separation signal (T);

[0050] S6.3, KCF tracking module: judge the separation signal in S6.2, if false, take the box in S6.2 as a new tracking box and input it together with the current image into the KCF algorithm, if true, continue to use the current tracking box and input it together with the current image into the KCF algorithm, and start counting in frames at the same time, when the count is greater than M, update the separation signal to false; get the result of KCF algorithm through the input data, the result contains the state S of this tracking, which indicates whether the tracking fails, 0 indicates failure, 1 indicates success, and the adjusted output rectangular frame B3;

[0051] S6.4, update tracking condition a: set tracking condition a to FALSE according to tracking state S=0, and set tracking condition to TRUE according to S=1.

[0052] The step S4 further comprises:

[0053] S4.1, a moving detection module is performed: a moving target detection is realized by using an image frame difference method with T frames apart, whether there is a moving target in the image is judged, and if there is, the result, i.e. a rectangular frame position, is reserved for selection and optimization; wherein the value of T can be configured as a configurable parameter for the user to configure, and is ensured to be a number greater than 1, including 3, 5, 6;

[0054] S4.2, a human body detection module is performed: a human body detection is performed on the current picture by using a Yolov3-tiny CNN convolutional neural network, and if a human body is detected, all the human body results, i.e. rectangular frames, are reserved for selection and optimization;

[0055] S4.3, a selection and optimization module is performed: the results of the moving detection and the human body detection are comprehensively selected and optimized to obtain the final tracking target.

[0056] In the step S4.1, the value of T is configured as 4 by default.

[0057] In the selection and optimization module of the step S4.3, the specific selection and optimization logic is as follows:

[0058] Case 1: when there is only a moving detection result, first, the width and height b_wh of the detection result rectangular frame are compared with the preset values min_wh and max_wh, the rectangular frame within the condition range is reserved, the largest area of the reserved rectangular frame is selected as the final rectangular frame, and the tracking condition b is set to TRUE;

[0059] Case 2: when there is only a human body detection result, because no moving detection result is detected, it is indicated that there is no moving target in the picture, and the position of the gimbal does not need to be adjusted, and the tracking condition b is set to FALSE;

[0060] Case 3: when there are both a human body detection result and a moving detection result, the moving detection structure rectangular frame and the human body detection rectangular frame are ANDed, and the human body detection frame with intersection and the intersection reaching the preset value is reserved as B1, otherwise the tracking condition b is set to FALSE; if the preset tracking mode is to prioritize tracking the historical target, B1 and the historical human body detection frame B2 are ANDed, the target with intersection and the largest intersection is taken as the final target detection frame, and the tracking condition b is set to TRUE, and if there is no intersection, the tracking condition b is set to FALSE; if the tracking mode is to prioritize tracking the moving human body target, the largest area in B1 is selected as the final human body result, and the tracking condition b is set to TRUE;

[0061] In the case 1, the experience values min_wh and max_wh are 50 and frame_h / 2 respectively; frame_h is the height of the whole frame image; in the case 3, the intersection after the AND operation reaching the preset value is 0.5.

[0062] The step S7 pan-tilt control link further comprises:

[0063] S7.1, a pan-tilt control signal calculation module: taking the rectangular frame B3 obtained in step S6, the deviation dx and dy of the center point of B3 from the center point of the image, wherein dx represents the deviation of the center point of B3 from the center point of the image on the horizontal axis, and dy represents the deviation of the center point of B3 from the center point of the image on the vertical axis, both in units of pixels;

[0064] S7.2, a motor execution module: converting dx and dy calculated in step S7.1 into motor control signal steps or rotation speed, and having a motor execution mechanism act;

[0065] S7.3, a motor state acquisition module: mainly acquiring a motor execution mechanism stop signal STOP, wherein STOP=1 indicates that the motor is in a stop state, and STOP=0 indicates that the motor is in a motion state;

[0066] S7.4, updating the motor stop state: according to the state acquired in step S7.3, setting the motor stop state to FALSE when STOP=0, and setting the motor stop state to TRUE when STOP=2.

[0067] Therefore, the application has the advantages that a new control flow method is proposed, and in particular, a method for human shape tracking in a multi-target complex scene is supported:

[0068] 1. In the application, Yolov3-tiny is used for human shape detection, which can efficiently and accurately detect human shapes.

[0069] 2. The application proposes a pre-tracking module, which can effectively reduce the tracking loss problem caused by the large difference between the tracking initialization result and the actual result caused by the low efficiency of CNN human shape detection on a low-power platform.

[0070] 3. The application proposes a fusion separation algorithm to solve the problem of switching tracking targets when multiple targets are occluded and overlapped.

[0071] 3. In the application, the KCF tracking algorithm is used, which can update the human shape result in real time, and the algorithm can be run on a low-power chip.

[0072] 4. In the application, the moving detection result is used as the target optimization and false detection human shape filtering, which can realize human shape tracking in a complex scene.

[0073] 5. In the application, the moving detection is used to effectively improve the target tracking when the human shape detection fails, and to improve the human shape loss situation. DETAILED DESCRIPTION

[0074] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application.

[0075] Figure 1 is a flow chart of the method of the present application.

[0076] Figure 2 is a schematic diagram of the local fusion separation module in the method of the present application. DETAILED DESCRIPTION

[0077] In order to enable a more clearly understanding of the technical content and advantages of the present application, the present application will be further described in detail in conjunction with the drawings.

[0078] As Figure 1 shown, the method of the present application relates to a solution for solving the multi-target complex scene of a low-power live broadcast and tracking pan-tilt, and the method comprises the following steps:

[0079] S1, acquiring video image data: acquiring real-time code stream data through a camera installed on a pan-tilt to obtain image data of each frame;

[0080] S2, tracking condition a: used for judging the state machine, the state machine indicates that the state has three kinds of detection state, tracking state and waiting state, if the current state machine is judged to be in the tracking state, the result is TRUE, if the result is TRUE, step S6 of tracking link is directly entered, otherwise step S3 of motor stop state judgment is entered;

[0081] S3, motor stop state: if TRUE, step S4 of detection link is entered, otherwise step S1 is continued to be executed;

[0082] S4, detection link: including a moving detection module, a human shape detection module and an optimization module;

[0083] S5, tracking condition b: a state identifier, the specific identifier of the state is updated by the optimization module in step S4, if tracking condition b is judged to be TRUE, step S6 of KCF tracking module is entered, otherwise step S1 of image data acquisition is continued to be operated;

[0084] S6, tracking link: including a KCF pre-tracking module, a local fusion separation module, a KCF tracking module and updating tracking condition a; wherein the local fusion separation module takes the output result of the KCF pre-tracking module in the previous step as input;

[0085] S7, pan-tilt control link: including a pan-tilt control signal calculation module, a motor execution module, a motor state acquisition module and updating motor stop state.

[0086] The step S6 further comprises:

[0087] S6.1, performing KCF pre-tracking module: the incoming rectangular frame and the current picture are taken as the input data of the module, and the module is triggered only when the tracking condition b is first met as TRUE. The module takes the incoming rectangular frame box as the reference, performs expansion operation in the width direction and the height direction to obtain new_box, expands a frame center as a circle point, expands the box width w and height h by kw and kh times, wherein kw is an empirical value of 2, and kh is an empirical value of 1.2, then the region of interest in the picture is deducted according to new_box and human body detection is performed, and the human body detection result person_box is taken as the output result of the module;

[0088] S6.2, performing local fusion separation module: taking the output result in the previous step S6.2 as the input; judging whether the number of input rectangular frames box is greater than 1,

[0089] if less than 1, entering a single target update module (T),

[0090] the single target update module: acquiring the motion state S (T-1) at the same time, wherein (T-1) represents the motion state S (T-1) of the (T-1)th frame, i.e. the motion state S (T-1) of the previous frame of the current frame; the motion state S value has S = 0, S < 0, S > 0; when S > 0, the box is adjusted, the center point is translated by a pixel Δx in the positive direction of X, Δx is an empirical value, which is 0.2 of the box width here, i.e. Δx = w * 0.2, at the same time, the width of the box is changed to 0.7 times of the original, i.e. new_w = 0.7 * w, and the separation start signal (T) is updated to true; when S < 0, the box is adjusted, the center point is translated by a pixel Δx in the negative direction of X, Δx is an empirical value, which is 0.2 of the box width here, i.e. Δx = w * 0.2, at the same time, the width of the box is changed to 0.7 times of the original, i.e. new_w = 0.7 * w, and the separation start signal (T) is updated to true; when S = 0, the box is not adjusted, and the separation start signal (T) is not updated;

[0091] if greater than 1, entering a multi-target update module (T), wherein (T) represents the (T)th frame;

[0092] the multi-target update module: recording the current tracking target frame as TB, calculating the IOU of TB and all boxes, when there are greater than 0 boxes with IOU > thr, thr is an empirical value of 0.5, taking the box with the largest area as the output of the multi-target update module. When the IOU of all boxes is not greater than thr, selecting the box with the largest area target frame as MB, updating the center point coordinates of TB to the center point coordinates of MB, keeping the width and height of TB unchanged, and taking TB as the output of the multi-target update module;

[0093] And if greater than 1, update the motion state when the following conditions are met simultaneously: when the input box meets the candidate condition,

[0094] Condition a: the height of the box is less than 1.2 of the height of the tracking target box TB height and greater than 0.8 of TB height;

[0095] Condition b: the number of boxes meeting condition a is greater than 1;

[0096] All boxes meeting the candidate box are judged as follows: the box center point coordinate x is recorded as box.x, the input image center point coordinate x is recorded as F.x, the state count variable is recorded as S, and all boxes are traversed, S is increased by 1 when box.x>F.x+7, S is decreased by 1 when box.x<F.x-7, and S=0 in other cases, and the traversal is ended in advance;

[0097] Further comprising: separation signal update, when the separation start signal is true, the separation signal is set to true, and the separation start signal is modified to false at the same time, to prevent repeated updating of the separation signal;

[0098] S6.3, KCF tracking module: judge the separation signal in S6.2, if false, the box in S6.2 is used as the new tracking box and the current image is used as the input data of KCF algorithm, if true, the current tracking box is used and the current image is used as the input data of KCF algorithm, and the counting is started in units of frames, and when the counting is greater than M, the separation signal is updated to false; the result of KCF algorithm is obtained through the input data, which contains the state S of this tracking, indicating whether the tracking fails, 0 indicating failure and 1 indicating success, and the adjusted output rectangle box B3; S6.4, update tracking condition a: set tracking condition a to FALSE according to tracking state S=0, and set tracking condition to TRUE according to S=1.

[0099] The step S4 further comprises:

[0100] S4.1, moving detection module: moving target detection is realized using image frame difference method with T frames apart, and it is judged whether there is a moving target in the image, if there is, the result, i.e. the rectangular frame position, is reserved for alternative optimization; wherein the value of T can be configured as a configurable parameter for user configuration, ensuring that it is a number greater than 1, including 3, 5 and 6;

[0101] S4.2, human body detection module: human body detection is performed on the current picture using Yolov3-tiny CNN convolutional neural network, and if human body is detected, all human body detection results, i.e. rectangular frames, are reserved for alternative optimization;

[0102] S4.3, optimization module: the results of moving detection and human body detection are integrated and optimized to obtain the final tracking target.

[0103] In the step S4.1, the value of T is configured as 4 by default.

[0104] In the step S4.3, the optimization logic is as follows:

[0105] Case 1: When only the mobile detection has a result, first compare the width and height b_wh of the detection result rectangular frame with the preset value min_wh < b_wh < max_wh, the rectangular frame within the condition range is retained, the retained rectangular frame with the largest area is selected as the final rectangular frame, and the tracking condition b is set to TRUE.

[0106] Case 2: When only the human body detection result is obtained, because no mobile detection result is detected, it indicates that there is no moving target in the picture, and the position of the gimbal does not need to be adjusted, and the tracking condition b is set to FALSE.

[0107] Case 3: When both the human body detection result and the mobile detection result are obtained, the mobile detection structure rectangular frame and the human body detection rectangular frame are ANDed, and the human body detection frame with intersection and reaching the preset value is retained as B1, otherwise the tracking condition b is set to FALSE. If the preset tracking mode is to track the historical target preferentially, the B1 and the historical human body detection frame B2 are ANDed, the target with intersection and the largest intersection is taken as the final target detection frame, and the tracking condition b is set to TRUE. If the tracking mode is to track the moving human body target preferentially, the largest area in B1 is selected as the final human body result, and the tracking condition b is set to TRUE.

[0108] In the case 1, the experience values min_wh and max_wh are 50 and frame_h / 2 respectively, and frame_h is the height of the whole frame image. In the case 3, the intersection after the AND operation and the preset value reaching 0.5.

[0109] The step S7 further includes:

[0110] S7.1, a cloud control signal calculation module: the rectangular frame B3 obtained in the step S6 is taken, and the deviation dx and dy of the center point of B3 from the center point of the image, i.e., the center point as the coordinate origin, are calculated, wherein dx represents the deviation of the center point of B3 from the center point of the image on the horizontal axis, and dy represents the deviation of the center point of B3 from the center point of the image on the vertical axis, both in units of pixels.

[0111] S7.2, a motor execution module: according to the dx and dy calculated in the step S7.1, the control signal step or rotation speed of the motor is converted, and the motor execution mechanism is actuated.

[0112] S7.3, obtaining motor state module: mainly obtains motor actuator stop signal STOP, STOP = 1 indicates that the motor is in a stop state, and STOP = 0 indicates that the motor is in a motion state;

[0113] S7.4, updating motor stop state: according to the state obtained in step S7.3, STOP = 0 sets the motor stop state to FALSE, and STOP = 2 sets the motor stop state to TRUE.

[0114] In particular, as shown in Figure 2 The local fusion separation module further comprises:

[0115] The module is used to implement a local fusion separation algorithm, and a rectangular frame box is input. If the number of input rectangular frames box is not greater than 1, a single target updating module (T) is entered. The single target updating module: obtains a motion state S (T-1) according to a motion state S (T-1) at the same time, wherein (T-1) represents the (T-1)th frame, that is, the motion state S (T-1) of the previous frame of the current frame. The motion state S value has S = 0, S < 0, and S > 0. When S > 0, the center point of the box is translated by a pixel Δx in the positive direction of X, Δx is an empirical value, which is 0.2 times the width of the box, that is, Δx = w * 0.2. At the same time, the width of the box is changed to 0.7 times the original width, that is, the new width new_w = 0.7 * w, and the separation start signal (T) is updated to true. When S < 0, the center point of the box is translated by a pixel Δx in the negative direction of X, Δx is an empirical value, which is 0.2 times the width of the box, that is, Δx = w * 0.2. At the same time, the width of the box is changed to 0.7 times the original width, that is, the new width new_w = 0.7 * w, and the separation start signal (T) is updated to true. When S = 0, the box is not adjusted, and the separation start signal (T) is not updated.

[0116] If it is greater than 1, a multi-target updating module (T) is entered, wherein (T) represents the Tth frame.

[0117] The multi-target updating module: records the current tracking target frame as TB, calculates the IOU of TB and all boxes, when there are greater than 0 boxes with IOU > thr, thr is an empirical value of 0.5, and the box with the largest area is selected as the output of the multi-target updating module. When the IOU of all boxes is not greater than thr, the target frame with the largest area in the box is selected and recorded as MB. The center point coordinates of TB are updated to the center point coordinates of MB, the width and height of TB are kept unchanged, and TB is taken as the output of the multi-target updating module.

[0118] If it is greater than 1, when the input box meets the following candidate conditions at the same time, the motion state is updated:

[0119] Condition a: the height of the box is less than 1.2 times the height of the tracking target frame TB and greater than 0.8 times the height of TB.

[0120] Condition b: the number of boxes satisfying condition a is greater than 1;

[0121] The following judgment is made on all boxes satisfying the candidate box, the box center point coordinate x is recorded as box.x, the input image center point coordinate x is recorded as F.x, the state count variable is recorded as S, all boxes are traversed, when box.x>F.x+7, S is added by 1, when box.x<F.x-7, S is reduced by 1, and in other cases, S=0, and the traversal is ended in advance;

[0122] Further comprising updating the separation start signal (T): when the separation start signal is true, the separation signal is set to true, and the separation start signal is modified to false at the same time, so as to prevent repeated updating of the separation signal. The specific operation of updating the separation signal is that if the last frame is the separation start signal (T-1), the current frame is updated to the waiting separation signal (T);

[0123] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application can be variously changed and modified. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A solution to the problem of low-power live tracking gimbal multi-target complex scene, characterized in that, The method comprises a local fusion separation module step: the module is used for realizing a local fusion separation algorithm, and a rectangular frame box is input, Firstly, it is judged whether the number of input rectangular frames box is greater than 1, When it is not greater than 1, a single target update module is entered; When it is greater than 1, a multi-target update module is entered, and when it is greater than 1, a candidate condition also needs to be met to update the motion state S(T) simultaneously, but only S(T-1) of the state S of the last frame enters the single target update module, and not the state of the current frame; The single target update module: the box entering the single target module is obtained, and the following processing is performed according to the motion state S(T-1) of the last frame, wherein T-1 represents the motion state S(T-1) of the last frame of the T-1th frame, that is, the motion state S of the current frame, and the motion state S value has S=0, S<0, and S>0: When S>0, the box is adjusted, the center point is translated by a pixel Δx in the positive direction of X, Δx is an empirical value, and Δx is taken as 0.2 of the width w of the box, that is, Δx=w*0.2, at the same time, the width w of the box is changed to 0.7 times of the original, that is, the new width new_w=0.7*w, and the separation start signal is updated to true; When S<0, the box is adjusted, the center point is translated by a pixel Δx in the negative direction of X, Δx is an empirical value, and Δx is taken as 0.2 of the width w of the box, that is, Δx=w*0.2, at the same time, the width w of the box is changed to 0.7 times of the original, that is, the new width new_w=0.7*w, and the separation start signal is updated to true; When S=0, the box is not adjusted, and the separation start signal is not updated; The multi-target update module: the current tracking target frame is recorded as TB, the intersection over union IOU of TB and all boxes is calculated, when there is a box with IOU>thr, wherein thr is an empirical value of 0.5 by default, the box with the largest area is taken as the output of the multi-target update module; when the IOU of all boxes is not greater than thr, the target frame with the largest area in the box is selected and recorded as MB, the center point coordinates of TB are updated to the center point coordinates of MB, the width and height of TB are kept unchanged, and TB is taken as the output of the multi-target update module; After the single target update module is completed, the update separation start signal is performed, when the separation start signal is true, the separation signal is set to true, and the separation start signal is modified to false to prevent repeated update of the separation signal; The method comprises the following steps: S1, acquiring video image data: real-time code stream data is acquired through a camera installed on a holder, and each frame of image data is obtained; S2, tracking condition a: used for judging the state machine, the state machine indicates that there are three states: a detection state, a tracking state, and a waiting state, if it is judged that the current state machine is in the tracking state, the result is TRUE, if the result is TRUE, step S6 of a tracking link is directly entered, otherwise, step S3 of a motor stop state judgment is entered; S3, motor stop state: if TRUE, step S4 of a detection link is entered, otherwise, step S1 is continuously executed; S4, detection link: including a moving detection module, a human body detection module, and an optimization module; S5, tracking condition b: for a state identifier, the specific identifier setting condition is updated by the optimization module in step S4, if it is judged that the tracking condition b is TRUE, enter the KCF tracking module of step S6, otherwise continue to step S1 to operate to obtain image data; S6, tracking link: including KCF pre-tracking module, local fusion separation module, KCF tracking module; and updating tracking condition a; wherein the local fusion separation module takes the output result of the KCF pre-tracking module in the previous step as input; S7, head control link: including calculating head control signal module, motor execution module, obtaining motor state module; and updating motor stop state.

2. The method of claim 1, wherein the method further comprises: Step S6 further comprises: S6.1, KCF pre-tracking module: the incoming rectangular box and the current picture are taken as the input data of the module, and the module is triggered only when the tracking condition b is TRUE for the first time; the module takes the incoming rectangular box as the reference, performs expansion operation in the width direction and the height direction to obtain new_box, expands a box center to a circle point, expands the box width w and height h by kw and kh times, wherein kw is an empirical value of 2, kh is an empirical value of 1.2, then the interested region in the picture is deducted according to new_box and human body detection is performed, and the human body detection result person_box is taken as the output result of the module; S6.2, local fusion separation module: taking the output result of the previous step S6.2 as input; determine whether the number of input rectangular boxes is greater than 1, if not greater than 1, enter the single target update module, the single target update module: simultaneously according to the motion state S(T-1), wherein T-1 represents the T-1 frame, i.e. the motion state S(T-1) of the previous frame of the current frame; the motion state S value has S=0, S<0, S>0; when S>0, adjust the box, the center point is translated by pixels Δx in the X positive direction, Δx is an empirical value, which is 0.2 of the box width here, i.e. Δx=w*0.2, at the same time, the width of the box is 0.7 times of the original, i.e. new_w=0.7*w, and the separation start signal is updated to true; when S<0, adjust the box, the center point is translated by pixels Δx in the X negative direction, Δx is an empirical value, which is 0.2 of the box width here, i.e. Δx=w*0.2, at the same time, the width of the box is 0.7 times of the original, i.e. new_w=0.7*w, and the separation start signal is updated to true; when S=0, the box is not adjusted, and the separation start signal is not updated; if greater than 1, enter the multi-target update module; The multi-target updating module: record the current tracking target box as TB, calculate the IOU of TB and all boxes, when there is more than 0 box with IOU>thr, thr is the default value of 0.5, take the box with the largest area as the output of the multi-target updating module; when the IOU of all boxes is not greater than thr, select the box with the largest area as MB, update the center point coordinates of TB to the center point coordinates of MB, keep the width and height of TB unchanged, and take TB as the output of the multi-target updating module; If greater than 1, when the input box meets the following candidate conditions, update the motion state: Condition a: the height of the box is less than 1.2 times the height of the tracking target box TB and greater than 0.8 times the height of TB; Condition b: the number of boxes meeting condition a is greater than 1; For all boxes meeting the candidate box, the following judgment is made, record the box center point coordinates x as box.x, the input image center point coordinates x as F.x, the state count variable as S, traverse all boxes, when box.x>F.x+7, S is increased by 1, when box.xF.x-7, S is reduced by 1, and in other cases, S=0, and the traversal is ended in advance; Further comprising: separate signal update, the specific operation is that if the last frame is a separate start signal T-1, the current frame is updated to a waiting separate signal; S6.3, KCF tracking module: judge the separate signal in S6.2, if false, take the box in S6.2 as a new tracking box and take the current image as input data of KCF algorithm, if true, continue to use the current tracking box and the current image as input data of KCF algorithm, and start counting by frame, when the count is greater than M, update the separate signal to false; get the result of KCF algorithm through the input data, the result contains the state S of this tracking, which indicates whether the tracking fails, 0 indicates failure, 1 indicates success, and the adjusted output rectangle B3; S6.4, update tracking condition a: set tracking condition a to FALSE according to tracking state S=0, and set tracking condition to TRUE according to S=1.

3. The method of claim 1, wherein the method further comprises: Step S4 further comprises: S4.1, moving detection module: moving target detection is realized by using image frame difference method with T frames apart, judge whether there is a moving target in the image, if there is, the result, i.e. the rectangular frame position, is reserved for optional optimization; wherein the value of T can be configured as a configurable parameter for users to configure, ensuring that it is a number greater than 1, including 3, 5 and 6; S4.2, human body detection module: use Yolov3-tiny CNN convolutional neural network to detect human body in the current image, if human body is detected, all human body detection results, i.e. rectangular frames, are reserved for optional optimization; S4.3, optimization module: the results of moving detection and human body detection are combined for optimization to obtain the final tracking target.

4. The method of claim 3, wherein, In step S4.1, the value of T is configured as 4 by default.

5. The method of claim 3, wherein the method further comprises: In the optimization module of step S4.3, the specific optimization logic is as follows: Case 1: when only the mobile detection has a result, first compare the width and height b_wh of the detection result rectangular frame with the preset value min_wh < b_wh < max_wh, the rectangular frame within the condition range is retained, the retained rectangular frame with the largest area is selected as the final rectangular frame, and the tracking condition b is set to TRUE; Case 2: when only the human body detection result is obtained, because no mobile detection result is detected, it indicates that there is no moving target in the picture, and the position of the holder does not need to be adjusted, the tracking condition b is set to FALSE; Case 3: when there are both human body detection results and mobile detection results, the mobile detection structure rectangular frame and the human body detection rectangular frame are ANDed, and the human body detection frame with intersection and reaching the preset value is retained as B1, otherwise the tracking condition b is set to FALSE, if the preset current tracking mode is to track the historical target preferentially, then B1 and the historical human body detection frame B2 are ANDed, the target with intersection and the largest intersection is selected as the final target detection frame, and the tracking condition b is set to TRUE, and if there is no intersection, the tracking condition b is set to FALSE; If the tracking mode is to track the moving human body target preferentially, the largest area in B1 is selected as the final human body result, and the tracking condition b is set to TRUE. In the case 1, the experience value min_wh is 50, and the experience value max_wh is frame_h / 2; frame_h is the height of the whole frame image; in the case 3, the intersection after the AND operation reaches the preset value of 0.

5.

6. The method of claim 1, wherein the method further comprises: The holder control link further comprises the following steps: S7.1, a cloud control signal calculation module: the rectangular frame B3 obtained in step S6 is taken, and the deviation dx, dy of the center point of B3 from the center point of the image, that is, the center point as the coordinate origin, wherein dx represents the deviation of the center point of B3 from the center point of the image on the horizontal axis, and dy represents the deviation of the center point of B3 from the center point of the image on the vertical axis, both in units of pixels; S7.2, a motor execution module: according to the dx and dy calculated in step S7.1, the control signal step or rotation speed of the motor is converted, and the motor execution mechanism is actuated; S7.3, a motor state acquisition module: mainly acquires the motor stop signal STOP, STOP=1 indicates that the motor is in a stop state, and STOP=0 indicates that the motor is in a moving state; S7.4, update the motor stop state: according to the state acquired in step S7.3, STOP=0 sets the motor stop state to FALSE, and STOP=2 sets the motor stop state to TRUE.

Images