Human body long-time tracking method and device based on visual perception

Abstract

The invention belongs to the technical field of visual perception processing, and particularly relates to a human body long-time tracking method and device based on visual perception. The method comprises the following steps: carrying out human body detection by using a deep learning detection model, and then screening out a tracking target; meanwhile, a target tracker and a feature library are created, the target tracker is used for matching the tracking target in real time, and the feature library is used for finding the tracking target after the tracking target is lost; executing target matching tracking and updating tracking target information in subsequent frame processing for determining the tracking target; target loss and rematching; and when the number of continuous unmatched tracking frames is greater than or equal to a frame number threshold value, triggering a target loss recovery mechanism, namely, realizing cross-frame identity re-association by comparing the historical appearance characteristics of the tracking target with the appearance characteristics of the detection target in the current frame. According to the invention, the problems of background interference in a high-complexity, strong-dynamic and unstructured environment, low reliability and efficiency of a human body following robot, and identity consistency maintenance of target detection and tracking in a complex dynamic environment are solved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of visual perception processing, and more particularly, to a human long-time tracking method and device based on visual perception. BACKGROUND

[0002] Under the driving of multiple factors such as consumption upgrading, resource tightening, and structural shortage of labor force, large-scale and intensive farming has become the mainstream trend, and intelligent and automated technology is the core driving force to promote the transformation and upgrading of this industry. Under this background, collaborative robots taking "human-robot enhanced collaboration" as the core concept show great application potential. Among them, human following robots, as an important carrier of this concept, can track the movement of specific workers in real time and perform auxiliary tasks, such as serving as an intelligent handling device in dead / diseased bird picking and transportation, as a mobile intelligent sensing and data hub in daily inspection, and as a mobile supply platform in group immunity and disinfection operations, thereby significantly improving work efficiency, reducing biosecurity risks, and improving fine management level.

[0003] Chinese patent document CN112288773A discloses a multi-scale human tracking method and device based on Soft-NMS, which includes (1) extracting human targets through a Yolov3 network, then performing non-maximum suppression based on the DIoU of the target frame, and then using a standard Kalman filter based on constant speed and linear observation model to predict the target motion state, using Mahalanobis distance to represent the correlation degree between the predicted frame and the next frame target frame, representing distance features; (2) using a deep learning model trained offline on the Mars dataset to extract human appearance features; (3) fusing the distance features and appearance features by scale, as the matching measurement standard for determining the next frame target, and performing target confirmation through cascading matching, constantly updating the input frame, and finally completing the tracking task.

[0004] In existing single-target human tracking technology, methods based on wireless communication rely on dedicated signal equipment and are easily affected by multipath effects and equipment deployment costs, resulting in significant precision decline in metal structure-intensive breeding houses; methods based on laser radar use point cloud data for target tracking, which has high accuracy in distance measurement but cannot provide target identity information, leading to identity confusion in multi-person scenarios, and the sensor cost is high; methods based on machine vision have become a research hotspot due to their low hardware cost, rich perception information, and high compatibility with deep learning technology. However, direct application of such general frameworks to livestock and poultry farming environments has the following problems: Poor real-time performance: general detection models have large computational load, making it difficult to meet real-time requirements on resource-limited embedded or edge computing devices.

[0005] Robustness is insufficient: the traditional appearance model updating strategy is easy to introduce noise after the tracking target is partially / fully occluded by other livestock or equipment, leading to model drift and tracking failure.

[0006] Long-time tracking ability is weak: once the target is lost, there is no effective recovery mechanism, and it is difficult to re-associate the correct identity. SUMMARY

[0007] The present application aims to overcome at least one of the above-mentioned defects of the prior art, and provides a long-time tracking method and device for human body based on visual perception, to solve the problems of low reliability and efficiency of human body following robots in complex, dynamic and unstructured environments, such as frequent target occlusion, complex light changes, dense obstacles and highly similar background interference, and the problem of maintaining the identity consistency of target detection and tracking in complex dynamic environments.

[0008] The detailed technical solutions of the present application are as follows: A long-time tracking method for human body based on visual perception, the method comprising: S1, using a deep learning detection model to detect human body in the first frame of image, and then screening out the tracking target; S2, creating a target tracker and a feature library based on the tracking target, the target tracker is used to match the tracking target in real time and record the tracking target information, and the feature library is used to find back the tracking target after it is lost; S3, detecting the human body target in the non-first frame image frame of the tracking target, when the number of consecutive unmatched tracking frames is less than the frame number threshold, performing matching tracking of the detected target and the tracking target; If the matching is successful, the tracking target information in the target tracker is updated; If the matching fails, the number of consecutive unmatched tracking frames is increased by 1, and the processing of the next frame of image is continued; S4, when the number of consecutive unmatched tracking frames is greater than or equal to the frame number threshold, the tracking target is lost and the target loss recovery mechanism is triggered: by comparing the historical appearance features of the tracking target in the feature library with the appearance features of the detected target in the current frame, the cross-frame identity re-association is realized.

[0009] Further, the human body detection using a deep learning detection model, and then screening out the tracking target specifically comprises: S1.1, obtaining an initial frame of image, and using a deep learning detection model to identify all human body categories in the initial frame of image, to obtain the tracking target detection result; S1.2, performing tracking target screening on the detected human body through confidence and spatial scale double criteria constraints, to obtain the final target set meeting the constraint conditions; S1.3, determining a tracking target from the final target set meeting the constraint condition in an automatic or interactive selection manner.

[0010] Further, the confidence and spatial scale dual criteria specifically include: S1.2.1, setting a confidence threshold , only retaining high-quality detection results with a confidence score greater than the confidence threshold ; S1.2.2, respectively calculating the detection bounding box area and the width-height ratio of each target in the detection result ; Then, excluding detection bounding box area , as a detection bounding box area threshold, and the width-height ratio abnormal and invalid detection, and storing those meeting the constraint condition into the final target set meeting the constraint condition .

[0011] Further, the S1.3 specifically includes: S1.3.1, in an automatic selection mode, selecting the human bounding box with the largest detection bounding box area as the tracking target from the ; S1.3.2, in an interactive selection mode, allowing the user to specify the target to be tracked through manual framing or clicking from the ; S1.3.3, if the is an empty set, i.e., no human category is detected or the detected human does not meet the target screening condition, returning to S1.1 to reacquire the image and identify and screen; otherwise, continuing to execute.

[0012] Further, the creating a target tracker and feature library specifically includes: S2.1, based on the determined tracking target , creating a target tracker; The target tracker includes tracking target bounding box information, appearance raw data, and continuous unmatched frame number information; the target bounding box information includes original bounding box and predicted trajectory information, the original bounding box information being the , , pixel coordinates of the upper left corner of the tracking target, respectively being the width and height of the tracking target bounding box; the predicted trajectory information is , wherein is the center point position pixel coordinates of the tracking target in the next frame predicted by the trajectory prediction algorithm, The length and width of the bounding box of the tracked target in the next frame predicted by the trajectory prediction algorithm; They are respectively The rate of change of all values ​​is 0. The original appearance data is for tracking targets. Original appearance image within the original bounding box ; The number of consecutive unmatched tracking frames That is, the number of consecutive frames in which the tracked target is not matched from the first unmatched frame. The initial value is 0, and it is incremented by 1 when the tracked target is not matched, and reset to zero when the tracked target is matched. S2.2 Construct a feature library for retrieving targets after they have been lost; The feature library includes original appearance images. and high-quality appearance library ; The original appearance image For the identified tracking target Original appearance image within the original bounding box Values ​​are assigned synchronously when the target tracker is created; The high-quality appearance library Used to store the original images of the tracking target that are evaluated as high quality by the appearance feature quality assessment method during target matching and long-term tracking; The high-quality appearance library Fixed storage capacity The original appearance data is dynamically updated, following the first-in, last-out principle. Original appearance data; The appearance feature quality assessment method is used when the size of the tracking target to be assessed is greater than a preset size threshold. And the confidence level is higher than the detection bounding box area threshold. At that time, the sample was determined to be a high-quality sample, that is... ,in, For the tracking target to be evaluated, For the tracking target to be evaluated confidence level The area of ​​the tracked target to be evaluated. ， The tracking targets to be evaluated are respectively The width and height of the bounding box.

[0013] Furthermore, in each subsequent frame processing after determining the tracking target, target matching tracking is performed, and the tracking target information is updated. Specifically, S3 includes: S3.1, using a detection model of deep learning, detecting human targets in non-first frame image frames of the tracking target to obtain a detection target set N is the total number of detected human targets, is the label number of the human target; each detected human target saves the bounding box information , wherein is the human target the pixel coordinates of the upper left corner of the bounding box, respectively, the width and height of the bounding box; whether the number of continuous unmatched tracking frames is greater than or equal to 15 frames; if yes, target loss and rematching are performed; otherwise, S3.2 is performed; S3.2, excluding detected human targets without intersection with the tracking target, and forming a candidate subset ; using the bounding box information in the predicted trajectory information and the candidate targets in the candidate subset to calculate IoU, and from the IoU cost vector , the detection target corresponding to the minimum cost is selected, and if the IoU value of the detection target corresponding to the minimum cost is less than or equal to the IoU threshold, the matching is successful, and the tracking target information is updated; If the matching fails, the ReID fusion matching is activated, the appearance feature vectors of all detection targets in the original appearance image of the tracking target are extracted, the cosine distance between the two is calculated, and the feature fusion is performed after normalization, the corresponding detection target is confirmed according to the fusion vector, and secondary matching is performed again, if the secondary matching is successful, the tracking target information is updated; if the secondary matching fails, the continuous unmatched tracking frame number information is increased by 1, and the processing of the next frame image is continued.

[0014] The application improves the matching accuracy through secondary matching. The specific ReID fusion matching method further reduces the possibility of misjudgment by fusing the appearance feature vectors of the tracking target and the detection target.

[0015] Further, the S3.2 specifically includes: S3.2.1, a spatial gate pre-filtering stage: excluding detection targets without intersection with the tracking target; First, the IoU value of the tracking target in the tracker and the detected human target in the current image frame is calculated ; (1); In formula (1): (2); (3); Then filtering, in Screening values greater than 0 as candidate targets for tracking, candidate targets are stored in candidate subset The number of candidate targets in the candidate subset is recorded as M : S3.2.2, IoU priority matching; First, using the bounding box information In the predicted trajectory information , calculate the IoU with the candidate targets in , generate the IoU cost vector : (4); Wherein: (5); (6); Then, from the IoU cost vector , select the detection target corresponding to the minimum cost : (7); If the IoU value Of the detection target corresponding to the minimum cost Is less than or equal to the IoU threshold , the matching is successful; if the IoU value Of the detection target corresponding to the minimum cost Is greater than the IoU threshold , the matching is successful or failed; If the matching is successful, execute S3.2.3; if the matching fails, execute S3.2.4; S3.2.3, update the tracking target information in the target tracker; If the IoU matching is successful, use the bounding box Corresponding to the detection target Update the Of the tracking target ; use the original appearance image data in the bounding box corresponding to the detection target Update the original appearance image Of the tracking target ; use the trajectory prediction algorithm, i.e. Kalman filter, to calculate the predicted trajectory information Of the tracking target in the next frame, which is used for target matching and tracking; using the appearance feature quality evaluation method corresponding appearance feature, update the high-quality appearance library : if the tracking target appearance image is high-quality, update to ; otherwise, do not update continuous unmatched tracking frame number information clear; return to S3.1, continue target recognition and detection of the next frame S3.2.4, ReID fusion matching (1) using the appearance feature extraction algorithm to extract the original appearance image of the tracking target appearance feature vector and the appearance feature vectors of all detection targets in the candidate subset (2) calculate the cosine distance between the appearance feature vector of the tracking target and the appearance feature vectors of all targets in the candidate subset , and perform normalization processing to form a normalized appearance feature distance vector : (3) respectively screen the elements in and , if the distance value is greater than the corresponding threshold value, set the distance value to 1: (8); (9); wherein represents the appearance distance threshold value, which is used to screen the maximum threshold value of the appearance feature cosine distance between the tracking target and the targets in the candidate subset calculated in (2); represents the normalized appearance feature distance vector.

[0016] (4) take the minimum value of the screened appearance distance vector and the IoU distance cost vector according to the element corresponding position, realize multi-modal data fusion, and generate a fusion vector : (5) in the fusion vector , select the detection target corresponding to the minimum fusion distance ; if the minimum distance is lower than the preset threshold value , it is determined that the matching is successful; otherwise, the matching fails.

[0017] ​​​If the matching is successful, S3.2.3 is executed, i.e. updating the tracking target information in the target tracker; otherwise, the continuous unmatched tracking frame number information Add 1, and execute S3.1.

[0018] Further, the feature library includes original appearance images and high-quality appearance libraries , and the S4 specifically includes: S41, feature library and current frame detection target feature extraction; The feature library of the tracking target is composed of the high-quality appearance library updated dynamically before the loss and the original appearance image, denoted as : (12); The appearance features in the tracking target original image and the appearance features of all targets in the current frame detection target set are extracted using the appearance feature extraction algorithm, wherein each feature in the current frame detection target set is denoted as , is the label number of the appearance feature, and : : (13); S42, feature matching; The cosine distances between the tracking target appearance features and the appearance features of all detection targets in the current frame detection target set are calculated, and normalization processing is performed to obtain a distance cost matrix , and the minimum distance value is selected from the distance cost matrix : (10); (11); If the minimum distance is lower than a threshold , it is determined that the target matching is successful, i.e. , and the corresponding detection target in the current frame detection target set is the i-th detection target; otherwise, the re-association fails, i.e. ; (14); S43, updating the tracker information; ​​​​If the reassociation succeeds, the tracking target is updated using the bounding box corresponding to the reassociated detection target updating the tracking target ; updating the tracking target using the original appearance image data of the bounding box corresponding to the reassociated detection target ; updating the prediction trajectory information using the updated tracking target , setting to 0; calculating the prediction trajectory information of the tracking target in the next frame using a trajectory prediction algorithm, i.e., a Kalman filter ; at this time, the number of consecutive unmatched tracking frames is cleared, and the subsequent processing of each frame is returned to S3; S44, if the reassociation fails, the number of consecutive unmatched tracking frames is incremented by 1, and the subsequent processing of each frame is returned to S3.

[0019] In another aspect of the present application, a device for implementing a human long-time tracking method based on visual perception is provided, and the device comprises: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform a human long-time tracking method based on visual perception as described above.

[0020] In another aspect of the present application, a computer readable storage medium storing executable instructions is also provided, and the instructions, when executed, cause the machine to perform a human long-time tracking method based on visual perception as described above.

[0021] Compared with the prior art, the present application has the following advantages: (1) The human long-time tracking method and device based on visual perception provided by the present application use an efficient target matching mechanism, a proposed spatial gating cascade matching strategy, and a coarse-to-fine IoU and ReID fusion matching to quickly exclude a large number of irrelevant interferences through pre-filtering, thereby greatly reducing the computational overhead while ensuring high matching accuracy, and the present application is particularly suitable for breeding environments with slow target movement and many scene interferences.

[0022] ​​​​(2) The human long-time tracking method and device based on visual perception provided by the application introduces a dynamic quality perception feature updating mechanism, which can effectively identify and filter low-quality, occluded or severely disturbed target images, avoid pollution and drift of the appearance model due to the introduction of noise in the updating process, and thus maintain the continuity and stability of identity recognition in typical livestock and poultry breeding scenes such as partial occlusion, complete occlusion and light mutation.

[0023] (3) The human long-time tracking method and device based on visual perception provided by the application has target long-time tracking and recovery capability. By maintaining a dynamically updated high-quality historical appearance feature library and starting cross-frame re-identification after the target is lost, the problem of target repositioning and identity re-association after the target is out of sight for a long time is effectively solved, greatly improving the continuous tracking capability and practical application value of the tracking system in complex scenes. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a flowchart of the human long-time tracking method based on visual perception described in the application.

[0025] Figure 2 is an initial frame image example containing two pedestrians in embodiment 1 of the application.

[0026] Figure 3 is a schematic diagram of the 2nd, 10th and 15th frame images containing two pedestrians in embodiment 1 of the application. DETAILED DESCRIPTION

[0027] The application will be further described below in conjunction with the drawings and embodiments.

[0028] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the application belongs.

[0029] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof.

[0030] The embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0031] Embodiment 1 ReferenceFigure 1 The embodiment provides a human long-time tracking method based on visual perception. S1, a human body is detected in a first frame image by using a deep learning detection model, and then a tracking target is screened out. Specifically, the S1 specifically includes the following steps. S1.1, human body detection using deep learning; Preferably, an initial frame of an image is acquired, and all human body categories in the initial frame are recognized by using a deep learning detection model. The detection model outputs an original bounding box of each human body category and a confidence score ; wherein is the pixel coordinate of the upper left corner of the bounding box, is the width and height of the bounding box, respectively; the deep learning detection model includes but is not limited to a YOLO series, an RT-DETR and the like.

[0032] The YOLOv8n model is used as the deep learning detection model, and it is assumed that two pedestrians are detected in the initial frame. As shown in Figure 2 , 1 is a pedestrian target , and 2 is a pedestrian target . The output information of each pedestrian is shown in the following table: Table 1: Output information of pedestrians

[0033] S1.2, target screening based on multi-criteria fusion; the detected human bodies are screened for tracking targets based on confidence and spatial scale.

[0034] Preferably, the S1.2 specifically includes the following steps. S1.2.1, confidence threshold screening, which can be adjusted according to the actual application scenario. The greater the value, the more reliable the screened detection target, and at the same time, the tracking target may be lost. In this embodiment, the confidence threshold is set to 0.1, and only high-quality detection results with a confidence score are retained . Comparing the data in the above table, that is, both can be retained.

[0035] S1.2.2, spatial scale constraint, invalid detection with an abnormal area or an aspect ratio of a detection bounding box is excluded from , and the detection meeting the constraint condition is stored in ; wherein, pixel2, pixel2 is a unit and can also be expressed as px2 ; , normal range is .

[0036] Set spatial scale constraint parameters: detect bounding box area threshold Pixel2, normal range of aspect ratio ; respectively calculate the detection bounding box area and aspect ratio of each target in : area A = 80 x 160 = 12800 pixels2, , meet the constraints; : area A = 85 x 165 = 14025 pixels2, , meet the constraints; The final target set that meets the constraint conditions .

[0037] S1.3, target selection mechanism; determine the tracking target in in an automatic or interactive selection manner , the specific mode is: S1.3.1, automatic selection mode, select the human body bounding box with the largest detection bounding box area from as the tracking target; S1.3.2, interactive selection mode, allow the user to specify the target to be tracked through manual framing or clicking from ; S1.3.3, if is an empty set, i.e. no human body category is detected or the detected human body does not meet the target screening condition, return to S1.1 to reacquire the image and identify and screen; otherwise, continue to execute; Preferably, the embodiment adopts the automatic selection mode, calculates the detection bounding box area of each target in : (12800 pixels2), (11625 pixels2), select the with the largest area as the tracking target , and the bounding box information is (1050, 440, 85, 165), and the confidence is 0.95.

[0038] S2, target tracker tracking and feature library construction; Based on creating a target tracker and a feature library, the target tracker is used for real-time matching tracking targets, and the feature library is used for finding back after the tracking target is lost, specifically including: S2.1, create a target tracker; Based on the determined tracking target Create a target tracker; the target tracker includes tracking target bounding box information, appearance raw data, and continuous unmatched frame number information; the target bounding box information includes raw bounding box and predicted trajectory information; the raw bounding box information is the tracking target of wherein is the pixel coordinate of the upper left corner of the tracking target, respectively, the width and height of the tracking target bounding box; the predicted trajectory information is wherein is the pixel coordinate of the center point of the tracking target bounding box in the next frame predicted by the trajectory prediction algorithm, is the length and width of the tracking target bounding box in the next frame predicted by the trajectory prediction algorithm; respectively change rate, the initial value is 0; the appearance raw data is the original appearance image in the tracking target raw bounding box; The trajectory prediction algorithm specifically uses a Kalman filter; The continuous unmatched tracking frame number information is the frame number of the tracking target from the beginning of the first unmatched to the continuous unmatched, the initial value is 0, and it is increased by 1 when the tracking target is not matched, and it is cleared when the tracking target is matched; Based on the tracking target Create a DCF (correlation filter) tracker based on fused Kalman filter, and the core information stored by the tracker is as follows: Raw bounding box information: ; Predicted trajectory information: in the initial state, the state vector of the Kalman filter is , that is wherein =1050+85 / 2=1092.5、 is the center point coordinate of the bounding box; Appearance raw data: the 85x165 pixel RGB image obtained by cropping the bounding box region from the initial frame ; Continuous unmatched frame number : initial value 0; The state equation and observation equation of the Kalman filter are set as follows: the state transition matrix F adopts an 8x8 dimension matrix, the process noise Q=diag([1e-4,1e-4,1e-4,1e-4,1e-3,1e-3,1e-3,1e-3]), and the observation noise R=diag([1e-2,1e-2,1e-2,1e-2]).

[0039] S2.2, feature library construction; constructing a feature library for finding a target after losing the target; the feature library includes original appearance features and a high-quality appearance library ; The original appearance features are original appearance images in original bounding boxes , which are assigned synchronously when creating a target tracker; the high-quality appearance library is used to save original appearance images of tracking targets evaluated as high quality by an appearance feature quality evaluation method in the process of target matching and long-time tracking; The high-quality appearance library stores a fixed capacity of appearance original data, dynamically maintains the latest pieces of appearance original data in accordance with the principle of first-in, first-out; and the appearance feature quality evaluation method is that when the size of a tracking target to be evaluated is greater than a preset size threshold = 200 pixels2and the confidence is higher than a threshold , the sample is determined as a high-quality sample, that is , wherein is the tracking target to be evaluated, is the area of the tracking target to be evaluated, ， are the width and height of the bounding box of the tracking target to be evaluated, respectively, and is the confidence of the tracking target.

[0040] S3, target matching and long-time tracking; detecting a human target in a non-first image frame of a tracking target, and performing matching tracking of the detected target and the tracking target when the number of consecutive unmatched tracking frames is less than a frame threshold; If the matching is successful, the tracking target information in the target tracker is updated, that is, the tracking target information is updated using the current detected target information; If the matching fails, the number of consecutive unmatched tracking frames is incremented by 1, and the processing of the next image frame is continued.

[0041] Specifically, S3 specifically includes: S3.1, human detection using deep learning; using the detection model using deep learning described in S1.1 to detect human targets in an image frame to obtain a detected target set ; each detected human target saves bounding box information​ , the original appearance image within the bounding box , confidence of ; For each frame image after the initial frame (denoted as the t-th frame, t≥2), the target matching tracking is performed in four sequential stages. This embodiment takes t=2 (the 2nd frame), t=10 (the 10th frame, the target is partially occluded), and t=15 (the 15th frame, the target reappears) as examples for illustration, as shown in FIG. 2. Figure 3 Figure 3 In FIG. 2, reference numeral 3 represents a pedestrian target , reference numeral 4 represents a pedestrian target , and reference numeral 5 represents an obstacle.

[0042] For the t-th frame image (t=2, 10, 15), the same YOLOv8n detection model as S1.1 is used for human body detection, and the detection target set , the bounding box, the confidence, and the appearance image information of each target are output. The detection results of each frame are as follows:

[0043] It is determined whether the number of consecutive unmatched tracking frames is greater than or equal to 15 frames; if yes, target loss and rematching, i.e., S4, are performed; otherwise, S3.2 is performed.

[0044] S3.2, cascade target matching tracking based on spatial gating; The detected human body targets that have no intersection with the tracking target are excluded, and a candidate subset is formed. The bounding box information in the predicted trajectory information is used to calculate the IoU with the candidate targets in the candidate subset . From the IoU cost vector , the detection target corresponding to the minimum cost is selected. If the IoU value of the detection target corresponding to the minimum cost is less than or equal to the IoU threshold, the matching is successful, and the tracking target information is updated. If the matching fails, ReID fusion matching is activated. The cosine distance between the appearance feature vector of the original appearance image of the tracking target and the appearance feature vectors of all detected targets is calculated. After normalization, the feature fusion is performed, the corresponding detection target is confirmed according to the fusion vector, and the matching is performed again. If the matching is successful, the tracking target information is updated. Otherwise, the number of consecutive unmatched tracking frames is incremented by 1, and S3.1 is performed.

[0045] ​Specifically, the spatial gating based cascade target matching tracking includes four sequentially executed stages of spatial gating pre-filtering, IoU priority matching, updating tracking target information, and ReID fusion matching.

[0046] S3.2.1, the spatial gating pre-filtering stage: excluding detection targets without intersection with tracking targets, aiming to reduce the amount of calculation, specifically including: First, taking t=2 frames as an example, the tracking target in the tracker (initial bounding box (1050, 440, 85, 165)) is calculated. The IoU value is calculated ; The formula is as follows: (1).

[0047] Among them: (2); (3).

[0048] Then, in , the values greater than 0 are selected as candidate targets of the tracking target, and the candidate targets are stored in , the number of candidate targets in is recorded as M :

[0049] The calculation result is: =0.82, =0.0. The candidate targets with >0 are screened, and ={ } is obtained, and M=1; At t=10 frames, the current bounding box is updated to (1100, 425, 82, 163), and the IoU with the detection target set in is 0.58, and the IoU value with d 2 is 0, ={ } is obtained, and M=1.

[0050] S3.2.2, IoU priority matching; Taking t=2 frames as an example, the bounding box of the predicted trajectory information output by the Kalman filter , is calculated. The IoU of the candidate target in is calculated, and the IoU cost vector is generated: (4); In this embodiment, ,but: Pixels 2 (5); Pixels 2 (6); Match successful. Execute S3.2.3 to update tracking information.

[0051] Taking t=10 frames as an example, in this embodiment, ,and Calculate IoU: Pixels 2 (15); Pixels 2 (16); Tracking target and candidate target Match successful. Execute S3.2.3 to update tracking information.

[0052] Then, from the IoU cost vector In the middle, select the detection target corresponding to the minimum cost. In this embodiment, : (7); If the detection target corresponds to the minimum cost IoU value Less than or equal to the IoU threshold If the match is successful, then the ReID fusion matching phase is activated; otherwise, the matching phase is activated. (17); If a match is found, proceed to step S3.2.3; otherwise, proceed to step S3.2.4. S3.2.3 Update tracking target information If the IoU match is successful, then the detection target is used. corresponding bounding box Update tracking target of Use detection target Update the tracking target using the original appearance image data within the corresponding bounding box. Original appearance image The trajectory prediction algorithm, namely the Kalman filter, is used to calculate the predicted trajectory information of the tracked target in the next frame. .

[0053] The appearance feature quality evaluation method is used to evaluate the detection target The corresponding appearance feature, update the high-quality appearance library If the tracking target appearance image is of high quality, update to Otherwise, do not update; the continuous unmatched tracking frame number information is cleared. After t=2 frame matching is successful, update the tracker information: The tracker bounding box: update the tracking target with (1060, 435, 86, 164) of the detection target ; ; The appearance data: replace with the 86x164 pixel image within the bounding box; The predicted trajectory: the Kalman filter calculates the Kalman predicted trajectory information of t=3 frame based on the new bounding box as ; Appearance library update: Area = 86x164 = 14104 pixels²≥200 pixels², s=0.94≥0.55, determine as a high-quality sample, store in and remove the earliest data (at this time only contains the frame data); Clear to 0.

[0054] After t=10 frame update, the bounding box is (1120, 420, 60, 120), and the Kalman predicted trajectory information is , Area = 60x120 = 7200≥200, s=0.78≥0.55, update to the high-quality appearance library ; Return to S3.1, continue target recognition and detection of the next frame.

[0055] S3.2.4, ReID fusion matching, in the ReID fusion matching stage, the target association task is completed by fusing multi-modal distance measurement.

[0056] This embodiment takes t=12 frame when the crowd is interlaced, causing the IoU with the candidate target to rise, verifying the effectiveness of the ReID fusion mechanism. t=12 frame , : (1130, 415, 70, 130), s=0.82; : (1140, 420, 68, 128), s=0.80; (400, 428, 79, 159), s = 0.91. Spatial gated pre-filtering. M=2.

[0057] IoU priority matching: ,and of =0.65, and of =0.62, all greater than =0.5, activate ReID fusion matching.

[0058] (1) Feature extraction: Extracted using the FastReID model Appearance features (2048-dimensional vector), and Features 、 ; (2) Distance calculation: = ≈0.12; =0.38; (3) Distance filtering: =0.25, =0.5, =0.12, =1; =0.65, =1; (4) Fusion computing: =min(0.12,0.65)=0.12; =min(1,1)=1; (5) Matching judgment: Minimum fusion distance 0.12 < =0.5, select To match the target, if the match is successful, execute S3.2.3 to update the information.

[0059] S4. Target loss and rematch.

[0060] When the number of consecutive unmatched tracking frames A frame count threshold of 15 frames or more, meaning no target is matched for 15 consecutive frames, indicates target loss and triggers a target loss recovery mechanism. The core of this mechanism is to achieve cross-frame identity re-association by comparing the historical appearance features of the tracked target in the feature library with the appearance features of the detected target in the current frame.

[0061] S41. Assuming the target is tracked at t=30 frames... Completely obscured by an obstacle, no match was found for 15 consecutive frames (t=16 to t=30). = 15, trigger target loss recovery mechanism; Tracking target feature library is dynamically updated before loss And the original appearance of the target when it is initially created That is, the original appearance image group of the first frame of the target, all of which are original appearance images of the tracking target, denoted as : (12); S42, in frame t = 31, detect human targets in image frames , : (1250, 400, 87, 167), s = 0.94; : (450, 420, 77, 157), s = 0.88; : (640, 400, 76, 155), s = 0.89; use FastReID model to extract features - , and features of 16 data in - ; (13).

[0062] S42, calculate the cosine distance between the appearance features of the tracking target and all detected target appearance features and normalize to obtain the distance cost matrix (16x3 dimensions), traverse the matrix to select the minimum distance of each detected target: = 0.18, = 0.85, = 0.92.

[0063] Set threshold , because = 0.18 < 0.25 = threshold , determine as a lost tracking target, re-association is successful, and the tracker information is updated.

[0064] S43, update the tracker information; In this embodiment, the target bounding box is updated: (1250, 400, 87, 167) is updated to ; Use the original appearance image data of the bounding box corresponding to the re-associated detected target to update the original appearance image of the tracking target ; ; Update the prediction trajectory information: update with the new bounding box =1250+87 / 2=1293.5, =400+167 / 2=483.5, =87, =167, the change rate is set to 0, and then the Kalman filter is used to calculate the prediction vector at t=32 frames; The number of consecutive unmatched tracking frames Clear, and return to S3 to perform subsequent processing of each frame.

[0065] S44, if the re-association fails, the number of consecutive unmatched tracking frames Add 1, and return to S3 to perform subsequent processing of each frame.

[0066] The embodiment quickly excludes a large number of irrelevant interference through pre-filtering, and uses IoU and ReID fusion matching from coarse to fine, thereby greatly reducing the calculation overhead while ensuring high matching accuracy, and is especially suitable for breeding environments with slow target movement and many scene interference objects. Meanwhile, a dynamic quality-aware feature updating mechanism is introduced, which can effectively identify and filter low-quality, occluded or severely disturbed target images, avoid pollution and drift of the appearance model due to the introduction of noise in the updating process, and thus still maintain the continuity and stability of identity recognition in typical livestock breeding scenes such as partial occlusion, complete occlusion and light mutation. Further, a dynamically updated high-quality historical appearance feature library is maintained, and cross-frame re-identification is started after the target is lost, effectively solving the problem of target repositioning and identity re-association after the target is out of sight for a long time, and greatly improving the continuous tracking ability and practical application value of the tracking system in complex scenes.

[0067] Embodiment 2 The embodiment provides a device for implementing the long-time tracking method of a human body based on visual perception provided by the application, and the device comprises: at least one processor; and a memory storing instructions, when the instructions are executed by the at least one processor, the at least one processor executes the long-time tracking method of a human body based on visual perception provided by the application as described above.

[0068] Embodiment 3 The embodiment also provides a computer-readable storage medium storing executable instructions, when the instructions are executed, the machine executes the long-time tracking method of a human body based on visual perception provided by the application as described above.

[0069] In particular, a system or apparatus can be provided with a readable storage medium on which is stored software program codes implementing the functions of any of the above-described embodiments, and a computer or processor of the system or apparatus is caused to read out and execute the instructions stored in the readable storage medium.

[0070] In this case, the program codes read from the readable medium can themselves implement the functions of any of the above-described embodiments, and thus the computer readable codes and the readable storage medium storing the computer readable codes constitute a part of the present specification.

[0071] Embodiments of the readable storage medium include floppy disks, hard disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program codes can be downloaded from a server computer or a cloud over a communication network.

[0072] Those skilled in the art will appreciate that embodiments of the present application can be supplied as a method, a system, or a computer program product. Thus, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) embodying computer readable program codes.

[0073] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present application. It is understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowcharts and / or block diagrams block or blocks. Figure 1 The flow or flows and / or blocks in a block or blocks Figure 1 An apparatus with a means for performing the function specified in the flow or flows and / or blocks in a block or blocks.

[0074] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flow or flows and / or blocks in a block or blocks. Figure 1 The flow or flows and / or blocks in a block or blocks Figure 1 An apparatus with a means for performing the function specified in the flow or flows and / or blocks in a block or blocks.

[0075] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, so that the instructions executed on the computer or other programmable data processing devices provide processes for implementing the functions specified in the flowchart Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0076] Obviously, the above embodiments of the present application are only examples for clearly illustrating the technical solutions of the present application, and are not intended to limit the specific embodiments of the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the claims of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A long-term human body tracking method based on visual perception, characterized in that, The method includes: S1. Use a deep learning detection model to perform human detection on the first frame image, and then select the target to track. S2. Create a target tracker and a feature library based on the target being tracked. The target tracker is used to match and record the target information in real time, and the feature library is used to retrieve the target after it is lost. S3. Detect human targets in non-first frame image frames of the tracked target. When the number of consecutive unmatched tracking frames is less than the frame number threshold, perform matching tracking between the detected target and the tracked target. If a match is found, update the tracking target information in the target tracker; If a match fails, the number of consecutive unmatched tracking frames is incremented by 1, and the processing of the next frame image continues. S4. When the number of consecutive unmatched tracking frames is greater than or equal to the frame number threshold, the tracked target is lost and the target loss recovery mechanism is triggered: Cross-frame identity re-association is achieved by comparing the historical appearance features of the tracked target in the feature library with the appearance features of the detected target in the current frame.

2. The long-term human body tracking method based on visual perception according to claim 1, characterized in that, The process of using a deep learning-based detection model to detect human bodies and then selecting targets for tracking specifically includes: S1.1 Obtain the initial frame of the image and use a deep learning detection model to identify all human body categories in the initial frame of the image to obtain the target detection results; S1.

2. The detected human body is subjected to tracking target screening based on the dual criteria of confidence level and spatial scale to obtain the final set of targets that meet the constraints. S1.3 Determine the tracking target from the final set of targets that meet the constraints, either automatically or interactively.

3. The long-term human body tracking method based on visual perception according to claim 1, characterized in that, The creation of the target tracker and feature library specifically includes: The target tracker includes: target bounding box information, raw appearance data, and information on the number of consecutive unmatched frames; the target bounding box information includes: raw bounding box and predicted trajectory information, wherein the raw bounding box information is the target being tracked. of , To track the pixel coordinates of the top-left corner of the target, These are the width and height of the target bounding box, respectively; the predicted trajectory information is... ,in These are the pixel coordinates of the center point of the bounding box of the tracked target in the next frame, predicted by the trajectory prediction algorithm. The length and width of the bounding box of the tracked target in the next frame predicted by the trajectory prediction algorithm; They are respectively The rate of change of all values ​​is 0. The original appearance data is for tracking targets. Original appearance image within the original bounding box ; The number of consecutive unmatched tracking frames That is, the number of consecutive frames in which the tracked target is not matched from the first unmatched frame. The initial value is 0, and it is incremented by 1 when the tracked target is not matched, and reset to zero when the tracked target is matched. S2.2 Construct a feature library for retrieving targets after they have been lost; The feature library includes original appearance features. and high-quality appearance library ; The original appearance features For the identified tracking target Original appearance image within the original bounding box Values ​​are assigned synchronously when the target tracker is created; The high-quality appearance library Used to store the original images of the tracking target that are evaluated as high quality by the appearance feature quality assessment method during target matching and long-term tracking; The high-quality appearance library Fixed storage capacity The original appearance data is dynamically updated, following the first-in, last-out principle. Original appearance data; The appearance feature quality assessment method is used when the size of the tracking target to be assessed is greater than a preset size threshold. And the confidence level is higher than the detection bounding box area threshold. At that time, the sample was determined to be a high-quality sample.

4. The long-term human body tracking method based on visual perception according to claim 1, characterized in that, S3 specifically includes: S3.1 Using a deep learning detection model, detect human targets in non-first frame image frames of the tracked target to obtain a set of detected targets. N is the total number of human targets detected. The label number for each detected human target; Save bounding box information ,in For human target The pixel coordinates of the top-left corner of the bounding box. These are the width and height of the bounding box, respectively; Determine the number of consecutive unmatched tracking frames Check if the value is greater than or equal to 15 frames; if yes, perform target loss and rematch; otherwise, perform S3.

2. S3.

2. Exclude detected human targets that do not intersect with the tracked target and form a candidate subset. ; The IoU is calculated using the bounding box information from the predicted trajectory information and the candidate targets in the candidate subset. The IoU cost vector is then used to calculate the IoU. In the process, the detection target corresponding to the minimum cost is selected. If the IoU value of the detection target corresponding to the minimum cost is less than or equal to the IoU threshold, the match is successful and the tracking target information is updated. If a match fails, activate ReID fusion matching to extract the tracked target. Original appearance image The appearance feature vectors of all detected targets in the appearance feature vector are used to calculate the cosine distance between them. After normalization, feature fusion is performed. The corresponding detected target is confirmed based on the fused vector, and a second matching is performed. If the second matching is successful, the tracking target information is updated; if the second matching fails, the number of consecutive unmatched tracking frames is incremented by 1, and the processing of the next frame image continues.

5. The long-term human body tracking method based on visual perception according to claim 4, characterized in that, S3.2 specifically includes: S3.2.1 Spatial Gating Pre-Filtering Stage: Exclude detection targets that have no intersection with the tracked target; First, the target is tracked in the computation tracker. Human targets detected in the current image frame Calculate IoU value ; （1）； In formula (1): （2）； （3）； Then, in Values ​​greater than 0 are selected as candidate targets for tracking, and these candidate targets are stored in a candidate subset. middle, The number of candidate targets is denoted as M: S3.2.2, IoU priority matching; First, use the predicted trajectory information Bounding box information in ,and Calculate the IoU of candidate targets and generate an IoU cost vector. : （4）； in: （5）； （6）； Then, from the IoU cost vector In the middle, select the detection target corresponding to the minimum cost. : （7）； If the detection target corresponds to the minimum cost IoU value Less than or equal to the IoU threshold If the minimum cost corresponds to the detection target, then the match is successful; IoU value Greater than the IoU threshold If the match fails, then the match is successful or not. If the match is successful, proceed to step S3.2.3; if the match fails, proceed to step S3.2.

4. S3.2.3 Update the tracking target information in the target tracker; If the IoU match is successful, then the detection target is used. corresponding bounding box Update tracking target of Use detection target Update the tracking target using the original appearance image data within the corresponding bounding box. Original appearance image The trajectory prediction algorithm, namely the Kalman filter, is used to calculate the predicted trajectory information of the tracked target in the next frame. ; Assessment and detection targets If the corresponding appearance feature is a high-quality image of the target being tracked, it will be updated to the high-quality appearance library. Otherwise, do not update; Number of consecutive unmatched tracking frames Clear to zero; return to S3.1 to continue target recognition and detection in the next frame; S3.2.4, ReID fusion matching; (1) Use appearance feature extraction algorithm to extract tracking target Original appearance image appearance feature vector and candidate subsets The appearance feature vectors of all detected targets ; (2) Calculate the appearance feature vector of the tracking target with candidate subsets All target appearance feature vectors The cosine distance is calculated and normalized to form a normalized appearance feature distance vector. : (3) For each and Filter the elements in the list; if the distance value is greater than its corresponding threshold, set the distance value to 1. （8）； （9）； in, represents the appearance distance threshold, used to filter the maximum threshold of the cosine distance of the appearance features between the tracked target and the target in the candidate subset calculated in (2); This is the filtered appearance distance vector. This is the filtered IoU distance cost vector. (4) Filter the appearance distance vector Distance cost vector with IoU By taking the minimum value at the corresponding position of each element, multimodal data fusion is achieved, generating a fusion vector. : (5) In the fusion vector In the middle, select the detection target corresponding to the smallest fusion distance. If the minimum distance Below the preset threshold If the match is successful, then the match is considered successful; otherwise, the match is considered unsuccessful. If a match is successful, then execute S3.2.3, i.e., update the tracking target information in the target tracker; otherwise, the number of consecutive unmatched tracking frames is updated. Add 1, then execute S3.

1.

6. The long-term human body tracking method based on visual perception according to claim 5, characterized in that, S4 specifically includes: S41. Feature library and target feature extraction for current frame detection; The feature library for tracking targets is dynamically updated from the appearance feature library before they are lost. and the original appearance of the target when it is initially created. The components are all original appearance images of the tracked target, denoted as... : The appearance feature extraction algorithm is used to extract the original image of the tracking target. appearance features and the set of detected targets in the current frame Appearance features of all targets ,in Each feature in the middle is denoted as , The label number represents the appearance characteristics. : S42, Feature matching; Calculate the appearance features of the tracked target and the set of detected targets in the current frame. The cosine distances of all detected target appearance features are calculated and normalized to obtain the distance cost matrix. Traverse the distance cost matrix Select the smallest distance value from them. : （10）； （11）； If minimum distance Below the threshold If the target is re-associated, then the set of detected targets in the current frame will be determined. The Middle One detection target; Otherwise, the re-association will fail; S43. Update tracker information; If the reassociation is successful, the bounding box information corresponding to the reassociated detected target will be used. Update tracking target Original bounding box information ; Update the tracking target using the original appearance image data within the bounding box corresponding to the re-associated detected target. Original appearance image ; Use the updated tracking target Original bounding box information is updated to predict trajectory information , Set to 0, They are respectively The rate of change; The trajectory prediction algorithm, namely the Kalman filter, is used to calculate and update the predicted trajectory information of the tracked target in the next frame. ; At this point, the number of consecutive unmatched tracking frames Clear the cache and return to S3 to perform processing for each subsequent frame; S44. If re-association fails, then the number of consecutive unmatched tracking frames... Increment by 1, and return to S3 to perform processing for each subsequent frame.

7. The long-term human body tracking method based on visual perception according to claim 2, characterized in that, Specifically, S1.3 includes: S1.3.1 Automatic selection mode, from the final set of targets that meet the constraints. Select the human bounding box with the largest detected bounding box area as the tracking target; S1.3.2 Interactive selection mode allows users to select options manually by boxing or clicking. Specify the target to be tracked; S1.3.3, if If the set is empty, meaning no human body category was detected or the detected human body does not meet the target filtering conditions, then return to S1.1 to reacquire images and perform identification and filtering; otherwise, continue execution.

8. An apparatus for implementing a long-term human body tracking method based on visual perception, characterized in that, The device includes: processor; A memory on which computer programs that can run on the processor are stored; When the computer program is executed by the processor, it implements the steps of a long-term human body tracking method based on visual perception as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images