Video target recognition method and device

Abstract

The present invention provides a video target recognition method and device, which relates to the technical field of video target recognition, and can solve the problems of difficult data labeling, high labeling cost, and lack of labeling data. The method includes: in response to inputting an original video, determining multiple key frames and frames to be processed, wherein the frames to be processed are continuous frames between any two adjacent key frames, and each key frame contains at least one target to be recognized. The target to be recognized is labeled to obtain a first target bounding box. According to the first target bounding box, a pre-trained target tracking model is used to identify the tracking target in the frame to be processed, and the similarity between the tracking target and the target to be recognized is determined based on the target overlap and the target category. A second target bounding box of the tracking target is determined. And according to the overlap between the first target bounding box and the second target bounding box, the target category in the original video is determined. The method of the present invention can reduce the workload of target labeling and improve the recognition efficiency and accuracy of video targets.

Description

Technical Field

[0001] The present invention relates to the technical field of video target recognition, and in particular to a video target recognition method and device. Background Art

[0002] Video fine-grained recognition is a specific task in the field of computer vision, which aims to finely identify and classify objects in videos. Different from ordinary video classification, video fine-grained recognition requires identifying objects at a finer category level.

[0003] However, in practical applications, since fine-grained video recognition often requires refined classification of targets in specific fields, the annotation process generally requires the participation of experts in related fields. At the same time, compared with images, targets in videos may have different postures, angles, occlusions, etc., and may be interfered by factors such as blur and motion, which brings uncertainty to annotation. Compared with images, video annotation also needs to deal with more target variability.

[0004] Since videos are composed of a large number of frames, the amount of data is much larger than that of images. Even if a single video is not long, the total workload accumulated is considerable. Therefore, the difficulty of data labeling, high labeling costs, and lack of labeled data have become bottleneck problems that need to be solved in the development and application of video fine-grained recognition technology. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the existing technical problems, the embodiments of the present invention provide a video object recognition method and device, which are used to at least partially solve the above technical problems.

[0007] (II) Technical solution

[0008] The present invention provides a video target recognition method, comprising: in response to inputting an original video, determining a plurality of key frames and frames to be processed, wherein the frames to be processed are continuous frames between any two adjacent key frames, and each key frame contains at least one target to be recognized. The target to be recognized is marked to obtain a first target bounding box. According to the first target bounding box, a pre-trained target tracking model is used to identify a tracking target in the frame to be processed, and the similarity between the tracking target and the target to be recognized is determined based on the target overlap and the target category. A second target bounding box of the tracking target is determined. And according to the overlap between the first target bounding box and the second target bounding box, the target category in the original video is determined.

[0009] Optionally, according to the first target bounding box, using a pre-trained target tracking model to identify the tracking target in the frame to be processed includes: according to the first target bounding box, using a tracker based on correlation filtering to predict the position and scale of the target in the adjacent frame to be processed in time sequence. According to the predicted position and scale of the target, an unlabeled sample is determined. And according to the unlabeled sample and the labeled target to be identified, the target tracking model is optimized to determine the tracking target.

[0010] Optionally, optimizing the target tracking model based on the unlabeled samples and the labeled targets to be identified to determine the tracking target includes: determining the second predicted target in the corresponding key frame based on the first predicted target in the previous frame to be processed adjacent to the key frame. Determining the similarity between the second predicted target and the labeled target in the corresponding key frame based on the target overlap and category similarity. And optimizing the target tracking model based on the first predicted target and the labeled targets in multiple key frames with the greatest similarity to the first predicted target to determine the tracking target.

[0011] Optionally, the video target recognition method further includes: when the similarity between the second predicted target and the annotated targets in multiple consecutive key frames is less than a preset threshold, deleting the second predicted target. When the similarity between the annotated target in the key frame and multiple second predicted targets is less than a preset threshold, reassociating the annotated target in the key frame using a newly created tracker.

[0012] Optionally, according to the first target bounding box, using a correlation filter-based tracker to predict the position and scale of the target in the adjacent to-be-processed frame in time sequence includes: initializing the tracker according to the first target bounding box to determine a third predicted target in the next to-be-processed frame adjacent to the key frame. According to the third predicted target, determining the predicted target with the largest response to the tracker frame by frame in time sequence.

[0013] Optionally, the first target bounding box includes a target category label, and determining the target category in the original video according to the overlap between the first target bounding box and the second target bounding box includes: determining multiple matching target bounding boxes when the overlap between the first target bounding box and the second target bounding box is greater than a preset threshold. Using a voting mechanism to determine the target category label with the largest number among the multiple matching target bounding boxes. And determining the target category in the original video according to the target category label with the largest number.

[0014] Optionally, before determining the multiple matching target bounding boxes, the video target recognition method further includes: cropping the original video frame by frame according to the first target bounding box and the second target bounding box to obtain multiple cropped areas, and the size of each cropped area is the same. Each frame of the cropped area is spliced ​​in time sequence to obtain the target video. And determining the target category in the original video according to the largest number of target category labels includes: determining the target category in the target video according to the largest number of target category labels.

[0015] Optionally, determining the multiple key frames and frames to be processed includes: splitting the original video to obtain multiple sub-video frames, extracting the multiple sub-video frames at fixed time intervals to obtain multiple key frames, and determining any sub-video frames between the multiple adjacent key frames as frames to be processed.

[0016] Optionally, labeling the target to be identified to obtain the first target bounding box includes: labeling the target to be identified in response to user input to obtain the first target bounding box. Or using a pre-trained deep learning model to label the target to be identified to obtain the first target bounding box.

[0017] The second aspect of the present invention provides a video target recognition device, including: a first determination module, which is used to determine multiple key frames and frames to be processed in response to an input original video, wherein the frames to be processed are continuous frames between any two adjacent key frames, and each key frame contains at least one target to be recognized. A labeling module, which is used to label the target to be recognized and obtain a first target bounding box. A recognition module, which is used to identify the tracking target in the frame to be processed based on the first target bounding box using a pre-trained target tracking model, and the similarity between the tracking target and the target to be recognized is determined based on the target overlap and the target category. A second determination module, which is used to determine a second target bounding box of the tracking target. And a third determination module, which is used to determine the target category in the original video based on the overlap between the first target bounding box and the second target bounding box.

[0018] (III) Beneficial effects

[0019] Compared with the prior art, the video object recognition method and device provided by the embodiments of the present invention have at least the following beneficial effects:

[0020] (1) The video target recognition method of the present invention reduces the workload of target labeling by extracting key frames from the original video and only labeling the key frames. At the same time, a pre-trained target tracking model is used to predict the position of the target in the frame to be processed, and the target is recognized in the original video based on the similarity between the predicted target and the labeled target and the overlap of the target bounding box, which greatly improves the recognition efficiency and accuracy of the video target.

[0021] (2) The video target recognition method of the present invention can determine the predicted target in each frame to be processed frame by frame through a tracker based on correlation filtering, and form unlabeled samples. The target tracking model is optimized by combining the samples formed by the unlabeled samples and the labeled targets to determine the tracking target, thereby ensuring the accuracy of target recognition when reducing the amount of annotation.

[0022] (3) The video target recognition method of the present invention matches the target similarity based on the frames to be processed adjacent to each key frame, which greatly reduces the amount of calculation and further improves the recognition efficiency of the video target. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0024] Figure 1 A flowchart of a video target recognition method according to an embodiment of the present invention is schematically shown;

[0025] Figure 2 A flowchart of a video target recognition method according to another embodiment of the present invention is schematically shown;

[0026] Figure 3 A diagram schematically shows a labeling result of an animal target in a key frame according to an embodiment of the present invention;

[0027] Figure 4 A diagram schematically shows the tracking result of an animal target in consecutive frames according to an embodiment of the present invention;

[0028] Figure 5 The structure block diagram of the video object recognition device according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0030] It should be noted that in the drawings or descriptions, similar or identical parts use the same figure numbers. The technical features in the various embodiments exemplified in the specification can be freely combined to form new solutions without conflict, and in the drawings, the shape or thickness of the embodiments can be expanded and simplified or conveniently indicated. Furthermore, the elements or implementations not shown or described in the drawings are in a form known to ordinary technicians in the relevant technical field. In addition, although demonstrations of parameters containing specific values ​​may be provided herein, it should be understood that the parameters do not need to be exactly equal to the corresponding values, but can be approximated to the corresponding values ​​within an acceptable error tolerance or design constraint.

[0031] Unless there are technical obstacles or contradictions, the above-mentioned various embodiments of the present invention can be freely combined to form other embodiments, and these other embodiments are all within the protection scope of the present invention.

[0032] Although the present invention is described in conjunction with the accompanying drawings, the embodiments disclosed in the drawings are intended to exemplify the preferred embodiments of the present invention and should not be construed as limiting the present invention. The size ratios in the drawings are merely illustrative and should not be construed as limiting the present invention.

[0033] Although some embodiments of the present general inventive concept have been shown and described, it will be appreciated by those skilled in the art that changes may be made to these embodiments without departing from the principles and spirit of the present general disclosed concept.

[0034] Figure 1 The flowchart of the video object recognition method according to an embodiment of the present invention is schematically shown.

[0035] According to an embodiment of the present invention, Figure 1 As shown, the video object recognition method includes, for example, operations S110 to S150.

[0036] In operation S110 , in response to inputting an original video, a plurality of key frames and frames to be processed are determined, wherein the frames to be processed are continuous frames between any two adjacent key frames, and each key frame contains at least one target to be recognized.

[0037] In operation S120 , the target to be recognized is labeled to obtain a first target bounding box.

[0038] In operation S130, a pre-trained target tracking model is used to identify a tracked target in a frame to be processed according to the first target bounding box, and a similarity between the tracked target and the target to be identified is determined based on a target overlap and a target category.

[0039] In operation S140, a second target bounding box of the tracking target is determined.

[0040] In operation S150 , a category of an object in the original video is determined according to an overlap degree between the first object bounding box and the second object bounding box.

[0041] In some embodiments, a user uploads a traffic surveillance video that includes multiple vehicles and pedestrians.

[0042] The video processing system first analyzes the video content and automatically selects frames containing obvious target changes as key frames, such as frames where a vehicle enters or leaves the screen. The continuous frames between these key frames, i.e. the frames to be processed, are the objects that the system needs to further analyze.

[0043] In the key frame, the system manually or automatically marks the target to be identified, such as a specific vehicle or pedestrian, and generates a first target bounding box.

[0044] Using a pre-trained target tracking model (such as a Siamese network model based on deep learning), the system identifies and tracks targets in the processing frame that are similar to the targets annotated in the key frame. The similarity of the tracked targets is comprehensively evaluated by calculating the target overlap (such as IOU, Intersection over Union) and the target category (determined by a classifier).

[0045] For each successfully tracked target, the system generates a second target bounding box to indicate its position in the frame to be processed.

[0046] Finally, the system compares the overlap between the first target bounding box and the second target bounding box. If the overlap exceeds a preset threshold, the category of the target in the original video (such as vehicle, pedestrian, etc.) can be determined.

[0047] In some embodiments, a user uploads a wildlife documentary that includes footage of multiple animals in action.

[0048] The system automatically selects frames containing obvious behavioral changes of animals as key frames, such as frames where animals appear, disappear, or have behavioral changes. The continuous frames between these key frames are used as frames to be processed.

[0049] In the key frame, the user uses the interface tool to mark the animal target of interest, such as a lion or an elephant, and generates a first target bounding box.

[0050] Using the target tracking model (such as the KCF algorithm based on correlation filtering), the system tracks the annotated target in the frame to be processed. By calculating the target overlap and category information, the similarity between the tracked target and the target to be identified is evaluated.

[0051] For successfully tracked animal targets, the system generates a second target bounding box to indicate its position in the frame to be processed.

[0052] By comparing the overlap between the first target bounding box and the second target bounding box, the system can determine the category of the animal in the original video.

[0053] The two embodiments respectively demonstrate the application of the video target recognition method in traffic monitoring and wildlife documentary analysis. By selecting key frames, marking the target to be identified, tracking using the target tracking model, and determining the target based on the overlap and category information, the method can efficiently identify and track specific targets in the original video.

[0054] According to an embodiment of the present invention, for example, through operations S231 to S233, a pre-trained target tracking model is used to identify a tracking target in a frame to be processed according to the first target bounding box.

[0055] In operation S231 , according to the first object bounding box, a tracker based on correlation filtering is used to predict the position and scale of the object in the adjacent to-be-processed frame in time sequence.

[0056] In operation S232, unlabeled samples are determined according to the position and scale of the predicted target.

[0057] In operation S233 , the target tracking model is optimized according to the unlabeled samples and the labeled targets to be identified to determine the tracking target.

[0058] In some embodiments, for example, a user uploads a documentary about wild animals on the African grasslands, which includes lions, elephants, antelopes and other animals.

[0059] The system automatically analyzes the video content and selects frames containing obvious animal behavior or position changes as key frames. For example, the moment a lion jumps out of the grass or the moment an elephant starts to move. The consecutive frames between these key frames are the frames to be processed.

[0060] In the keyframe, the user uses the interface tool to mark the animal target of interest, such as a specific lion, and generates a first target bounding box. This bounding box accurately indicates the position and size of the lion in the keyframe.

[0061] The system uses a correlation filter-based tracker (such as the KCF algorithm) as the target tracking model. The tracker uses the information in the first target bounding box (such as the color, texture and other features of the target) to initialize the tracking model. Then, the position and scale of the target in the adjacent frames to be processed are predicted in time sequence. This is because the movement of animal targets between adjacent frames is usually continuous and smooth, so the information of the previous frame can be used to predict the possible position and size of the target in the next frame.

[0062] Based on the prediction results of the tracker, the system determines the approximate position and scale of the target in the frame to be processed and generates unlabeled samples. Although these unlabeled samples have not been manually labeled, based on the prediction of the tracker, they have a high degree of similarity with the labeled targets in the key frame.

[0063] The system combines unlabeled samples with labeled targets to be identified to optimize the target tracking model. This involves, for example, fine-tuning the parameters inside the tracker to improve its accuracy in tracking targets in subsequent frames. The optimization process may include updating the filter coefficients of the tracker, adjusting the learning rate, etc.

[0064] Through continuous optimization and iteration, the target tracking model can gradually adapt to the appearance changes of the target (such as those caused by changes in lighting, angle or posture) and accurately track the target in subsequent frames to be processed.

[0065] After the optimized target tracking model is used for recognition and tracking, the system determines the second target bounding box of the tracked target in each frame to be processed. These bounding boxes accurately mark the continuous motion trajectory of the animal target in the video.

[0066] By comparing the overlap between the first target bounding box and the second target bounding box, the system can determine whether tracking is successful and determine the category of the animal in the original video. Further analysis of the changes in these bounding boxes can also infer the animal's behavior patterns, such as movement speed and direction changes.

[0067] This example demonstrates how to use a correlation filter-based tracker to identify and track specific animal targets in wildlife documentaries. By combining labeled information and unlabeled samples to optimize the tracking model, this method can achieve accurate target tracking in complex natural environments.

[0068] According to an embodiment of the present invention, for example, operations S3331 to S3333 are performed to optimize the target tracking model according to the unlabeled samples and the labeled targets to be identified, so as to determine the tracking target.

[0069] In operation S3331, a second prediction target in a corresponding key frame is determined according to a first prediction target in a previous frame to be processed that is adjacent to the key frame.

[0070] In operation S3332, based on the target overlap and the category similarity, the similarity between the second predicted target and the labeled target in the corresponding key frame is determined.

[0071] In operation S3333, the target tracking model is optimized according to the first predicted target and the annotated targets in the plurality of key frames having the greatest similarity to the first predicted target to determine the tracking target.

[0072] In some embodiments, for example, a user uploads a documentary about wild animals in a forest and annotates several key frames of animal targets, such as bears, deer, etc. The animal target in each key frame is accurately annotated with a bounding box.

[0073] The system uses the target tracking model to start initial tracking of the target from the previous frame to be processed that is adjacent to the key frame. Based on the information of the previous frame, the system predicts the position and scale of the target in the current frame and generates the first predicted target.

[0074] For each key frame, the system determines the second predicted target in the corresponding key frame by interpolation or extrapolation based on the first predicted target in the previous frame to be processed adjacent to the key frame. This second predicted target is a preliminary estimate of the target position and scale in the key frame.

[0075] The system calculates the similarity between the second predicted target and the labeled target in the corresponding keyframe. The similarity is evaluated based on two main indicators: target overlap (such as IOU) and category similarity.

[0076] Object overlap: By comparing the degree of overlap between the bounding box of the second predicted object and the bounding box of the labeled object, the system can evaluate the similarity of their spatial positions.

[0077] Category similarity: The system also considers the category information of the target. If the second predicted target is of the same or similar category as the labeled target (for example, both are mammals), their similarity will be higher.

[0078] By comprehensively considering these two indicators, the system can determine the similarity between the second predicted target and the labeled target.

[0079] According to the first predicted target and the annotated targets in multiple key frames with the greatest similarity to the first predicted target, the system optimizes the target tracking model. For example, it includes:

[0080] Update filter parameters: The system uses annotated targets with high similarity to update the filter parameters in the tracker, making the tracker more adaptable to changes in the target’s appearance.

[0081] Learning dynamic features: By analyzing the changes of the target in consecutive frames, the system learns the dynamic features of the target, such as movement speed, direction, etc., to further improve the tracking accuracy.

[0082] Through continuous optimization, the target tracking model can better adapt to the target's appearance and motion changes, and improve the accuracy of tracking the target in subsequent frames.

[0083] The optimized target tracking model continues to track the target in the subsequent frames to be processed, generating accurate second target bounding boxes. These bounding boxes mark the continuous motion trajectory of the animal target in the video.

[0084] By combining unlabeled samples and labeled targets to optimize the target tracking model and considering the target's category information and spatial position relationship during the optimization process, this method can achieve more accurate and robust target tracking in complex natural environments.

[0085] According to an embodiment of the present invention, the video object recognition method, for example, further includes operations S410 to S420.

[0086] In operation S410, when the similarities between the second predicted target and the labeled targets in a plurality of consecutive key frames are all less than a preset threshold, the second predicted target is deleted.

[0087] In operation S420, when the similarities between the annotated target in the key frame and the plurality of second predicted targets are all less than a preset threshold, the annotated target in the key frame is re-associated using the newly created tracker.

[0088] In some embodiments, for example, a user uploads a documentary about wild animals in a forest and labels animal targets in some key frames, such as wolves, deer, etc.

[0089] The system begins to initially track the target according to the previous steps and generates a second predicted target in each keyframe.

[0090] For each second predicted target, the system calculates the similarity between it and the labeled targets in multiple consecutive key frames. If the similarities are all less than a preset threshold (for example, IOU is less than 0.3 or the category similarity is lower than a certain value), it means that the match between the predicted target and the real target is very low.

[0091] Case 1: Delete the second prediction target.

[0092] If the similarity between the second predicted target and the labeled target in multiple consecutive key frames is lower than the threshold, the system determines that these predicted targets may be mistracked or noise, so it will delete these second predicted targets and stop further tracking of these targets.

[0093] Case 2: The keyframe annotation target is not similar to multiple predicted targets.

[0094] On the contrary, if within a certain key frame, the similarities between the labeled target and multiple second predicted targets are all lower than a preset threshold, this means that the current tracker may not be able to accurately associate the labeled target with the predicted target.

[0095] In this case, the system re-associates the labeled objects in the keyframes using a newly created tracker. The newly created tracker can be a tracker based on a different algorithm or feature, or a re-initialized original tracker.

[0096] The reassociation process includes, for example:

[0097] Use the newly created tracker to search for and track the annotated target again in the keyframe and subsequent frames.

[0098] By comparing the output of the newly created tracker with the annotated target, the similarity between the new predicted target and the annotated target is determined.

[0099] If the similarity between the output result of the newly created tracker and the labeled target is higher than the preset threshold, the result is accepted as the new tracking target and tracking continues.

[0100] By removing unmatched predicted targets and reassociating annotated targets, the system can optimize tracking performance and reduce mistracking and target loss. The optimized tracking model will continue to identify and track targets in subsequent frames to be processed.

[0101] Finally, the system outputs the position, trajectory, and possible category information of each identified animal target in the video for further analysis and use by the user.

[0102] Through this embodiment, it can be seen that in the video target recognition method, by combining similarity evaluation and threshold comparison, the system can effectively handle the situation where the predicted target does not match the labeled target, and optimize the tracking performance by deleting or reassociating the target, thereby improving the accuracy and robustness of target recognition.

[0103] According to an embodiment of the present invention, for example, through operations S5311 to S5312, a correlation filter-based tracker is used to predict the position and scale of the target in the adjacent to-be-processed frame in time sequence according to the first target bounding box.

[0104] In operation S5311, a tracker is initialized according to the first target bounding box to determine a third predicted target in a next frame to be processed adjacent to the key frame.

[0105] In operation S5312, based on the third prediction target, a prediction target having the largest response to the tracker is determined frame by frame in time sequence.

[0106] In some embodiments, for example, there is a documentary about a lion running in the wild. In this video, the first frame (key frame) is annotated in detail, where the position of the lion is accurately marked by a target bounding box.

[0107] Based on this first target bounding box, a correlation filter-based tracker is initialized. This tracker will learn the appearance features of the lion in the first frame, such as color, texture, and shape.

[0108] Using the initialized tracker, we start predicting the position of the lion in the next frame (the second frame) adjacent to the key frame. By calculating the correlation between the filter template and the target candidate region, we get a response output. The maximum position of this output is the predicted position of the lion in the second frame, which is called the third predicted target. At the same time, the tracker also predicts the scale of the lion in the second frame based on its appearance and size.

[0109] With the third predicted target in the second frame, the tracker continues to work frame by frame in time. For each frame, the tracker searches for the target location in the current frame based on the predicted target in the previous frame. By continuously calculating the correlation between the filter template and the candidate region, the tracker is able to find the predicted target with the largest filter response, which is the most likely location of the lion in the current frame.

[0110] At the same time, the tracker will dynamically adjust its predicted scale according to the target's appearance changes and motion state. For example, if the lion gradually moves away from the camera while running, the tracker will predict a smaller target scale; conversely, if the lion approaches the camera, the tracker will predict a larger target scale.

[0111] In practical applications, the target may be affected by factors such as occlusion, appearance change or scale change, which may cause the tracker to lose the target or make inaccurate predictions. To address these challenges, some optimization strategies can be used, such as combining deep learning features to improve the robustness of the tracker, or using multi-frame information to assist in predicting the position and scale of the target.

[0112] By combining a correlation filter-based tracker with a time series prediction method, it is possible to effectively track the target in consecutive video frames and predict changes in its position and scale. This has wide application value in areas such as wildlife documentary analysis and security monitoring.

[0113] According to an embodiment of the present invention, the first target bounding box includes a target category label. For example, operations S651 to S653 are performed to determine the target category in the original video according to the overlap between the first target bounding box and the second target bounding box.

[0114] In operation S651 , when the overlap between the first target bounding box and the second target bounding box is greater than a preset threshold, a plurality of matching target bounding boxes are determined.

[0115] In operation S652, a voting mechanism is used to determine the target category label with the largest number among the multiple matching target bounding boxes.

[0116] In operation S653, the object category in the original video is determined according to the object category label with the largest number.

[0117] In some embodiments, for example, a user uploads a wildlife documentary about the African grasslands and annotates some animals in key frames. Each annotation includes a target bounding box and a target category label, such as "lion", "elephant", etc.

[0118] Based on the first target bounding box (annotated in the key frame), the tracker starts to predict the position and scale of the target in subsequent frames and generate second target bounding boxes. These second target bounding boxes indicate the position changes of the target in consecutive frames.

[0119] For each second target bounding box, the system calculates the overlap between it and the first target bounding box (for example, using IOU, i.e., intersection over union). If the overlap is greater than a preset threshold (such as IOU greater than 0.5), the second target bounding box is considered to match the first target bounding box, i.e., they correspond to the same target.

[0120] After multiple matching target bounding boxes are determined, the system uses a voting mechanism to determine the target category labels in these bounding boxes. Specifically, each matching target bounding box carries the category label that is predicted or inherited from the first target bounding box. The system counts the category labels of all matching target bounding boxes and finds the label that appears the most times.

[0121] For example, if there are 5 matching object bounding boxes, 3 of which have the label "lion" and 2 have the label "cheetah", then "lion" is the label that appears the most times.

[0122] Based on the target category label with the largest number of votes, the system finally determines the category of the target in the original video. In the above example, the system determines the target as "lion".

[0123] If multiple category labels appear the same number of times and the most, the system can further consider other factors (such as the size and shape of the bounding box) to make a judgment, or simply mark it as an "uncertain" category.

[0124] If the overlap of the bounding boxes of the unmatched target exceeds a preset threshold, the system can consider that the tracking has failed or the target has disappeared in subsequent frames. At this time, strategies such as reinitializing the tracker and using other tracking algorithms can be adopted as needed.

[0125] Through this method, the system can determine and maintain the target's category information while tracking the target, providing an important basis for subsequent tasks such as target analysis and behavior understanding.

[0126] According to an embodiment of the present invention, before determining a plurality of matching target bounding boxes, the video target recognition method further includes operations S710 to S720.

[0127] In operation S710, the original video is cropped frame by frame according to the first target bounding box and the second target bounding box to obtain a plurality of cropping regions, and the size of each cropping region is the same.

[0128] In operation S720, each frame of cropped area is spliced ​​in time sequence to obtain a target video.

[0129] Determining the object category in the original video according to the object category label with the largest number includes operation S810.

[0130] In operation S810, an object category within a target video is determined according to the object category label with the largest number.

[0131] In some embodiments, for example, a user uploads a documentary about wildlife activities in a forest and annotates target animals in key frames, each annotation including a target bounding box and a target category label.

[0132] The system uses a tracker to track the target in the key frame, predict the position and scale of the target in the subsequent frames, and generate a second target bounding box. These second target bounding boxes indicate the position changes of the target in consecutive frames.

[0133] Before determining multiple matching target bounding boxes, the system first crops the original video frame by frame based on the first target bounding box and the second target bounding box. The purpose of cropping is to obtain the area closely related to the target and remove the background and other irrelevant information.

[0134] The cropping process is as follows: the system first determines the position and size of the first target bounding box and the second target bounding box in their respective frames. Then, based on the sizes of these bounding boxes, the system determines a fixed cropping area size to ensure that the target is fully contained while minimizing the background. Then, the system crops the original video frame by frame to obtain multiple cropping areas of the same size.

[0135] After obtaining the cropped areas of each frame, the system splices these cropped areas in time sequence to generate a new video, namely the target video. This target video only contains the areas related to the target, which is convenient for subsequent analysis and processing.

[0136] After the target video is stitched together, the system calculates the overlap between the second target bounding box and the first target bounding box. By comparing the overlap with a preset threshold, the system determines multiple matching target bounding boxes that correspond to the targets that are stably tracked in consecutive frames.

[0137] For multiple matching target bounding boxes, the system uses a voting mechanism to determine the most common target category label. Each matching target bounding box carries the category label that it predicts or inherits from the first target bounding box. The system counts the category labels of all matching target bounding boxes and finds the label that appears the most times.

[0138] Based on the target category label with the largest number of votes, the system finally determines the category of the target in the target video. This category label not only reflects the attributes of the target in the key frame, but also maintains consistency in the entire target video through the tracking and cropping process.

[0139] Finally, the system outputs the target video and its corresponding category label for users to further analyze and apply. For example, users can perform more detailed feature extraction, behavior analysis, or compare the target video with other datasets.

[0140] Through this method, the system can generate a video closely related to the target through cropping and splicing operations while identifying the target, and determine the category of the target. This not only improves the accuracy and efficiency of target recognition, but also facilitates subsequent tasks such as target analysis and behavior understanding.

[0141] According to the embodiment of the present invention, for example, a plurality of key frames and frames to be processed are determined through operations S911 to S913.

[0142] In operation S911, the original video is split to obtain a plurality of sub video frames.

[0143] In operation S912, multiple sub-video frames are extracted at fixed time intervals to obtain multiple key frames.

[0144] In operation S913, any sub-video frames between a plurality of adjacent key frames are determined as frames to be processed.

[0145] In some embodiments, for example, a user uploads a documentary about African wildlife, which may be several hours long and contain a large number of frames.

[0146] The system first splits the original video into a series of sub-video frames. This is usually achieved by cutting the video into multiple independent frames according to a fixed duration (for example, one frame per second or one frame per half a second). In this way, the original long video is converted into a collection of a large number of sub-video frames.

[0147] Next, the system will extract key frames from these sub-video frames at a fixed time interval. For example, the system may set a frame to be extracted every 5 seconds or every 10 seconds as a key frame. The purpose of this is to select representative key frames that can reflect changes in video content from a large number of sub-video frames.

[0148] During this process, the system may consider some additional factors to optimize the selection of key frames, such as the content difference between frames, image clarity, the significance of animal behavior, etc. These factors can be automatically evaluated by the algorithm or set by the user according to specific needs.

[0149] After determining the key frames, the system can easily determine the frames to be processed. The frames to be processed refer to the sub-video frames located between any two adjacent key frames. Although these frames are not selected as key frames, they still contain a lot of useful information, which is very important for tasks such as target tracking and behavior analysis.

[0150] Finally, the system outputs the determined key frames and frames to be processed. These frames can be used in subsequent tasks such as target recognition, tracking, and behavior analysis.

[0151] After determining the key frames in this way, researchers can perform more efficient target recognition and behavior analysis on these key frames. For example, the location and category of wild animals can be marked in the key frames, and then the tracking algorithm can be used to track the movement trajectory of these animals in the frames to be processed. In addition, the behavior patterns and interaction relationships of animals can be inferred by analyzing the content changes between key frames.

[0152] Determining key frames and frames to be processed in this way not only improves the efficiency of video processing, but also makes subsequent target recognition and behavior analysis more accurate and comprehensive.

[0153] According to an embodiment of the present invention, for example, the target to be identified is labeled through operations S1021 to S1022 to obtain a first target bounding box.

[0154] In operation S1021, in response to user input, the target to be identified is marked to obtain a first target bounding box.

[0155] In operation S1022, a pre-trained deep learning model is used to label the target to be identified to obtain a first target bounding box.

[0156] In some embodiments, in a wildlife documentary, for example, researchers are interested in a particular animal (such as a lion) and want to track and identify the animal. At this time, the researchers can determine the first target bounding box by following the steps below:

[0157] Step 1: The researcher opens the video annotation tool and loads the documentary video to be processed.

[0158] Step 2: As the video plays, the researchers pause at the key frame where the lion first appears.

[0159] Step 3: Using the annotation tool, the researcher draws a rectangular box on the key frame, making sure that the rectangular box tightly surrounds the outline of the lion. This rectangular box is the first target bounding box.

[0160] Step 4: The annotation tool saves the position and size information of the first target bounding box for subsequent use.

[0161] In this way, the position of the target animal in the key frame can be accurately determined, providing an important basis for subsequent target tracking and identification.

[0162] In addition to manual annotation by users, you can also use a pre-trained deep learning model to automatically determine the first target bounding box. This method is usually suitable for large-scale video processing or scenarios that require efficient annotation. The following are the specific steps:

[0163] Step 1: Prepare a large amount of labeled training data, which includes the position and size information of the target animal (such as a lion) in the video.

[0164] Step 2: Use this training data to train a deep learning model that can automatically detect and label target animals.

[0165] Step 3: When a new documentary video needs to be processed, the pre-trained deep learning model is applied to the video.

[0166] Step 4: The model automatically detects the target animal in the video and generates the corresponding target bounding box on each key frame. These target bounding boxes are used as the first target bounding boxes for subsequent target tracking and recognition.

[0167] The advantage of this method is its high degree of automation and fast processing speed, but the prerequisite is that there must be enough labeled data to train an accurate model.

[0168] Regardless of the method used to determine the first target bounding box, its purpose is to provide an accurate starting point in the subsequent target tracking and recognition process. By accurately marking the position and size of the target, the accuracy and stability of the tracking algorithm can be greatly improved, thus providing strong support for fields such as wildlife behavior analysis and ecological research.

[0169] Figure 2 The following schematically shows a flow chart of a video object recognition method according to another embodiment of the present invention.

[0170] According to an embodiment of the present invention, Figure 2 As shown, the video object recognition method includes, for example, steps 1 to 6.

[0171] Step 1: Get the original video, pre-process it, and split it into a set of sub-frames.

[0172] Step 2: Set key frames at fixed intervals, and annotate the targets of interest in the key frames manually or through a preset deep learning algorithm to form a target bounding box.

[0173] For example, the target bounding box information contains the target's position information, size information, and fine-grained species information in the video, which is used for subsequent tracking and labeling processes.

[0174] Step 3: For the video sequence between two adjacent key frames, the semi-supervised target tracking algorithm is used to predict the position change of the target bounding box in the continuous video frames and record it.

[0175] The position change of the target bounding box in consecutive video frames is predicted by a semi-supervised target tracking algorithm, for example:

[0176] Initialize the tracker: Use the target bounding box annotated in the keyframe to initialize the correlation filter-based tracker. A correlation filter is learned in the feature space to describe the model of the target appearance, and the properties of the circulant matrix are used to efficiently calculate the filter response in the frequency domain to calculate the score of the potential target location.

[0177] Target search and positioning: In adjacent continuous video frames, the target position of the previous frame is taken as the center and the target search is performed in the search area around it. The tracker based on correlation filtering calculates the response map of the current frame and the filter and finds the position with the largest response as the new position of the target.

[0178] Let the i-th frame image be x i , the relevant filter is w, then the filter response y i It can be expressed as:

[0179] y i =F -1 (F(x i )⊙F(w)) (1)

[0180] Target status update: Update the target's position and size information based on the tracking results of the current frame.

[0181] Let the target position of the i-th frame be p i , scale is s i , then the target position p of the i+1th frame is i+1 and scales i+1 It can be updated by the following formula:

[0182] p i+1 =p i +Δp i s i+1 =s i *Δs i(2)

[0183] Among them, Δp i and Δs i They represent the changes in position and scale respectively, which can be estimated by the tracker.

[0184] Collection of semi-supervised training data: During the tracking process, the tracking results are used as unlabeled samples, together with the key frame labeled samples to form semi-supervised training data. The target appearance model is updated by online learning. By calculating the confidence or response strength of the current frame and the model, the learning rate is dynamically adjusted to adapt to changes in the target appearance.

[0185] Step 4: Associate and compare the target annotated in the key frame with the tracked target, and perform online optimization and update of the semi-supervised target tracking model.

[0186] The specific example of the correlation comparison is:

[0187] Target position prediction: For each tracked target, the Kalman filter algorithm is used to predict its position in the current key frame based on its motion information (position, speed, etc.) in the previous continuous frame.

[0188] Let the target state in the i-th key frame be x i , including position and velocity information, then the predicted state of the i+1th key frame is It can be calculated by the following formula:

[0189]

[0190] Where A is the state transition matrix, which describes how the target state is transferred from the current moment to the next moment. B is the control input matrix, which describes how the external control input (if any) affects the target state. If there is no external control, B can be a zero matrix or omitted. i is the control input at the i-th moment. If there is no external control, then u i Is a zero vector or omitted.

[0191] Construct a multi-target similarity bipartite graph: The two vertex sets of the bipartite graph are the target set annotated by the key frame and the target set to be tracked, and the weight of the edge is the similarity between the two targets. The calculation of the similarity comprehensively considers the overlap between the targets, such as IOU (Intersection over Union), and category similarity.

[0192] Let the key frame annotation target be t i , the tracking target is o j , the IOU of the two is iou(t i , o j ), the category similarity is sim(ti , o j ), then the total similarity s(t i , o j ) can be calculated by the following formula:

[0193] s(t i , o j )=α*iou(t i , o j )+β*sim(t i , o j ) (4)

[0194] Wherein, α and β are weight coefficients, satisfying α+β=1.

[0195] Similar target association: Use the Hungarian algorithm to find the maximum weighted match in the bipartite graph and find the optimal association between the keyframe annotated target and the tracked target.

[0196] The online optimization and update of the target tracking model is, for example:

[0197] For each successfully associated tracking target, the tracking model is optimized online using the annotation information of the target in the past M key frames. The optimization method may be a gradient descent method, etc., to adjust the predicted position of the tracking model to the annotation position.

[0198] Assume that the tracking model parameters are θ and the loss function in the kth key frame is L k (θ), then the parameter update can be achieved through the following gradient descent formula:

[0199]

[0200] For the unassociated tracking target, if it fails to be associated within N consecutive key frames, it is determined as a tracking failure and is deleted.

[0201] For the keyframe annotated targets that cannot be associated, add a new tracker to the tracking model for tracking. For example, you can change the target ID and update the starting frame for initializing the tracker.

[0202] Step 5: Repeat steps 3 and 4 until the end of the video. For each target recorded by the ID, the target area is captured from the original video frame in sequence according to the bounding box information, and they are stitched together in sequence to form a target video with fine-grained annotations.

[0203] In order to ensure that the target video size is fixed, the method for capturing the target area is to use the upper left corner of the target bounding box as the starting position, and the maximum size recorded by the target bounding box as the reference size to set the captured area. If the captured area exceeds the video frame boundary, the video boundary is set as the end position of the captured area, and the starting position is modified to adapt to the captured area.

[0204] Step 6: Calculate the Intersection over Union (IoU) between the captured area and the historical key frame annotation box, introduce a voting mechanism, and use the label with the largest number in the matching key frame annotation box to specify the category of the target video.

[0205] Specifically, let the cutout area corresponding to the i-th target video frame be R i , the label box in the kth key frame is B k , their IoU is:

[0206]

[0207] If IoU(R i , B k ) is greater than a preset threshold (such as 0.5), the key frame annotation box is considered to match the target in the video frame. Among all the matching annotation boxes, the category label with the most occurrences is designated as the target category in the video frame.

[0208] Suppose the class label set is {c1, c2, ..., c m}, the number of times the j-th category label appears is n j , then the category c of the target video * It can be determined by the following formula:

[0209]

[0210] Determining the category of the target video by majority voting can improve the accuracy and robustness of annotation and compensate for the deviations and errors that may occur during the tracking process.

[0211] In some embodiments, step 1: obtain the original video and pre-process it, obtain a 14-second video from the ecological monitoring camera, and decode it into a 3840×2160 resolution RGB image sequence with a frame rate of 15fps. Intercept the first 13 seconds of the video where the target exists, and extract key frames from the image sequence, extracting one frame every 1 second as a key frame starting from the first second, and extracting a total of 13 key frames.

[0212] Figure 3 The diagram schematically shows the labeling result of an animal target in a key frame according to an embodiment of the present invention.

[0213] Step 2: Label the animal targets in the key frames, and use manual methods to label the animal targets in each key frame. The labeling content includes, for example:

[0214] The position and size of the target are represented by a rectangular bounding box, and the coordinates of the upper left corner (x, y) and the width and height (width, height) of the bounding box are recorded.

[0215] The target species' scientific name, e.g. Figure 3 As shown, such as Anas platyrhynchos (mallard), etc.

[0216] The annotation results are saved in VOC XML format. Each keyframe generates an XML file with the file name corresponding to the frame number of the keyframe. An example of an XML file is as follows:

[0217] <annotation>

[0218] <folder / >

[0219] <filename> NoLabeled_0_00000_20230422-220151_13720_00030.xml< / filename>

[0220] <size>

[0221] <width> 3840< / width>

[0222] <height> 2160< / height>

[0223] <depth> 3< / depth>

[0224] < / size>

[0225] <object>

[0226] <name> Anas_platyrhynchos< / name>

[0227] <pose> undefine< / pose>

[0228] <truncated> undefine< / truncated>

[0229] <difficute> 0< / difficute>

[0230] <bndbox>

[0231] <xmin> 1197< / xmin>

[0232] <ymin> 1108< / ymin>

[0233] <xmax> 1350< / xmax>

[0234] <ymax> 1230< / ymax>

[0235] < / bndbox>

[0236] < / object>

[0237] < / annotation>

[0238] Step 3: For the video sequence between two adjacent key frames, the semi-supervised target tracking algorithm is used to predict the position change of the target bounding box in the continuous video frames and record it.

[0239] Specifically, the CSRT (Channel and Spatial Reliability Tracker) tracker is used to track the target. The CSRT tracker is a tracking algorithm based on discriminant correlation filters. It learns a discriminant model in the target area and uses the model to locate the target in subsequent frames. The main principles of the CSRT tracker are as follows:

[0240] In the initial frame, the features of the target area are extracted according to the bounding box of the target, and a discriminative correlation filter model is trained. The goal of the discriminative correlation filter model is to learn a filter w that maximizes the response value of the target area and minimizes the response value of the background area. The learning of the filter can be achieved by minimizing the following loss function:

[0241]

[0242] Among them, x i represents the feature vector of the i-th sample, y i represents the label of the i-th sample (the target area is 1 and the background area is 0), f(x i )=w T x i Represents sample x i The response value of , λ is the regularization coefficient.

[0243] Figure 4 The figure schematically shows the tracking result of the animal target in the continuous frames according to the embodiment of the present invention.

[0244] In subsequent frames, the learned filter model is used to score the candidate areas, and the area with the largest response value is obtained as the predicted position of the target. In order to adapt to changes in the target's appearance, the CSRT tracker adopts an adaptive update strategy to adjust the update rate of the model according to the target's reliability score. For each key frame, the CSRT tracker is initialized according to the annotated target bounding box. For each frame between two adjacent key frames, the CSRT tracker is used to predict the position of the target in the current frame and record the predicted bounding box position (x, y, width, height). If the tracker's confidence is lower than the preset threshold (such as 0.5), the tracking is considered to have failed and tracking is stopped. The target tracking results are shown in Figure 2. Figure 4 shown.

[0245] Step 4: Associate and compare the target annotated in the key frame with the tracked target, and perform online optimization and update of the semi-supervised target tracking model. The specific steps are as follows:

[0246] For each keyframe, the manually annotated target bounding box is associated and compared with the target bounding box predicted by the tracker. First, based on the target information in the adjacent frames, a Kalman filter is used to predict its position in the next keyframe.

[0247] Assume the state vector of the target is x = [x, y, v x , v y ] T , where (x, y) is the center coordinate of the target, (v x , v y ) is the speed of the target. The state transfer equation and observation equation are as follows:

[0248] x t+1 =Fx t +w t z t =Hx t +v t (9)

[0250] Among them, F is the state transfer matrix, H is the observation matrix, and w t and z t are process noise and observation noise, respectively.

[0251] Then, for n manually annotated targets and m tracker predicted targets, an n×m cost matrix C is constructed, where C ij The cost of associating the i-th manually annotated target with the j-th tracker predicted target is measured using the IoU metric. The IoU calculation formula is as follows:

[0252]

[0253] Among them, B1 and B2 are two bounding boxes, Area(B1∩B2) represents the intersection area of ​​the two bounding boxes, and Area(B1∪B2) represents the union area of ​​the two bounding boxes.

[0254] For manually annotated targets that are not successfully associated, they are considered new targets, a new target ID is assigned to them, and a new tracker is initialized for tracking.

[0255] For a tracker prediction target that is not successfully associated, if the duration of its unassociated state exceeds a preset threshold (eg, 5 frames), it is considered that the target has left the video screen and is deleted from the tracker list.

[0256] The successfully associated target pairs are used to optimize and update the tracker model.

[0257] Specifically, for the CSRT tracker, the successfully associated target pairs can be used to update the discriminative correlation filter model. The manually annotated target bounding box is used as a positive sample, and some background areas around the target are sampled as negative samples. The discriminative correlation filter model is retrained to adapt to the changes in the target appearance.

[0258] Step 5 and step 6 are the same as those in the above embodiment and will not be described again here.

[0259] Based on the above method, the present invention also provides a video target recognition device. Figure 5 The video object recognition device is described in detail.

[0260] Figure 5 The structure block diagram of the video object recognition device according to an embodiment of the present invention is schematically shown.

[0261] like Figure 5 As shown, the video object recognition device 500 of this embodiment includes, for example: a first determination module 510 , a labeling module 520 , a recognition module 530 , a second determination module 540 and a third determination module 550 .

[0262] The first determination module 510 is used to determine a plurality of key frames and frames to be processed in response to the input original video, wherein the frames to be processed are continuous frames between any two adjacent key frames, and each key frame contains at least one target to be identified. In one embodiment, the first determination module 510 can be used to perform the operation S110 described above, which will not be described in detail herein.

[0263] The marking module 520 is used to mark the target to be identified and obtain a first target bounding box. In one embodiment, the marking module 520 can be used to perform the operation S120 described above, which will not be described in detail here.

[0264] The recognition module 530 is used to identify the tracking target in the frame to be processed using a pre-trained target tracking model according to the first target bounding box, and the similarity between the tracking target and the target to be identified is determined based on the target overlap and the target category. In one embodiment, the recognition module 530 can be used to perform the operation S130 described above, which will not be repeated here.

[0265] The second determination module 540 is used to determine a second target bounding box of the tracked target. In one embodiment, the second determination module 540 can be used to perform the operation S140 described above, which will not be described in detail herein.

[0266] The third determination module 550 is used to determine the target category in the original video according to the overlap between the first target bounding box and the second target bounding box. In one embodiment, the third determination module 550 can be used to perform the operation S150 described above, which will not be described in detail here.

[0267] According to an embodiment of the present invention, any multiple modules of the first determination module 510, the labeling module 520, the identification module 530, the second determination module 540 and the third determination module 550 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the first determination module 510, the labeling module 520, the identification module 530, the second determination module 540 and the third determination module 550 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the first determination module 510, the labeling module 520, the identification module 530, the second determination module 540 and the third determination module 550 may be at least partially implemented as a computer program module, which may perform a corresponding function when executed.

[0268] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present invention. The attached method embodiment provides the elements of various steps in an exemplary order and is not intended to be limited to a specific order or hierarchy.

[0269] It should also be noted that the directional terms mentioned in the embodiments, such as "upper", "lower", "front", "back", "left", "right", etc., are only reference directions of the drawings and are not intended to limit the scope of protection of the present invention. Throughout the drawings, the same elements are represented by the same or similar reference numerals. Conventional structures or constructions will be omitted when they may cause confusion in the understanding of the present invention. In addition, the shapes, sizes, and positional relationships of the components in the drawings do not reflect the actual sizes, proportions, and actual positional relationships.

[0270] In the foregoing detailed description, various features are grouped together in a single embodiment to simplify the invention. This disclosure method should not be interpreted as reflecting such an intention.

[0271] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. With respect to the term "comprising" used in the specification, the word is covered in a manner similar to the term "including". Any term "or" used in the specification is intended to mean "non-exclusive or".

[0272] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A video target recognition method, characterized in that: include: In response to inputting an original video, a plurality of key frames and frames to be processed are determined, wherein the frames to be processed are continuous frames between any two adjacent key frames, and each of the key frames contains at least one target to be identified; Marking the target to be identified to obtain a first target bounding box; According to the first target bounding box, a pre-trained target tracking model is used to identify the tracking target in the frame to be processed, and the similarity between the tracking target and the target to be identified is determined based on the target overlap and the target category; Determine a second target bounding box of the tracked target; as well as Determining the target category in the original video according to the overlap between the first target bounding box and the second target bounding box; The step of identifying the tracking target in the frame to be processed by using a pre-trained target tracking model according to the first target bounding box includes: According to the first target bounding box, using a correlation filter-based tracker to predict the position and scale of the target in the adjacent frame to be processed in time sequence; Determine unlabeled samples based on the location and scale of the predicted target; and Optimizing the target tracking model according to the unlabeled samples and the labeled target to be identified to determine the tracking target; The step of optimizing the target tracking model according to the unlabeled sample and the labeled target to be identified to determine the tracking target includes: Determine a second prediction target in a corresponding key frame according to a first prediction target in a previous frame to be processed that is adjacent to the key frame; Determining the similarity between the second predicted target and the labeled target in the corresponding key frame based on the target overlap and the category similarity; and The target tracking model is optimized according to the first predicted target and the labeled targets in the plurality of key frames having the greatest similarity to the first predicted target to determine the tracking target.

2. The method according to claim 1, characterized in that Also includes: When the similarity between the second predicted target and the labeled targets in a plurality of consecutive key frames is less than a preset threshold, deleting the second predicted target; When the similarities between the annotated target in the key frame and the plurality of second predicted targets are all less than a preset threshold, the annotated target in the key frame is re-associated using a newly created tracker.

3. The method according to claim 1, characterized in that The step of predicting the position and scale of the target in the adjacent frame to be processed in time sequence using a tracker based on correlation filtering according to the first target bounding box includes: Initializing the tracker according to the first target bounding box to determine a third predicted target in a next frame to be processed adjacent to the key frame; According to the third prediction target, the prediction target with the maximum response to the tracker is determined frame by frame in time sequence.

4. The method according to claim 1, characterized in that: The first target bounding box includes a target category label, and determining the target category in the original video according to the overlap between the first target bounding box and the second target bounding box includes: When the overlap between the first target bounding box and the second target bounding box is greater than a preset threshold, determining a plurality of matching target bounding boxes; Using a voting mechanism to determine the target category labels with the largest number among the multiple matching target bounding boxes; and The target category in the original video is determined according to the target category labels with the largest number.

5. The method according to claim 4, characterized in that Before determining a plurality of matching target bounding boxes, the method further includes: According to the first target bounding box and the second target bounding box, the original video is cropped frame by frame to obtain a plurality of cropped areas, and the size of the cropped areas in each frame is the same; splicing the cropped areas of each frame in time sequence to obtain a target video; and Determining the target category in the original video according to the target category label with the largest number includes: The target category in the target video is determined according to the target category labels with the largest number.

6. The method according to claim 1, characterized in that Determining a plurality of key frames and frames to be processed comprises: Splitting the original video to obtain multiple sub-video frames; Extracting the plurality of sub-video frames at fixed time intervals to obtain the plurality of key frames; and Determine any sub-video frame between the plurality of adjacent key frames as the frame to be processed.

7. The method according to claim 1, characterized in that The marking of the target to be identified to obtain a first target bounding box includes: In response to user input, marking the target to be identified to obtain the first target bounding box; or A pre-trained deep learning model is used to label the target to be identified to obtain the first target bounding box.

8. A video target recognition device, characterized in that: include: A first determination module is used to determine a plurality of key frames and frames to be processed in response to an input original video, wherein the frames to be processed are continuous frames between any two adjacent key frames, and each of the key frames contains at least one target to be identified; A labeling module, used for labeling the target to be identified to obtain a first target bounding box; an identification module, configured to identify the tracking target in the frame to be processed by using a pre-trained target tracking model according to the first target bounding box, wherein the similarity between the tracking target and the target to be identified is determined based on target overlap and target category; A second determination module, used to determine a second target bounding box of the tracked target; as well as A third determination module, configured to determine a target category in the original video according to an overlap between the first target bounding box and the second target bounding box; The step of identifying the tracking target in the frame to be processed by using a pre-trained target tracking model according to the first target bounding box includes: According to the first target bounding box, using a correlation filter-based tracker to predict the position and scale of the target in the adjacent frame to be processed in time sequence; Determine unlabeled samples based on the location and scale of the predicted target; and Optimizing the target tracking model according to the unlabeled samples and the labeled target to be identified to determine the tracking target; The step of optimizing the target tracking model according to the unlabeled sample and the labeled target to be identified to determine the tracking target includes: Determine a second prediction target in a corresponding key frame according to a first prediction target in a previous frame to be processed that is adjacent to the key frame; Determining the similarity between the second predicted target and the labeled target in the corresponding key frame based on the target overlap and the category similarity; and The target tracking model is optimized according to the first predicted target and the labeled targets in the plurality of key frames having the greatest similarity to the first predicted target to determine the tracking target.

Images