An image target labeling method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose an image target labeling method and device, electronic equipment and storage medium. Embodiments of the present application can obtain a video frame sequence to be labeled of a road camera; for any current frame in the video frame sequence, a preset target detection model is used for detection to generate a detection box for a target detection object; a prediction box in the current frame is obtained, the prediction box being generated by a preset target tracking model according to a motion state of a target tracking object in a previous frame or multiple frames and a position of the detection box; the detection box in the current frame is matched with the prediction box, if the matching is successful, a historical identity of the target tracking object corresponding to the prediction box is assigned to a target detection object corresponding to the detection box, and if the matching fails, a new identity is assigned to the target detection object corresponding to the detection box. Thus, based on the spatiotemporal coherence of the video, the target is tracked and labeled efficiently and accurately.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image target labeling method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the rapid development of smart cities and autonomous driving technology, the demand for automatic and high-precision labeling of traffic monitoring videos is increasingly urgent. Currently, video labeling technology mainly relies on computer vision models to identify and segment targets in video frames. Among them, the general segmentation model based on large-scale data training (such as Segment Anything Model 2, SAM2) has attracted attention due to its performance in a wide range of scenarios. However, when facing the specific field of traffic monitoring, SAM2 still exposes the following technical defects:

[0003] (1) Lack of domain adaptability. Although the data set used by SAM2 is large in scale, the sample coverage for urban traffic scenarios (especially high-density road environments) is limited, resulting in poor generalization ability of the model in actual traffic scenarios;

[0004] (2) Lack of temporal consistency guarantee, unable to determine whether the object in the next frame is the same as the one in the previous frame;

[0005] (3) Poor robustness to complex scenes such as occlusion, slight changes in perspective, occlusion, or appearance may lead to classification errors, reducing the usability of labeled data.

[0006] Therefore, there is an urgent need for an automatic labeling method that integrates target detection and tracking for traffic monitoring scenarios to improve the accuracy, temporal consistency, and scene robustness of labeling results. SUMMARY

[0007] The embodiments of the present application provide an image target labeling method, device, electronic equipment and storage medium, which can efficiently and accurately track and label targets based on the spatiotemporal coherence of videos.

[0008] The embodiments of the present application provide an image target labeling method, comprising:

[0009] Obtaining a sequence of video frames to be labeled of a road camera;

[0010] For any current frame in the sequence of video frames, detecting based on a preset target detection model to generate a detection box for a target detection object;

[0011] Obtaining a prediction box in the current frame, the prediction box being generated by a preset target tracking model according to the motion state of a target tracking object in the previous frame or multiple frames and the position of the detection box;

[0012] matching the detection box in the current frame with the prediction box, if matching succeeds, assigning a historical identity of the target tracking object corresponding to the prediction box to the target detection object corresponding to the detection box, and if matching fails, assigning a new identity to the target detection object corresponding to the detection box.

[0013] Optionally, the target tracking object is multiple, and the matching the detection box in the current frame with the prediction box comprises:

[0014] respectively calculating an overlap degree of the detection box of the target detection object and the prediction box of each target tracking object;

[0015] extracting an appearance feature vector of the target detection object in the detection box, and respectively calculating a similarity of the appearance feature vector and a historical feature vector of each target tracking object;

[0016] matching the detection box with the prediction box based on the overlap degree and the similarity.

[0017] Optionally, the method further comprises:

[0018] in the case of successful matching, updating the target tracking model based on a position difference between the detection box and the prediction box;

[0019] fusing the appearance feature vector of the target detection object in the detection box with the historical feature vector of the target tracking object to obtain an updated historical feature vector of the target tracking object.

[0020] Optionally, after obtaining the video frame sequence to be labeled, the method further comprises:

[0021] for a first frame in the video frame sequence, performing picture segmentation on the first frame based on a preset semantic segmentation model to determine a region of interest;

[0022] for any current frame in the video frame sequence, performing detection based on a preset target detection model to generate a detection box for a target detection object, comprising:

[0023] for any current frame in the video frame sequence, performing detection based on a preset target detection model to obtain multiple initial detection boxes;

[0024] determining one or more initial detection boxes located in the region of interest as the detection box for the target detection object.

[0025] Optionally, after detecting, based on the preset target detection model, for any current frame in the sequence of video frames, to generate a detection frame for a target detection object, the method further comprises:

[0026] For any current frame in the sequence of video frames, in a case where the deviation index of the detection frame is greater than a first deviation threshold, performing picture segmentation on the current frame based on a preset semantic segmentation model to redetermine a region of interest;

[0027] The deviation index is a ratio of the initial detection frame located outside the region of interest to a number of all the initial detection frames.

[0028] Optionally, after detecting, based on the preset target detection model, for any current frame in the sequence of video frames, to generate a detection frame for a target detection object, the method further comprises:

[0029] For any current frame in the sequence of video frames, in a case where the deviation index of the detection frame is greater than a second deviation threshold, generating a warning information, the warning information being used to remind a monitoring personnel that the road camera starts to rotate;

[0030] In a case where the deviation index of the detection frame of the current frame is greater than a third deviation threshold, performing picture segmentation on the current frame based on a preset semantic segmentation model to redetermine a region of interest;

[0031] The deviation index is a ratio of the initial detection frame located outside the region of interest to a number of all the initial detection frames.

[0032] Optionally, the method further comprises:

[0033] Determining a bottom center point coordinate of the initial detection frame;

[0034] If the bottom center point coordinate of the initial detection frame is located outside a polygonal contour of the region of interest, it is determined that the initial detection frame is located outside the region of interest;

[0035] If the bottom center point coordinate of the initial detection frame is located inside the polygonal contour of the region of interest, it is determined that the initial detection frame is located inside the region of interest.

[0036] Embodiments of the present application also provide an image target labeling device, comprising:

[0037] A video frame acquisition module is configured to acquire a sequence of video frames to be labeled of a road camera;

[0038] a target detection module configured to, for any current frame in the sequence of video frames, perform detection based on a preset target detection model to generate a detection box for a target detection object;

[0039] a target tracking module configured to obtain a prediction box in the current frame, the prediction box being generated by a preset target tracking model based on a motion state of a target tracking object in a previous frame or multiple frames and a position of the detection box;

[0040] a target labeling module configured to match the detection box with the prediction box in the current frame, if the matching is successful, assign a historical identity of the target tracking object corresponding to the prediction box to a target detection object corresponding to the detection box, and if the matching fails, assign a new identity to the target detection object corresponding to the detection box.

[0041] Embodiments of the present application also provide an electronic device comprising a memory storing a plurality of instructions; and a processor loading the instructions from the memory to perform steps in any of the image target labeling methods provided by the embodiments of the present application.

[0042] Embodiments of the present application also provide a computer-readable storage medium storing a plurality of instructions, the instructions being adapted to be loaded by a processor to perform steps in any of the image target labeling methods provided by the embodiments of the present application.

[0043] Embodiments of the present application can obtain a sequence of video frames to be labeled by a road camera; for any current frame in the sequence of video frames, perform detection based on a preset target detection model to generate a detection box for a target detection object; obtain a prediction box in the current frame, the prediction box being generated by a preset target tracking model based on a motion state of a target tracking object in a previous frame or multiple frames and a position of the detection box; match the detection box with the prediction box in the current frame, if the matching is successful, assign a historical identity of the target tracking object corresponding to the prediction box to a target detection object corresponding to the detection box, and if the matching fails, assign a new identity to the target detection object corresponding to the detection box.

[0044] Thus, the detection boxes in different frames are associated by the tracking model to assign a stable and unchanged identity to the same target, and the problems such as short-time occlusion, target overlap and motion blur are effectively handled by the matching mechanism of the prediction box and the detection box. When a target disappears temporarily and then reappears, the target can be recaptured and associated back to the original identity based on the motion prediction and historical appearance information by the tracking model, instead of being misjudged as a new target, thereby realizing the time sequence consistency of the target, greatly reducing the missed detection and false detection, and improving the quality of the labeled data. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to make the technical solutions in the embodiments of the present application clearer, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0046] Figure 1 is a flowchart of an image target labeling method provided by an embodiment of the present application;

[0047] Figure 2 is a flowchart of a reset judgment provided by an embodiment of the present application;

[0048] Figure 3 is a structural diagram of an image target labeling device provided by an embodiment of the present application;

[0049] Figure 4 is a structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort belong to the scope of protection of the present application.

[0051] With the rapid development of smart cities and autonomous driving technologies, there is an increasing demand for automatic and high-precision labeling of traffic monitoring videos. Currently, video labeling technology mainly relies on computer vision models to identify and segment targets in video frames. Among them, the general segmentation model based on large-scale data training (such as Segment Anything Model 2, i.e. SAM2) has attracted attention due to its performance in a wide range of scenarios. However, when facing the specific field of traffic monitoring, SAM2 still exposes the following technical defects:

[0052] (1) Insufficient domain adaptability. Although the data set used by SAM2 is large in scale, the coverage of samples for urban traffic scenarios (especially high-density road environments) is limited, resulting in poor generalization ability of the model in actual traffic scenarios. In actual testing, the model often misidentifies and segments large areas of vehicles and pedestrians in traffic congestion situations, such as failing to accurately distinguish adjacent vehicle outlines in vehicle-dense areas, or incorrectly segmenting a group of adhering pedestrians as a single entity, which seriously affects the reliability of the labeling results.

[0053] (2) Lack of timing consistency guarantee. The targets in traffic monitoring videos (such as vehicles) have continuous spatio-temporal representation characteristics, requiring the labeling model not only to complete single-frame recognition, but also to have the ability to judge the identity of the same target across frames. Although SAM2 supports video labeling, its architecture lacks an efficient multi-target tracking mechanism, which cannot associate the same target in adjacent frames, resulting in the inability to assign a stable identity (ID) to the vehicle. For example, when the target vehicle moves or changes appearance between consecutive frames, the model cannot associate it as the same entity, resulting in a broken track or identity switching.

[0054] (3) Poor robustness to complex scenes such as occlusion. In highway scenarios, vehicles often have temporary feature disappearance or distortion due to mutual occlusion, light changes, motion blur, etc. SAM2 is prone to serious misidentification in such scenarios, such as misidentifying two cars as one medium or large truck after occlusion, or losing the target segmentation mask due to partial occlusion. Such errors significantly reduce the usability of labeled data, which cannot meet the needs of high-precision trajectory analysis, behavior prediction, and other downstream tasks.

[0055] In view of this, the present application provides an image target labeling method to solve this problem.

[0056] Specifically, please refer to Figure 1 The specific process of the image target labeling method can be as follows:

[0057] S110, obtaining a video frame sequence to be labeled of a road camera.

[0058] In this embodiment, the road camera refers to a fixed or movable image acquisition device deployed in traffic scenes such as highways, intersections, tunnels, and bridges. The video frame sequence refers to a series of static image units obtained by decoding the continuous video stream collected by the road camera in chronological order.

[0059] Specifically, first, the original video stream collected by the road camera is accessed. The video stream can be packaged in an efficient compression encoding format (such as H.264 or H.265) to reduce transmission bandwidth and storage space.

[0060] Subsequently, the compressed video stream is fully decoded to convert the entire video content into a sequence of static images arranged in chronological order. This process is performed without skipping frames, i.e., each frame in the video stream is fully restored to ensure the integrity and continuity of the timing information. Because any skipping behavior can miss critical motion information (such as sudden lane changes or sudden braking), causing the subsequent target tracking algorithm to produce track breaks or identity switching errors due to data loss.

[0061] Finally, in the generated video frame sequence, each frame of image will be accompanied by timestamp information and bound to a global frame number, providing stable, continuous and high-fidelity data input for subsequent target labeling, and ensuring high consistency of labeling results in the time dimension.

[0062] After obtaining the video frame sequence to be labeled of the road camera, one or more target categories to be focused on in this labeling task are first determined. Specifically, the first frame image of the video can be displayed to the user through an interactive interface. The user needs to check one or more target categories to be focused on in this labeling task from a pre-defined category library (such as vehicles, pedestrians, non-motor vehicles, traffic signs, etc.). For example, if the user only checks "sedan" and "truck", other categories such as pedestrians and traffic lights will be automatically ignored in subsequent processing, and only "sedan" and "truck" will be focused on, thereby improving processing speed and reducing unnecessary interference detection.

[0063] After confirming the target categories, the first frame processing procedure is started. This procedure includes two parallel or serial computing tasks:

[0064] 1. Semantic segmentation to generate road mask: For the first frame in the video frame sequence, a pre-installed semantic segmentation model is used to perform picture segmentation on the first frame to determine the region of interest.

[0065] The semantic segmentation model is a computer vision algorithm whose core task is to classify input images at the pixel level. The semantic segmentation model assigns a specific category label (such as "road", "vehicle", "sky", "building", etc.) to each pixel point in the image, thereby achieving fine deconstruction and understanding of the image content. For example, the semantic segmentation model can be DeepLabV3, U-Net, PSPNet, etc.

[0066] The region of interest refers to a region in the image that is specially delineated and related to the current task.

[0067] For example, by calling a pre-trained semantic segmentation model, the first frame image is analyzed at the pixel level to accurately divide the passable road region and generate a binary mask (for example, white pixels represent roads and black pixels represent backgrounds or other objects). This mask serves as the "region of interest" for subsequent target detection, and detection is only performed within this region, thereby effectively filtering out interference from sidewalks, green belts, sky, etc. and improving the relevance and accuracy of labeling.

[0068] 2. First frame target detection and verification: For the first frame in the video frame sequence, a pre-installed target detection model is used for detection to obtain multiple initial detection boxes; one or more initial detection boxes located in the region of interest are determined as detection boxes for target detection objects.

[0069] wherein the target detection model is a trained computer vision algorithm (e.g., YOLOv10, Faster R-CNN, etc.) whose core task is to process a single input image and output the locations and classes of all targets of interest in the image.

[0070] Specifically, based on the target class selected by the user and the generated road mask, the target detection model performs inference on the first frame. The detection result (i.e., the target detection box) will be superimposed on the image for the user to visually verify. The user needs to confirm whether all detection boxes are accurately selected and located within the road mask area (i.e., the region of interest), so as to avoid errors caused by model mis-detection or missed detection in the first frame from being propagated and amplified in the entire video sequence.

[0071] When the user confirms that the first frame labeling result is correct, the initialization is considered ready. At this point, the labeling system has the task target (class), working area (road mask), and correct initial state (first frame detection result), and can start the subsequent full-automatic labeling.

[0072] S120, for any current frame in the video frame sequence, based on the preset target detection model, a detection box for the target detection object is generated.

[0073] wherein the target detection object refers to an independent entity identified by the target detection model in the current frame. It is a "momentary" concept, which only represents that the target detection model has found a target (such as a car) that meets a certain class in the current frame, but has not been associated with any historical track.

[0074] The detection box is a rectangular area output by the target detection model to represent the position of the target detection object in the image. In some embodiments, the detection box can be represented by a quaternion (x, y, width, height), where (x, y) is the top-left corner coordinate (or center point coordinate) of the detection box, and width and height are the width and height of the detection box. The detection box represents the "observation" of the current frame.

[0075] Specifically, the process includes: for any current frame in the video frame sequence, based on the preset target detection model, a plurality of initial detection boxes are obtained; and one or more initial detection boxes located within the region of interest are determined as the detection box for the target detection object.

[0076] In this embodiment, after the region of interest is labeled in the video frame before the current frame, the detection boxes detected in the current frame can be filtered and selected based on the region of interest, and the detection boxes located outside the region of interest are discarded to filter non-road targets, thereby reducing the burden of subsequent target tracking and the situation of false tracking.

[0077] In some embodiments, the process mainly includes: determining the bottom center point coordinate of the initial detection box; if the bottom center point coordinate of the initial detection box is located outside the polygonal contour of the region of interest, determining that the initial detection box is located outside the region of interest; if the bottom center point coordinate of the initial detection box is located inside the polygonal contour of the region of interest, determining that the initial detection box is located inside the region of interest.

[0078] In this embodiment, the bottom center point of the initial detection box is used to represent the position of the target detection object on the road. In addition, the center point or the corner point of the initial detection box can also be used to represent the position of the target detection object on the road, which can be selected according to actual conditions.

[0079] Specifically, the contour finding algorithm (such as the findContours function in OpenCV) can be used to extract the polygonal contour representing the road region of interest from the current frame. Then, the point-in-polygon detection is performed: for each initial detection box output by the target detection model, the bottom center point P(x, y) is extracted, and it is judged whether the point P is located inside the road polygonal contour. If the point P is inside the polygonal contour, it means that the initial detection box is located in the road region and is a valid target, which is retained and sent to the subsequent target tracking model for tracking. If the point P is outside the polygonal contour, it means that the initial detection box is located in the non-road region, which is determined as an “off-region marker box” and discarded directly.

[0080] S130, obtaining a prediction box in the current frame, the prediction box being generated by a preset target tracking model according to the motion state and the position of the detection box of the target tracking object in the previous frame or multiple frames.

[0081] The target tracking model is a trained computer vision algorithm (such as StrongSORT, DeepSORT, etc.), and its core task is to maintain the identity of the target across the sequence of video frames. That is, a unique identity (ID) is assigned and maintained for the same target in different frames, realizing cross-frame tracking. It does not directly detect the target, but receives the detection result of each frame, and based on the motion, appearance and other information, it associates these discrete detections to form a continuous trajectory.

[0082] The prediction box is a rectangular region generated by a motion prediction module (such as a Kalman filter) in the target tracking model, which estimates the position of the target in the current frame. It is the result predicted based on the motion state (such as position, speed) of the target in the previous frame or multiple frames, representing an “estimated value”.

[0083] A target tracking object refers to a tracked target maintained by a target tracking model. It is a "persistent" concept, containing historical information of the target (such as past position, velocity, appearance feature, identity, etc.). Its goal is to associate multiple target detection objects as the same entity across time.

[0084] A motion state is a mathematical representation of the motion information of a target tracking object. For example, in Kalman filtering, the motion state includes position, velocity, acceleration, etc. The motion state is predicted according to the target tracking model, and is updated according to new detection results.

[0085] In this embodiment, the target tracking model estimates the possible positions of the target tracking objects in the current frame according to the motion states of the target tracking objects in the previous frame or multiple frames and the positions of the detection boxes, providing prior information for subsequent data association.

[0086] Specifically, first, all target tracking objects currently being tracked (each target tracking object has a unique identity) are traversed. For each target tracking object, based on its motion state (including position, velocity, acceleration, etc.) in the previous frame (or multiple frames) and a known dynamic model (such as a uniform speed model, a uniform acceleration model, etc.), an optimal, uncertain prediction of the position and state of the target tracking object in the current frame is made. The output of the target tracking model is a prediction box, representing the position and size where the target tracking object should most appear in the current frame. At the same time, this prediction box is also a benchmark for matching with actual detection results in the current frame. By comparing the differences between the prediction box and the detection box, it can be efficiently and accurately determined which detection results correspond to which existing tracking targets.

[0087] S140, match the detection box in the current frame with the prediction box, if the matching is successful, assign the historical identity of the target tracking object corresponding to the prediction box to the target detection object corresponding to the detection box, if the matching fails, assign a new identity to the target detection object corresponding to the detection box.

[0088] The historical identity is a unique and unchanging number (such as ID: 103) assigned by the target tracking model to each target tracking object. This ID represents the same physical entity (such as that red truck) throughout the video sequence.

[0089] The embodiment links the discrete and identityless detection results with the continuous and identified tracking trajectories by data association. The association needs two sets of data: one set of data represents the observation values of the current frame (i.e. the detection boxes generated by the target detection model), and the other set of data represents the prediction values based on the historical information (i.e. the prediction boxes generated by the target tracking model for each existing tracking target). Then, the target labeling and tracking are realized by finding a most matched prediction box (i.e. an existing tracking target) for each detection box.

[0090] In the matching process, if a detection box of the current frame is successfully paired with a prediction box of an existing target tracking object, it means that they represent the same physical entity. Therefore, the historical identity carried by the target tracking object is "given" to the detection box of the current frame.

[0091] If a detection box fails to match with the prediction box of any existing target tracking object, it means that a new target enters the monitoring field of view. Therefore, a new target tracking object is initialized for this detection box, and a brand new and unique identity is assigned to it, and the tracking process of the target is started.

[0092] In addition, if the prediction box of an existing target tracking object fails to match with any detection box, it means that a temporary loss of an existing target (such as being blocked or driving out of the picture). At this time, the loss of the target is recorded once, and if it is still not captured again after a certain number of consecutive losses, it is determined that it leaves and the tracking is terminated.

[0093] Specifically, assuming that there are multiple target tracking objects, the process of matching the detection box in the current frame with the prediction box includes: calculating the overlap degree of the detection box of the target detection object with the prediction box of each target tracking object; extracting the appearance feature vector of the target detection object in the detection box, and calculating the similarity of the appearance feature vector with the historical feature vector of each target tracking object; and matching the detection box with the prediction box based on the overlap degree and the similarity.

[0094] In the embodiment, when matching the detection box in the current frame with the prediction box, two factors need to be considered: one is the overlap degree of the detection box and the prediction box, and the other is the similarity in appearance between the target detection object in the detection box and the historical target tracking object.

[0095] wherein the overlap degree is a geometric metric index quantifying the degree of spatial position coincidence of the two bounding boxes in the two-dimensional image plane. The appearance feature vector is a high-dimensional, numerical vector, which is a highly abstracted and compressed representation of the visual appearance information (such as color, texture, shape, local structure) of the image block (such as a car, a truck) in the target detection box. The historical feature vector refers to a collection of multiple appearance feature vectors extracted at different time instants (different frames) in the entire life cycle of a target tracking object. It constitutes the dynamic appearance library of the target, recording the appearance change history of the target in the past period of time.

[0096] In a specific implementation, the overlap degree can be measured by the intersection over union (IoU). For each detection box and each prediction box, the intersection over union between them is calculated, which is the ratio of the intersection area to the union area, ranging from 0 to 1. The higher the IoU value, the larger the overlap area of the detection box and the prediction box in the physical space, meaning the higher the possibility of the detection target continuing to move along its historical trajectory.

[0097] The appearance feature vector can be extracted by a deep learning model (for example, a convolutional neural network CNN). The appearance feature vector of the target detection object in each detection box is extracted by a pre-trained Re-Identification (Re-ID) deep neural network. At the same time, each target tracking object maintains a dynamic appearance library, which stores the appearance feature vectors (i.e. historical feature vectors) corresponding to the successful association of the target in the past several frames. Then, the cosine similarity between the appearance feature vector of the current detection box and all historical feature vectors in the appearance library of the target tracking object is calculated, and the maximum or average value is taken as the measure of the appearance similarity between the two.

[0098] Finally, the overlap degree and the similarity are considered comprehensively to match the detection boxes.

[0099] For example, suppose that the existing target tracking objects include a white car (identity: J-101, whose prediction box in the current frame is located at (x1, y1) predicted by the target tracking model) and a red truck (identity: K-103, whose prediction box in the current frame is located at (x2, y2) predicted by the target tracking model). In the current frame, a detection box of a white car (located at (x1', y1')) and a detection box of a black car (located at (x2', y2')) are detected by the target detection model.

[0100] When matching, if the overlap between the detection frame position (x1', y1') of the white car and the prediction frame position (x1, y1) is greater than the overlap threshold, and the similarity between the two in appearance is greater than the similarity threshold, it is considered that the two are matched successfully and belong to the same physical entity, and thus the detection frame of the white car is labeled as J-101.

[0101] If the overlap between the detection frame position (x2', y2') of the black car and the prediction frame position (x2, y2) is greater than the overlap threshold, but the similarity between the two in appearance is much less than the similarity threshold, it is considered that the matching fails, indicating that the black car is a newly appearing target, and thus a brand new identity is assigned to it for subsequent tracking, such as J-102.

[0102] Thus, by comprehensively measuring the similarity of the "overlap" (spatial information) and the "appearance feature vector" (visual information) and comparing them with the "historical feature vector" (temporal information), the target tracking capability with high precision and strong robustness in complex scenes is realized.

[0103] In addition, after successful matching, it also includes:

[0104] In the case of successful matching, the target tracking model is updated based on the position difference between the detection frame and the prediction frame, and the appearance feature vector of the target detection object in the detection frame is fused with the historical feature vector of the target tracking object to obtain the updated historical feature vector of the target tracking object.

[0105] In this embodiment, if the detection frame and a certain prediction frame are matched successfully, the latest information (position, appearance) of the detection frame is used to update the state of the target tracking object corresponding to the prediction frame, such as updating the Kalman filter of the target tracking model (the Kalman filter will correct its state estimation, including speed estimation, according to the difference between the two in proportion to the Kalman gain), so that it can better predict the position of the target tracking object in the next frame. At the same time, the appearance feature of the current detection frame is fused and updated with the historical feature of the target tracking object, so that the appearance expression can better adapt to the changes of the target (such as changes in lighting).

[0106] Thus, through the cycle of "prediction-observation-correction", the motion information of the target can be quickly learned from zero with only one or two frames of data, and accurate motion prediction can be continuously provided in subsequent tracking. This ability enables it to effectively deal with temporary occlusion - within a few frames of target disappearance, it can reasonably predict the possible position of the target based on the last known speed and direction, maintaining high-efficiency target tracking capability.

[0107] The embodiments of the present application associate the detection boxes in different frames through a tracking model, assign a stable and unchanged identity label to the same target, and effectively handle problems such as short-term occlusion, target overlap, and motion blur through a matching mechanism of the prediction box and the detection box. When the target disappears and reappears, it can be recaptured and associated back to the original identity label based on motion prediction and historical appearance information through the tracking model, instead of being misjudged as a new target, realizing the time sequence consistency of the target, greatly reducing the missed detection and false detection, and improving the quality of the labeled data.

[0108] In addition, while performing target detection and tracking on each frame, the above method further comprises:

[0109] For any current frame in the video frame sequence, if the deviation index of the detection box is greater than the first deviation threshold, the current frame is segmented based on the preset semantic segmentation model, and the region of interest is re-determined;

[0110] The deviation index is the ratio of the initial detection box located outside the region of interest to the number of all initial detection boxes.

[0111] In actual scenarios, the road camera may be rotated or zoomed due to pan-tilt control, wind influence, or manual adjustment. Such fundamental changes in scene structure will make the “region of interest” defined in the initialization stage completely invalid, leading to the collapse of subsequent detection and tracking. To this end, the present application also traverses all initial detection boxes generated by the target detection model in the current frame when performing target detection and tracking on each frame, and judges their spatial positional relationship with the currently used region of interest to identify scene changes and adjust in time. For example, the judgment basis can be whether the bottom center point of the detection box is located inside the polygon contour of the region of interest. This point best represents the contact position of the target and the ground, avoiding misjudgment caused by changes in the size of the frame or the inclination of the vehicle.

[0112] Specifically, first, the number of detection boxes located outside the region of interest (Noutside) and the total number of detection boxes (Ntotal) are counted, and the ratio of the two is calculated, that is, the deviation index (Deviation Index, DI): DI=Noutside / Ntotal. This index quantitatively reflects the degree of deviation between the current frame observation and the expected working area.

[0113] Then, the index is compared with a preset first deviation threshold (e.g., 50%). If the deviation index is less than or equal to the first deviation threshold, it is considered that the deviation of a small number of detection boxes is caused by normal traffic behaviors or slight shaking, such as vehicle line pressing, temporary crossing, etc., and the current defined region of interest is continued to be used for target detection and tracking. If the deviation index is greater than the first deviation threshold, it means that more than half of the observed targets are outside the expected region, which is extremely unlikely to be accidental traffic behavior, and strongly suggests that the underlying scene itself has changed in structure (i.e., the camera is rotating). Therefore, it is considered that the case is abnormal, and a reset operation is triggered to pause the standard target detection and tracking process, take the current frame as a new reference frame, and send it to the pre-trained semantic segmentation model for analysis to segment the road region under the current view and convert it into a new binary mask, which is a completely new region of interest that accurately matches the current picture.

[0114] At the same time, to avoid the identity (ID) of the trajectories of the new and old scenes from being cross-contaminated, all existing tracking targets can be emptied, and the detection and tracking list is reinitialized based on the new region of interest starting from the current frame (i.e., the steps S120-S140 are repeated), to ensure the absolute purity and temporal consistency of the subsequent labeled data.

[0115] The embodiment can automatically recover from scene changes without human intervention, continue to output high-quality labeled data, and realize a fully automated labeling pipeline.

[0116] Please refer to Figure 2 In another implementation manner, the method further includes:

[0117] S210, for any current frame in the video frame sequence, if the deviation index of the detection box is greater than the second deviation threshold, a warning information is generated, and the warning information is used to remind a monitoring personnel that a road camera starts to rotate.

[0118] When the deviation index is greater than the second deviation threshold (e.g., 20%-30%), it is indicated that a considerable proportion of targets have appeared in the abnormal region, and the scene may be changing (e.g., the camera starts to rotate slowly). It is determined as a suspected rotating event, and a warning information is automatically generated. The warning information includes a timestamp, a camera ID, a deviation index value, and other key data, and notifies a monitoring personnel at a remote end through a visual interface pop-up window, a sound reminder, or a log record, etc. At this time, the labeling process of the system itself does not stop, but continues to run. Through the warning, the monitoring personnel can confirm whether the camera rotation is planned or the device is malfunctioning, and make preparations for observation or subsequent intervention, realizing an upgrade from "completely unattended" to "automated with supervision".

[0119] S220, in a case where the deviation index of the detection frame of the current frame is greater than the third deviation threshold, performing picture segmentation on the current frame based on a preset semantic segmentation model, and re-determining the region of interest;

[0120] The deviation index is a ratio of the initial detection frame located outside the region of interest to the number of all initial detection frames.

[0121] When the deviation index is greater than the second deviation threshold (for example, 20%-30%) but less than the third deviation threshold (for example, 60%-70%), it is considered that the current scene change is within a tolerable range or the rotation is not yet completed, and therefore the target detection and tracking are continued with the existing region of interest.

[0122] When the deviation index is greater than the third deviation threshold, it indicates that the deviation is very serious, most of the targets have overflowed, and the existing region of interest and tracking context have completely failed. The camera rotation may be close to completion or have a large jump. Therefore, a reset process is triggered immediately. This process is independent of the early warning and is executed as soon as the condition is met: pause the current target detection and tracking process, take the current frame as a reference, call the semantic segmentation model to re-segment the picture, generate a completely new region of interest, and clear the existing tracking list to completely avoid ID pollution, and then re-initialize the entire labeling process based on the new region of interest (i.e., repeat steps S120-S140 described above).

[0123] The embodiment divides the process of scene change into a pre-warning stage and a forced reset stage by setting two different levels of deviation thresholds (the second deviation threshold and the third deviation threshold), and realizes early perception and differentiated response to the camera rotation event. This not only guarantees the automatic performance of the target labeling system, but also gives the monitoring personnel the key right to know and intervene, and embodies the design concept of "automation as the main, and man-machine cooperation as the auxiliary".

[0124] In order to better implement the above method, an image target labeling device is further provided in the embodiment of the application. The device can be specifically integrated in an electronic device, which can be a terminal, a server, or the like. The terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a notebook computer, a personal computer, or the like. The server can be a single server or a server cluster composed of multiple servers.

[0125] For example, in the embodiment, the image target labeling device is specifically integrated in a server, and the method of the embodiment of the application is described in detail.

[0126] For example, as shown in Figure 3 The image target labeling device can include a video frame acquisition module 310, a target detection module 320, a target tracking module 330, and a target labeling module 340, as follows:

[0127] The video frame acquisition module 310 is configured to acquire a video frame sequence to be labeled of a road camera.

[0128] The target detection module 320 is configured to, for any current frame in the video frame sequence, perform detection based on a preset target detection model to generate a detection box for a target detection object.

[0129] The target tracking module 330 is configured to acquire a prediction box in the current frame, the prediction box being generated by a preset target tracking model based on a motion state of a target tracking object in a previous frame or multiple frames and a position of the detection box.

[0130] The target labeling module 340 is configured to match the detection box with the prediction box in the current frame, if the matching is successful, assign a historical identity of the target tracking object corresponding to the prediction box to a target detection object corresponding to the detection box, and if the matching fails, assign a new identity to the target detection object corresponding to the detection box.

[0131] Optionally, the target labeling module 340 comprises:

[0132] The overlap detection unit is configured to calculate an overlap degree of the detection box of the target detection object with the prediction box of each target tracking object respectively.

[0133] The appearance similarity detection unit is configured to extract an appearance feature vector of the target detection object in the detection box, and calculate a similarity of the appearance feature vector with a historical feature vector of each target tracking object respectively.

[0134] The matching unit is configured to match the detection box with the prediction box based on the overlap degree and the similarity.

[0135] Optionally, the apparatus further comprises:

[0136] The position updating module is configured to, in a case where the matching is successful, update the target tracking model based on a position difference between the detection box and the prediction box.

[0137] The appearance feature fusion module is configured to fuse the appearance feature vector of the target detection object in the detection box with the historical feature vector of the target tracking object to obtain an updated historical feature vector of the target tracking object.

[0138] Optionally, the apparatus further comprises:

[0139] The region of interest determination module is configured to, for a first frame in the video frame sequence, perform picture segmentation on the first frame based on a preset semantic segmentation model to determine a region of interest.

[0140] The target detection module 320 comprises:

[0141] An initial detection unit is configured to, for any current frame in the sequence of video frames, perform detection based on a preset target detection model to obtain a plurality of initial detection boxes;

[0142] A screening unit is configured to determine one or more initial detection boxes located within the region of interest as detection boxes for the target detection object.

[0143] Optionally, the device further comprises:

[0144] A first region-of-interest resetting module is configured to, for any current frame in the sequence of video frames, perform picture segmentation on the current frame based on a preset semantic segmentation model to re-determine the region of interest in a case where the deviation index of the detection boxes is greater than a first deviation threshold.

[0145] The deviation index is a ratio of the initial detection boxes located outside the region of interest to the number of all the initial detection boxes.

[0146] Optionally, the device further comprises:

[0147] A second region-of-interest resetting module is configured to, for any current frame in the sequence of video frames, generate a warning information in a case where the deviation index of the detection boxes is greater than a second deviation threshold, the warning information being used to remind a monitoring personnel that a road camera starts to rotate.

[0148] In a case where the deviation index of the detection boxes of the current frame is greater than a third deviation threshold, performing picture segmentation on the current frame based on a preset semantic segmentation model to re-determine the region of interest.

[0149] The deviation index is a ratio of the initial detection boxes located outside the region of interest to the number of all the initial detection boxes.

[0150] Optionally, the device further comprises:

[0151] A detection box position determining module is configured to determine a bottom center point coordinate of the initial detection box.

[0152] A position judging module is configured to, if the bottom center point coordinate of the initial detection box is located outside the polygonal contour of the region of interest, determine that the initial detection box is located outside the region of interest.

[0153] If the bottom center point coordinate of the initial detection box is located inside the polygonal contour of the region of interest, determine that the initial detection box is located within the region of interest.

[0154] In implementation, the above various units can be implemented as independent entities, or can be combined as the same or several entities, and the specific implementation of the above various units can be referred to the method embodiments above, which will not be described here.

[0155] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0156] The embodiments of the present application also provide an electronic device, which can be a terminal, a server, etc. Wherein, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a notebook computer, a personal computer, etc.; the server can be a single server, or a server cluster composed of multiple servers, etc.

[0157] In some embodiments, the image target labeling apparatus can also be integrated in multiple electronic devices, for example, the image target labeling apparatus can be integrated in multiple servers, and the multiple servers can implement the image target labeling method of the present application.

[0158] In the present embodiment, as shown in Figure 4 The electronic device structure related to the embodiments of the present application is shown, and specifically:

[0159] The electronic device can include a processor 410 with one or more processing cores, a memory 420 with one or more computer readable storage media, a power supply 430, an input module 440, a communication module 450, etc. Those skilled in the art can understand that the electronic device structure shown in Figure 4 The electronic device structure shown in the present embodiment does not constitute a limitation on the electronic device, and can include more or fewer components than shown, or combine certain components, or different component arrangements. Among them:

[0160] The processor 410 is the control center of the electronic device, which connects all parts of the electronic device through various interfaces and lines, executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 420, and calling data stored in the memory 420. In some embodiments, the processor 410 can include one or more processing cores; in some embodiments, the processor 410 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 410.

[0161] The memory 420 can be used to store software programs and modules, and the processor 410 can execute various function applications and data processing by running the software programs and modules stored in the memory 420. The memory 420 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 420 can include a high-speed random access memory, and can also include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 420 can also include a memory controller to provide the processor 410 with access to the memory 420.

[0162] The electronic device also includes a power supply 430 for powering the various components, which in some embodiments can be logically connected to the processor 410 through a power management system, so that the power management system can be used to manage charging, discharging, and power consumption management, etc. The power supply 430 can also include one or more direct current or alternating current power sources, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, etc.

[0163] The electronic device can also include an input module 440, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0164] The electronic device can also include a communication module 450, which in some embodiments can include a wireless module, and the electronic device can use the wireless module of the communication module 450 for short-range wireless transmission, thereby providing the user with wireless broadband Internet access. For example, the communication module 450 can be used to help the user send and receive emails, browse web pages, and access streaming media, etc.

[0165] Although not shown, the electronic device can also include a display unit, etc., which will not be described here. In particular, in the present embodiment, the processor 410 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 420 according to the following instructions, and the processor 410 will run the application programs stored in the memory 420, thereby implementing the various functions in the above image target labeling method.

[0166] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by instructions controlling related hardware, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0167] To this end, an embodiment of the present application provides a computer readable storage medium, which stores a plurality of instructions capable of being loaded by a processor to execute steps in any of the image target labeling methods provided by the embodiments of the present application.

[0168] The storage medium can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0169] According to an aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the electronic device to perform the image target labeling method provided in the above embodiments.

[0170] Since the instructions stored in the storage medium can execute the steps in any of the image target labeling methods provided by the embodiments of the present application, the beneficial effects of any of the image target labeling methods provided by the embodiments of the present application can be achieved, which are described in detail in the above embodiments and will not be described here again.

[0171] The above describes in detail the image target labeling method, device, electronic device and storage medium provided by the embodiments of the present application. The principles and implementation manners of the present application are described by applying specific examples in this paper. The above embodiment descriptions are only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, the specific implementation manners and application ranges will be changed according to the idea of the present application. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. An image target labeling method, characterized in that, The method comprises: acquiring a video frame sequence to be labeled of a road camera; for a first frame in the video frame sequence, performing picture segmentation on the first frame based on a preset semantic segmentation model to determine a region of interest, and for any current frame in the video frame sequence, performing detection based on a preset target detection model to obtain a plurality of initial detection boxes, and determining one or more initial detection boxes located in the region of interest as detection boxes for a target detection object; acquiring a prediction box in the current frame, the prediction box being generated by a preset target tracking model according to a motion state of a target tracking object in a previous frame or a plurality of frames and a position of a detection box; matching the detection box in the current frame with the prediction box, if the matching is successful, assigning a historical identity of the target tracking object corresponding to the prediction box to a target detection object corresponding to the detection box, and if the matching fails, assigning a new identity to the target detection object corresponding to the detection box; further comprising: for any current frame in the video frame sequence, if a deviation index of the detection box is greater than a deviation threshold, performing picture segmentation on the current frame based on the preset semantic segmentation model to redetermine the region of interest, and based on the redetermined region of interest, determining the detection box of the target detection object for the video frame after the current frame; wherein the deviation index is a ratio of the initial detection box located outside the region of interest to a number of all the initial detection boxes.

2. The image object labeling method of claim 1, wherein, The target tracking object has a plurality of target tracking objects, and the matching of the detection box in the current frame with the prediction box comprises: respectively calculating an overlap degree of the detection box of the target detection object and the prediction box of each target tracking object; extracting an appearance feature vector of the target detection object in the detection box, and respectively calculating a similarity of the appearance feature vector and a historical feature vector of each target tracking object; based on the overlap degree and the similarity, matching the detection box with the prediction box.

3. The image object labeling method of claim 2, wherein, The method further comprises: in the case of successful matching, updating the target tracking model based on a position difference between the detection box and the prediction box; fusing the appearance feature vector of the target detection object in the detection box with the historical feature vector of the target tracking object to obtain an updated historical feature vector of the target tracking object.

4. The image object labeling method of claim 1, wherein After the detection based on the preset target detection model is performed for any current frame in the video frame sequence to generate the detection box for the target detection object, the method further comprises: for any current frame in the video frame sequence, if a deviation index of the detection box is greater than a second deviation threshold, generating an early warning information, the early warning information being used to remind a monitoring personnel that the road camera starts to rotate; in the case that the deviation index of the detection box of the current frame is greater than a third deviation threshold, performing picture segmentation on the current frame based on the preset semantic segmentation model to redetermine the region of interest; wherein the deviation index is a ratio of the initial detection box located outside the region of interest to a number of all the initial detection boxes.

5. The image object labeling method of claim 1 or 4, wherein The method further comprises: determining a bottom center point coordinate of the initial detection frame; if the bottom center point coordinate of the initial detection frame is located outside the polygonal contour of the region of interest, determining that the initial detection frame is located outside the region of interest; if the bottom center point coordinate of the initial detection frame is located inside the polygonal contour of the region of interest, determining that the initial detection frame is located inside the region of interest.

6. An image target labeling apparatus characterized by comprising: The device comprises: a video frame acquisition module configured to acquire a video frame sequence to be labeled of a road camera; a region of interest determination module configured to, for a first frame in the video frame sequence, perform picture segmentation on the first frame based on a preset semantic segmentation model to determine a region of interest; an initial detection unit configured to, for any current frame in the video frame sequence, perform detection based on a preset target detection model to obtain a plurality of initial detection frames; a screening unit configured to determine one or more initial detection frames located inside the region of interest as detection frames for target detection objects; a target tracking module configured to acquire a prediction frame in the current frame, the prediction frame being generated by a preset target tracking model based on a motion state of a target tracking object in a previous frame or a plurality of frames and a position of a detection frame; 7. An electronic device, comprising: a target labeling module configured to match the detection frame in the current frame with the prediction frame, if the matching is successful, assign a historical identity of the target tracking object corresponding to the prediction frame to a target detection object corresponding to the detection frame, and if the matching fails, assign a new identity to the target detection object corresponding to the detection frame; 8. A computer-readable storage medium, characterized in that, a region of interest resetting module configured to, for any current frame in the video frame sequence, in a case where a deviation index of a detection frame is greater than a first deviation threshold, perform picture segmentation on the current frame based on the preset semantic segmentation model to redetermine a region of interest, and based on the redetermined region of interest, determine detection frames of the target detection objects for the video frames after the current frame; wherein the deviation index is a ratio of initial detection frames located outside the region of interest to a number of all initial detection frames. The device comprises a processor and a memory, the memory stores a plurality of instructions; the processor loads the instructions from the memory to execute the steps in the image target labeling method according to any one of claims 1-5. The computer readable storage medium stores a plurality of instructions, the instructions are suitable for being loaded by a processor to execute the steps in the image target labeling method according to any one of claims 1-5.

Images