Methods for labeling training samples and electronic devices and storage media for the model

Abstract

This application discloses a method for labeling model training samples and an electronic device and storage medium, relating to the field of computer technology. The method includes: obtaining a current reference video frame from consecutive video frames included in the initial model training samples; tracking target objects in consecutive initial video frames following the reference video frame based on the reference bounding boxes, obtaining tracking video frames labeled with tracking bounding boxes; and during the tracking of target objects in the initial video frames, detecting whether the initial video frames meet the labeling calibration conditions; if a reference video frame meeting the labeling calibration conditions is detected, performing target detection on target objects in target video frames following the reference video frame, obtaining detection video frames labeled with detection bounding boxes; and determining the detection video frame as the next current reference video frame, until the target model training samples are obtained.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method for labeling model training samples and an electronic device and storage medium. Background Technology

[0002] In related technologies, before training a model, it is usually necessary to label the model training samples. The specific method is usually to have labelers label the targets in each image to obtain model training samples.

[0003] However, the training samples used to train the model are usually massive, and the efficiency of manual annotation is extremely low. The manual annotation method is far from meeting the requirements of model training. In addition, manual annotation also has problems such as annotation omissions and annotation errors.

[0004] No effective solution has yet been proposed to address the technical issues, such as the low efficiency of labeling training samples in related technologies. Summary of the Invention

[0005] This application provides a method for labeling model training samples, as well as an electronic device and storage medium, to at least solve the technical problem of low labeling efficiency of model training samples in related technologies.

[0006] According to one embodiment of this application, a method for labeling model training samples is provided, including:

[0007] Obtain the current benchmark video frame from the continuous video frames included in the initial model training samples, wherein the benchmark video frame has a benchmark bounding box of the target object, and the confidence of the benchmark bounding box is greater than or equal to a first threshold.

[0008] Based on the reference bounding box, the target object in the consecutive initial video frames following the reference video frame is tracked to obtain the tracking video frame marked with the tracking bounding box. In the process of tracking the target object in the initial video frame, it is detected whether the initial video frame meets the annotation calibration condition, wherein the annotation calibration condition is used to indicate that the confidence level corresponding to the initial video frame is less than or equal to a second threshold.

[0009] When a reference video frame that meets the calibration conditions is detected, target detection is performed on the target object in the target video frame that is after the reference video frame in the consecutive video frames to obtain a detection video frame with a detection bounding box, wherein the confidence of the detection bounding box is greater than or equal to the first threshold.

[0010] The detected video frame is determined as the next current reference video frame until the target model training sample is obtained, wherein the target model training sample includes the consecutive video frames with the bounding boxes of the target object.

[0011] According to another embodiment of the present application, a model training sample annotation device is also provided, comprising:

[0012] The acquisition module is used to acquire the current reference video frame from the continuous video frames included in the initial model training samples, wherein the reference video frame has a reference bounding box of the target object, and the confidence of the reference bounding box is greater than or equal to a first threshold.

[0013] The tracking module is used to track the target object in consecutive initial video frames following the reference video frame based on the reference bounding box, to obtain a tracking video frame with a tracking bounding box. During the tracking of the target object in the initial video frame, the module detects whether the initial video frame meets the annotation calibration condition, wherein the annotation calibration condition is used to indicate that the confidence level corresponding to the initial video frame is less than or equal to a second threshold.

[0014] The target detection module is used to perform target detection on the target object in the target video frame following the reference video frame in the consecutive video frames when a reference video frame that meets the annotation calibration conditions is detected, and to obtain a detection video frame with a detection annotation box, wherein the confidence of the detection annotation box is greater than or equal to the first threshold.

[0015] A determination module is used to determine the detected video frame as the next current reference video frame until the target model training sample is obtained, wherein the target model training sample includes the consecutive video frames labeled with bounding boxes of the target object.

[0016] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the step of labeling training samples of any of the above-described models.

[0017] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the annotation method for training samples of any of the above-described models.

[0018] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the annotation method for training samples of any of the above-described models.

[0019] This application provides a method for labeling model training samples. Starting from a baseline video frame already labeled with a high-confidence bounding box, this method efficiently performs target tracking on subsequent consecutive initial video frames. The key is that during the tracking process, the system continuously detects whether the initial video frames meet the labeling calibration condition, which indicates that the tracking confidence has decreased. When this condition is met, the system initiates a high-precision target detection, ensuring that the obtained detection bounding box also has high confidence. Finally, the video frame that has undergone high-confidence detection is determined as the next current baseline video frame to begin a new round of tracking. This solution addresses the technical problem of low annotation efficiency for model training samples by replacing most frame-by-frame detection with "target tracking," a computationally less demanding operation than target detection. This significantly reduces the overall computational load and improves processing speed. Furthermore, by setting "annotation calibration conditions" and periodically updating the "baseline video frame" using high-confidence "target detection" results, this solution constructs an adaptive feedback calibration loop. This loop effectively suppresses the accumulated errors caused by simple tracking, ensuring the long-term accuracy of the annotation results. Ultimately, it achieves an optimized balance between efficiency and accuracy, thereby improving the annotation efficiency of model training samples and solving the technical problem of low annotation efficiency. Attached Figure Description

[0020] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a hardware structure block diagram of a computer device for a method of labeling model training samples according to an embodiment of this application;

[0022] Figure 2 This is a flowchart of a method for labeling model training samples according to an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of a model training sample annotation process according to an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of a target detection process according to an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of an upper branch process and a lower branch process according to an embodiment of this application;

[0026] Figure 6This is a schematic diagram illustrating the process of feature extraction by a feature extraction module according to an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of a feature fusion module according to an embodiment of this application;

[0028] Figure 8 This is a schematic diagram of the FNN module according to an embodiment of this application;

[0029] Figure 9 This is a schematic diagram of the internal execution steps of an Attention module according to an embodiment of this application;

[0030] Figure 10 This is a schematic diagram of a tracking operation process according to an embodiment of this application;

[0031] Figure 11 This is a structural block diagram of a model training sample annotation device according to an embodiment of this application;

[0032] Figure 12 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0034] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0035] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of a computer device for a model training sample annotation method according to an embodiment of this application. For example... Figure 1As shown, the server device may include one or more ( Figure 1 Only one is shown in the image. A processor 102 (which may include, but is not limited to, a central processing unit (CPU), microprocessor (MCU), or programmable logic device (FPGA), etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0037] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the model training sample annotation method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0038] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0039] This embodiment provides a method for labeling model training samples. Figure 2 This is a flowchart of a method for labeling model training samples according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0040] Step S12: Obtain the current reference video frame from the continuous video frames included in the initial model training samples, wherein the reference video frame has a reference bounding box of the target object, and the confidence of the reference bounding box is greater than or equal to a first threshold.

[0041] Step S14: Based on the reference bounding box, track the target object in the consecutive initial video frames following the reference video frame in the consecutive video frames to obtain the tracking video frame marked with the tracking bounding box. In the process of tracking the target object in the initial video frame, detect whether the initial video frame meets the annotation calibration condition, wherein the annotation calibration condition is used to indicate that the confidence level corresponding to the initial video frame is less than or equal to the second threshold.

[0042] Step S16: When a reference video frame that meets the calibration conditions is detected, target detection is performed on the target object in the target video frame following the reference video frame in the consecutive video frames to obtain a detection video frame with a detection bounding box, wherein the confidence of the detection bounding box is greater than or equal to the first threshold.

[0043] Step S18: Determine the detected video frame as the next current reference video frame until the target model training sample is obtained, wherein the target model training sample includes the consecutive video frames with the bounding boxes of the target object.

[0044] Optionally, in this embodiment, a method for labeling model training samples is provided. This method starts from a reference video frame that has been labeled with a high-confidence reference bounding box and efficiently performs target tracking on subsequent consecutive initial video frames. The key is that during the tracking process, the system detects in real time whether the initial video frame meets the labeling calibration condition, which indicates that the tracking confidence has decreased. When the condition is met, the system initiates a high-precision target detection and ensures that the obtained detection bounding box also has high confidence. Finally, the video frame that has undergone high-confidence detection is determined as the next current reference video frame to start a new round of tracking. This solution addresses the technical problem of low annotation efficiency for model training samples by replacing most frame-by-frame detection with "target tracking," a computationally less demanding operation than target detection. This significantly reduces the overall computational load and improves processing speed. Furthermore, by setting "annotation calibration conditions" and periodically updating the "baseline video frame" using high-confidence "target detection" results, this solution constructs an adaptive feedback calibration loop. This loop effectively suppresses the accumulated errors caused by simple tracking, ensuring the long-term accuracy of the annotation results. Ultimately, it achieves an optimized balance between efficiency and accuracy, thereby improving the annotation efficiency of model training samples and solving the technical problem of low annotation efficiency.

[0045] As an optional approach, detecting whether the initial video frame meets the annotation calibration conditions includes:

[0046] S21, based on the video frame parameters of the initial video frame, detect whether the confidence level corresponding to the initial video frame is less than or equal to the second threshold;

[0047] S22, if the confidence level is detected to be less than or equal to the second threshold, the initial video frame is determined to be the reference video frame that satisfies the annotation calibration conditions.

[0048] Optionally, in this embodiment, the video frame parameters may, but are not limited to, indicate one or more values ​​extracted from a video frame or its processing to quantify the current state, such as the number of frames tracked since the last calibration, the size of the tracking box, the tracking score, etc.

[0049] Optionally, in this embodiment, the confidence level can be, but is not limited to, an indicator: a quantitative metric used to represent the degree of confidence the algorithm has in the accuracy of the current output result (e.g., a tracking bounding box). In this scheme, when certain parameters of a video frame indicate that the tracking quality may deteriorate, the system considers the corresponding confidence level to have decreased.

[0050] Optionally, in this embodiment, the calibration conditions can be, but are not limited to, an indication: a logical condition set within the system, which, when met, indicates that the current tracking results may no longer be reliable and a high-precision target detection operation needs to be initiated for verification and calibration.

[0051] Optionally, in this embodiment, the reference video frame may be, but is not limited to, the first video frame detected to meet the "annotation calibration conditions" during continuous tracking. It serves as a trigger signal, indicating that the system needs to switch from an efficient tracking mode to a precise calibration mode.

[0052] Optionally, in this embodiment, the above scheme can be explained using, but is not limited to, the following scenario: In a video analysis system for urban intelligent traffic management, the system needs to continuously annotate the trajectory of a blue car turning left at an intersection. At frame 100 of the video sequence, the system determines the car's position through high-precision detection and uses it as the baseline video frame. Starting from frame 101, the system enters an efficient tracking mode, continuously outputting the tracking bounding box of the car. At this time, the system internally sets multiple "video frame parameters" for monitoring tracking quality. These parameters are calculated and updated in real time, such as "number of consecutive tracks," "distance between the tracking box and the image edge," "fluctuation rate of the tracking score," and "degree of change in the target appearance model." At frame 125, a bus suddenly appears in the video, partially obscuring the blue car. At this time, although the tracking bounding box output by the tracking algorithm is still on the car, its internally calculated "tracking score" (a form of confidence representation) begins to drop sharply, while the "degree of change in the target appearance model" also increases dramatically. The system monitors abnormal changes in these "video frame parameters" in real time and, based on internally defined logic (e.g., when the tracking score is below 0.6 or the appearance model change is greater than 0.5), determines that the parameters have fallen below a preset second threshold. According to this scheme's logic, the system determines that the "label calibration condition" is met at frame 125 and immediately marks frame 125 as a "reference video frame" requiring calibration. This top-level framework does not concern itself with which specific parameter triggered the calibration; it only defines the core logic of "detecting confidence through parameters, and initiating calibration when the confidence is low," laying the foundation for more specific calibration strategies in subsequent examples. The system will then, based on this signal, initiate a complete object detection after frame 125 (e.g., frame 126) to obtain an absolutely accurate bounding box to correct for tracking deviations that may be caused by occlusion.

[0053] This embodiment achieves the following technical effects: it provides a flexible and scalable top-level framework for the calibration mechanism of the entire annotation method, making calibration no longer fixed but adaptively triggered according to specific circumstances, laying the foundation for the introduction of subsequent specific calibration strategies, and improving the intelligence level of the entire annotation process and its adaptability to complex scenarios.

[0054] As an optional approach, the step of detecting whether the confidence level corresponding to the initial video frame is less than or equal to the second threshold based on the video frame parameters of the initial video frame includes:

[0055] S31, Detect the number of consecutive tracking of the initial video frame, wherein the number of consecutive tracking is the number of the initial video frames that have been continuously tracked after the reference video frame, and the video frame parameters include the number of consecutive tracking;

[0056] S32, if the number of consecutive tracking is greater than or equal to the tracking number threshold, determine that the confidence level corresponding to the detected initial video frame is less than or equal to the second threshold;

[0057] S33, if the number of consecutive tracking is less than the tracking number threshold, determine that the confidence level corresponding to the detected initial video frame is greater than the second threshold.

[0058] Optionally, in this embodiment, the number of consecutive tracks can be, but is not limited to, indicating the number of video frames for which tracking operations have been performed consecutively since the last high-precision target detection and establishment of the reference video frame. It is a counter that starts counting from 0, incrementing by one for each successfully tracked frame.

[0059] Optionally, in this embodiment, the tracking number threshold can be, but is not limited to, an integer preset by the system, such as 20. It defines the maximum number of frames the system is allowed to continuously track between two high-precision detections and is a core parameter of this calibration strategy.

[0060] Optionally, in this embodiment, the above scheme can be explained using, but is not limited to, the following scenario: In an airport tower monitoring video annotation task for air traffic control, the system needs to perform long-term, precise annotation of a Boeing 747 aircraft taxiing on the runway to generate samples for training an air traffic control AI model. At frame 500 of the video, the system successfully locks onto the aircraft through a complete target detection, obtains its precise baseline annotation box, and sets it as the baseline video frame. At this point, the system's internal "continuous tracking count" counter is reset to 0. Internally, to handle targets like aircraft with relatively stable movement and obvious features, the system sets a "tracking count threshold" of 25. Starting from frame 501, the system activates a high-speed target tracking module. For frame 501, the system successfully tracks the aircraft, and the counter becomes 1; for frame 502, successful tracking occurs, and the counter becomes 2; this process continues. Because the background (runway, grass) is relatively simple when the aircraft is taxiing on the runway, and the aircraft's appearance remains largely unchanged, the tracking annotation boxes output by the tracking algorithm remain very stable and accurate. Nevertheless, when processing frame 525, the system detected that the "number of consecutive tracks" had reached 25, equal to the preset "tracking number threshold." According to the logic of this scheme, the system determined that the confidence level at this point was already below the second threshold, satisfying the annotation calibration conditions. Therefore, the system paused tracking and immediately initiated a complete high-precision target detection operation on frame 525. The result of this detection might be almost identical to the prediction result of the tracking algorithm in frame 525, but the significance of this operation lies in its use of an absolutely precise "anchor point" to correct for any subtle, imperceptible positioning deviations that might have accumulated over the past 25 frames. After confirming the detection result of frame 525, the system established it as the new baseline video frame and reset the "number of consecutive tracks" counter to 0. This process of "tracking 25 frames, detecting 1 frame" continuously looped throughout the aircraft's taxiing process, ensuring that even after thousands of consecutive tracking frames, the bounding boxes would not exhibit significant drift, guaranteeing the high quality of the final training samples.

[0061] This embodiment achieves the following technical effects: by setting a fixed "check-up" cycle, the tracking results are actively and periodically corrected with high precision, which can effectively avoid the problem of the final target being lost or seriously deviated due to the accumulation of tracking errors over a long period of time, thereby ensuring the annotation accuracy of the method when processing long-term videos.

[0062] As an optional approach, the step of detecting whether the confidence level corresponding to the initial video frame is less than or equal to the second threshold based on the video frame parameters of the initial video frame includes:

[0063] S41, Detect the annotation deviation parameter of the initial video frame, wherein the annotation deviation parameter is used to indicate the degree of positional deviation between the tracking annotation box marked in the initial video frame and the target object, and the larger the annotation deviation parameter is, the higher the degree of positional deviation;

[0064] S42, if the labeled deviation parameter is greater than or equal to the deviation parameter threshold, determine that the confidence level corresponding to the detected initial video frame is less than or equal to the second threshold;

[0065] S43, if the labeled deviation parameter is less than the deviation parameter threshold, determine that the confidence level corresponding to the detected initial video frame is greater than the second threshold.

[0066] Optionally, in this embodiment, the deviation parameter can be, but is not limited to, an indicator: a quantitative metric used to measure the degree of difference between the tracking bounding boxes predicted by the tracking algorithm and the detection bounding boxes independently calculated by the object detection algorithm on the same frame image. A commonly used calculation method is the Intersection over Union (IoU), and the deviation parameter can be defined as (1-IoU). The smaller the IoU value, the lower the overlap between the two boxes, and the larger the deviation parameter.

[0067] Optionally, in this embodiment, the above scheme can be explained using, but is not limited to, the following scenario: Continuing the above embodiment of airport monitoring, during the annotation of a Boeing 747 aircraft. In frame 525, the system initiates target detection because it has reached the "tracking quantity threshold" of 25 frames. At this time, the system has two bounding boxes for the aircraft in frame 525: one is the "tracking bounding box" predicted by the tracking algorithm from frame 524, and the other is the "detection bounding box" independently analyzed by the target detection algorithm in frame 525. Ideally, these two boxes should highly overlap. The system then calculates the "annotation deviation parameter" for these two boxes. Assuming that due to heat wave disturbance on the runway, the tracking algorithm's prediction has a slight cumulative error, causing its predicted "tracking bounding box" to be shifted upwards by 10 pixels compared to the "detection bounding box". The system calculates the IoU between the two boxes to be 0.92, therefore the "annotation deviation parameter" is (1-0.92=0.08). The "deviation parameter threshold" set internally by the system is 0.5. Since 0.08 is much smaller than 0.5, the system considers this deviation to be within an acceptable range and a normal cumulative error. Therefore, it updates the baseline with the detection box and continues tracking. However, in another scenario, suppose that during the tracking process from frame 600 to frame 625, a ground support vehicle quickly passes in front of the aircraft, severely interfering with the tracking algorithm. This causes the tracking algorithm to experience "target drift" in frame 625, where its predicted "tracking bounding box" incorrectly selects the ground support vehicle. In the periodic detection of frame 625, the detection algorithm still accurately locates the original aircraft. At this point, the system calculates the "label deviation parameter" between the "tracking bounding box" (on the ground support vehicle) and the "detection bounding box" (on the aircraft). Since the two boxes have almost no overlap, their IoU is close to 0, and the calculated deviation parameter is 0.99, much larger than the threshold of 0.5. Based on this, the system determined that the tracking had failed severely and the corresponding confidence level was extremely low. It immediately triggered a deeper calibration mechanism, such as completely resetting the tracker model and using the accurate detection result of frame 625 as a new and reliable starting point to begin the next round of tracking.

[0068] This embodiment achieves the following technical effects: it provides a "circuit breaker" mechanism or "fail-safe" guarantee for the system. Once a significant deviation between the tracking result and the actual detection result is detected, it is determined that the tracking has suffered a catastrophic failure and is immediately reset, thereby preventing the continued propagation of seriously erroneous annotation results and greatly enhancing the reliability and robustness of the method in the face of sudden interference and complex scenarios.

[0069] As an optional approach, the step of tracking the target object in consecutive initial video frames following the reference video frame based on the reference bounding box, to obtain tracking video frames marked with tracking bounding boxes, includes:

[0070] S51, extract the appearance features of the target object in the reference video frame according to the reference annotation box, wherein the appearance features are used to characterize the appearance information of the target object;

[0071] S52, construct an object filter for identifying the target object based on the appearance features;

[0072] S53, within the preset search area of ​​the initial video frame, the tracking bounding box of the target object is identified by the object filter, and a tracking video frame marked with the tracking bounding box is obtained.

[0073] Optionally, in this embodiment, appearance features may be, but are not limited to, indications: a digital and vectorized description of the visual characteristics of a target object, which includes information such as the target's outline, texture, and color, used to uniquely represent the target in a computer.

[0074] Optionally, in this embodiment, the object filter may, but is not limited to, indicating that, in a correlation filtering tracking algorithm, it is a mathematical model (typically a matrix represented in the frequency domain) trained based on the appearance features of the target. It is designed to produce a very strong response signal (peak) when correlated with an image region containing the target.

[0075] Optionally, in this embodiment, the preset search area may, but is not limited to, indicating a local range within the current video frame to be processed where the target object is most likely to appear. Subsequent matching calculations will be limited to this area to improve efficiency.

[0076] Optionally, in this embodiment, the above solution can be explained using, but is not limited to, the following scenario: In a video annotation task for warehouse logistics management, the system needs to continuously annotate a specific model of blue forklift moving between shelves in a video. At frame 80, the system locks the forklift through object detection and uses it as the reference video frame. Step 1: Extract appearance features. The system selects the image region of the blue forklift based on the reference annotation box. Then, the system runs a feature extraction algorithm to calculate and encode information such as its unique blue body color distribution, the edge contours formed by the wheels and forks, and the marking textures on the body, ultimately converting it into a high-dimensional feature vector composed of a large number of values, which is the "appearance feature" of the forklift. Step 2: Construct an object filter. The system uses this appearance feature vector as a positive sample and uses the surrounding shelves, ground, and other background areas as negative samples, and trains a "filter" specifically for the "blue forklift" through a series of complex mathematical operations (including Fourier transform, etc.). Its mathematical expression can be simplified as follows:

[0077]

[0078]

[0079] Where Z: Multi-channel feature matrix extracted from the target bounding box (including HOG, RGB, etc.). M: Foreground mask matrix, with values ​​between 0 and 1, used to highlight the target region and suppress the background. G: Ideal Gaussian response map. A Gaussian function with a peak value at the target center and dimensions of H×W. The coordinates are the position. and The center position of a dimension H×W The standard deviation of the Gaussian distribution. f(): denotes the Fourier transform of the matrix within the parentheses. ¯(): denotes taking the complex conjugate. • denotes element-wise multiplication of matrices. Regularization parameter (a very small positive number) prevents the denominator from being zero and avoids overfitting. The final constructed filter (represented in the frequency domain) can be viewed as an optimal matching template that has "learned" how to distinguish between the forklift and the background. The third step is to identify the tracking bounding box. When processing frame 81, the system first determines a "preset search region" centered on the forklift's position in frame 80. Then, the system performs a correlation operation between this "object filter" and the image features within the search region, mathematically expressed as R = inv_f(f(z_new)·H), where z_new is the feature of the new region and inv_f is the inverse Fourier transform. The result of this operation is a response map R. Most areas in the map (such as shelves and the ground) have low response values, but a very sharp response peak appears at the new location where the forklift actually appears. The system uses an algorithm to locate this peak and generates a new "tracking bounding box" centered on it, thus efficiently tracking the forklift in frame 81. This process of "feature extraction - filter construction - matching recognition" is repeated continuously in subsequent initial video frames, achieving rapid and continuous annotation of the target.

[0080] This embodiment achieves the following technical effects: it provides a highly efficient, real-time target tracking framework. Correlation filtering is a classic tracking algorithm that achieves a good balance between speed and accuracy. It transforms complex convolution operations into efficient element-wise multiplication through frequency domain operations, greatly improving the matching speed. It is a key technical support for achieving the core objective of "high-efficiency" labeling in the entire scheme.

[0081] As an optional approach, extracting the appearance features of the target object from the reference video frame based on the reference bounding box includes:

[0082] S61, extract the shape features of the target object and extract the color features of the target object, wherein the shape features are features used to characterize the contour and texture information of the target object, and the color features are features used to characterize the color information of the target object;

[0083] S62, the shape feature and the color feature are fused to obtain the appearance feature.

[0084] Optionally, in this embodiment, the shape feature may include, but is not limited to, an indicator: a feature that primarily describes the local edges and orientation information of the image, such as a histogram of gradient orientations (HOG). It is insensitive to changes in illumination and can stably characterize the contours and structure of the target.

[0085] Optionally, in this embodiment, the color feature can be, but is not limited to, an indicator: a feature that directly describes the color distribution of an image, such as raw RGB pixel values ​​or a color histogram. It is very effective for identifying targets with specific colors.

[0086] Optionally, in this embodiment, the above scheme can be explained using, but is not limited to, the following scenario: continuing the task of labeling the blue forklift in the above warehousing and logistics scenario. When the system extracts the appearance features of the forklift within the reference labeling box of frame 80, it performs two operations in parallel. Operation 1: Extracting shape features. The system first converts the color image of the forklift area into a grayscale image to eliminate color interference. Then, it calculates the gradient magnitude and direction of each pixel in the image, mathematically expressed as Gx(x,y)=I(x+1,y)-I(x-1,y) and Gy(x,y)=I(x,y+1)-I(x,y-1), gradient magnitude M(x,y)=sqrt(Gx^2+Gy^2), and gradient direction theta(x,y)=arctan2(Gy,Gx). Next, the gradient direction is quantized into multiple bins and statistically analyzed within multiple small image blocks to form a gradient direction histogram (HOG). The final generated HOG feature vector v accurately describes the forklift's square body outline, rounded tire edges, and straight fork structure; this is the "shape feature." Operation Two: Extracting Color Features. Simultaneously, the system directly extracts the RGB pixel values ​​I_rgb from the original color image of the region and normalizes these values ​​(e.g., dividing by 255). This normalized RGB pixel matrix constitutes the forklift's "color feature," clearly recording information such as the body being a specific blue and the warning lights being yellow. Final Step: Fusion. The system concatenates the HOG feature vector v and the normalized RGB pixel matrix I_rgb / 255.0 in a dimensional manner, mathematically expressed as z=concat(v,I_rgb / 255.0), forming a longer, more informative fused feature vector z. This fused vector is the "appearance feature" ultimately used to construct the object filter. In subsequent tracking, assuming the forklift traveled to a dimly lit corner of the warehouse, the blue of its body became less vibrant in the image (color features were affected), but its overall square outline and fork structure remained clear (shape features remained stable). Because the appearance features included stable shape features, the tracker was still able to successfully lock onto the forklift, demonstrating the robustness brought by multi-feature fusion.

[0087] This embodiment achieves the following technical effects: by fusing the complementary features of shape and color, a more comprehensive and reliable target model is constructed, thereby significantly improving the stability and robustness of the tracking algorithm in complex scenarios such as changes in lighting, partial occlusion, and interference from similar colored objects, and reducing the probability of tracking failure.

[0088] As an optional approach, before identifying the tracking bounding box of the target object within a preset search area of ​​the initial video frame using the object filter, the method further includes:

[0089] S71, obtain the position information and rotation angle information of the reference annotation box, wherein the position information is used to indicate the position of the reference annotation box in the reference video frame, and the rotation angle information is used to indicate the rotation posture of the reference annotation box in the reference video frame;

[0090] S72, determine the preset search area based on the position information and the rotation angle information.

[0091] Optionally, in this embodiment, the location information may include, but is not limited to, indicating: the coordinates of the center point of the target's bounding box in the previous frame, which provides a reference position for searching in the new frame.

[0092] Optionally, in this embodiment, the rotation angle information may, but is not limited to, indicate the tilt angle of the target relative to the vertical or horizontal direction of the image in the previous frame, which describes the rotational posture of the target.

[0093] Optionally, in this embodiment, the above scheme can be explained using, but is not limited to, the following scenario: In a parking lot monitoring video annotation task for the development of an assisted driving system, the system is tracking a silver sedan performing a reverse parking maneuver in a "non-standard" shape. In frame 300, the car body is at a 45-degree angle to the vertical direction of the video frame. The system obtains the reference bounding box of the car through target detection and records its center point coordinates as (500, 600), size as (120, 60), and a key "rotation angle information": 45 degrees. When processing frame 301, the system needs to first determine a "preset search area" for efficient tracking. Step 1: Obtain position and rotation information. The system reads that the position of the previous frame is (500, 600), and the rotation angle is 45 degrees. Step 2: Determine the search area. Without rotation information, the system would only be able to draw a large, square rectangle (e.g., 200x200) around (500, 600) as the search area to ensure coverage of the tilted vehicle. However, this solution does the following: First, it creates a basic search box slightly larger than (120, 60), for example, (150, 75), at the origin. Then, it rotates this (150, 75) rectangle by 45 degrees, transforming it into a tilted parallelogram. Finally, it translates the center of this tilted search area to (500, 600). The resulting tilted area, tightly encompassing the vehicle's potential movement and rotation, is the "preset search area." Because this area is much smaller than a traditional square search box and excludes a large number of irrelevant background pixels (such as adjacent parking lines and the ground), subsequent object filter matching calculations become faster and less susceptible to interference, resulting in more efficient and robust tracking of the rotating car.

[0094] This embodiment achieves the following technical effects: by constructing a more "close-fitting" search area, the number of background pixels that need to be processed is reduced, the amount of computation is lowered, and efficiency is improved; at the same time, less background interference also makes target localization more accurate, especially when the target rotates, which can significantly improve the tracking success rate and the fit of the bounding box.

[0095] As an optional approach, the step of performing target detection on the target object in the target video frame following the reference video frame in the consecutive video frames to obtain a detection video frame marked with a detection bounding box includes:

[0096] S81, extract the features of the target video frame layer by layer according to different spatial resolutions to obtain multiple layer feature maps. Among them, different layer feature maps have feature data of the target video frame at different spatial resolutions. The layer feature map with higher spatial resolution includes more local features of the target object, and the layer feature map with lower spatial resolution includes more overall features of the target object.

[0097] S82, the different hierarchical feature maps of the multiple hierarchical feature maps are fused to obtain multiple fused feature maps, wherein the fused feature maps simultaneously include the local features and the overall features of the target object;

[0098] S83, output the candidate bounding boxes in the target video frame according to the multiple fused feature maps, and obtain the candidate video frames marked with the candidate bounding boxes;

[0099] S84, Detect the confidence level of the candidate label boxes;

[0100] S85, if the confidence level of the candidate annotation box is greater than or equal to the first threshold, the candidate annotation box is determined as the detection annotation box, and the candidate video frame is determined as the detection video frame.

[0101] Optionally, in this embodiment, the hierarchical feature map can, but is not limited to, indicate the feature data output by the deep neural network at different layers (depths). Typically, shallow feature maps have high spatial resolution and contain rich local features such as edges and textures; deep feature maps have low spatial resolution but contain higher-level, overall features about what the object is.

[0102] Optionally, in this embodiment, the fused feature map can be, but is not limited to, a new feature map generated by combining information from feature maps of different levels through upsampling, addition, or concatenation. It is designed to have both the semantic understanding capability of deep maps and the precise localization capability of shallow maps.

[0103] Optionally, in this embodiment, the alternative label box may indicate, but is not limited to, candidate detection results initially output by the detection model that have not yet been verified with final confidence.

[0104] Optionally, in this embodiment, the above scheme can be explained using, but is not limited to, the following scenario: In a complex urban street scene video frame, the system needs to simultaneously annotate a double-decker bus occupying nearly half the frame in the foreground and a cyclist in the background that is only a few dozen pixels in size. Step 1: Layer-by-layer feature extraction. The input 640x640 pixel image is processed by a deep convolutional network. In the shallow layers of the network, an 80x80 high-resolution "hierarchical feature map" is generated. Due to its high resolution, this map clearly preserves the minute contour details of the cyclist in the distance (local features). In the deep layers of the network, a 20x20 low-resolution "hierarchical feature map" is generated. Although the details are blurred, its feature vectors strongly express the concept that "this is a large vehicle" (overall features). Step 2: Feature fusion. The system upsamples the 20x20 deep layer map (a kind of magnification operation) and merges it with information from a higher-resolution hierarchical map (e.g., a 40x40 image). Through this top-down and bottom-up multi-path information integration, the system constructs a series of "fusion feature maps." One fusion map is specifically designed for detecting large targets, knowing both "this is a car" and its precise outline; another is specifically designed for detecting small targets, knowing both "this is a human-shaped object" and the precise pixel boundaries of this tiny human figure. The third step: Outputting candidate bounding boxes. The system runs multiple detection heads on these fusion feature maps, initially outputting a large number of "candidate bounding boxes." For example, for a double-decker bus, a candidate bounding box with a confidence score of 0.99 is output; for a cyclist, a candidate bounding box with a confidence score of 0.95 is output; and it may also output a candidate bounding box with a confidence score of only 0.45 for a roadside mailbox. The fourth and fifth steps: Confidence detection and determination. The system sets the first threshold to 0.9. The candidate bounding boxes for the double-decker bus and the cyclist both have confidence scores greater than 0.9, therefore they are determined as the final "detection bounding boxes," and the video frames they belong to are also determined as "detection video frames." The mailbox candidate box was automatically discarded by the system because its confidence level was below 0.9 and it will not appear in the final annotation results. Through this complete process, the system successfully annotated two targets with significantly different sizes simultaneously in a complex image.

[0105] This embodiment achieves the following technical effects: enabling the detection network to simultaneously "see" both the macroscopic concept and the microscopic details of an image, thus allowing it to accurately identify large targets nearby and clearly locate small targets in the distance, much like the human eye. This greatly improves the applicability and overall accuracy of target detection and ensures the reliability of the calibration phase.

[0106] As an optional approach, the feature extraction of the target video frame is performed layer by layer according to different spatial resolutions to obtain multiple layer feature maps, including:

[0107] S91, while reducing the spatial resolution of the target video frame, extract features used to characterize the target object to obtain initial sampling features;

[0108] S92, the initial sampling features are split into multiple branch features, at least one of the multiple branch features is enhanced to obtain an enhanced feature, and the enhanced feature is merged with the remaining branch features to obtain the hierarchical feature map.

[0109] Optionally, in this embodiment, the initial sampling features may, but are not limited to, indicate: input features obtained by downsampling from the previous stage or the original image at the beginning of a feature extraction stage.

[0110] Optionally, in this embodiment, the branching feature may, but is not limited to, indicate: the initial sampled feature is divided into multiple parts along the channel dimension (which can be understood as the “thickness” of the feature).

[0111] Optionally, in this embodiment, the above scheme can be explained using, but is not limited to, the following scenario: In the street scene detection network described above, when processing to a certain intermediate level, the input "initial sampling feature" is a feature map with dimensions [256, 80, 80]. According to this scheme, the system will perform the following operations to generate the final output of this level. Step 1: Splitting. The system splits this [256, 80, 80] feature map in two along the channel dimension (256 channels), obtaining two "branch features" with dimensions [128, 80, 80], which we call branch A and branch B. Step 2: Enhancement. The system chooses to perform deep processing on branch A, feeding it into a complex network structure composed of a self-attention module and a feedforward network for processing. This processing will extract higher-level and more discriminative information from branch A, ultimately outputting an "enhanced feature" with the same dimensions [128, 80, 80]. In this process, branch B is completely preserved without any processing, like an information "highway" or "shortcut". Step 3: Merging. The system reassembles the processed "enhanced features" (from branch A) and the preserved original "branch features" B along the channel dimension, forming a new "hierarchical feature map" with dimensions [256, 80, 80] as the final output of this level. In this way, the output feature map contains both the high-level semantic information extracted through deep processing (from the enhanced features) and fully retains some of the original detailed information from the previous stage (from the preserved branch B), thus making the feature representation ability of the entire network stronger and the gradient propagation smoother.

[0112] This embodiment achieves the following technical effects: it solves the common gradient vanishing and information bottleneck problems in deep neural networks. Through the "feature reuse" mechanism, it leverages the powerful feature extraction capabilities of deep networks while preserving valuable original details, resulting in higher quality feature maps and thus improving detection accuracy and network training stability.

[0113] As an optional approach, the step of outputting the candidate bounding boxes in the target video frame based on multiple fused feature maps to obtain the candidate video frames labeled with the candidate bounding boxes includes:

[0114] S101, the coordinate information of the target object is detected from the fused feature map through the first convolutional network, and the type information of the target object is identified from the fused feature map through the second convolutional network, wherein the coordinate information is used to indicate the position and size of the target object in the target video frame, and the type information is used to indicate the type to which the target object belongs;

[0115] S102, Generate the candidate annotation box based on the coordinate information and the type information.

[0116] Optionally, in this embodiment, the first convolutional network may, but is not limited to, be a network specifically designed to regress the precise coordinates of the target, typically employing standard convolutions. This network structure can preserve spatial information to the maximum extent and ensure positioning accuracy.

[0117] Optionally, in this embodiment, the second convolutional network may be, but is not limited to, a network specifically designed to identify target categories. It typically employs lightweight designs such as depthwise separable convolutions because its spatial accuracy requirements are not as high as those for localization tasks. Using a lightweight network can significantly reduce computational load while ensuring classification accuracy.

[0118] Optionally, in this embodiment, the above scheme can be explained using, but is not limited to, the following scenario: In the street scene detection described above, when the system needs to output candidate bounding boxes about a distant cyclist from a fused feature map of [256, 80, 80], it will simultaneously feed this fused feature map into two parallel processing flows. Flow 1: Generate coordinate information through a first convolutional network. The fused feature map is first fed into a network composed of multiple "standard convolutional" layers. This network structure is complex and computationally intensive, but its advantage is that it can preserve the precise spatial location information in the feature map to the maximum extent and is very sensitive to pixel-level deviations. After processing by this network, a set of values ​​is finally output, such as (510.3, 320.8, 14.9, 36.2). After decoding, these values ​​correspond to the precise "coordinate information" (center point x, y, width, height) of the cyclist. Flow 2: Generate type information through a second convolutional network. At the same time, the fused feature map is also fed into a network composed of multiple "depth-separable convolutional" layers. This type of network requires far less computation than standard convolutional networks, resulting in faster processing speeds. Its task is not precise localization, but accurate content recognition. After processing by this network, a probability distribution vector is output, for example, {pedestrian: 0.85, bicycle: 0.95, background: 0.01}. The system selects the "bicycle" (or combined as "cyclist") with the highest probability as the "type information." The final step is to generate candidate bounding boxes. The system combines the coordinate information from step one with the type information from step two, and incorporates their confidence levels to generate a complete "candidate bounding box": {type: cyclist, coordinates: (510.3, 320.8, 14.9, 36.2), confidence level: 0.95}. Through this "divide and conquer" design, matching the most suitable tool to the two different tasks of "localization" and "classification," the system optimizes overall computational efficiency while ensuring localization accuracy.

[0119] This embodiment achieves the following technical effects: it "tailor-makes" the most suitable tools for different sub-tasks, enabling the most accurate results for positioning tasks with extremely high precision requirements, while allowing classification tasks that are sensitive to computational efficiency to be completed at a faster speed. This "divide and conquer" design improves the performance of critical tasks without significantly increasing the total computational cost, achieving a better balance between annotation accuracy and efficiency.

[0120] As an optional approach, obtaining the current benchmark video frame from the continuous video frames included in the initial model training samples includes:

[0121] S111, if the reference video frame is not present in the continuous video frames included in the initial model training samples, target detection is performed on the continuous video frames frame by frame according to the time order of the continuous video frames to obtain candidate video frames with candidate bounding boxes labeled with the target object;

[0122] S112, Detect the confidence level of the candidate bounding boxes;

[0123] S113, if the confidence level of the candidate annotation box is greater than or equal to the first threshold, determine the reference video frame according to the annotation box size of the candidate annotation box;

[0124] S114, if the confidence level of the candidate bounding box is less than the first threshold, continue to perform the target detection on the next video frame of the candidate video frame.

[0125] Optionally, in this embodiment, candidate video frames / candidate bounding boxes may indicate, but are not limited to, preliminary detection results obtained by the system through frame-by-frame detection during the initialization phase, which have not yet undergone any verification.

[0126] Optionally, in this embodiment, the above scheme can be explained using, but is not limited to, the following scenario: An automated annotation system begins processing a brand-new nighttime surveillance video of a city street, at which point there are no "reference video frames" in the system. Step 1: Frame-by-frame detection. The system performs a complete target detection for each frame, starting from the first frame of the video. Assume that from frames 1 to 10, due to the dim lighting, the system does not detect any targets with sufficiently high confidence. In frame 11, a car with its headlights on enters the frame, and the system detects it, outputting a "candidate bounding box." However, because the vehicle is partially in shadow, its confidence is only 0.7, lower than the system's first threshold of 0.9. According to the logic of this scheme, the system determines this is an unreliable detection, discards it, and continues target detection for the next video frame (frame 12). Step 2: Confidence detection. In frame 20, the car has completely entered the streetlights and becomes very clear. The system detects it again, and this time the output "candidate bounding box" has a confidence of 0.98, far exceeding the first threshold of 0.9. This result passed the first round of screening, and the system temporarily marked frame 20 as a "candidate video frame." The third step: size determination. The system then checks the size of this candidate bounding box with a confidence level of 0.98 and uses this size to ultimately determine whether to adopt it as the baseline video frame. This dual-screening mechanism ensures that the system only ends its costly frame-by-frame search process and initiates the subsequent efficient tracking process when a target that is both clearly identifiable (high confidence) and sufficiently stable (appropriate size) appears in the video. If the size of the high-confidence target remains unacceptable, the system will remain in frame-by-frame detection mode until it finds the perfect starting point.

[0127] This embodiment achieves the following technical effects: it establishes a strict, dual quality control checkpoint at the starting point of the entire annotation process. It effectively filters out low-quality, unreliable detection results and further selects suitable targets for stable tracking, ensuring that only the most "qualified" targets are chosen as the starting point. This significantly improves the robustness and final success rate of the entire automated annotation process.

[0128] As an optional approach, determining the reference video frame based on the size of the candidate annotation box includes:

[0129] S121, compare the size of the candidate annotation box with a preset size threshold;

[0130] S122, if the size of the annotation box is smaller than the preset size threshold, continue to perform the target detection on the next video frame of the candidate video frame;

[0131] S123, if the size of the annotation box is greater than or equal to the preset size threshold, the candidate video frame corresponding to the candidate annotation box is determined as the reference video frame, and the candidate annotation box is determined as the reference annotation box.

[0132] Optionally, in this embodiment, the preset size threshold can be, but is not limited to, indicating: a predefined value (e.g., a minimum pixel area of ​​30x30, or a minimum side length of 30) to determine whether a detected target has sufficient size and sharpness for stable tracking. This threshold setting is based on a common-sense technical principle: targets that are too small have sparse and highly unstable features, making tracking algorithms prone to failure.

[0133] Optionally, in this embodiment, the above scheme can be explained using, but is not limited to, the following scenario: Continuing with the example of nighttime surveillance video, when the system detects a car with a confidence level of 0.98 in frame 20 and enters the size judgment process, it performs the following comparison. Step 1: Compare size with threshold. The system compares the size of the candidate bounding box (assumed to be 200x80 pixels) with a preset size threshold (assumed to be 30x30 pixels) within the system. Step 2: Execute judgment logic. The system determines that 200x80 is much larger than 30x30, i.e., "the bounding box size is greater than or equal to the preset size threshold." According to the logic of this scheme, the system performs the operation of "determining the candidate video frame corresponding to the candidate bounding box as the reference video frame, and determining the candidate bounding box as the reference bounding box." Thus, frame 20 is officially established as the reference, and its corresponding bounding box is also established as the reference bounding box. The entire initialization process is successfully completed, and the system will enter efficient tracking mode starting from frame 21. In another example, suppose in a drone aerial video, the system detects a pedestrian on the ground in frame 50 with a confidence level of 0.95, passing the confidence screening. However, due to the high shooting altitude, the bounding box size corresponding to the pedestrian in the image is only 15x25 pixels. The system compares this to the same 30x30 pixel size threshold and determines that its "bounding box size is smaller than the preset size threshold." According to the logic of this scheme, the system performs the operation of "continuing to perform the target detection on the next video frame of the candidate video frame," that is, abandoning frame 50 as the tracking starting point and continuing to detect frame by frame in frames 51 and 52 until the pedestrian gets closer or another larger target appears, and its bounding box size meets the requirements.

[0134] This embodiment achieves the following technical effects: After the "validity" screening, a "feasibility" screening is added. This eliminates targets that, while credible, are too small to be stably tracked, preventing the tracking process from failing at the outset due to excessively high target risk. This is a proactive risk mitigation strategy that significantly improves the robustness and success rate of the entire automated annotation process.

[0135] Optionally, in this embodiment, in order to better understand the process of labeling the above model training samples, the labeling process of the above model training samples will be described below in conjunction with optional embodiments, but it is not intended to limit the technical solution of the embodiments of this application.

[0136] This embodiment provides a method for labeling model training samples. Figure 3 This is a schematic diagram of a model training sample annotation process according to an embodiment of this application, such as... Figure 3 As shown, the main steps include the following:

[0137] STEP 3-1: Acquire video data captured by the camera;

[0138] STEP 3-2: Divide the video data into images according to time sequence;

[0139] Optionally, during image segmentation, the video frame order is not disrupted, and images are acquired sequentially according to the set image acquisition interval. For example, for a 30FPS video, if the acquisition frequency is set to 3 times per second, an image is acquired every 10 frames and then sorted sequentially.

[0140] STEP 3-3: Preprocess the segmented images, such as scaling the image size to a specific dimension;

[0141] Optionally, this specific size can be set to 640. 640 pixels. To avoid image distortion during scaling, the original aspect ratio of the image can be maintained by scaling the longest side to 640 pixels, while the shorter sides are filled with grayscale.

[0142] STEP 3-4: Input the preprocessed image into the object detection network;

[0143] STEP 3-5: Determine whether the target to be detected exists in the image using the object detection network. If not, return to STEP 3-1; if it exists, continue execution.

[0144] STEP 3-6: If a target exists, obtain its bounding box coordinates, angle in the image, and confidence level;

[0145] Optionally, the angle of the target in the image can be the target rotation angle measured with the top left corner of the image as the origin.

[0146] STEP 3-7: Determine whether the confidence level of the acquired target box is greater than the set threshold. If it is greater, continue execution; if it is less, return to STEP 3-1.

[0147] STEP 3-8: For target bounding boxes that meet the confidence level, further determine whether their size is greater than the set size threshold;

[0148] Furthermore, if the target bounding box size is larger than this size threshold (i.e., coarse-grained annotation is used), then proceed to the next step to enter the target tracking process;

[0149] If the target bounding box size is not larger than the size threshold (i.e., fine-grained annotation is used), the target bounding box position and category are directly output, and the target detection process continues for the next frame (i.e., return to STEP 3-4).

[0150] STEP 3-9: If the target tracking process is entered, obtain the confirmed target bounding box and angle of the current frame, and prepare to process the next image to be labeled;

[0151] STEP 3-10: Enter the target tracking process and process subsequent images to obtain new target bounding boxes and angle values;

[0152] STEP 3-11: After tracking N more images, input the image back into the target detection network for calibration;

[0153] Optionally, depending on the image category being labeled, N can be set to a value between 10 and 30. Periodic detection by the object detection model improves both the accuracy of the detection and the efficiency of the labeling process.

[0154] STEP 3-12: Determine whether the deviation between the target bounding box obtained by the tracking process and the target bounding box obtained by the detection network is greater than a set threshold. If yes, return to STEP 3-4 to restart the detection. If no, proceed to the next step.

[0155] STEP 3-13: Continuously track the target;

[0156] STEP 3-14: Output the target bounding box location and target category information;

[0157] STEP 3-15: Program ends.

[0158] To better understand the object detection mentioned in this application, the object detection process is described below. Figure 4This is a schematic diagram of a target detection process according to an embodiment of this application, such as... Figure 4 As shown, the "target detection" operation described in this application can be mainly divided into three core parts: feature extraction, feature fusion, and detection head output.

[0159] I. Multi-level Feature Extraction (Backbone):

[0160] This section corresponds to the process of extracting features from the target video frame layer by layer according to different spatial resolutions, resulting in multiple layers of feature maps. Its purpose is to extract different levels of information from the original image, ranging from those containing local detail features (high spatial resolution) to those containing overall conceptual features (low spatial resolution).

[0161] STEP 4-1: Obtain the preprocessed image and get the feature values ​​[1,3,640,640].

[0162] STEP 4-2: Input the above feature values ​​into the Conv network to obtain feature values ​​of [1,64,320,320].

[0163] STEP 4-3: Input the above feature values ​​into the Conv network to obtain feature values ​​of [1,128,160,160].

[0164] STEP 4-4: Input the above feature values ​​into the feature extraction module to obtain the feature values ​​[1,128,160,160].

[0165] STEP 4-5: Input the above feature values ​​into the Conv network to obtain feature values ​​of [1,256,80,80].

[0166] STEP 4-6: Input the above feature values ​​into the feature extraction module to obtain the hierarchical feature map F1 of [1,256,80,80].

[0167] STEP 4-7: Input the above feature values ​​into the Conv network to obtain the feature values ​​[1,512,40,40].

[0168] STEP 4-8: Input the above feature values ​​into the feature extraction module to obtain the hierarchical feature map F2 of [1,512,40,40].

[0169] STEP 4-9: Input the above feature values ​​into the Conv network to obtain the feature values ​​[1,512,20,20].

[0170] STEP 4-10: Input the above feature values ​​into the feature extraction module to obtain the hierarchical feature map F3 of [1,512,20,20].

[0171] II. Cross-level feature fusion and multi-scale detection (Neck & Head):

[0172] This part corresponds to the process of fusing different hierarchical feature maps from multiple hierarchical feature maps to obtain multiple fused feature maps, and outputting candidate bounding boxes in the target video frame based on the multiple fused feature maps. Its core lies in combining deep overall conceptual features with shallow local detail features to generate fused features that contain both rich semantics and precise location information, and performing detection on fused features at different scales.

[0173] 1. Generation and detection of high-resolution fusion features:

[0174] STEP 4-11: Upsample the hierarchical feature map F3 obtained in STEP 4-10 to obtain feature values ​​of [1,512,40,40].

[0175] STEP 4-12: Combine the above feature values ​​with the hierarchical feature map F2 obtained from STEP 4-8 to obtain the feature values ​​[1,1024,40,40].

[0176] STEP 4-13: Input the above feature values ​​into the feature extraction module to obtain the intermediate fusion feature F4 of [1,512,40,40].

[0177] STEP 4-14: Upsample the above intermediate fusion feature F4 to obtain feature values ​​of [1,512,80,80].

[0178] STEP 4-15: Combine the above feature values ​​with the hierarchical feature map F1 obtained in STEP 4-6 to obtain the feature values ​​[1,768,80,80].

[0179] STEP 4-16: Input the above feature values ​​into the feature extraction module to obtain a high-resolution fused feature map of [1,256,80,80].

[0180] STEP 4-17: Input the high-resolution fused feature map into Detect1 for processing.

[0181] 2. Generation and detection of medium-resolution fused features:

[0182] STEP 4-18: Input the output features of STEP 4-16 into the Conv network to obtain feature values ​​of [1,256,40,40].

[0183] STEP 4-19: Combine the above eigenvalues ​​with the intermediate fusion feature F4 from STEP 4-13 to obtain the eigenvalues ​​[1,768,40,40].

[0184] STEP 4-20: Input the above feature values ​​into the feature extraction module to obtain a medium-resolution fused feature map of [1,512,40,40], and input it into the second detection head (Detect2) for processing.

[0185] 3. Generation and detection of low-resolution fusion features:

[0186] STEP 4-21: Input the output features of STEP 4-20 into the Conv network to obtain the feature values ​​[1,512,20,20].

[0187] STEP 4-22: Combine the above feature values ​​with the hierarchical feature map F3 from STEP 4-10 to obtain the feature values ​​[1,1024,20,20].

[0188] STEP 4-23: Input the above feature values ​​into the feature extraction module to obtain a low-resolution fused feature map of [1,512,20,20], and input it into Detect3 for processing.

[0189] III. Detailed Description of the Decoupling Detection Head:

[0190] The internal structures of detection heads one, two, and three are identical, such as Figure 4 As shown in the figure, this design adopts the "divide and conquer" approach, decoupling the detection task into two independent sub-tasks: localization and classification.

[0191] Localization task: This is handled through a separate upper branch path. This path uses multiple standard convolutional networks (Conv) (i.e., a first-order convolutional network), whose powerful feature extraction and spatial information preservation capabilities help to more accurately regress the coordinate information of the target object from the fused feature map.

[0192] Classification task: This is handled through a separate lower branch path. This path uses a less computationally intensive depthwise separable convolution (DWConv) (i.e., a second type of convolutional network) that focuses on extracting features to distinguish object categories, thereby efficiently identifying the type information of the target object.

[0193] Furthermore, the internal structures of the first, second, and third detection heads are identical, such as... Figure 4 As shown, this design employs the "divide and conquer" approach, decoupling the detection task into two independent sub-tasks: localization and classification. Figure 5 This is a schematic diagram of an upper branch process and a lower branch process according to an embodiment of this application, as shown below. Figure 5 As shown, the detailed steps of the upper and lower branch processes are as follows:

[0194] STEP 5-1: Feed the input fused feature map into the upper branch path and the lower branch path for processing.

[0195] Upward branch process (locating tasks):

[0196] This pathway aims to more accurately regress the coordinate information of the target object from the fused feature map.

[0197] STEP 5-1.1: Process the features through a standard convolutional (Conv) network to obtain new features.

[0198] STEP 5-1.2: Process the above features again through a Conv network.

[0199] STEP 5-1.3: The processed network features are finally passed through a Conv network to output the detected target bounding box (i.e., coordinate information).

[0200] The multiple standard convolutional (Conv) networks here constitute a first convolutional network, whose powerful feature extraction and spatial information preservation capabilities help generate more accurate bounding box coordinates.

[0201] Sub-branch process (categorized tasks):

[0202] This pathway is designed to efficiently identify the type information of the target object.

[0203] STEP 5-2.1: Process the features through a depthwise separable convolutional (DWConv) network to obtain new features.

[0204] DWConv is a depthwise separable network. This network first performs depthwise spatial convolution (for extracting spatial features), and then performs pointwise convolution (for channel blending). This can significantly reduce the amount of computation and parameters while maintaining the model's expressiveness, thereby improving efficiency.

[0205] STEP 5-2.2: Process the above features again through a DWConv network.

[0206] STEP 5-2.3: The processed network features are finally passed through a Conv network to output the detection category (i.e., type information).

[0207] The multiple depthwise separable convolutional (DWConv) networks here constitute a second convolutional network, which has less computational cost and focuses on extracting features to distinguish object categories, thus achieving higher operating efficiency.

[0208] STEP 5-3: Process complete.

[0209] Optionally, the solution in this application extracts features through a feature extraction module. Figure 6 This is a schematic diagram illustrating the feature extraction process of a feature extraction module according to an embodiment of this application, as shown below. Figure 6 As shown, this module implements an efficient feature reuse mechanism (i.e., splitting the initial sampled features into multiple branch features, enhancing at least one of the branch features to obtain an enhanced feature, and merging the enhanced feature with the remaining branch features), the main steps of which are as follows:

[0210] STEP 6-1: The input features first pass through a Conv network for preliminary feature extraction.

[0211] STEP 6-2: Make two copies of the newly obtained features (i.e., the initial sampling features), and label them F1 and F2 respectively.

[0212] STEP 6-3: Divide feature F1 into equal parts by channel using a split network to obtain new features F3 and F4. (This step splits the initial sampled features into multiple branch features).

[0213] STEP 6-4: Process feature F3 (i.e., one of the multiple branch features) through multiple feature fusion modules to obtain a new enhanced feature. (This step enhances at least one of the multiple branch features to obtain the enhanced feature).

[0214] Furthermore, Figure 7 This is a schematic diagram of the feature fusion module according to an embodiment of this application, as shown below. Figure 7 As shown, the feature fusion module includes an Attention module and an FNN module.

[0215] The Attention module primarily employs a self-attention mechanism, which allows the model to autonomously focus on important regions, thereby enhancing the representational ability of features.

[0216] in, Figure 8 This is a schematic diagram of the FNN module according to an embodiment of this application, as shown below. Figure 8 As shown, the FNN module contains two Conv networks. The first Conv network stabilizes deep training through residual connections (which allow the input features and output features to be added bit by bit), while the second Conv network is used to output the final result.

[0217] STEP 6-5: Merge the enhanced features obtained in the previous step with the retained features F2 and F4 to obtain new features. (This step merges the enhanced features with the remaining branch features.)

[0218] STEP 6-6: The merged features are then processed through a Conv model for final integration and output.

[0219] STEP 6-7: Process complete.

[0220] Optionally, the Attention module is described below. Within the "Feature Extraction Module," the Attention module is used to enhance features. Figure 9 This is a schematic diagram of the internal execution steps of an Attention module according to an embodiment of this application, as shown below. Figure 9 As shown, the main steps are as follows:

[0221] 1. Input encoding and normalization:

[0222] STEP 9-1 & 9-2: Obtain the input data and encode it to obtain the initial feature F1.

[0223] STEP 9-3: Perform root mean square normalization (RMS Normalization) on feature F1 to obtain feature F2. Using RMS normalization is more computationally efficient than traditional layer normalization, which helps improve overall computational efficiency.

[0224] 3. Calculation of multi-head self-attention:

[0225] STEP 9-4: Copy the normalized feature F2 into three copies: Q (Query), K (Key), and V (Value).

[0226] STEP 9-5: Perform Rotary Positional Encoding on Q and K respectively. This step incorporates the absolute position information into the vector spaces of Key and Query in the form of complex rotations, enabling the model to more accurately capture the relative positional relationships between features and solving the bottleneck of positional modeling in the traditional Transformer architecture.

[0227] STEP 9-6: Multiply the rotated encoded Q and K positionally to calculate the attention score.

[0228] STEP 9-7: Perform a softmax operation on the attention score to normalize it into weights.

[0229] STEP 9-8: Multiply the normalized weights by the feature V positionally to obtain the weighted features.

[0230] Optionally, to increase the model's adaptability and expressiveness, the self-attention computation process (i.e., one "head") from STEP 9-4 to STEP 9-7 can be copied M times and executed in parallel. Finally, the results of the M times can be merged to form a multi-head self-attention mechanism.

[0231] 4. Residual Connections and Feedforward Networks:

[0232] STEP 9-9: Add the output obtained after self-attention calculation to the initial feature F1 in step 2 by position to form the first residual connection.

[0233] STEP 9-10: Perform root mean square normalization on the summed feature F4 again.

[0234] STEP 9-11: The normalized features are transformed nonlinearly through an FFN (feedforward network) to further enhance the expressive power of the features.

[0235] STEP 9-12: Add the output of FFN to feature F4 again positionally to form a second residual connection, and obtain the final output feature.

[0236] STEP 9-13: Process complete.

[0237] Optionally, the "tracking" operation in this application will be described below. Figure 10 This is a schematic diagram of a tracking operation flow according to an embodiment of this application, such as... Figure 10 As shown, the "tracking" operation can be mainly divided into four core parts: appearance feature extraction, object filter construction, target localization and confirmation, and model update and optimization.

[0238] 1. Feature Extraction:

[0239] This section corresponds to the process of extracting the appearance features of the target object from the reference video frame based on the reference annotation box. Its purpose is to establish a digital "identity profile" for the target object that can characterize its appearance information.

[0240] STEP 10-1: Obtain the coordinates of the reference bounding box, the rotation angle value, and the initial video frame to be processed, as determined by the object detection process.

[0241] STEP 10-2: Extract features from the target object within the reference bounding box of the reference video frame. Optionally, a feature fusion approach can be used, combining shape and color features.

[0242] STEP 10-2.1 - 10-2.4 (Shape Feature Extraction): This process is achieved by calculating the gradient orientation histogram (HOG).

[0243] First, convert the color image to grayscale.

[0244] Next, calculate the gradient magnitude and direction for each pixel. The formula is:

[0245]

[0246]

[0247]

[0248]

[0249] in, At pixel Pixel value at;

[0250] in, For at pixel The gradient value in the x-direction;

[0251] in, For at pixel The gradient value in the y-direction;

[0252] in, For at pixel Gradient magnitude;

[0253] in, For at pixel The gradient direction is specified here. Furthermore, arctan2 is used to calculate the gradient direction, rather than the more common arctan method. The main advantages are: automatic handling of all quadrants; and the range of the arctan function is limited to... (i.e., the first and fourth quadrants), it is impossible to directly distinguish the directions in the second and third quadrants (because ( The ratio may be the same in both quadrants. However, arctan2, by receiving only the two parameters Gx and Gy, can accurately determine the quadrant of the vector based on their signs, thus returning a value within the range of... The correct angle.

[0254] Handling division by zero exceptions: When When the gradient direction is perpendicular, This will result in a division by zero error. If handled correctly, it will return π / 2 or -π / 2 (depending on the sign of Gy) without crashing the program.

[0255] STEP 10-2.3: Segmentation along the gradient direction: Segment all of the image Divide into C bins (number of Hog channels):

[0256]

[0257] STEP 10-2.4: Construct a histogram:

[0258] STEP 10-2.4.1: Divide the image into many small blocks, such as 16x16 pixel blocks.

[0259] STEP 10-2.4.2: Within each block, calculate the weighted gradient magnitude for each bin in each direction:

[0260]

[0261] Where c = 0, 1, 2, ..., C;

[0262] STEP 10-2.4.3: Normalize each small block to increase the robustness of feature matching;

[0263] Multiple small blocks are combined into a large block, and the pixel values ​​are normalized to obtain the pixel value v of each small block again;

[0264] STEP 10-2.5: Combine all small feature blocks and fuse them with the original image features:

[0265]

[0266] in, The feature values ​​of each extracted small block are concatenated;

[0267] in, The pixel values ​​of an RGB image;

[0268] STEP 10-2.6 (Boundary Effect Suppression): Weights each channel of the fused feature z to suppress computational effects that may occur near image boundaries. The formula is:

[0269]

[0270]

[0271]

[0272] Where W and H are the width and height of the target bounding box;

[0273] 2. Object Filter Construction:

[0274] This section corresponds to the process of constructing an object filter for identifying the target object based on the appearance features. The aim is to train a mathematical model specifically designed to match the target based on its appearance features.

[0275] STEP 10-3: Calculate the spatial reliability mask M. This step calculates the Euclidean distance between the target features and the background region using a sliding window to generate a foreground mask matrix M, which is used to highlight the target region and suppress background interference in subsequent training.

[0276] STEP 3.1: Create a sliding window from the top left to the bottom right of the image to be detected, extract the pixel values ​​within the window, and calculate the sum of all pixel values ​​within the window. The Euclidean distance is set such that if the distance is greater than the set threshold, all elements in the sliding window are set to 1; otherwise, they are set to 0.

[0277]

[0278]

[0279] Where i and j represent the image to be labeled, divided into i groups based on the size of the target bounding box. j blocks to be labeled;

[0280] To set a threshold;

[0281] For the first The location of each block to be labeled;

[0282] STEP 10-4: Construct a correlation filter The filter is represented in the frequency domain, and its calculation formula is:

[0283]

[0284] Where Z: multi-channel feature matrix extracted from the target bounding box (including HOG, RGB, etc.).

[0285] M: Foreground mask matrix, with values ​​between 0 and 1, used to highlight the target area and suppress the background.

[0286] G: Ideal Gaussian response map. A Gaussian function with a peak value at the center of the target, with dimensions H×W, where is the coordinate position, and is the center position of dimension H×W, and σ is the standard deviation of the Gaussian distribution.

[0287] f(): indicates performing a Fourier transform on the matrix within the parentheses.

[0288] ¯( ): indicates taking the complex conjugate.

[0289] • Represents element-wise multiplication of matrices.

[0290] Regularization parameter (a very small positive number) prevents the denominator from being zero and avoids overfitting.

[0291] The final filter constructed (represented in the frequency domain).

[0292] 3. Target localization and confirmation:

[0293] This part corresponds to the process of identifying the tracking bounding box of the target object through the object filter within the preset search area of ​​the initial video frame.

[0294] STEP 10-5: Determine the preset search area. This step corresponds to obtaining the position and rotation angle information of the reference frame, and determining the preset search area based on the position and rotation angle information. Specifically, the system determines the search center based on the coordinate position of the reference frame, and constructs a compact search area based on its rotation angle and size.

[0295] STEP 10-6: Calculate the response map. Within the preset search region, the system utilizes the filter constructed in the previous step. Perform relevant calculations to compute the response function R. The formula is: ,in, The size selected around the center point is W. The feature value of H. The point with the largest value on the response map R is determined as the center point of the target object in the current frame.

[0296] STEP 10-7: Confirm the target area size. To handle changes in target size, the system will test multiple scaling ratios around the determined center point (e.g., 10 values ​​between 0.6 and 2.0) and calculate the response value at each ratio. Finally, the scaling factor with the largest response value (largest R value) is selected and multiplied by the original box width and height (w, h) to obtain the new target box size (Wt, Ht).

[0297] 4. Model Update and Optimization:

[0298] This section is used to smooth the output results and update the model to adapt to the slow changes in the target's appearance.

[0299] STEP 10-8: Bounding Box Optimization. To avoid noise-induced jitter in the annotation boxes, the newly determined target box dimensions are smoothed and optimized. The formula is:

[0300]

[0301]

[0302] in, For smoothing coefficients, The width of the target bounding box. The height of the target bounding box.

[0303] STEP 10-9: Obtain the final target coordinates and annotation frame.

[0304] STEP 10-10: Update the filter. To adapt to changes in the target's appearance (such as slow rotation or changes in lighting), the system recalculates the appearance feature z based on the new target position and size, and updates it smoothly. The formula is: ,in, The feature value of the target box detected in the previous frame. This is the smoothing coefficient.

[0305] STEP 10-11: Process complete.

[0306] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0307] Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0308] This embodiment also provides a labeling device for model training samples, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0309] Figure 11 This is a structural block diagram of a model training sample annotation device according to an embodiment of this application; as shown... Figure 11 As shown, it includes:

[0310] The acquisition module 1102 is used to acquire the current reference video frame from the continuous video frames included in the initial model training samples, wherein the reference video frame has a reference bounding box of the target object, and the confidence of the reference bounding box is greater than or equal to a first threshold.

[0311] The tracking module 1104 is used to track the target object in the consecutive initial video frames following the reference video frame according to the reference annotation box, to obtain the tracking video frame with the tracking annotation box marked, and during the tracking of the target object in the initial video frame, to detect whether the initial video frame meets the annotation calibration condition, wherein the annotation calibration condition is used to indicate that the confidence level corresponding to the initial video frame is less than or equal to a second threshold.

[0312] The target detection module 1106 is used to perform target detection on the target object in the target video frame following the reference video frame in the consecutive video frames when a reference video frame that meets the annotation calibration conditions is detected, and to obtain a detection video frame with a detection annotation box, wherein the confidence of the detection annotation box is greater than or equal to the first threshold.

[0313] The determination module 1108 is used to determine the detected video frame as the next current reference video frame until the target model training sample is obtained, wherein the target model training sample includes the continuous video frames with annotation boxes labeled with the target object.

[0314] In one exemplary embodiment, the tracking module includes:

[0315] The first detection unit is used to detect whether the confidence level corresponding to the initial video frame is less than or equal to the second threshold based on the video frame parameters of the initial video frame.

[0316] The first determining unit is configured to determine the initial video frame as the reference video frame that satisfies the annotation calibration conditions when the confidence level is detected to be less than or equal to the second threshold.

[0317] In an exemplary embodiment, the first detection unit is further configured to:

[0318] The number of consecutive tracking of the initial video frame is detected, wherein the number of consecutive tracking is the number of the initial video frames that have been continuously tracked after the reference video frame, and the video frame parameters include the number of consecutive tracking.

[0319] If the number of consecutive tracks is greater than or equal to the tracking number threshold, it is determined that the confidence level corresponding to the detected initial video frame is less than or equal to the second threshold.

[0320] If the number of consecutive tracks is less than the tracking number threshold, it is determined that the confidence level corresponding to the initial video frame is greater than the second threshold.

[0321] In an exemplary embodiment, the first detection unit is further configured to:

[0322] The annotation deviation parameter of the initial video frame is detected, wherein the annotation deviation parameter is used to indicate the degree of positional deviation between the tracking bounding box marked in the initial video frame and the target object, and the larger the annotation deviation parameter is, the higher the degree of positional deviation.

[0323] If the labeled deviation parameter is greater than or equal to the deviation parameter threshold, it is determined that the confidence level corresponding to the detected initial video frame is less than or equal to the second threshold;

[0324] If the labeled deviation parameter is less than the deviation parameter threshold, it is determined that the confidence level corresponding to the detected initial video frame is greater than the second threshold.

[0325] In one exemplary embodiment, the tracking module includes:

[0326] An extraction unit is configured to extract the appearance features of the target object in the reference video frame based on the reference bounding box, wherein the appearance features are used to characterize the appearance information of the target object;

[0327] The construction unit is used to construct an object filter for identifying the target object based on the appearance features;

[0328] The identification unit is used to identify the tracking bounding box of the target object within a preset search area of ​​the initial video frame using the object filter, thereby obtaining a tracking video frame labeled with the tracking bounding box.

[0329] In one exemplary embodiment, the extraction unit is further configured to:

[0330] Extract the shape features of the target object and extract the color features of the target object, wherein the shape features are features used to characterize the contour and texture information of the target object, and the color features are features used to characterize the color information of the target object;

[0331] The shape feature and the color feature are fused together to obtain the appearance feature.

[0332] In one exemplary embodiment, the tracking module further includes:

[0333] The acquisition unit is configured to acquire the position information and rotation angle information of the reference annotation box before identifying the tracking annotation box of the target object through the object filter within a preset search area of ​​the initial video frame, wherein the position information is used to indicate the position of the reference annotation box in the reference video frame, and the rotation angle information is used to indicate the rotation attitude of the reference annotation box in the reference video frame.

[0334] The second determining unit is used to determine the preset search area based on the position information and the rotation angle information.

[0335] In one exemplary embodiment, the target detection module includes:

[0336] The second extraction unit is used to extract the features of the target video frame layer by layer according to different spatial resolutions to obtain multiple layer feature maps. The different layer feature maps have feature data of the target video frame at different spatial resolutions. The layer feature map with higher spatial resolution includes more local features of the target object, and the layer feature map with lower spatial resolution includes more overall features of the target object.

[0337] A fusion unit is used to fuse different hierarchical feature maps from multiple hierarchical feature maps to obtain multiple fused feature maps, wherein the fused feature maps simultaneously include the local features and the overall features of the target object;

[0338] The output unit is used to output candidate bounding boxes in the target video frame based on multiple fused feature maps, so as to obtain candidate video frames labeled with the candidate bounding boxes.

[0339] The third detection unit is used to detect the confidence level of the candidate label boxes;

[0340] The third determining unit is configured to determine the candidate annotation box as the detection annotation box and the candidate video frame as the detection video frame when the confidence level of the candidate annotation box is greater than or equal to the first threshold.

[0341] In one exemplary embodiment, the second extraction unit is further configured to:

[0342] While reducing the spatial resolution of the target video frame, features used to characterize the target object are extracted to obtain initial sampling features;

[0343] The initial sampling features are split into multiple branch features, at least one of the multiple branch features is enhanced to obtain an enhanced feature, and the enhanced feature is merged with the remaining branch features to obtain the hierarchical feature map.

[0344] In one exemplary embodiment, the output unit is further configured to:

[0345] The first convolutional network detects the coordinate information of the target object from the fused feature map, and the second convolutional network identifies the type information of the target object from the fused feature map. The coordinate information is used to indicate the position and size of the target object in the target video frame, and the type information is used to indicate the type to which the target object belongs.

[0346] The alternative label boxes are generated based on the coordinate information and the type information.

[0347] In one exemplary embodiment, the acquisition module includes:

[0348] The fourth detection unit is used to perform target detection on the continuous video frames frame by frame according to the time sequence of the continuous video frames, to obtain candidate video frames with candidate bounding boxes labeled with the target object, and when the reference video frame is not present in the continuous video frames included in the initial model training samples;

[0349] The fifth detection unit is used to detect the confidence level of the candidate bounding boxes;

[0350] The fourth determining unit is configured to determine the reference video frame based on the size of the candidate annotation box when the confidence level of the candidate annotation box is greater than or equal to the first threshold.

[0351] The continuation unit is configured to continue the target detection for the next video frame of the candidate video frame if the confidence level of the candidate bounding box is less than the first threshold.

[0352] In one exemplary embodiment, the fourth determining unit is further configured to:

[0353] Compare the size of the candidate annotation box with a preset size threshold;

[0354] If the size of the annotation box is smaller than the preset size threshold, the target detection continues for the next video frame of the candidate video frame;

[0355] If the size of the annotation box is greater than or equal to the preset size threshold, the candidate video frame corresponding to the candidate annotation box is determined as the reference video frame, and the candidate annotation box is determined as the reference annotation box.

[0356] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0357] For a description of the features in the embodiment corresponding to the annotation device for model training samples, please refer to the relevant description of the embodiment corresponding to the annotation method for model training samples, which will not be repeated here.

[0358] Embodiments of this application also provide an electronic device. Figure 12 This is a schematic diagram of an electronic device according to an embodiment of this application, such as... Figure 12 As shown, the electronic device includes a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to perform the steps in any of the above embodiments of the method for labeling model training samples.

[0359] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0360] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0361] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the method for labeling model training samples when running.

[0362] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0363] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0364] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods described in various embodiments of this application; the computer program product further includes a non-volatile computer-readable storage medium storing the computer program, which, when executed by a processor, implements the steps of the model training sample annotation method described in various embodiments of this application.

[0365] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0366] The above provides a detailed description of a method for labeling model training samples provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for labeling a model training sample, characterized in that, The method comprises: obtaining a current reference video frame from continuous video frames included in an initial model training sample, wherein a reference annotation box of a target object has been labeled in the reference video frame, and a confidence of the reference annotation box is greater than or equal to a first threshold value; tracking the target object in continuous initial video frames after the reference video frame in the continuous video frames according to the reference annotation box to obtain tracking video frames labeled with tracking annotation boxes, and detecting whether the initial video frames satisfy annotation calibration conditions in the process of tracking the target object in the initial video frames, wherein the annotation calibration conditions are used to indicate that a confidence corresponding to the initial video frames is less than or equal to a second threshold value; in a case where a reference video frame satisfying the annotation calibration conditions is detected, performing target detection on the target object in target video frames after the reference video frame in the continuous video frames to obtain detection video frames labeled with detection annotation boxes, wherein a confidence of the detection annotation box is greater than or equal to the first threshold value; determining the detection video frames as next current reference video frames until a target model training sample is obtained, wherein the target model training sample comprises the continuous video frames labeled with the annotation boxes of the target object.

2. The method of claim 1, wherein the detection of whether the initial video frames satisfy annotation calibration conditions comprises: detecting, according to a video frame parameter of the initial video frame, whether the confidence corresponding to the initial video frame is less than or equal to the second threshold value; in a case where it is detected that the confidence is less than or equal to the second threshold value, determining that the initial video frame is the reference video frame satisfying the annotation calibration conditions.

3. The method of claim 2, wherein the detection of whether the confidence corresponding to the initial video frame is less than or equal to the second threshold value according to the video frame parameter of the initial video frame comprises: detecting a continuous tracking number of the initial video frame, wherein the continuous tracking number is a number of the initial video frames that have been continuously tracked after the reference video frame, and the video frame parameter comprises the continuous tracking number; in a case where the continuous tracking number is greater than or equal to a tracking number threshold value, it is determined that the confidence corresponding to the initial video frame is detected to be less than or equal to the second threshold value; in a case where the continuous tracking number is less than the tracking number threshold value, it is determined that the confidence corresponding to the initial video frame is detected to be greater than the second threshold value.

4. The method of claim 2, wherein the detection of whether the confidence corresponding to the initial video frame is less than or equal to the second threshold value according to the video frame parameter of the initial video frame comprises: detecting an annotation deviation parameter of the initial video frame, wherein the annotation deviation parameter is used to indicate a position deviation degree between the tracking annotation box labeled in the initial video frame and the target object, and the greater the annotation deviation parameter is, the higher the position deviation degree is. determining that the confidence corresponding to the initial video frame is less than or equal to the second threshold value, in a case where the annotation deviation parameter is greater than or equal to a deviation parameter threshold value; determining that the confidence corresponding to the initial video frame is greater than the second threshold value, in a case where the annotation deviation parameter is less than the deviation parameter threshold value.

5. The method of claim 1, wherein the tracking the target object in the continuous initial video frame after the reference video frame according to the reference annotation box to obtain a tracking video frame annotated with a tracking annotation box comprises: extracting appearance features of the target object in the reference video frame according to the reference annotation box, wherein the appearance features are used to represent appearance information of the target object; constructing an object filter used to identify the target object according to the appearance features; identifying the tracking annotation box of the target object in the initial video frame within a preset search region by using the object filter to obtain the tracking video frame annotated with the tracking annotation box.

6. The method of claim 5, wherein the extracting appearance features of the target object in the reference video frame according to the reference annotation box comprises: extracting shape features of the target object and color features of the target object, wherein the shape features are features used to represent contour and texture information of the target object, and the color features are features used to represent color information of the target object; and fusing the shape features and the color features to obtain the appearance features.

7. The method of claim 5, wherein before the identifying the tracking annotation box of the target object in the initial video frame within a preset search region by using the object filter, the method further comprises: obtaining position information and rotation angle information of the reference annotation box, wherein the position information is used to indicate a position of the reference annotation box in the reference video frame, and the rotation angle information is used to indicate a rotation posture of the reference annotation box in the reference video frame; and determining the preset search region according to the position information and the rotation angle information.

8. The method of claim 1, wherein the target detection of the target object in the target video frame after the reference video frame to obtain a detection video frame annotated with a detection annotation box comprises: extracting features of the target video frame layer by layer according to different levels of spatial resolution to obtain a plurality of level feature maps, wherein different level feature maps have feature data of the target video frame at different levels of spatial resolution, the level feature map with higher spatial resolution includes more local features of the target object, and the level feature map with lower spatial resolution includes more overall features of the target object; and ​ ​ ​ ​ fusing different hierarchical feature maps of the plurality of hierarchical feature maps to obtain a plurality of fused feature maps, wherein the fused feature maps comprise both the local features and the global features of the target object; outputting the candidate bounding boxes in the target video frame according to the plurality of fused feature maps to obtain a candidate video frame labeled with the candidate bounding boxes; detecting a confidence of the candidate bounding boxes; in a case where the confidence of the candidate bounding boxes is greater than or equal to the first threshold, determining the candidate bounding boxes as the detection bounding boxes and determining the candidate video frame as the detection video frame.

9. The method of claim 8, wherein the extracting features of the target video frame layer by layer according to different hierarchical spatial resolutions of the target video frame to obtain a plurality of hierarchical feature maps comprises: extracting initial sampling features for representing the target object while reducing the spatial resolution of the target video frame; splitting the initial sampling features into a plurality of branch features, enhancing at least one of the branch features to obtain enhanced features, and merging the enhanced features with the remaining branch features to obtain the hierarchical feature maps.

10. The method of claim 8, wherein the outputting the candidate bounding boxes in the target video frame according to the plurality of fused feature maps to obtain the candidate video frame labeled with the candidate bounding boxes comprises: detecting coordinate information of the target object from the fused feature maps by a first convolutional network and identifying type information of the target object from the fused feature maps by a second convolutional network, wherein the coordinate information is used to indicate the position and size of the target object in the target video frame, and the type information is used to indicate the type of the target object; generating the candidate bounding boxes according to the coordinate information and the type information.

11. The method of claim 1, wherein the obtaining a current reference video frame from continuous video frames included in the initial model training samples comprises: in a case where the reference video frame does not exist in the continuous video frames included in the initial model training samples, performing target detection on the continuous video frames frame by frame according to the time sequence of the continuous video frames to obtain a candidate video frame labeled with a candidate bounding box of the target object; detecting a confidence of the candidate bounding box; in a case where the confidence of the candidate bounding box is greater than or equal to the first threshold, determining the reference video frame according to the bounding box size of the candidate bounding box; in a case where the confidence of the candidate bounding box is less than the first threshold, continuing to perform the target detection on a next video frame of the candidate video frame.

12. The method of claim 11, wherein the determining the reference video frame according to the bounding box size of the candidate bounding box comprises: ​ ​ ​ ​ compare the bounding box size of the candidate bounding box with a preset size threshold; if the bounding box size is smaller than the preset size threshold, continue to perform the target detection on a next video frame of the candidate video frames; if the bounding box size is greater than or equal to the preset size threshold, determine the candidate video frame corresponding to the candidate bounding box as the reference video frame, and determine the candidate bounding box as the reference bounding box.

13. An electronic device, comprising: comprising: a memory for storing a computer program; a processor for implementing the steps of the annotation method of the model training sample according to any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium, and when the computer program is executed by the processor, the steps of the annotation method of the model training sample according to any one of claims 1 to 12 are implemented.

15. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the annotation method of the model training sample according to any one of claims 1 to 12.

Images