Video annotation method, system, device and medium based on continuous insertion

Abstract

The application discloses a video labeling method, system, device and medium based on continuous insertion, relates to the technical field of data processing, and comprises the following steps: acquiring a target video and a preset labeling object set, acquiring a time axis, acquiring a current frame, a historical frame and a subsequent frame; generating a first recognition result; if the first recognition result matches the preset labeling object set, a second recognition result is generated, and a first difference degree is acquired; if the first difference degree exceeds a first preset threshold, initial labeling content is inserted into the current frame at the ith time point, target recognition is performed on the subsequent frame at the ith time point to generate a third recognition result, and a second difference degree is acquired according to the third recognition result and the first recognition result; if the second difference degree exceeds a second preset threshold, the initial labeling content is updated according to the third recognition result, and frame-by-frame identification and labeling updating are continuously performed to the end point of the target video. The application has the advantages of accurate judgment, adaptive adjustment and lightweight operation.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a video annotation method, system, device, and medium based on continuous insertion. Background Technology

[0002] In video annotation tasks within the field of computer vision, such as intelligent surveillance or autonomous driving environment analysis, real-time annotation of dynamic objects in video streams is required to assist in target detection and behavior recognition. However, existing technologies often cannot efficiently handle the sudden appearance, continuous movement, and morphological changes of objects, resulting in insufficient annotation accuracy, severe computational redundancy, and a degraded user experience.

[0003] Specifically, when an object in a target video first appears in a specific frame, existing methods rely on independent identification and annotation for each frame, lacking collaborative analysis of historical and subsequent frames. This leads to delayed detection of new objects or incorrect annotation insertion, such as objects being overlooked when they emerge from the edge of the field of view. Simultaneously, subtle changes in position and shape during object movement may go undetected, causing annotation boxes or text to lag or drift, especially in high-speed motion scenes, resulting in misaligned annotations or user misjudgments. Furthermore, when objects disappear or undergo abrupt changes in shape, existing methods struggle to adaptively eliminate or update annotations, instead mechanically repeating the processing of all frames, increasing computational overhead (e.g., increased CPU load) and causing performance bottlenecks in long videos. More critically, these methods lack intelligent decision-making mechanisms based on differences, failing to distinguish between stable states and points of change in objects, thus wasting resources on invalid annotations and overall reducing real-time performance, accuracy, and efficiency. An adaptive continuous annotation scheme is urgently needed to optimize the smoothness and reliability of video analysis. Summary of the Invention

[0004] In view of the technical problems pointed out in the background art, the present invention provides a video annotation method, system, device and medium based on continuous insertion.

[0005] A video annotation method based on continuous insertion includes: acquiring a target video and a preset set of annotation objects, and acquiring a corresponding timeline based on the target video; acquiring the current frame of the target video at the i-th time point on the timeline, as well as the historical frames and subsequent frames adjacent to the current frame; performing target recognition on the current frame at the i-th time point and generating a first recognition result; if the first recognition result matches the preset set of annotation objects, performing target recognition on the historical frames at the i-th time point and generating a second recognition result, and obtaining a first difference degree based on the second recognition result and the first recognition result; if the first difference degree exceeds a first preset threshold, inserting initial annotation content into the current frame at the i-th time point, performing target recognition on the subsequent frames at the i-th time point and generating a third recognition result, and obtaining a second difference degree based on the third recognition result and the first recognition result; if the second difference degree exceeds a second preset threshold, updating the initial annotation content based on the third recognition result, and continuing to identify and update annotations frame by frame towards the end of the target video.

[0006] Optionally, performing target recognition and generating a first recognition result for the current frame at the i-th time point includes: generating contour information containing the morphological data and position data of the target based on the boundary contour of the target in the current frame, and defining the contour information as the first recognition result, wherein the morphological data includes the contour geometric features of the target, and the position data includes the coordinates of the target center point in the plane of the current frame.

[0007] Optionally, if the first identification result matches the preset annotation object set, it includes: comparing and matching the morphological data in the first identification result with the reference morphological templates stored in the preset annotation object set; when the morphological data in the first identification result meets the similarity requirement with any reference morphological template, it is determined that the first identification result matches the preset annotation object set.

[0008] Optionally, obtaining the first difference degree based on the second identification result and the first identification result includes: comparing the target's position data in the second identification result and the first identification result to generate position difference features; comparing the target's morphological data in the second identification result and the first identification result to generate morphological difference features; and adding the position difference features to the morphological difference features to obtain the first difference degree.

[0009] Optionally, inserting initial annotation content into the current frame at the i-th time point includes: determining the positioning reference point of the annotation content based on the target's position data in the first recognition result; extracting the matching target's identifier text and annotation box boundary from a preset annotation object set; and overlaying the initial annotation content consisting of the annotation box boundary and identifier text at the positioning reference point.

[0010] Optionally, updating the initial annotation content based on the third recognition result includes: moving the positioning reference point of the initial annotation content based on the position data of the third recognition result, so that the boundary of the annotation box and the label text are synchronously shifted to the new target position; when the third recognition result indicates that the target is out of the identifiable range, the initial annotation content in the current frame of the video is eliminated.

[0011] A video annotation system based on continuous insertion is also provided. The system includes: an acquisition module, used to acquire a target video and a preset set of annotation objects, and to acquire the corresponding timeline based on the target video, and to acquire the current frame of the target video at the i-th time point on the timeline, as well as the historical frames and subsequent frames adjacent to the current frame; a data processing module, used to perform target recognition on the current frame at the i-th time point and generate a first recognition result, and if the first recognition result matches the preset set of annotation objects, to perform target recognition on the historical frames at the i-th time point and generate a second recognition result, and to obtain a first difference degree based on the second recognition result and the first recognition result; a first annotation module, used to insert initial annotation content into the current frame at the i-th time point when the first difference degree exceeds a first preset threshold, and to perform target recognition on the subsequent frames at the i-th time point and generate a third recognition result, and to obtain a second difference degree based on the third recognition result and the first recognition result; and a second annotation module, used to update the initial annotation content based on the third recognition result when the second difference degree exceeds a second preset threshold, and to continue to identify and update the annotation frame by frame towards the end of the target video.

[0012] Optionally, the data processing module is further configured to: generate contour information containing morphological data and position data of the target based on the boundary contour of the target in the current frame, and define the contour information as a first recognition result, wherein the morphological data includes the contour geometric features of the target, and the position data includes the coordinates of the target center point in the plane of the current frame.

[0013] An electronic device is also provided, comprising: a memory having a computer program stored thereon; and a processor for executing the computer program in the memory to implement a video annotation method based on continuous insertion.

[0014] A non-transitory computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements a video annotation method based on continuous insertion.

[0015] The beneficial effects of this invention are reflected in: In the entire video annotation method based on continuous insertion, the spatiotemporal continuous analysis window constructed based on three-frame slices, combined with the filtering mechanism of morphological template matching, effectively solves the problem of misjudgment of new targets. By comparing the position / morphological differences between the current frame and historical frames (first difference degree), the timing of the new appearance of the object is strictly determined, which avoids repeated annotation of continuously moving targets and ensures timely identification of targets emerging from the edge. Secondly, the dynamic update strategy of difference degree threshold control accurately optimizes resource allocation. The annotation focus is triggered only when the displacement or deformation between the later frame and the current frame exceeds the threshold (such as the synchronous movement of the annotation box after a cat jumps), while in a stable state (such as a pet...). The cat's small steps skip invalid updates, and combined with the automatic elimination mechanism when the target disappears, it significantly reduces most of the repetitive computation load in long videos (especially reducing GPU rendering pressure). Finally, the closed-loop rolling advancement mechanism formed by the whole process (taking the updated next frame as the new current frame) achieves on-demand scheduling of computing resources while maintaining spatiotemporal consistency of annotation. In high-speed scenes, the low threshold ensures that micro-movements are updated (the annotation box follows the vehicle in real time when it turns), and in static scenes, the high threshold filters jitter interference (indoor furniture avoids flickering). To a certain extent, this solves the problems of annotation lag, drift accumulation and performance bottleneck caused by full-frame processing in existing methods. Attached Figure Description

[0016] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0017] Figure 1 This is a partial flowchart of S1 to S4 in the video annotation method based on continuous insertion of the present invention; Figure 2 This is a partial flowchart of S2 in the video annotation method based on continuous insertion of the present invention; Figure 3 This is a partial flowchart of S3 in the video annotation method based on continuous insertion of the present invention; Figure 4 This is a partial flowchart of S4 in the video annotation method based on continuous insertion of the present invention; Figure 5 This is a schematic diagram illustrating the steps of the video annotation method based on continuous insertion of the present invention; Figure 6 This is a schematic diagram of part of step S2 in the video annotation method based on continuous insertion of the present invention; Figure 7 This is a schematic diagram of another part of the steps in S2 of the video annotation method based on continuous insertion of the present invention; Figure 8 This is a schematic diagram of a portion of step S3 in the video annotation method based on continuous insertion of the present invention; Figure 9 This is a schematic diagram of a portion of step S4 in the video annotation method based on continuous insertion of the present invention; Figure 10 This is a block diagram illustrating an electronic device according to an embodiment of the present invention.

[0018] Figure label: 700 - Electronic device; 701 - Processor; 702 - Memory; 703 - Multimedia component; 704 - I / O interface; 705 - Communication component. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0020] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0021] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0022] like Figures 1 to 5 As shown, a video annotation method based on continuous insertion is provided. In one embodiment, the method includes: S1. Obtain the target video and the set of preset labeled objects, and obtain the corresponding timeline according to the target video. Obtain the current frame of the target video at the i-th time point of the timeline, as well as the historical frames and subsequent frames adjacent to the current frame. S2. Perform target recognition on the current frame at the i-th time point and generate a first recognition result. If the first recognition result matches the preset labeled object set, perform target recognition on the historical frame at the i-th time point and generate a second recognition result. Obtain a first difference degree based on the second recognition result and the first recognition result. S3. If the first difference exceeds the first preset threshold, insert the initial annotation content into the current frame at the i-th time point, perform target recognition on the subsequent frame at the i-th time point and generate the third recognition result, and obtain the second difference based on the third recognition result and the first recognition result. S4. If the second difference exceeds the second preset threshold, the initial annotation content is updated according to the third recognition result, and the annotation is updated frame by frame towards the target video endpoint.

[0023] In this embodiment, it should be noted that in S1, the raw data required for analysis and the basic temporal analysis framework are prepared. Specifically, the target video data stream to be processed and a predefined set of labeled objects need to be acquired first. The target video is the main body of the analysis, containing continuous image frames arranged in a time sequence, recording the dynamic changes of the scene. The predefined set of labeled objects is a key filtering condition; it stores morphological feature templates of specific target categories that need to be focused on and labeled (e.g., vehicles of specific shapes, pedestrian outlines, animals in specific postures, etc.), essentially a "target library." Next, a corresponding logical timeline is constructed based on the temporal characteristics of the target video itself. This timeline is not physical time, but rather a logical index system that divides the video frame sequence, so that each frame corresponds to a precise time point (time index i). This step is crucial, as it provides the coordinate basis for subsequently locating frames at specific "moments." Then, based on this timeline, precise frame localization is performed for the specific time point i to be analyzed: the "current frame" (frame i) at this precise time point is obtained, along with its immediately preceding frame (i.e., the "historical frame," frame i-1 or earlier but ensuring it is adjacent) and its immediately following frame (i.e., the "next frame," frame i+1). This forms a small, continuous three-frame slice centered on the current frame (historical frame - current frame - next frame). Obtaining these three frames is the basic unit for subsequent analysis processes.

[0024] For example, in a street surveillance video captured by an autonomous vehicle, the minute-level video stream file is first loaded, along with a pre-defined "urban traffic object library" (containing morphological templates for cars, buses, pedestrians, bicycles, etc.). Next, the video's frame rate (e.g., 30 frames per second) is analyzed to establish a logical timeline (dividing the video into continuous index points in 1 / 30-second units). When processing time point i (e.g., frame 315 corresponding to 10.5 seconds) begins, it precisely reads frame 315 from the video stream at this logical time point as the current frame (potentially showing the front half of a car), while simultaneously reading its preceding frame (frame 314, potentially showing the car's initial state closer to the edge of the field of view) as a historical frame, and its following frame (frame 316, potentially showing the car's clearer and more complete form) as a subsequent frame. This prepares a continuous visual context at a specific time point for the collaborative comparative analysis in step S2.

[0025] Furthermore, the design of S1, which constructs a "three-frame slice," is specifically targeted at addressing the annotation lag and redundancy issues caused by isolated frame analysis (lacking historical and future context) in background techniques. By actively extracting the current time point i and its directly adjacent preceding and following frames, a miniature spatiotemporal analysis unit is established from the outset. This unit enables effective dynamic comparison of an object at time point i with its most recent past (historical frame) and most recent future (following frame) in steps S2 and S3. The acquired target video is the data source for analysis, the pre-defined set of annotated objects is the focus of analysis (filtering out a large amount of irrelevant background information and focusing only on the predetermined target), the timeline provides the logical coordinates for accurately locating each frame, and the three frames (historical frame / current frame / following frame) constitute the smallest effective continuous analysis window around the current time point i. It is particularly important to emphasize that "adjacent" not only refers to the continuity in time index (i.e., i-1, i, i+1), but also means that these frames have a high degree of continuity and correlation in the target's motion state. This continuity is the basis for tracking position changes and judging morphological evolution, and provides a physically meaningful comparison benchmark for subsequent difference calculation.

[0026] For example, when processing a smart home surveillance video clip (with a preset object set including "a specific breed of pet cat"), in the three frames acquired at time point i: the historical frame might show a blurry figure partially appearing at the edge of the sofa (difficult to initially identify); the current frame (frame i) might capture a clearer outline of the cat's face; and the subsequent frame might capture a more relaxed full-body posture of the cat. At this point, S1 has completed its work, possessing video data, a target library (cat morphological templates), a time index (locating 315 frames), and three consecutive visual state snapshots around frame i (blurry figure, clear cat face, complete cat body). This "three-frame slice" is ready, signifying that the data preparation is complete for initiating targeted target recognition at time point i and subsequent cross-frame difference calculations. The entire analysis process can revolve around the current core frame and its direct context, avoiding invalid processing of irrelevant distant frames.

[0027] In S2, a collaborative verification mechanism between historical frames and the current frame addresses the crucial question of whether an object is a newly emerging target, while also providing a quantitative basis for annotation decisions in subsequent steps. Specifically, target identification is first performed on the current frame at time point i, extracting the contour geometric features of key objects and their center point coordinates as the first identification result. The boundary contour recognition method here (such as edge detection or semantic segmentation) can effectively distinguish the target subject from complex backgrounds. For example, in a traffic monitoring scene, after capturing a clear outline of the front half of a car in the current frame (frame 315), the geometric features of its hood and windshield are extracted as morphological data, and the coordinates of the car's geometric center in the image are calculated.

[0028] Subsequently, the morphological data from the first recognition result is matched with a preset set of labeled objects (such as a "city traffic object library"): the geometric features of the car outline extracted in the current frame are compared with the reference templates in the library (such as the side profile of a standard car). When the feature overlap rate reaches the set requirement, the match is considered successful. This matching process is essentially an efficient filtering mechanism that ensures that only preset target categories are considered. For example, in a smart home scenario, if the cat's facial outline in the current frame highly matches the ear-pointing features and facial proportions of a "specific cat breed template," the next step of analysis is triggered. If the match is successful, the process immediately backtracks to the corresponding historical frame (frame 314) at the same time point and executes the same recognition process to generate a second recognition result. For example, historical frames in a traffic scene may only show the blurry outline of the headlight edges, while historical frames in a home scene may only record the fluffy, blurry blocky area of ​​the sofa edge. At this point, based on the differential comparison between the recognition results of historical frames and the current frame, the first difference score is calculated: the positional difference feature reflects the degree of displacement of the target's center point coordinates between two adjacent frames (quantifying the motion amplitude as a relative displacement ratio), while the morphological difference feature simply records whether the geometric structure of the target's outline has undergone a fundamental change (0 indicates stability, 1 indicates abrupt change). In the traffic case, the geometric structure difference between the clear car outline in the current frame and the blurry headlight outline in the historical frame is significant (morphological difference feature = 1), and the displacement ratio of the car's center point from the edge of the image to the center is relatively large (positional difference feature value is high), resulting in a high first difference score. When this difference score exceeds a preset threshold (e.g., 0.8), it means that there is no identifiable target in the historical frame or the target shape is incomplete, thus determining that the target in the current frame is a newly appearing object—this is the core condition for triggering annotation insertion. The collaborative comparison mechanism fundamentally avoids the risk of false alarms in isolated frame judgment. For example, when a continuously moving bus in a traffic video has been partially labeled in the previous frame (historical frame), the positional difference feature value between the current frame recognition result and the historical frame recognition result is low and the morphological difference feature is 0. The first difference degree is lower than the threshold, so the label will not be inserted again, thus significantly reducing redundant operations.

[0029] Furthermore, the first layer uses morphological template matching to lock in a preset target category (solving the "what to label"), while the second layer uses difference analysis between historical frames and the current frame to determine whether the target is a newly appearing entity (solving the "when to label"). For example, in a pet monitoring scenario, after the current frame (frame i) identifies a clear cat face outline and matches it with a preset cat template, the historical frame (frame i-1) identifies a fluffy, blurry block outline. If the center point of the cat face in the current frame has a significantly larger displacement ratio compared to the blurry block in the historical frame (e.g., moving from the edge of the sofa to the center), and the geometric structure abruptly changes from a featureless clumpy shape to a recognizable feline face (morphological difference feature = 1), then a high first difference score is generated. If this value exceeds a threshold, it confirms that the cat has appeared completely for the first time in the current frame, at which point the initial labeling insertion of S3 is triggered.

[0030] Conversely, if a complete cat outline already exists in a historical frame (morphological difference feature = 0), regardless of the magnitude of positional movement, the first difference score will be lowered by morphological stability, suppressing annotation insertion (determining that the target is not newly appearing). This mechanism precisely avoids two typical errors: first, it prevents mistaking blurry or partially occluded targets in historical frames for new objects (e.g., in a traffic scene, if a historical frame only shows headlights but the current frame shows the complete front of the vehicle, morphological difference feature = 1 will reasonably trigger annotation); second, it avoids repeated annotation of stable moving targets (e.g., a bus that is fully displayed for several consecutive frames). Simultaneously, the difference score threshold mechanism provides flexible control over motion sensitivity: a low threshold is suitable for high-speed scenes (micro-movements can trigger annotation updates), while a high threshold is suitable for static scenes (reducing jitter interference), thus supporting a balance between the timeliness and accuracy of annotations at the underlying level.

[0031] In S3, newly identified targets from S2 undergo intelligent initial labeling and insertion, and the initial verification of the target's changing state in subsequent frames is immediately initiated. The strategy for determining the first preset threshold needs to consider both the target's motion speed and morphological stability requirements. If the target moves frequently but deforms little in the application scenario (e.g., street vehicles), a lower threshold (e.g., 0.6-0.7) is set so that slight positional displacement triggers labeling insertion. If the target's shape is highly variable but its displacement is slow (e.g., deformable gesture recognition), the threshold is increased to above 0.8 to ensure that morphological abrupt changes become the primary triggering condition. For example, in traffic monitoring, a low threshold (0.65) is used for newly appearing vehicles, while a high threshold (0.85) is used for newly appearing furniture in static scenes.

[0032] Specifically, when the first difference calculated by S2 exceeds a preset first threshold (indicating that the target in the current frame is significantly different from historical frames and is highly likely to be newly appearing), it is determined that the target needs to be labeled for the first time at time point i. The labeling insertion process then begins: based on the target's position data (especially the target's center point coordinates) in the generated first recognition result, the positioning reference point for the label content is determined, ensuring that the label (whether a box, line, or text) can be accurately attached to the core position of the target object; next, the corresponding identifier text (such as the target category name) and recommended label box boundary information (such as the default box size ratio) are retrieved from the preset label object set; then, in the current frame (frame i), with the positioning reference point as a reference, the initial label content composed of the retrieved label box boundaries and identifier text is overlaid and displayed, thus visually marking the newly recognized target object in that video frame.

[0033] For example, in a traffic monitoring scenario, S2 has identified the front half of a newly appearing car in frame 315 (the current frame) (high first degree of difference). Using the center point of the car's hood as the positioning reference point, it extracts the text identifier "car" and a rectangular bounding box matching the car's size from the "Urban Traffic Object Database," and overlays a rectangular bounding box containing the text "car" at that location, completing the initial annotation. Immediately afterwards, to preliminarily predict possible changes to the target in the next moment and determine whether the annotation needs immediate updating, the same target recognition process is immediately executed on the following frame (i.e., the frame after time point i, such as frame 316), generating a third recognition result (including morphological and positional data of the following frame) describing the target's state information in the following frame.

[0034] Then, based on the third identification result, a differential comparison is performed with the first identification result (i.e., the state baseline of frame i) to calculate the second difference degree. Its calculation logic is the same as that of the first difference degree in S2, that is, it includes binary features of the target position movement magnitude and whether the morphological structure has changed. This step essentially constructs a future prediction window with an extremely short delay to capture significant displacements or morphological changes that the newly labeled target may undergo after the current frame.

[0035] Furthermore, the timing of annotation insertion is precisely tied to the moment when the target significantly reappears (guaranteed by the difference threshold mechanism of S2), and a "quick check" of the next frame is initiated simultaneously, providing instantaneous basis for the dynamic update decision of S4. For example, in a smart home scenario where a cat appears, after the current frame i (clear cat face) is annotated (displaying the "cat" icon), the subsequent frame (displaying the cat's full body) is immediately identified. If the cat makes a large turn or jump in the subsequent frame (significant change in morphology resulting in morphological difference feature = 1), or moves rapidly from the center of the image to a corner (high positional difference feature value), the calculated second difference degree is likely to exceed the preset second threshold, which will directly trigger the annotation update operation of S4 (such as moving the annotation box position or updating the morphological label).

[0036] Conversely, if the cat only slightly turns its head (morphological difference feature = 0) or takes small steps in the following frame (positional difference feature value is below the second threshold), the annotation will not be updated immediately. It will be judged again as needed in the subsequent sequence (S4), which effectively avoids excessive annotation adjustment caused by slight changes in a single frame.

[0037] This design addresses two issues: first, the timeliness of labeling newly emerging targets, ensuring that targets are marked as soon as they become identifiable; and second, by rapidly assessing differences in subsequent frames, it provides a quantitative basis for decision-making regarding whether the labeling is stable in the next frame and whether it needs to immediately adapt to target changes. This builds the potential for dynamic adaptation from the initial labeling stage, reducing the risk of labeling misalignment caused by high-speed moving or rapidly deforming targets, and avoiding unnecessary computational overhead from updating every frame. The entire S3 step, in coordination with S2, completes a closed loop from confirming the emergence of a new target to labeling initialization and instantaneous future state checks.

[0038] In S4, based on significant changes in the target in subsequent frames, the initial annotation content inserted in S3 is dynamically adjusted or eliminated, and a continuous frame-by-frame tracking mechanism oriented towards the video endpoint is established. The value of the second preset threshold is set in stages according to the update sensitivity requirements: a low threshold (0.3-0.5) is used for high-speed motion scenes (vehicle tracking), where even slight displacement triggers annotation position updates; a high threshold (0.7-0.9) is used for low-speed scenes (indoor monitoring) to filter out minor movement interference caused by lighting fluctuations. For example, a low threshold (0.4) is set for driving vehicles in autonomous driving videos to ensure the annotation box follows the vehicle in real time when turning, while a high threshold (0.8) is used for furniture annotation updates in smart homes to avoid invalid updates caused by breathing movements.

[0039] Specifically, when the second difference calculated by S3 (reflecting the change in the target between the current frame i and the next frame i+1) exceeds the preset second threshold (indicating that the target's position or shape has changed significantly in the next frame), the annotation update process is initiated. This update consists of two operations: First, based on the target's position data in the third recognition result generated by S3 for the next frame, the positioning reference point of the initial annotation content is moved (i.e., the reference anchor points of the annotation box and text in the image are adjusted), so that the annotation box boundary and the labeled text are synchronously shifted to the new position of the target in the next frame as a whole, maintaining the spatial consistency between the annotation and the target entity; Second, when the third recognition result indicates that the target has left the identifiable range (such as moving out of the image or being completely occluded), the initial annotation content in the current frame is directly eliminated to avoid displaying invalid markers.

[0040] For example, in a traffic monitoring scenario, if S3 has already inserted a "sedan" label box for a newly appearing car in frame 315 (current frame i), and the subsequent frame (frame 316) shows the car accelerating towards the right edge of the screen, causing the second difference to exceed the threshold (significant positional displacement), then based on the new coordinates of the car's rear center point in frame 316, the label box and text as a whole are shifted to the right to the corresponding position. If the car completely leaves the screen in the subsequent frame 320, and the third recognition result cannot detect a valid target, then the label box for that frame is removed.

[0041] After completing the update or elimination operation of the next frame (i+1 frame), the processing focus is automatically shifted towards the end of the video: the original next frame (i+1 frame) is taken as the new "current frame", and the S1 to S4 process is repeated (obtaining its adjacent historical frames and the new next frame, identifying and matching, calculating the difference, and deciding whether to insert or update the annotation), thus forming an adaptive, rolling forward continuous annotation maintenance mechanism that covers the complete life cycle of the target from its appearance to its disappearance.

[0042] Furthermore, by using the second difference threshold as the sole quantitative basis for update decisions, computationally intensive annotation relocation operations are only performed when the target undergoes significant changes (displacement exceeding limits or drastic shape changes). When the target is in a stable state, redundant updates are skipped, significantly reducing computational overhead (e.g., reducing the number of GPU renderings). The introduction of the elimination mechanism precisely solves the persistent problem of invalid annotations remaining after the target disappears. For example, in a smart home cat monitoring scenario, S3 has already inserted a "cat" annotation box in the i-th frame (clear cat face). If the subsequent frame (i+1 frame) shows the cat jumping from the sofa to the ground, causing deformation and significant displacement (high second difference), the annotation box is immediately moved according to the new position of the cat's torso center point after landing. When the cat subsequently crawls under the bed and becomes completely invisible, the annotation is cleared.

[0043] Subsequently, the next frame (originally frame i+1) is automatically used as the new starting point to reposition its adjacent frames. If the cat's form reappears in a subsequent frame (e.g., frame i+100) (even if the position and form are different), a new round of annotation insertion may still be triggered as long as it matches the preset template and differs sufficiently from the previous valid frame. More importantly, the difference threshold provides adjustable update sensitivity: a low second threshold can be used for high-speed moving targets (e.g., fast vehicles on the street) to ensure that even slight displacement triggers the bounding box to follow; a high second threshold can be used for low-speed or static targets (e.g., indoor furniture) to suppress invalid updates caused by lighting jitter. This adaptive continuous tracking mechanism neither interrupts the tracking potential due to temporary target loss (e.g., temporary occlusion) (annotations can be reconstructed in subsequent frames) nor wastes resources when the target is in a stable state, ultimately achieving a balance between annotation accuracy, computational efficiency, and resource consumption.

[0044] In summary, the spatiotemporal continuity analysis window constructed based on three-frame slices, combined with a morphological template matching filtering mechanism, effectively solves the problem of misjudgment of new targets. By strictly determining the timing of an object's new appearance by comparing the positional / morphological differences between the current frame and historical frames (first difference degree), it avoids repeated annotation of continuously moving targets while ensuring timely identification of targets emerging from edges. Secondly, the dynamic update strategy of difference degree threshold control precisely optimizes resource allocation. Marker repositioning is only triggered when the displacement or deformation between the current and subsequent frames exceeds the threshold (e.g., the annotation box moves synchronously after a cat jumps), while in a stable state... (For example, when a pet cat takes small steps) invalid updates are skipped. Combined with the automatic elimination mechanism when the target disappears, this reduces most of the repetitive computation load in long videos (especially reducing GPU rendering pressure). Finally, the closed-loop rolling advancement mechanism formed by the whole process (taking the updated next frame as the new current frame) achieves on-demand scheduling of computing resources while maintaining spatiotemporal consistency of annotation. In high-speed scenes, the low threshold ensures that micro-movements are updated (the annotation box follows the vehicle in real time when it turns), and in static scenes, the high threshold filters jitter interference (indoor furniture avoids flickering). To a certain extent, this solves the problems of annotation lag, drift accumulation and performance bottleneck caused by full-frame processing in existing methods.

[0045] like Figure 6 As shown, in one embodiment, S2 involves performing target recognition on the current frame at the i-th time point and generating a first recognition result, which includes: S21. Based on the boundary contour of the target in the current frame, generate contour information containing the target's morphological data and position data, and define the contour information as the first recognition result. The morphological data includes the target's contour geometric features, and the position data includes the coordinates of the target's center point in the current frame plane.

[0046] In this embodiment, it should be noted that in S21, in the current frame at time point i, the significant geometric features and position coordinates of the target are extracted using boundary contour recognition technology. Specifically, a geometric structure description is constructed based on the target's contour edge point set, and its center point coordinates on the image plane are calculated. For example, when processing street surveillance video, polygon fitting is performed on the hood contour of a car in the current frame, its corner features are extracted as morphological data, and the center point coordinates of the car body are used as the position reference.

[0047] It should also be noted that the generation of the second and third recognition results can be carried out in the same way as the generation of the first recognition result, only the object of data processing changes accordingly.

[0048] like Figure 6 As shown, in one embodiment, step S2, if the first identification result matches a preset labeled object set, includes: S22. Compare and match the morphological data in the first recognition result with the reference morphological templates stored in the preset labeled object set; S23. When the morphological data in the first recognition result meets the similarity requirement with any reference morphological template, it is determined that the first recognition result matches the preset annotation object set.

[0049] In this embodiment, it should be noted that in S22, the target shape data generated in S21 is compared geometrically with the reference template in the preset labeled object set. This matching process evaluates similarity through the alignment of contour feature points. For example, in a smart home scenario, the overlap between the angle ratio of the cat's face and ear tips and the cat's contour features in the template library is calculated. The subsequent process is triggered only when the geometric fit meets the preset requirements.

[0050] In S23, similarity is set as the decision boundary. If the feature overlap rate between the current target contour features (such as the tilt of a car windshield) and any reference template (such as the side profile of a standard vehicle) exceeds a threshold, the match is considered successful. For example, in pet monitoring, if the ear spacing and nose bridge ratio of a cat's face contour deviate from the "specific cat breed template" by less than the tolerance value, it is confirmed as a valid target.

[0051] like Figure 7 As shown, in one embodiment, obtaining the first difference degree based on the second identification result and the first identification result in S2 includes: S24. Compare the target's location data in the second identification result with the target's location data in the first identification result, and generate location difference features; S25. Compare the morphological data of the target in the second recognition result with the morphological data in the first recognition result, and generate morphological difference features; S26. Add the positional difference feature to the morphological difference feature and obtain the first difference degree.

[0052] In this embodiment, it should be noted that in S24, the offset of the target center point coordinates is compared between the historical frame and the current frame. The influence of resolution difference is eliminated by calculating the relative displacement ratio of the center point between the two frames. For example, in a traffic scene, the offset of the vehicle headlight position in the historical frame and the center of the vehicle front in the current frame is represented by the percentage value of the maximum diagonal of the screen. Specifically, the movement distance of the x-axis and y-axis coordinate points of the position data in the first and second recognition results is used as the offset, and then the offset is divided by the distance of the diagonal of the current frame to obtain the position difference feature.

[0053] In S25, the contour geometry of the target in two frames is binarized for judgment: if the core target remains unchanged, such as the number / distribution of car wheels not changing substantially, the morphological difference feature is marked as 0; if the target changes, such as a cat changing from curled up to stretched out, causing a change in the contour topology, it is marked as 1.

[0054] In S26, the position difference ratio is linearly superimposed with the binary morphological features to generate the first difference degree. For example, the blurred headlight outline of the historical frame (morphological difference = 1) and the offset of the complete front of the vehicle in the current frame (position difference 0.6) are combined to generate a difference degree of 1.6. When this value exceeds a threshold (e.g., 1.0), the annotation insertion mechanism is triggered. The difference degree synthesis module realizes the coupling effect of displacement amplitude and morphological change in decision-making.

[0055] It should also be noted that the second degree of difference can be obtained in the same way as the first degree of difference, only the object of data processing changes accordingly.

[0056] like Figure 8 As shown, in one implementation, inserting initial annotation content into the current frame at the i-th time point in S3 includes: S31. Based on the target's location data in the first identification result, determine the positioning reference point of the labeled content; S32. Extract the identifier text and annotation box boundaries of the matching target from the preset annotation object set; S33. Display the initial annotation content consisting of the annotation box boundary and the label text overlaid at the positioning reference point.

[0057] In this embodiment, it should be noted that in S31, the core point for attaching the annotation content is determined based on the target location data (especially the center point coordinates) generated in S21. For example, in a traffic scenario, the anchor point of the annotation frame is set according to the center point coordinates of the car hood to ensure that the annotation frame can move synchronously with the target and maintain spatial consistency.

[0058] In step S32, the system retrieves the label text associated with the current matching target and the recommended label box size from a preset set of labeled objects. For example, in a smart home scenario, after identifying a cat, the system extracts the text label "pet cat" and the bounding box parameters suitable for the cat's body size from the library.

[0059] In S33, using the positioning reference point set in S31 as a reference, a visual marker combining a label box and identification text is overlaid and displayed on the current frame. For example, in an autonomous driving video, the "car" label and a rectangle are overlaid on the center of the car's hood, completing the initial visual annotation of the new target.

[0060] like Figure 9 As shown, in one embodiment, updating the initial annotation content based on the third recognition result in S4 includes: S41. Based on the location data of the third recognition result, move the positioning reference point of the initial annotation content so that the boundary of the annotation box and the label text are synchronously moved to the new target position. S42. When the third recognition result indicates that the target is out of the identifiable range, the initial annotation content in the current frame of the video is removed.

[0061] In this embodiment, it should be noted that in S41, when the second difference generated in S3 exceeds the threshold, the original annotation reference point is moved according to the new target position data in the next frame (e.g., frame 316). For example, after a cat jumps from the sofa to the ground, the entire annotation box is shifted according to the center point coordinates of the landing posture to maintain the synchronization between the annotation and the entity's position.

[0062] In S42, when the third recognition result indicates that the target has moved out of the frame or is severely occluded, the annotation content of the current frame is directly cleared. For example, in traffic videos, the corresponding annotation box is removed immediately after a car leaves the frame to avoid invalid information interfering with the user's judgment and to reduce the rendering computation load.

[0063] A video annotation system based on continuous insertion is also provided, the system including: The acquisition module is used to acquire the target video and a set of preset labeled objects, and to acquire the corresponding timeline based on the target video, and to acquire the current frame of the target video at the i-th time point of the timeline, as well as the historical frames and subsequent frames adjacent to the current frame; The data processing module is used to perform target recognition on the current frame at the i-th time point and generate a first recognition result. If the first recognition result matches the preset labeled object set, the module performs target recognition on the historical frame at the i-th time point and generates a second recognition result. The module also obtains a first difference degree based on the second recognition result and the first recognition result. The first annotation module is used to insert initial annotation content into the current frame at the i-th time point when the first difference exceeds the first preset threshold, and to perform target recognition on the subsequent frame at the i-th time point and generate a third recognition result, and to obtain the second difference based on the third recognition result and the first recognition result. The second annotation module is used to update the initial annotation content according to the third recognition result when the second difference exceeds the second preset threshold, and continue to recognize and update the annotation frame by frame towards the target video endpoint.

[0064] In one embodiment, the data processing module is further configured to: generate contour information containing morphological data and position data of the target based on the boundary contour of the target in the current frame, and define the contour information as a first recognition result, wherein the morphological data includes the contour geometric features of the target, and the position data includes the coordinates of the target center point in the plane of the current frame.

[0065] In one implementation, the calling module is also used to: when there is no matching item in the cached tag library, match the data tag corresponding to the calling request with the full tag library, directly index the operation and maintenance data in the database from the full tag library and return it.

[0066] In this embodiment, it should be noted that the specific method of performing the operation in the above-mentioned video annotation system based on continuous insertion has been described in detail in the embodiments of the video annotation method based on continuous insertion, and will not be elaborated here.

[0067] Figure 10 This is a block diagram of an electronic device according to an exemplary embodiment of a video annotation method based on continuous insertion. Figure 10 As shown, the electronic device 700 may include: a processor 701 and a memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an I / O interface 704 (input / output interface), and a communication component 705.

[0068] The processor 701 controls the overall operation of the electronic device 700 to complete all or part of the steps in the aforementioned video annotation method based on continuous insertion. The memory 702 stores various types of data to support the operation of the electronic device 700. This data may include, for example, instructions for any application or method operating on the electronic device 700, and application-related data such as contact data, sent and received messages, pictures, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 703 may include a screen and audio components. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 702 or transmitted via communication component 705. The audio component also includes at least one speaker for outputting audio signals. I / O interface 704 provides an interface between processor 701 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or a combination thereof, is not limited here. Therefore, the corresponding communication component 705 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0069] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the aforementioned video annotation method based on continuous insertion.

[0070] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the video annotation method based on continuous insertion described above. For example, the computer-readable storage medium may be the memory 702 including the program instructions described above, which may be executed by the processor 701 of the electronic device 700 to complete the video annotation method based on continuous insertion described above.

[0071] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described video annotation method based on continuous insertion when executed by the programmable device.

[0072] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0073] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0074] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A video annotation method based on continuous insertion, characterized in that, include: Obtain the target video and a set of preset labeled objects, and obtain the corresponding timeline based on the target video. Obtain the current frame of the target video at the i-th time point on the timeline, as well as the historical frames and subsequent frames adjacent to the current frame. For the current frame at the i-th time point, target recognition is performed and a first recognition result is generated. If the first recognition result matches the preset labeled object set, then target recognition is performed on the historical frame at the i-th time point and a second recognition result is generated. The first difference degree is obtained based on the second recognition result and the first recognition result. If the first difference exceeds the first preset threshold, the initial annotation content is inserted into the current frame at the i-th time point, and the target is identified in the subsequent frame at the i-th time point and a third identification result is generated. The second difference is obtained based on the third identification result and the first identification result. If the second difference exceeds the second preset threshold, the initial annotation content is updated according to the third recognition result, and the annotation is updated frame by frame towards the target video endpoint. The second and third identification results are generated using the same target identification method as the first identification result, with only the identified video frames being changed to the historical frame and the subsequent frame, respectively. The second difference degree is obtained using the same difference degree acquisition method as the first difference degree, and only the recognition result involved in data processing is changed to the third recognition result and the first recognition result; When the second difference does not exceed the second preset threshold, the initial annotation content is not updated immediately, and the judgment continues in subsequent frames; The step of updating the initial annotation content based on the third recognition result further includes: when the third recognition result indicates that the target is out of the identifiable range, the initial annotation content in the current frame of the video is eliminated; After completing the annotation update judgment for the following frame, the following frame is taken as the new current frame, and the historical frames adjacent to the new current frame and the new following frame are obtained. The above steps are repeated until the target video ends.

2. The video annotation method based on continuous insertion according to claim 1, characterized in that, The step of performing target recognition on the current frame at the i-th time point and generating the first recognition result includes: Based on the boundary contour of the target in the current frame, contour information containing the target's morphological data and position data is generated, and the contour information is defined as the first recognition result. The morphological data includes the geometric structural features of the target's contour, and the position data includes the coordinates of the target's center point in the plane of the current frame.

3. The video annotation method based on continuous insertion according to claim 1, characterized in that, The condition that the first identification result matches the preset labeled object set includes: The morphological data in the first recognition result is compared and matched with the reference morphological templates stored in the preset labeled object set; When the morphological data in the first recognition result meets the similarity requirement with any reference morphological template, it is determined that the first recognition result matches the preset labeled object set.

4. The video annotation method based on continuous insertion according to claim 1, characterized in that, The step of obtaining the first difference degree based on the second identification result and the first identification result includes: Compare the target's location data in the second identification result with those in the first identification result to generate location difference features; By comparing the morphological data of the target in the second recognition result with that in the first recognition result, morphological difference features are generated; The positional difference features are combined with the morphological difference features to obtain the first difference degree.

5. The video annotation method based on continuous insertion according to claim 1, characterized in that, The insertion of initial annotation content into the current frame at the i-th time point includes: Based on the target's location data in the first identification result, determine the positioning reference point for the labeled content; Extract the identifier text and annotation box boundaries of the matching target from the preset set of labeled objects; The initial annotation content, consisting of the annotation box boundary and the label text, is overlaid at the positioning reference point.

6. The video annotation method based on continuous insertion according to claim 5, characterized in that, The step of updating the initial annotation content based on the third identification result includes: Based on the location data of the third identification result, the positioning reference point of the initial annotation content is moved, so that the boundary of the annotation box and the label text are synchronously shifted to the new target position.

7. A video annotation system based on continuous insertion, characterized in that, The system is used to implement the video annotation method based on continuous insertion as described in any one of claims 1 to 6, the system comprising: The acquisition module is used to acquire the target video and a set of preset labeled objects, and to acquire the corresponding timeline based on the target video, and to acquire the current frame of the target video at the i-th time point of the timeline, as well as the historical frames and subsequent frames adjacent to the current frame; The data processing module is used to perform target recognition on the current frame at the i-th time point and generate a first recognition result. If the first recognition result matches the preset labeled object set, the module performs target recognition on the historical frame at the i-th time point and generates a second recognition result. The module also obtains a first difference degree based on the second recognition result and the first recognition result. The first annotation module is used to insert initial annotation content into the current frame at the i-th time point when the first difference exceeds the first preset threshold, and to perform target recognition on the subsequent frame at the i-th time point and generate a third recognition result, and to obtain the second difference based on the third recognition result and the first recognition result. The second annotation module is used to update the initial annotation content according to the third recognition result when the second difference exceeds the second preset threshold, and continue to recognize and update the annotation frame by frame towards the target video endpoint.

8. The video annotation system based on continuous insertion according to claim 7, characterized in that, The data processing module is also used for: Based on the boundary contour of the target in the current frame, contour information containing the target's morphological data and position data is generated, and the contour information is defined as the first recognition result. The morphological data includes the geometric structural features of the target's contour, and the position data includes the coordinates of the target's center point in the plane of the current frame.

9. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the video annotation method based on continuous insertion as described in any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the video annotation method based on continuous insertion as described in any one of claims 1 to 6.

Images