Video labeling method, electronic device, and storage medium

Abstract

This application provides a video annotation method, electronic device, and storage medium. The method includes: acquiring the original video and user-annotated information for the first frame of the original video; generating an initial contour mask based on the SAM3 model and the first frame and its annotation information; determining whether the number of frames in the original video exceeds a preset frame count threshold; if so, dividing the original video into multiple sub-videos; performing temporal propagation annotation on each sub-video sequentially based on the initial contour mask to obtain the initial contour mask for each frame in the original video; and generating the annotation result of the original video based on the original video and the initial contour masks for each frame in the original video. This application generates the initial contour mask based on the SAM3 model, requiring only minimal human interaction to automatically locate the target and identify its state, significantly reducing the workload of manual frame-by-frame annotation and lowering labor costs.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and more specifically, to a video annotation method, an electronic device, and a storage medium. Background Technology

[0002] With the rapid popularization of industrial visual inspection and intelligent manufacturing, visual models such as YOLO are widely used for factory equipment identification, safety monitoring, and equipment status inspection. Training these models requires massive amounts of high-precision labeled data. Industrial scenarios contain a large number of non-standard special equipment, self-made tooling, and irregularly shaped parts. These targets are difficult to accurately describe with natural language, and the on-site monitoring videos are mostly long-term time-series videos, which place strict requirements on automatic labeling efficiency, segmentation accuracy, long video processing, and recognition of minute states.

[0003] Existing annotation methods are divided into two categories: manual annotation and intelligent automatic annotation. Manual annotation typically relies on manually selecting targets frame by frame, manually assigning labels, and then training a model after accumulating samples. Existing automatic annotation methods generate pre-annotated results based on multimodal large models, cue words, and semantic understanding.

[0004] However, manual annotation is time-consuming, labor-intensive, and costly. Long-term operation can easily lead to mis-annotation or omissions, and manual contour selection has poor accuracy. Multimodal semantic annotation schemes have poor adaptability to non-standard special equipment and are difficult to output reliable annotations stably. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a video annotation method, electronic device, and storage medium to solve the problems of high time cost and poor accuracy in the prior art.

[0006] To achieve the above objectives, the technical solution adopted in this application is as follows: Firstly, this application provides a video annotation method, the method comprising: Obtain the original video and the user's annotation information for the first frame of the original video. The annotation information includes forward annotation information and reverse annotation information. Based on the SAM3 model, an initial contour mask is generated according to the first frame and its annotation information. Determine whether the number of frames in the original video exceeds a preset frame threshold. If so, divide the original video into multiple sub-videos, with each adjacent sub-video having a preset number of overlapping frames. Based on the initial contour mask, temporal propagation annotation is performed on each sub-video sequentially to obtain the initial contour mask of each frame in the original video. The annotation result of the original video is generated based on the original video and the initial contour mask of each frame in the original video.

[0007] Optionally, the step of sequentially performing temporal propagation annotation on each sub-video according to the initial contour mask to obtain the initial contour mask of each frame in the original video includes: Using the initial contour mask as the reference mask for the first sub-video, the first sub-video is subjected to temporal propagation annotation to obtain the initial contour mask for each frame in the first sub-video. The initial contour mask of the last preset number of frames in the first sub-video is used as the reference mask of the second sub-video. The second sub-video is then time-propagated and labeled according to the reference mask to obtain the initial contour mask of each frame of the second sub-video. This process is repeated until the initial contour mask of all frames in the original video is obtained.

[0008] Optionally, after obtaining the initial contour mask of each frame of the second sub-video, the method further includes: releasing the video memory occupied by the first sub-video.

[0009] Optionally, generating the annotation result of the original video based on the original video and the initial contour mask of each frame in the original video includes: Based on the original video and the initial contour mask of each frame, determine the processed frame corresponding to each frame; Each processed frame and a pre-set classification sample are input into the feature extraction model to obtain the state feature vector corresponding to each processed frame. The classification sample includes a preset number of images corresponding to each state and a state label corresponding to each image. The state feature vectors corresponding to each processed frame are input into the classification model to obtain the annotation results of the original video. The annotation results include the initial contour mask, device status label and confidence score of each frame.

[0010] Optionally, determining the processed frame corresponding to each frame based on the original video and the initial contour mask of each frame includes: Noise filtering is performed on the initial contour mask of each frame to obtain the target contour mask of each frame. Determine the bounding contour of the target contour mask for each frame; The original video is cropped based on the outer bounding contour of the target contour mask to obtain the processed frames corresponding to each frame. The pixels included in the processed frames are the pixels within the outer bounding contour range of each frame.

[0011] Optionally, each processed frame and pre-set classification samples are input into a feature extraction model to obtain the state feature vector corresponding to each processed frame, including: Each processed frame and a pre-set classification sample are input into the feature extraction model, and the feature extraction model outputs the initial vector corresponding to each processed frame. The initial vector is standardized to obtain the state feature vector.

[0012] Optionally, the step of inputting the state feature vectors corresponding to each processed frame into the classification model to obtain the annotation results of the original video includes: The state feature vectors corresponding to each processed frame are input into the classification model, and the classification model outputs the classification results corresponding to each processed frame. According to the time sequence and the preset sliding window length, the classification results corresponding to each processed frame are voted on by a sliding window to obtain intermediate labeling results; Based on the preset state hysteresis threshold strategy, the annotation result of the original video is determined according to the confidence level of each frame in the intermediate annotation results.

[0013] Optionally, after generating the initial contour mask, the method further includes: In the human-computer interaction interface, the initial contour mask is displayed on the first frame.

[0014] Secondly, this application provides a video annotation device, the device comprising: The acquisition module is used to acquire the original video and the user's annotation information for the first frame of the original video. The annotation information includes forward annotation information and reverse annotation information. The generation module is used to generate an initial contour mask based on the SAM3 model, according to the first frame and the annotation information of the first frame; The judgment module is used to determine whether the number of frames of the original video exceeds a preset frame number threshold. If so, the original video is divided into multiple sub-videos, and each adjacent sub-video has a preset number of overlapping frames. The annotation module is used to sequentially perform temporal propagation annotation on each sub-video according to the initial contour mask to obtain the initial contour mask of each frame in the original video. The output module is used to generate the annotation result of the original video based on the original video and the initial contour mask of each frame in the original video.

[0015] Thirdly, this application provides an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the video annotation method described above.

[0016] Fourthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the video annotation method described above.

[0017] The beneficial effects of this application are as follows: By acquiring the positive and negative annotation information of the first frame and generating an initial contour mask based on the SAM3 model, the initial target localization can be completed with only a small amount of manual interaction, significantly reducing the workload of manual frame-by-frame annotation, lowering labor costs, and avoiding mislabeling and omissions caused by long-term operations. Utilizing the precise segmentation capabilities of the SAM3 model, the annotation accuracy of the target contour is improved, requiring no prompts and adapting to scenarios such as the identification of non-standard special equipment. Through long video segmentation and the design of overlapping adjacent sub-videos, peak memory usage is controlled, achieving stable annotation of long videos. Furthermore, through temporal propagation annotation, automated annotation of the original video is achieved, significantly improving annotation efficiency. Finally, structured annotation results are generated, providing reliable data support for subsequent model training or equipment status recognition, adapting to the actual needs of industrial scenarios. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a video annotation method provided in an embodiment of this application; Figure 2 This is a schematic diagram of annotation information provided in an embodiment of this application; Figure 3 This is a schematic diagram of a process for generating annotation results of the original video according to an embodiment of this application; Figure 4 This is a flowchart illustrating how to determine the processed frame corresponding to each frame, as provided in an embodiment of this application. Figure 5 This is a schematic diagram of a process for obtaining annotation results of the original video according to an embodiment of this application; Figure 6 This is a schematic diagram of the system architecture of a video annotation system provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a video annotation device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0021] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0023] Existing annotation methods are divided into two categories: manual annotation and intelligent automatic annotation. However, manual annotation is time-consuming, labor-intensive, and costly. Long-term operation is prone to errors and omissions, and manual contour selection has poor accuracy. Multimodal semantic annotation schemes have poor adaptability to non-standard special equipment and are difficult to output reliable annotations stably.

[0024] Based on this, this application generates an initial contour mask based on the Segment Anything Model 3 (SAM3) model and the user's annotation information on the first frame of the original video. It also reduces the workload of manual frame-by-frame annotation based on the temporal propagation and long video segmentation relay mechanism. Furthermore, it uses the SAM3 model to accurately generate the target contour mask without the need for prompt words, thereby improving the standard accuracy and making it suitable for non-standard special equipment scenarios.

[0025] Figure 1 This is a flowchart illustrating a video annotation method provided in an embodiment of this application. Next, refer to... Figure 1 The process of video annotation methods will be introduced.

[0026] S101. Obtain the original video and the user's annotation information for the first frame of the original video. The annotation information includes forward annotation information and reverse annotation information.

[0027] Optionally, the original video may be related to industrial scenes, frame sets, and files, which can be uploaded to the system by the front end or a script.

[0028] Specifically, the system first completes session creation and data access, receiving the original video uploaded by the user through the front-end or script interface, while initializing the session and maintaining key session-related data, such as session identifier and global inference state.

[0029] As an alternative implementation method, users can positively select a specified target and negatively select an exclusion area on the first frame of the original video by using a box selection or a point selection based on the prompts on the human-computer interaction interface. Figure 2 This is a schematic diagram illustrating an annotation information provided in an embodiment of this application. For example... Figure 2 As shown, the user selects a welder's cape by clicking, where green stars represent positive annotation information, red stars represent negative annotation information, and the red box is the target's bounding box. Positive and negative annotation information are used to achieve accurate separation between the target and the background.

[0030] As an alternative implementation, users can also annotate via a HyperText Transfer Protocol (HTTP) request.

[0031] Optionally, the system writes the annotation information to the current session in real time, while the SAM3 service provides feedback to the user on the target selected by the user in the form of a highlight mask, which is used to confirm the annotation information.

[0032] S102. Based on the SAM3 model, generate an initial contour mask according to the first frame and its annotation information.

[0033] Specifically, the system first performs a model loading step, loading the SAM3 model and completing the inference environment initialization. After initialization, the first frame image and the user-confirmed annotation information are input into the SAM3 model. The model accurately identifies the annotation intent through multimodal resolution capabilities, eliminates interference from reverse annotation regions, extracts the target contour, and generates an initial contour mask.

[0034] After generation, the initial contour mask is highlighted and displayed to the user in the human-computer interaction interface for the user to confirm the accuracy of the annotation. At the same time, the initial contour mask is written to the current session, completing the generation and storage of the initial annotation result.

[0035] S103. Determine whether the number of frames in the original video exceeds the preset frame threshold. If so, divide the original video into multiple sub-videos, with each adjacent sub-video having a preset number of overlapping frames.

[0036] Specifically, after creating a session and accessing the original video, the number of frames in the original video is counted and compared with a preset frame count threshold. This preset frame count threshold can be flexibly adjusted according to the performance of the computing device. If the number of frames in the original video exceeds the preset frame count threshold, the original video is segmented, and the global frame range of each segment is recorded to ensure that there is a preset number of overlapping frames between adjacent segments. For example, the preset frame count threshold is 3000 frames, and the preset number is, for example, 10 frames.

[0037] After segmentation is completed, the system writes the segment information into the session, synchronously records the initial state of each segment, and maintains the relevant data of the segment through session state storage.

[0038] Optionally, if the number of frames in the original video does not exceed a preset frame rate threshold, there is no need to segment the original video. Instead, the original video can be treated as a sub-video and the following steps can be performed.

[0039] S104. Based on the initial contour mask, perform temporal propagation annotation on each sub-video sequentially to obtain the initial contour mask of each frame in the original video.

[0040] Optionally, the initial contour mask is used as the reference mask for the first sub-video. The initial contour mask is automatically transferred and matched between frames of the sub-video to accurately track the target's motion trajectory and generate the initial contour mask for all frames of the sub-video. For subsequent sub-videos, the initial contour mask of the last preset number of overlapping frames of the previous sub-video is extracted and used as the reference mask for the current sub-video. Automatic annotation and temporal propagation are performed without user interaction.

[0041] After each sub-video is annotated, the system can release the video memory occupied by that sub-video, further controlling the peak video memory usage.

[0042] Throughout the propagation process, the system updates the initial contour mask of each frame in real time through session state storage, while simultaneously tracking the propagation progress to ensure a stable propagation process until the mask generation of each frame of the entire original video is completed.

[0043] S105. Generate the annotation results of the original video based on the original video and the initial contour mask of each frame in the original video.

[0044] Specifically, the initial contour mask is processed, and the bounding box and rotated bounding box corresponding to the target are generated simultaneously, ensuring that the initial contour mask, bounding box, and rotated bounding box of the same object in the same frame maintain index alignment. The bounding box and rotated bounding box are used to represent the boundary position corresponding to the target.

[0045] Based on the initial contour mask, bounding box, and rotated bounding box of each frame, the processed frames are determined, and then the state feature vectors of each processed frame are generated. These state feature vectors are then input into the classification model to obtain the annotation results of the original video. The annotation results can include the frames of the original video, the initial contour mask, the bounding box, the rotated bounding box, and the device status label of the target.

[0046] Optionally, the annotation results can be output via query and export functions as needed, simultaneously outputting model, session, and device operating status information. The annotation results can be exported in JSON, PNG, or MP4 format.

[0047] The annotation results can be directly used for YOLO (You Only Look Once) model training.

[0048] In this embodiment, by acquiring the positive and negative annotation information of the first frame and generating an initial contour mask based on the SAM3 model, the initial target localization can be completed with only a small amount of manual interaction, significantly reducing the workload of manual frame-by-frame annotation, lowering labor costs, and avoiding mislabeling and omissions caused by long-term operations. Utilizing the precise segmentation capabilities of the SAM3 model improves the annotation accuracy of the target contour, eliminating the need for prompts and adapting to scenarios such as the identification of non-standard special equipment. Through long video segmentation and the design of overlapping adjacent sub-videos, peak memory usage is controlled, achieving stable annotation of long videos. Furthermore, through time-series propagation annotation, automated annotation of the original video is achieved, significantly improving annotation efficiency. Finally, structured annotation results are generated, providing reliable data support for subsequent model training or equipment status recognition, adapting to the actual needs of industrial scenarios.

[0049] Next, we will introduce the specific process of performing temporal propagation annotation on each sub-video according to the initial contour mask in step S104 above to obtain the initial contour mask of each frame in the original video.

[0050] Optionally, the initial contour mask is used as the reference mask for the first sub-video, and temporal propagation annotation is performed on the first sub-video to obtain the initial contour mask for each frame in the first sub-video.

[0051] Optionally, after completing the original video segmentation, the system uses the initial contour mask generated by the SAM3 model as the reference mask for the first sub-video, synchronously transmits the initial contour mask, accurately tracks the motion trajectory and contour changes of the target in each frame of the first sub-video, and automatically generates the initial contour mask for all frames of the sub-video.

[0052] Optionally, the initial contour mask of the last preset number of frames in the first sub-video is used as the reference mask of the second sub-video, and the second sub-video is time-propagated and labeled according to the reference mask to obtain the initial contour mask of each frame of the second sub-video. This process is repeated until the initial contour mask of all frames in the original video is obtained.

[0053] Optionally, after the temporal propagation annotation of the first sub-video is completed, the system automatically extracts the initial contour masks of the last preset number of overlapping frames of that sub-video and uses them as the reference mask for the second sub-video. Based on this reference mask, temporal propagation annotation is performed on the second sub-video, in the same process as the first sub-video, automatically transferring the mask and target boundary information to generate the initial contour masks for each frame of that sub-video. This logic is followed to process all subsequent sub-videos until the initial contour masks for all frames of the entire original video are generated.

[0054] In this embodiment, the initial contour mask and the overlapping frame reference mask are used as a basis to achieve seamless connection of temporal propagation annotation of each sub-video, ensuring the continuity and accuracy of the annotation of the whole video, without requiring users to interact segment by segment, and further improving the efficiency of automated annotation of long videos.

[0055] Optionally, after obtaining the initial contour mask of each frame of the second sub-video, the video memory occupied by the first sub-video is released.

[0056] Specifically, after the system completes the generation of the initial contour masks for each frame of the second sub-video, it determines that the annotation task for the first sub-video is complete. Only the initial contour masks of the last preset number of overlapping frames are retained for subsequent sub-videos; no other data for this sub-video needs to be retained. The system automatically clears the non-overlapping frame data, initial contour mask cache, and related temporary session data corresponding to the first sub-video, releasing the GPU from its memory usage. The freed-up GPU memory can be directly used for time-series propagation annotation and other computational consumption in subsequent sub-videos. The entire process is automatically triggered and executed by the system without any manual user intervention.

[0057] Subsequent sub-videos will follow the same process as the second sub-video in terms of memory release.

[0058] Figure 3 This is a schematic diagram illustrating a process for generating annotation results for original videos, provided in an embodiment of this application. For example... Figure 3 As shown, the specific steps in step S105 above to generate the annotation result of the original video based on the original video and the initial contour mask of each frame in the original video are as follows: S301. Based on the original video and the initial contour mask of each frame, determine the corresponding processed frame for each frame.

[0059] Specifically, noise filtering is first applied to the initial contour mask of each frame, removing simply connected regions with fewer than 32 pixels and retaining the main connected regions to obtain the target contour mask. Then, the circumscribed contour of the target contour mask is determined. Subsequently, based on this circumscribed contour, the corresponding frames of the original video are cropped and uniformly scaled to 200×200 to obtain the processed frames for each frame. The processed frames can retain target features to the maximum extent and remove background interference.

[0060] S302. Input each processed frame and the pre-set classification samples into the feature extraction model to obtain the state feature vector corresponding to each processed frame. The classification samples include a preset number of images corresponding to each state and the state label corresponding to each image.

[0061] Specifically, classification samples are prepared in advance, including a preset number of images corresponding to each device state and a state label for each image. For example, the classification samples may include 20 images of the switch being turned on and 20 images of the switch being turned off.

[0062] Optionally, each processed frame and the classification samples are input together into a predefined feature extraction model. The feature extraction model can be a mid-to-late stage network structure of ConvNeXt-Tiny. The feature extraction model only outputs a 1024-dimensional initial vector and does not directly output the classification result.

[0063] The initial vector is then L2 normalized, and then z-score normalization based on training set statistics is performed to obtain the state feature vectors corresponding to each processed frame, ensuring the consistency and comparability of the feature vectors.

[0064] S303. Input the state feature vectors corresponding to each processed frame into the classification model to obtain the annotation results of the original video. The annotation results include the initial contour mask, device status label and confidence score of each frame.

[0065] Specifically, the standardized state feature vector is input into the classification model. The classification model can be a binary support vector machine (SVM).

[0066] The classification model outputs the device status label and corresponding confidence score for each processed frame based on the preset kernel function and parameters. Specifically, the model first performs discrimination according to the preset kernel function strategy, prioritizing the use of a linear kernel. Simultaneously, it searches for the optimal linear kernel parameter C within the range of 0.1, 1, 10, 50, and 100. If the linear kernel cannot obtain a sufficiently stable classification boundary, it switches to a radial basis function (RBF) kernel. In this case, the parameters are preferably selected within the range of C∈{1,10,50,100} and gamma∈{1e-4,5e-4,1e-3,5e-3,1e-2} to ensure classification accuracy.

[0067] The system associates the status label and confidence level with the initial contour mask of each frame, integrates them to form the complete annotation result of the original video, and writes the annotation result into the session for storage, so as to facilitate subsequent query and export.

[0068] In this embodiment, automatic identification of device status is achieved through post-frame extraction, feature vector generation, and classification, supplementing the shortcomings of simply annotating contours. Simultaneously, the feature extraction model only extracts and outputs state feature vectors, improving the accuracy of subtle state identification and resolving misjudgments when different state vectors of the same device are close together and pure geometric features are insufficient, thus reducing the workload of manual state verification.

[0069] Figure 4 This is a flowchart illustrating how to determine the processed frame corresponding to each frame, as provided in an embodiment of this application. Figure 4 As shown, the process of determining the processed frame corresponding to each frame based on the original video and the initial contour mask of each frame in step S301 above will be described next.

[0070] S401. Noise filtering is performed on the initial contour mask of each frame to obtain the target contour mask of each frame.

[0071] Specifically, a single-connected noise filtering operation is performed on the initial contour mask of each frame. Specifically, single-connected regions with fewer than a preset noise threshold are removed from the mask, retaining only the main connected regions that accurately represent the target contour. After this processing, the target contour mask for each frame is obtained. This target contour mask can accurately correspond to the target object, avoiding subsequent cropping deviations caused by noise.

[0072] S402. Determine the outer bounding contour of the target contour mask for each frame.

[0073] The circumscribed contour is the smallest closed contour surrounding the target contour mask.

[0074] Specifically, based on the noise-filtered target contour mask, a geometric calculation algorithm is used to generate the smallest closed contour that can completely enclose the target contour mask, i.e., the circumscribed contour. This contour can accurately define the spatial extent of the target object.

[0075] S403. Cropping the original video based on the outer bounding contour of the target contour mask to obtain the processed frames corresponding to each frame. The pixels included in the processed frames corresponding to each frame are the pixels within the outer bounding contour range of each frame.

[0076] Specifically, based on the bounding contour of each frame, the pixel region within the contour range is precisely cropped from each frame of the original video. Simultaneously, the cropped target region is uniformly scaled to a preset fixed size, ultimately yielding the processed frames corresponding to each frame. The original video frames can be 1920×1080, and the preset fixed size can be 200×200.

[0077] The processed frame contains only pixels within the bounded outline area, minimizing background interference and preserving the target's outline, color, and other features.

[0078] In this embodiment, noise filtering is used to optimize the mask quality and avoid interference from small, invalid areas. The target range is defined by the circumscribed contour, and the target area is accurately cropped and its size is standardized. This effectively removes background interference, preserves the core features of the target, and ensures the accuracy of subsequent processing. It is suitable for the target labeling and state recognition needs in complex backgrounds in industrial scenarios.

[0079] Next, we will introduce the process in step S302 of inputting each processed frame and the pre-set classification sample into the feature extraction model to obtain the state feature vector corresponding to each processed frame.

[0080] Optionally, each processed frame and a pre-set classification sample are input into the feature extraction model, and the feature extraction model outputs the initial vector corresponding to each processed frame.

[0081] Optionally, the initial vectors output by the feature extraction model for each processed frame can be 1024-dimensional. Not directly outputting the classification results ensures the purity of feature extraction, thus preserving complete feature information.

[0082] Optionally, the initial vector can be standardized to obtain the state feature vector.

[0083] Optionally, a two-step standardization process is performed on the initial vector output by the feature extraction model: First, L2 normalization is applied to the initial vector. By calculating the L2 norm of the initial vector, it is scaled to the unit norm range, eliminating differences in vector magnitude. Second, based on training set statistics, z-score standardization is applied to the L2-normalized vector to ensure that the feature vector follows a standard normal distribution, further optimizing feature consistency. After these two steps, the state feature vectors corresponding to each processed frame are obtained.

[0084] In this embodiment, an initial vector is obtained through a feature extraction model, and then standardized to eliminate numerical differences and unify the scale, so that the state feature vectors have consistency and comparability, avoiding classification bias caused by differences in the scale of the initial vectors, providing accurate input for subsequent classification model discrimination, and ensuring the accuracy and stability of device status recognition.

[0085] Figure 5 This is a schematic diagram illustrating a process for obtaining annotation results of the original video, provided in an embodiment of this application. For example... Figure 5 As shown, the specific steps in step S303 above, which involve inputting the state feature vectors corresponding to each processed frame into the classification model to obtain the annotation results of the original video, will be described next.

[0086] S501. Input the state feature vector corresponding to each processed frame into the classification model, and the classification model outputs the classification result corresponding to each processed frame.

[0087] Specifically, the state feature vectors corresponding to each processed frame are input into a preset classification model. The model calculates and judges based on preset parameters, and outputs the classification results corresponding to each processed frame, namely the initial device state label and corresponding confidence score of each frame. The initial device state label of each frame can be, for example, on state, off state, etc.

[0088] S502. According to the time sequence and the preset sliding window length, the classification results corresponding to each processed frame are voted on by sliding window to obtain intermediate annotation results.

[0089] The sliding window length is a preset range of the number of frames used to vote on the classification results of consecutive frames. The sliding window length can be set according to the video frame rate.

[0090] Specifically, according to the time sequence of each frame in the original video, a preset sliding window length is set, the window is slid sequentially, and the classification results of consecutive frames in each window are statistically voted. The initial device status label with the highest frequency of occurrence in the window and the corresponding average confidence are selected as the intermediate annotation results of each frame in that window, thus completing the preliminary calibration of the single frame result and reducing the impact of single frame misjudgment on the overall annotation.

[0091] S503. Based on the preset state hysteresis threshold strategy, determine the annotation result of the original video according to the confidence of each frame in the intermediate annotation result.

[0092] The preset state hysteresis threshold strategy can define a state hysteresis range, which can be set according to the state characteristics of industrial equipment.

[0093] Specifically, the confidence level of each frame in the intermediate annotation results is assessed: when the confidence level is higher than the upper limit of the state hysteresis range, the corresponding device state label is confirmed as the initial device state label; when the confidence level is lower than the lower limit of the state hysteresis range, the opposite label of the corresponding initial device state label is used as the device state label. For example, all device state labels include green and red; if the initial device state label is green, then the corresponding device state label is red. When the confidence level is within the state hysteresis range, the device state label of the previous frame is maintained to avoid frequent state switching between frames. After calibration using this strategy, the continuity and stability of state recognition are ensured.

[0094] In this embodiment, a preliminary classification result is output through a classification model. The impact of misjudgment in a single frame is reduced by sliding window voting. Then, the state hysteresis threshold strategy is used to stabilize the state of continuous frames, avoid frequent state switching, improve the stability and accuracy of device state recognition, ensure that the output original video annotation results are reliable, and adapt to the needs of video quality fluctuations and slight differences in target state in industrial scenarios.

[0095] Optionally, after generating the initial contour mask, the method further includes: displaying the initial contour mask on the first frame in the human-computer interaction interface.

[0096] Optionally, after generating the initial contour mask using the SAM3 model, the initial contour mask is highlighted and overlaid on the first frame image in the first frame display area of ​​the interface to present the contour range of the target object. For example, when annotating a welder's cape, the cape's contour is highlighted. Simultaneously, the user's forward annotations, reverse annotations, and automatically calculated minimum bounding boxes can also be displayed, allowing users to intuitively verify whether the initial contour mask accurately matches the target object.

[0097] If the user finds that the initial contour mask is incorrect, they can re-annotate the first frame through the human-computer interaction interface. The system will then regenerate the initial contour mask and update the display in real time until the user confirms that it is correct.

[0098] This display process does not require manual triggering by the user; it is completed automatically by the system. At the same time, the display status is synchronously written to the session, providing a basis for verification in subsequent annotation processes and ensuring the accuracy of the initial annotation results.

[0099] Optionally, Figure 6 This is a schematic diagram of the system architecture of a video annotation system provided in an embodiment of this application. For example... Figure 6 As shown, the video annotation system includes an input and calling side, an annotation service side, and an output and runtime observation side.

[0100] The input and calling side includes a data input module, which is used to receive the original video uploaded by the user, and at the same time receive the annotation information of the first frame of the original video by the user.

[0101] The annotation service side includes a model loading module, a session management module, a user prompt injection module, a temporal propagation module, and a geometric post-processing module. The model loading module loads various models and trackers, binds them to the target computing device, and completes the inference environment initialization. The session management module creates and maintains annotation sessions, records and stores key session data such as session_id, inference_state, segments, and video_segments, and manages session creation, operation, and closure. The user prompt injection module receives annotation information from the user on the first frame, passes the annotation intent to the SAM3 model, and writes the user's annotation information into the current session. The temporal propagation module automatically propagates the mask between video frames based on the initial contour mask. For long video segments, a relay method is used to achieve temporal propagation of each sub-video, ensuring the continuity of annotation throughout the video. The geometric post-processing module optimizes the initial contour mask, generating structured data such as bounding boxes and rotated bounding boxes corresponding to the target, ensuring that the masks, bounding boxes, and rotated bounding boxes of the same object in the same frame maintain index alignment, and performs classification processing based on the original video and the initial contour mask.

[0102] The runtime observation side includes a result export module and a runtime status monitoring module. The result export module outputs annotation results as needed, including annotation results for specified frames, original frames, mask images, overlay images, annotation files, and videos, while also outputting model, session, and device runtime status information. The runtime status monitoring module tracks the runtime status of the entire annotation process in real time, including model runtime status, session status, memory usage status, and time-series propagation progress, providing timely feedback on anomalies to ensure the stable execution of the annotation method.

[0103] Based on the same inventive concept, this application also provides a video annotation device corresponding to the video annotation method. Since the principle of the device in this application is similar to that of the video annotation method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0104] Reference Figure 7 The diagram shown is a structural schematic of a video annotation device provided in an embodiment of this application, wherein: The acquisition module 701 is used to acquire the original video and the annotation information of the user on the first frame of the original video. The annotation information includes forward annotation information and reverse annotation information. The generation module 702 is used to generate an initial contour mask based on the SAM3 model, according to the first frame and the annotation information of the first frame; The judgment module 703 is used to determine whether the number of frames of the original video exceeds a preset frame number threshold. If so, the original video is divided into multiple sub-videos, and each adjacent sub-video has a preset number of overlapping frames. The annotation module 704 is used to sequentially perform temporal propagation annotation on each sub-video according to the initial contour mask to obtain the initial contour mask of each frame in the original video. The output module 705 is used to generate the annotation result of the original video based on the original video and the initial contour mask of each frame in the original video.

[0105] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0106] This application also provides an electronic device, such as... Figure 8 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application, including a processor 801, a memory 802, and a bus. The memory 802 stores machine-readable instructions executable by the processor 801. When the computer device is running, the processor 801 and the memory 802 communicate via the bus, and the processor 801 executes the machine-readable instructions to perform the processing of the aforementioned video annotation method.

[0107] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the video annotation method described above.

[0108] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0109] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0110] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A video annotation method, characterized in that, The method includes: Obtain the original video and the user's annotation information for the first frame of the original video. The annotation information includes forward annotation information and reverse annotation information. Based on the SAM3 model, an initial contour mask is generated according to the first frame and its annotation information. Determine whether the number of frames in the original video exceeds a preset frame threshold. If so, divide the original video into multiple sub-videos, with each adjacent sub-video having a preset number of overlapping frames. Based on the initial contour mask, temporal propagation annotation is performed on each sub-video sequentially to obtain the initial contour mask of each frame in the original video. The annotation result of the original video is generated based on the original video and the initial contour mask of each frame in the original video.

2. The video annotation method according to claim 1, characterized in that, The step of sequentially performing temporal propagation annotation on each sub-video based on the initial contour mask to obtain the initial contour mask for each frame in the original video includes: Using the initial contour mask as the reference mask for the first sub-video, the first sub-video is subjected to temporal propagation annotation to obtain the initial contour mask for each frame in the first sub-video. The initial contour mask of the last preset number of frames in the first sub-video is used as the reference mask of the second sub-video. The second sub-video is then time-propagated and labeled according to the reference mask to obtain the initial contour mask of each frame of the second sub-video. This process is repeated until the initial contour mask of all frames in the original video is obtained.

3. The video annotation method according to claim 2, characterized in that, After obtaining the initial contour mask of each frame of the second sub-video, the method further includes: releasing the video memory occupied by the first sub-video.

4. The video annotation method according to claim 1, characterized in that, The step of generating the annotation result of the original video based on the original video and the initial contour mask of each frame in the original video includes: Based on the original video and the initial contour mask of each frame, determine the processed frame corresponding to each frame; Each processed frame and a pre-set classification sample are input into the feature extraction model to obtain the state feature vector corresponding to each processed frame. The classification sample includes a preset number of images corresponding to each state and a state label corresponding to each image. The state feature vectors corresponding to each processed frame are input into the classification model to obtain the annotation results of the original video. The annotation results include the initial contour mask, device status label and confidence score of each frame.

5. The video annotation method according to claim 4, characterized in that, The step of determining the processed frame corresponding to each frame based on the original video and the initial contour mask of each frame includes: Noise filtering is performed on the initial contour mask of each frame to obtain the target contour mask of each frame. Determine the bounding contour of the target contour mask for each frame; The original video is cropped based on the outer bounding contour of the target contour mask to obtain the processed frames corresponding to each frame. The pixels included in the processed frames are the pixels within the outer bounding contour range of each frame.

6. The video annotation method according to claim 4, characterized in that, Each processed frame and pre-set classification samples are input into the feature extraction model to obtain the state feature vector corresponding to each processed frame, including: Each processed frame and a pre-set classification sample are input into the feature extraction model, and the feature extraction model outputs the initial vector corresponding to each processed frame. The initial vector is standardized to obtain the state feature vector.

7. The video annotation method according to claim 4, characterized in that, The step of inputting the state feature vectors corresponding to each processed frame into the classification model to obtain the annotation results of the original video includes: The state feature vectors corresponding to each processed frame are input into the classification model, and the classification model outputs the classification results corresponding to each processed frame. According to the time sequence and the preset sliding window length, the classification results corresponding to each processed frame are voted on by a sliding window to obtain intermediate labeling results; Based on the preset state hysteresis threshold strategy, the annotation result of the original video is determined according to the confidence level of each frame in the intermediate annotation results.

8. The video annotation method according to claim 1, characterized in that, After generating the initial contour mask, the method further includes: In the human-computer interaction interface, the initial contour mask is displayed on the first frame.

9. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when the electronic device is running, are executed by the processor to perform the steps of the video annotation method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the video annotation method as described in any one of claims 1 to 8.

Images