Video stream object tracking and detection method, device, apparatus and storage medium

Abstract

The application relates to the field of artificial intelligence, and provides a video stream object tracking and detection method, device, equipment and storage medium. The method comprises the following steps: acquiring a video stream, and determining a semantic description of a target object to be tracked and detected; selecting a detection method combination and corresponding parameters according to a preset scene adaptive configuration; generating an initial bounding box of the target object in a video frame according to the selected detection method combination, and taking the initial bounding box as a prompt input into an open domain visual detection and segmentation model to perform pixel-level segmentation on the video frame, so as to obtain a first mask sequence corresponding to the target object; performing pixel-level depth estimation on the video frame to obtain a depth map, and filtering the first mask sequence to obtain a second mask sequence; and introducing an artificial correction mechanism to correct the second mask sequence, so as to generate a corrected second mask sequence. The technical scheme provided by the application improves the accuracy and robustness of video stream object tracking and detection, and expands the range of applicable scenes.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, device, and storage medium for tracking and detecting objects in a video stream. Background Technology

[0002] Object tracking and detection in video streams is one of the core tasks in computer vision, widely used in scenarios such as intelligent surveillance, autonomous driving, industrial quality inspection, and robot interaction. With the development of deep learning technology, object detection and segmentation models based on convolutional neural networks and Transformer architectures have made significant progress. However, in practical applications, video streams often face complex dynamic environments, including problems such as changes in lighting, target occlusion, scale changes, motion blur, and the coexistence of multiple targets.

[0003] In existing technologies, commonly used video object tracking and detection methods are mainly divided into two categories: one is traditional methods based on template matching or optical flow, which have poor robustness and are difficult to handle non-rigid deformations and abrupt changes in appearance; the other is end-to-end methods based on deep learning, such as YOLO and Mask R-CNN, which, although highly accurate, usually require a large amount of labeled data to train specific categories, have limited generalization ability, and cannot flexibly adapt to different detection requirements in different scenarios. Furthermore, most methods rely on only a single image feature, ignoring the important role of depth information in distinguishing foreground from background, leading to false positives and false negatives in complex backgrounds. At the same time, when automated processing results are unreliable, the lack of effective human intervention mechanisms makes the system unreliable in critical tasks. Summary of the Invention

[0004] Based on this, it is necessary to address the shortcomings of the existing technologies, such as limited generalization ability, poor scene adaptability, and lack of deep information fusion and human intervention mechanisms, and propose a video stream object tracking and detection method, device, equipment, and storage medium.

[0005] Firstly, a method for tracking and detecting objects in a video stream is provided, the method comprising: Acquire the video stream and determine the semantic description of the target object to be tracked and detected; Based on the preset scene adaptive configuration, select the combination of detection methods and corresponding parameters. The combination of detection methods includes a combination based on a multimodal large language model and an open-domain visual detection and segmentation model. Based on the selected combination of detection methods, the semantic description is input into a multimodal large language model to generate an initial bounding box of the target object in the video frame. The initial bounding box is then used as a prompt to input an open-domain visual detection and segmentation model to perform pixel-level segmentation on the video frame, thereby obtaining a first mask sequence corresponding to the target object. When the detection method combination further includes a combination of a depth estimation model and a rule model, the depth estimation model is used to perform pixel-level depth estimation on the video frame to obtain a depth map, and the depth information in the depth map and at least one of the area threshold, confidence threshold and depth range in the rule model are used to filter the first mask sequence to obtain a second mask sequence. When the confidence level of the automated processing result is lower than the quality threshold, at least one point annotation input by the user is received on a single frame of video. The point annotations are used as prompts to re-enter the open-domain visual detection and segmentation model, and pixel-level segmentation and filtering of the first mask sequence are performed again to generate a corrected second mask sequence.

[0006] Secondly, a video stream object tracking and detection apparatus is provided, the apparatus comprising: The acquisition module is used to acquire the video stream and determine the semantic description of the target object to be tracked and detected; The selection module is used to adaptively configure the detection method combination and corresponding parameters according to the preset scene. The detection method combination includes a combination based on a multimodal large language model and an open-domain visual detection and segmentation model. The first generation module is used to input the semantic description into a multimodal large language model according to the selected combination of detection methods, generate the initial bounding box of the target object in the video frame, and use the initial bounding box as a prompt to input an open domain visual detection and segmentation model to perform pixel-level segmentation on the video frame to obtain a first mask sequence corresponding to the target object. The second generation module is used to perform pixel-level depth estimation on the video frame using the depth estimation model when the detection method combination further includes a combination based on a depth estimation model and a rule model, to obtain a depth map, and to filter the first mask sequence using the depth information in the depth map and at least one of the area threshold, confidence threshold and depth range in the rule model to obtain a second mask sequence. The receiving module is used to receive at least one point annotation input by the user on a single frame of video when the confidence level of the automated processing result is lower than the quality threshold. The correction module is used to re-input the point annotations as prompts into the open domain visual detection and segmentation model, and to perform pixel-level segmentation and filter the first mask sequence again to generate a corrected second mask sequence.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the video stream object tracking and detection method described above.

[0008] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described video stream object tracking and detection method.

[0009] As can be seen from the technical solution provided in this application: On the one hand, by acquiring video streams and determining the semantic description of target objects, semantically driven precise localization is achieved. Based on this, according to preset scene adaptive configuration, the combination of detection methods and corresponding parameters are flexibly selected, enabling the system to automatically adapt to the optimal detection strategy for different video scenes (e.g., indoor / outdoor, small / large targets, high-speed motion / stillness, etc.), thereby significantly improving the versatility and scene adaptability of the method. On the other hand, by inputting the semantic description into a multimodal large language model to generate initial bounding boxes, and using these as prompts to guide the open-domain visual detection and segmentation model to perform pixel-level segmentation, the semantic understanding capability of the large language model and the open-domain model are fully utilized. The first aspect is the zero-shot segmentation capability, which enables accurate segmentation of any target without pre-training for specific categories, reducing development costs. Secondly, by utilizing the depth map obtained from the depth estimation model and combining it with multi-dimensional constraints such as area thresholds, confidence thresholds, and depth ranges in the rule model, the first mask sequence is filtered, effectively eliminating false masks caused by background interference, depth ambiguity, or segmentation errors. This significantly improves the accuracy and stability of target localization. When the automated processing result falls below the quality threshold, manual correction is introduced in a timely manner, ensuring the system's reliability and fault tolerance in extreme cases. The final output second mask sequence balances segmentation accuracy and noise suppression, providing high-quality foundational data for subsequent video analysis and decision-making. In summary, the technical solution of this application, while maintaining a high level of automation, improves the accuracy and robustness of video stream object tracking and detection and expands the scope of applicable scenarios through multi-source information fusion and adaptive configuration. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart of a video stream object tracking and detection method in one embodiment; Figure 2 This is a structural block diagram of a video stream object tracking and detection device in one embodiment; Figure 3 This is a structural block diagram of a computer device in one embodiment. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] In existing technologies, commonly used video object tracking and detection methods are mainly divided into two categories: one is traditional methods based on template matching or optical flow, which have poor robustness and are difficult to handle non-rigid deformations and abrupt changes in appearance; the other is end-to-end methods based on deep learning, such as YOLO and Mask R-CNN, which, although highly accurate, usually require a large amount of labeled data to train specific categories, have limited generalization ability, and cannot flexibly adapt to different needs for detection methods in different scenarios (e.g., text boxes, bounding boxes, dot cues, etc.). Furthermore, most methods rely only on single image features (e.g., RGB color or texture), ignoring the important role of depth information in distinguishing foreground from background, leading to false positives and false negatives in complex backgrounds. Simultaneously, when automated processing results are unreliable, the lack of effective human intervention mechanisms makes the system unreliable in critical tasks.

[0013] To address the aforementioned problems in existing technologies, this application proposes a video stream object tracking and detection method, the main process of which is as follows: Figure 1 As shown, it mainly includes steps S1 to S6, which are detailed below: Step S1: Acquire the video stream and determine the semantic description of the target object to be tracked and detected.

[0014] In this embodiment, the video stream can originate from various sources such as a camera, storage device, or network transmission. The video stream can be a continuous sequence of frames or a set of frames sampled at intervals. The semantic description of the target object refers to the characteristics of the target object expressed in natural language, such as "red robotic arm," "circular workpiece on the left," or "defect with surface cracks." This semantic description can be provided by the user through text input, voice input, or a preset configuration file. The purpose of the semantic description is to provide clear target guidance for the subsequent multimodal large language model, enabling it to understand the user's intent. In this embodiment, the semantic description can use concise natural language statements or structured phrases, such as "Category: robotic arm; Attribute: maximum; Position: center of the screen."

[0015] Step S2: Select the combination of detection methods and corresponding parameters according to the preset scene adaptive configuration. The combination of detection methods includes a combination based on multimodal large language model and open domain visual detection and segmentation model.

[0016] In practical applications of video stream object tracking and detection, the requirements for detection methods vary greatly across different scenarios (e.g., industrial quality inspection, human tracking, animal monitoring, etc.). Existing technologies typically employ fixed pipelines, such as using only one detection model, like the Segment Anything Model (SAM), which cannot flexibly adapt to scene changes. To address this issue, this application introduces a scene-adaptive configuration mechanism. This involves pre-setting combinations of detection methods and parameter configurations for different video scenarios (e.g., indoor / outdoor, small / large targets, high-speed motion / stationary, etc.), enabling the system to automatically match the optimal detection strategy during runtime. Specifically, the combination of detection methods is not a single model but an organic fusion of multiple models. For example, for scenarios requiring precise semantic understanding, a combination based on a Multimodal Large Language Model (MLLM) and an open-domain visual detection and segmentation model can be selected: MLLM is responsible for understanding semantics and generating coarse localization, while the open-domain model handles fine segmentation. For scenarios requiring spatial constraints, a combination based on a depth estimation model and a rule-based model can be selected: the depth estimation model provides depth information, and the rule-based model performs post-processing filtering. Of course, all four models can be enabled simultaneously to form a complete fusion pipeline. Parameters, including but not limited to model weights, thresholds, and propagation directions, can all be configured according to the scenario. This adaptive configuration allows the same system to adapt to different business needs with zero code, greatly improving versatility and deployment efficiency.

[0017] It should be noted that the open-domain visual detection and segmentation model mentioned in the above embodiments refers to a deep learning model that can accept input prompts such as bounding boxes, points or text, and output pixel-level segmentation masks of the corresponding targets, and has zero-shot segmentation capability. The rule model refers to a set of interpretable post-processing rules, including area threshold, confidence threshold, depth range and temporal consistency threshold, etc., which are used for automatic screening and quality control of segmentation masks.

[0018] As an embodiment of this application, the selection of detection method combinations and corresponding parameters according to the preset scene adaptive configuration can be achieved through the following steps S21 and S22: Step S21: Based on the preset Session group configuration, select one of the five detection methods—text, bbox, bbox_point, point, and xanylabeling—as the detection method for the current video stream.

[0019] Session group configuration refers to dividing video data into different groups (Session Groups) based on dimensions such as business scenario, device, and viewpoint. Each Session Group can be configured with an independent detection method. Five detection methods represent different levels of automation and human intervention requirements: the text method is fully automated, relying only on semantic description; the bounding box method requires user-provided bounding box hints; the bbox_point method combines bounding boxes and point hints; the point method requires only a few point annotations; and the xanylabeling method serves as a fallback, allowing users to perform arbitrary forms of annotation. The choice of method depends on the characteristics of the target object, the complexity of the scene, and the user's accuracy requirements. For example, for targets with regular shapes and clear semantics (e.g., standard workpieces), the text method can be chosen; for targets with irregular shapes or ambiguous semantics (e.g., animals in natural scenes), the bounding box or point methods can be chosen.

[0020] Step S22: Based on the parameter hierarchical inheritance mechanism, apply the global default parameters, group-level coverage parameters, and task-level coverage parameters in sequence to determine the specific parameters corresponding to the selected detection method.

[0021] The hierarchical parameter inheritance mechanism is designed for fine-grained management in large-scale, multi-scenario deployments. The system maintains three levels of parameters: global default parameters apply to all session groups; group-level overriding parameters are adjusted locally for specific session groups; and task-level overriding parameters are temporarily modified for individual video streams or single tasks. When performing detection, the system first loads the global default parameters, then checks for the existence of group-level overriding parameters, and if found, overrides the corresponding items; finally, it checks for task-level overriding parameters, and if found, overrides them again. This mechanism avoids redundant configuration while allowing for flexible adjustments. For example, the global default confidence threshold is 0.5; an industrial scenario group can increase the threshold to 0.7 to ensure quality, while a specific task can further reduce it to 0.6 to recall more targets.

[0022] Furthermore, in another embodiment of this application, step S2 may also include an adaptive selection strategy based on the characteristics of the target object. For example, the aspect ratio of the target object is calculated; if the aspect ratio is greater than a preset threshold, the bbox_point detection method is selected; otherwise, the text detection method is selected. This is because slender targets (such as wires or cracks) tend to contain too much background within the bounding box, making point cues more effective for segmentation; while compact targets (such as spheres or parts) can be directly segmented using the text method. This adaptive selection based on geometric features further enhances the intelligence level of the configuration.

[0023] Step S3: Based on the selected combination of detection methods, input the semantic description into the multimodal large language model to generate the initial bounding box of the target object in the video frame. Use the initial bounding box as a prompt to input the open domain visual detection and segmentation model to perform pixel-level segmentation of the video frame. Use a bidirectional propagation mechanism to propagate the segmentation results forward and backward along the video time sequence to obtain the first mask sequence corresponding to the target object.

[0024] In video object detection, while relying solely on open-domain visual detection and segmentation models (e.g., SAM) can achieve zero-shot segmentation, their ability to understand natural language is limited, making it impossible to accurately locate targets based on semantic descriptions such as "red robotic arm." Conversely, multimodal large language models (MLLM) excel at understanding semantics and generating coarse localization information (e.g., bounding boxes), but their segmentation accuracy is insufficient for industrial-grade requirements. Existing technologies either rely solely on MLLM, resulting in coarse segmentation, or rely solely on open-domain models, requiring precise human prompts, neither of which can achieve fully automated semantic-driven segmentation.

[0025] To resolve this contradiction, this application organically links two models: first, it utilizes the semantic understanding capabilities of MLLM to generate an initial bounding box for the target object; then, it uses this bounding box as input to an open-domain visual detection and segmentation model, which performs pixel-level precise segmentation. This "semantic → coarse localization → fine segmentation" pipeline leverages both the semantic advantages of MLLM and the segmentation accuracy of the open-domain model, achieving fully automated conversion from natural language to precise masks.

[0026] In one specific implementation, step S3 can be achieved through steps S31 and S32: Step S31: Input the semantic description and the current video frame into the multimodal large language model, and add contextual cues to indicate the generation of the target object bounding box to generate the initial bounding box.

[0027] Multimodal large language models can process both text and images simultaneously. In this step, the semantic description determined in step S1 (e.g., "the largest robotic arm") is input into the MLLM along with the image data of the current video frame. Simultaneously, to guide the model to output bounding boxes instead of other forms of results, a contextual cue—a predefined instruction text—is appended to guide the multimodal large language model to output results in a specific format. For example, "Please mark the location of this target with a bounding box, in the format [x_min, y_min, x_max, y_max]". The MLLM understands the meaning of the target based on the semantic description and infers the target's location region by combining it with the image content, outputting an initial bounding box containing coordinates. This bounding box may not be precise enough (e.g., it may include some background or omit edges), but its purpose is to provide initial localization for subsequent fine-grained segmentation; therefore, the accuracy requirement is relatively relaxed.

[0028] Step S32: Use the initial bounding box as a rectangular cue input to the open-domain visual detection and segmentation model, and use a bidirectional propagation mechanism to propagate the segmentation results forward and backward along the video time sequence to generate the first mask sequence.

[0029] Open-domain visual detection and segmentation models, such as the Segment Everything Everywhere All at Once (SEEM) model, support segmentation using bounding boxes, points, or text as cues. In this step, the initial bounding box generated in step S31 is input to the model as a rectangular cue. The model performs pixel-level segmentation within the bounding box region, outputting a precise mask, i.e., the pixel-level outline of the target object in the current frame. However, videos consist of consecutive frames, and processing only a single frame cannot meet the requirements of temporal tracking. Therefore, this application employs a bidirectional propagation mechanism: starting from the current frame, the segmentation result is propagated forward (towards the end of the video) and backward (towards the beginning of the video). Specifically, in one embodiment of this application, the generation of the first mask sequence by propagating the segmentation results forward and backward along the video time sequence using a bidirectional propagation mechanism can be as follows: propagating backward along the video time sequence from the current frame to the end of the video to obtain a first forward mask sequence; propagating forward along the video time sequence from the current frame to the beginning of the video to obtain a first reverse mask sequence; and performing frame-by-frame Intersection over Union (IoU) merging of the first forward mask sequence and the first reverse mask sequence to remove duplicate and conflicting mask regions to obtain the first mask sequence. The advantage of this bidirectional propagation is that unidirectional propagation (forward only) may lead to missed detections in subsequent frames due to sudden changes in target appearance or brief occlusion, while reverse propagation can compensate for these omissions. Through frame-by-frame IoU merging, the system automatically selects the higher-quality mask from the two directions and removes redundancy in overlapping areas, ultimately obtaining a temporally consistent first mask sequence. This mask sequence is a temporal set of pixel-level segmentation masks covering all or part of the video frames, i.e., covering all frames in the entire video where the target appears, providing a foundation for subsequent depth filtering and rule verification.

[0030] It should be noted that, in the embodiments of this application, the bidirectional propagation mechanism specifically uses the segmentation mask of the current frame as a reference to propagate the mask to adjacent frames. In one embodiment, an optical flow-based propagation method is used, that is, the optical flow field between adjacent frames is calculated, and the mask of the current frame is mapped to the adjacent frames according to the optical flow field to obtain the propagated mask. In another embodiment, a feature matching-based propagation method is used, that is, high-dimensional features are extracted within the mask region of the current frame, matching feature points are searched in adjacent frames, and a propagation mask is generated based on the matching results. Forward propagation starts from the current frame and processes frame by frame until the end of the video; backward propagation starts from the current frame and processes frame by frame until the beginning of the video.

[0031] Furthermore, to reduce computational complexity, in another embodiment of this application, bidirectional propagation is not performed on all frames, but rather the video is uniformly sampled (e.g., 1 frame per second), and propagation and segmentation are performed only on the sampled frames. Masks for intermediate frames are generated through interpolation. In another implementation, when the video length exceeds a preset threshold (e.g., 500 frames), the video is divided into multiple segments, and bidirectional propagation is performed independently within each segment.

[0032] In some cases, the initial bounding boxes generated by MLLM may have low confidence, such as when the target semantics are ambiguous or when multiple similar objects exist in the image. In such cases, this application provides an adaptive adjustment strategy: when the confidence of the initial bounding boxes generated by the multimodal large language model is lower than a preset threshold, the initial bounding boxes are converted into point cues; these point cues are then used instead of bounding box cues as input to the open-domain visual detection and segmentation model. Point cues refer to using the center point of the bounding box or a user-specified point as the input. The advantage of point cues is that they avoid the problem of the bounding box potentially containing too much background, and are especially suitable for situations where the target size is small or the shape is irregular. This flexible cue conversion mechanism further improves the robustness of segmentation.

[0033] Through step S3, this application successfully obtained a first mask sequence covering the entire video, starting from the semantic description. However, this mask sequence may still contain false masks caused by background interference, depth ambiguity, or segmentation errors, requiring subsequent steps for cleanup.

[0034] To address the potential issue of insufficient confidence when multimodal large language models generate initial bounding boxes, this application provides a flexible cue conversion mechanism. In one implementation, step S3 further includes: when the confidence of the initial bounding box generated by the multimodal large language model is lower than a preset threshold, converting the initial bounding box into a point cue; and using the point cue instead of the bounding box cue as input to the open-domain visual detection and segmentation model. Specifically, multimodal large language models typically include a confidence score along with the output bounding box, indicating the model's confidence in the accuracy of the bounding box. When this score is lower than a preset threshold (e.g., 0.6), it indicates that the model is not confident enough in the target localization. If the bounding box is still used as a cue in this case, a large amount of background region may be introduced into the segmentation process, leading to a decrease in mask quality. The solution is to convert the bounding box into a point cue: taking the center point of the bounding box (or another point specified by the user) as the new cue. The advantage of a point cue is that it does not carry region information, only indicating "the target is near this location." The open-domain model will perform segmentation based on the local features around the point, thereby avoiding background contamination caused by inaccurate bounding boxes. This conversion mechanism is particularly effective when the target size is small, the boundaries are blurred, or there are multiple similar targets in the image.

[0035] Step S4: When the detection method combination also includes a combination of depth estimation model and rule model, the depth estimation model is used to perform pixel-level depth estimation on the video frame to obtain a depth map, and the depth information in the depth map and at least one of the area threshold, confidence threshold and depth range in the rule model are used to filter the first mask sequence to obtain the second mask sequence.

[0036] Although the first mask sequence obtained after step S3 has achieved a preliminary mapping from semantics to pixel level, it still faces serious challenges in practical applications. Existing technologies that rely solely on RGB image features for post-segmentation processing (e.g., morphological filtering, connected component analysis) have fundamental flaws: when the background texture is similar to the target, RGB features cannot effectively distinguish between foreground and background, resulting in a large number of false masks remaining. For example, in industrial scenarios, a robotic arm and a metal support in the background have similar grayscale distributions, making it difficult to eliminate background interference based solely on RGB information. Furthermore, single confidence threshold filtering cannot solve the depth ambiguity problem—a high-confidence mask may actually be located on the wrong depth plane. To address these issues, this application introduces a depth estimation model and a rule model to form a multi-dimensional filtering mechanism.

[0037] In one specific implementation, the depth estimation model employs a monocular depth estimation network to perform pixel-level depth estimation for each frame of video image, outputting a depth map. The value of each pixel in the depth map represents the relative distance from that point to the camera. Depth information provides crucial spatial constraints for distinguishing foreground from background. For example, if a target object is located within 1 to 3 meters of the camera, pixels outside this range in the depth map can be considered background interference. In this embodiment, the aforementioned monocular depth estimation network can adopt a MiDaS (Mixed Depth and Semantic) network architecture, which takes a single RGB image as input and outputs the corresponding depth map through an encoder-decoder structure. The encoder uses ResNet or Vision Transformer as the backbone network, and the decoder recovers spatial resolution through upsampling and skip connections. The monocular depth estimation network is pre-trained on the MIX-6 dataset and can output relative depth values ​​in both indoor and outdoor scenes. Alternatively, the monocular depth estimation network can also adopt a DPT (Dense Prediction Transformer) architecture.

[0038] In one embodiment of this application, filtering the first mask sequence using depth information can be achieved through steps S41 and S42, as detailed below: Step S41: Extract the pixel region within the depth range corresponding to the target object from the depth map to generate a depth mask.

[0039] First, based on the estimated physical size of the target object or known prior knowledge, a depth range (e.g., 1 meter to 3 meters) is set. Then, each pixel in the depth map is traversed, and its depth value is determined to fall within this range. All pixels within the range are marked as 1, and those outside the range are marked as 0, thus generating a binary depth mask. This depth mask represents the area in space that may contain the target object. The depth range can be flexibly adjusted through the scene adaptive configuration in step S2. For example, for close-up person detection, it can be set to 0.5 meters to 2 meters; for distant building detection, it can be set to 10 meters to 50 meters.

[0040] Step S42: Perform a frame-by-frame pixel-level intersection operation between the depth mask and the first mask sequence, and retain the regions that are simultaneously located within the depth mask and the first mask.

[0041] The depth mask generated in step S41 is then subjected to a pixel-by-pixel logical AND operation with each frame mask in the first mask sequence. Specifically, for each pixel location, the pixel is retained in the result only if it is marked as a target in the first mask and within a valid depth range in the depth mask; otherwise, the pixel is discarded. This step effectively removes false masks (such as poster patterns in the background) that appear as targets on the RGB image but are actually located in the wrong depth plane. The physical meaning of this operation is that the target object must simultaneously satisfy both the conditions of "appearing like a target" and "being located at a reasonable depth," neither of which can be omitted.

[0042] After depth information filtering is completed, the rule model further plays its role. The rule model is a set of interpretable post-processing rules used for automatic filtering and quality control of multimodal outputs. In one specific implementation, filtering the first mask sequence using at least one of the area threshold, confidence threshold, and depth range in the rule model further includes the following technical solutions exemplified by steps S43 and S44: Step S43: Calculate the pixel area of ​​each retained region, and retain only the region with the largest area.

[0043] In many real-world scenarios, users are often concerned with the largest target object in the scene (e.g., the "largest robotic arm"). Therefore, a rule-based model can set an area filtering rule: calculate the number of pixels contained in each retained mask region, retain only the largest region, and discard all smaller regions. This rule can effectively remove small fragments caused by noise or missegmentation. Of course, the area rule can also be configured in reverse (retaining the smallest region) to adapt to different scenario requirements.

[0044] Step S44: Calculate the confidence score for each retained region, and retain only regions with scores higher than the confidence threshold.

[0045] Open-domain visual detection and segmentation models typically include a confidence score when outputting a mask, reflecting the model's confidence level in that mask. Rule-based models set a confidence threshold (e.g., 0.7), and only masks with scores higher than this threshold are retained. This rule filters out low-quality outputs where the model is uncertain. The confidence threshold can also be adjusted through scene-adaptive configuration: it can be set to 0.9 in high-precision industrial scenarios, while 0.5 can be set in fast preview scenarios.

[0046] Furthermore, to further enhance the stability of the results, the rule model in the above embodiments can also perform filtering based on temporal consistency. Specifically, this includes: calculating the intersection-over-union (IoU) ratio of corresponding masks in adjacent video frames; if the IoU ratio is lower than a preset temporal consistency threshold, the mask is determined to be an abnormal result and is discarded. This is because in consecutive video frames, masks of the same target usually have a high degree of overlap (large IoU) between adjacent frames. If the IoU between the mask of a certain frame and the corresponding mask of the previous frame suddenly becomes very small (e.g., less than 0.3), it is very likely that the mask is an abnormal result caused by momentary false detection or occlusion. Through this temporal consistency filtering, randomly jittery false masks can be effectively eliminated, making the tracking results smoother and more reliable.

[0047] Through step S4, this application achieves a seamless transition from automated processing to manual correction, ensuring high-quality detection and segmentation results can be produced in any scenario.

[0048] Furthermore, this application provides a specific strategy to address the common challenge of occlusion. In one embodiment, step S4 may further include: analyzing the gradient of the depth map; if the depth gradient of the target region is greater than a preset gradient threshold, the region is determined to be occluded, and the depth filtering step is skipped, relying solely on the rule model for filtering. The depth gradient refers to the rate of change of depth values ​​in the image space. When the depth gradient of the target region is greater than the preset gradient threshold, it indicates that there may be a sudden change in depth in the region, at which point the reliability of the depth estimation result decreases. Therefore, the system reduces the weight of depth filtering and relies more on the rule model for filtering. Specifically, the filtering weight of depth information is reduced from 1.0 to 0.3, while the confidence threshold of the rule model is increased from 0.7 to 0.8 to offset or reduce the risk caused by unreliable depth information. The preset gradient threshold in the above embodiment can be set to 0.5 meters / pixel without loss of generality. This preset gradient threshold is determined by statistically analyzing the depth gradient distribution at the occlusion boundary on a standard test dataset.

[0049] It should be noted that while depth filtering can effectively distinguish foreground from background under normal circumstances, depth estimation at the occlusion boundary is often unreliable when the target is occluded. Continuing to use depth filtering in this situation may result in the erroneous removal of target pixels from the occluded portion. Therefore, this application temporarily skips depth filtering when occlusion is detected, relying solely on a rule-based model for quality control. In other words, when depth information is unreliable, depth constraints are abandoned to avoid erroneous deletion, and other constraints (e.g., area, confidence, temporal consistency, etc.) are used to ensure quality; depth filtering automatically resumes once the occlusion is resolved.

[0050] When a target is occluded by other objects, a significant depth jump typically occurs at the occlusion boundary, manifested as a sharp increase in the depth gradient value. For example, when a person stands in front of a car, the depth value of the area where the person is located is relatively small, while the depth value of the area where the car is located is relatively large, resulting in a very large depth gradient at the boundary between the two. In this case, the output of the depth estimation model near the occlusion boundary is often unreliable because the occlusion causes a loss or confusion of depth information. If depth filtering is still used, the occluded target pixels may be incorrectly rejected. Therefore, by calculating the maximum or average value of the depth gradient within the target area and comparing it with a preset gradient threshold (e.g., 0.5 m / pixel), the system can determine whether occlusion exists in the current frame. If occlusion exists, the depth filtering step is temporarily skipped, and quality control is performed solely based on the area threshold and confidence threshold in the rule model. This adaptive strategy ensures that targets are not incorrectly filtered out in the case of occlusion, thus maintaining the integrity of the detection.

[0051] To further improve the temporal stability of tracking, this application also introduces a motion prediction mechanism. In one embodiment, after step S4, the method may further include: predicting the centroid position of each frame mask in the second mask sequence using a Kalman filter, and matching the prediction result with the actual detection result of the next frame to correct the temporal drift of the mask. The Kalman filter is a classic linear recursive filter that can predict future states based on historical observations. In this application, the system inputs the centroid position of each frame mask (i.e., the two-dimensional coordinates of the target center point) as the observation value into the Kalman filter, and the filter outputs the predicted value of the centroid position of the next frame. When processing the next frame, the system first obtains the actual detection result of that frame, and then compares the actual centroid position with the Kalman prediction value. If the deviation between the two is too large (exceeding a preset threshold), it is considered that the actual detection result may have drifted, and the predicted value is used to correct it. This method effectively suppresses the mask position jump caused by slight jitter of the segmentation model, making the tracking trajectory smoother and more natural.

[0052] Step S5: When the confidence level of the automated processing result is lower than the quality threshold, receive at least one point annotation input by the user on a single frame of video.

[0053] After the aforementioned multi-dimensional filtering, a second mask sequence is obtained (i.e., the temporal set of the final segmentation masks obtained after filtering by the depth estimation model and the rule model, eliminating false masks), and its quality has been significantly improved. However, automated processing is not foolproof. When encountering extreme situations (e.g., the target is completely occluded, there are drastic changes in lighting, semantic ambiguity, etc.), the automated processing result may still be lower than the preset quality threshold. In this case, this application introduces a manual correction mechanism, which includes receiving at least one point annotation input by the user on a single frame of video when the confidence level of the automated processing result is lower than the quality threshold. Here, the point annotation is the pixel coordinate point generated by the user clicking on the target object with a mouse or touch screen. Optionally, the user clicks a point in the center area or a salient feature area of ​​the target object. When the target object has a complex shape or is occluded, the user can click multiple points in multiple different parts of the target object. Specifically, the system will display the frame where the automated processing failed on the interface and prompt the user to click a point (point annotation) on the target object in that frame.

[0054] Step S6: Use the point annotations as prompts to re-enter the open domain visual detection and segmentation model, and execute steps S3 and S4 again to generate the corrected second mask sequence.

[0055] In this embodiment, the user only needs to click once, and the system will use that point annotation as a new prompt input to the open-domain visual detection and segmentation model, re-executing the entire process starting from step S3. Because the point annotation provides more accurate localization information, the segmentation result after re-execution can usually correctly identify the target. This mechanism minimizes manual intervention—requiring only single-frame, single-point annotation, rather than frame-by-frame annotation, improving efficiency by tens of times.

[0056] It should be noted that the quality threshold in the above embodiments can be dynamically adjusted according to the application scenario. Specifically, in industrial quality inspection scenarios requiring high reliability, the quality threshold can be set to a higher value (e.g., a quality threshold of 0.85), prioritizing output quality even if it means triggering more manual corrections. In real-time monitoring scenarios requiring high efficiency, the quality threshold can be set to a lower value (e.g., a quality threshold of 0.6), triggering manual intervention only in cases of obvious failure. In one implementation, the system automatically adjusts the quality threshold based on the historical frequency of manual corrections, maintaining the manual correction rate between 5% and 10%.

[0057] Furthermore, in another embodiment of this application, after generating the corrected second mask sequence, the process may further include: using the temporal propagation mechanism of the open-domain visual detection and segmentation model to propagate the point annotations of a single frame to all frames of the entire video sequence, generating a globally consistent corrected mask sequence. In other words, when a user clicks a point on a frame, the system not only corrects that frame but also automatically propagates the correction effect to all frames of the entire video, ensuring that the tracking results of the entire video sequence are corrected. Specifically, the above-mentioned propagation of the point annotations of a single frame to the entire video sequence can be as follows: using the labeled frame as a reference, performing mask propagation forward and backward respectively; during the propagation process, using optical flow tracing or feature point matching to map the position of the labeled point to adjacent frames, and re-inputting the mapped point to the open-domain visual detection and segmentation model on the adjacent frames as a prompt to generate the mask for that frame; iterating in this way until the entire video sequence is covered; when encountering occlusion that causes optical flow tracing to fail, pausing the propagation and prompting the user to supplement the annotations near that frame.

[0058] After the multi-dimensional filtering in step S4 and the manual corrections in steps S5 and S6, the resulting second mask sequence possesses high-quality characteristics: it contains the precise pixel-level contours of the target object, maintains good temporal consistency, and eliminates background interference and abnormal results. The system outputs the second mask sequence as the final tracking and segmentation result. The output format can be flexibly customized according to the needs of downstream applications: for example, it can be output as a video overlay (drawing the mask contours on the original video), as a JSON-formatted list of mask coordinates, or as a frame-by-frame sequence of binary mask images. The output results can be directly used for subsequent tasks such as robot manipulation, behavior analysis, and quality inspection.

[0059] In one embodiment of this application, Figure 1 The example method may also include: feeding back the tracking and segmentation results output by the system to the scene adaptive configuration system, and automatically adjusting the combination of detection methods and corresponding parameters based on the quality statistics of the results. This step embodies the closed-loop optimization concept of this application. Specifically, the system performs a quality assessment on the output second mask sequence, and statistically analyzes indicators such as average confidence, false positive rate, false negative rate, and mask area fluctuation. After feeding back these statistical information to the scene adaptive configuration system in step S2, the configuration system can automatically adjust the combination of detection methods and parameters accordingly. For example, if it is found that depth filtering in the current scene leads to too many false negatives, the system can automatically reduce the strictness of the depth range or temporarily disable depth filtering; if it is found that the confidence threshold is too high, resulting in insufficient recall, the system can automatically lower the threshold. This feedback mechanism enables the system to continuously self-optimize and adapt to environmental changes during long-term operation without human intervention.

[0060] Furthermore, in one embodiment of this application, automatically adjusting the corresponding parameters based on the quality statistics of the results can be achieved by: statistically analyzing the mean and standard deviation of the target mask area in the most recent N frames; and dynamically setting the area filtering threshold for the current frame based on the mean and standard deviation. Traditional area filtering thresholds are fixed values ​​and cannot adapt to changes in target size over time (e.g., the target gradually moves closer to or further away from the camera). By statistically analyzing the mean and standard deviation of the area in the most recent N frames, the system can calculate the typical size range of the target in the current scene and dynamically adjust the area filtering threshold accordingly. For example, if the mean area of ​​the most recent N frames is 1000 pixels and the standard deviation is 200 pixels, the area threshold can be set to the mean minus twice the standard deviation (600 pixels), and areas below this value are considered noise and removed. This method allows the filtering rules to adapt to changes in target scale, improving robustness.

[0061] The feedback mechanism described above, which feeds back the tracking and segmentation results from the system to the scene adaptive configuration system, is not limited to adjusting the filtering threshold; it can also be used to optimize the selection of detection methods. For example, if the system statistics show that the success rate of using the text method in the current scene is consistently low (below 60%), the feedback system can automatically switch the detection method to the bbox_point method to improve the overall quality. This statistically based automatic switching avoids the tedious process of manual trial and error.

[0062] In addition, the feedback mechanism can also be used to update Session group configurations. Once the system has accumulated enough quality statistics in a particular Session group, it can automatically generate new group-level configuration parameters and write them to the configuration library for use by subsequent tasks in the same scenario. This gives the system a "learning" ability, making it more accurate with use.

[0063] To address the selection of detection methods in scene adaptive configurations, this application provides an adaptive strategy based on the geometric features of the target object. In one implementation, selecting the combination of detection methods and corresponding parameters according to the preset scene adaptive configuration further includes: calculating the aspect ratio of the target object; if the aspect ratio is greater than a preset threshold, the bbox_point detection method is selected; otherwise, the text detection method is selected. This strategy stems from in-depth observation of real-world application scenarios: slender targets (e.g., wires, cracks, links of robotic arms, etc.) typically have a large aspect ratio (e.g., width much greater than height or vice versa), and the bounding boxes occupied by these targets in images often contain a large amount of background. If the text method is used directly (i.e., segmentation relies solely on semantic description), open-domain models may misclassify the background as part of the target, leading to a decrease in segmentation accuracy. The bbox_point method combines bounding boxes and point cues, allowing users or the system to specify a key point within the bounding box, guiding the model to focus on the target body, thereby effectively suppressing background interference. Conversely, for compact targets (e.g., spherical workpieces, circular parts, faces, etc.), whose aspect ratio is close to 1 and have less background within the bounding box, the text method is sufficient. By calculating the aspect ratio and setting a threshold (e.g., 3.0), the system can automatically select the optimal detection method without manual intervention. This adaptive strategy based on geometric features further enriches the connotation of scene adaptive configuration, enabling the system to make intelligent decisions based on the target shape.

[0064] From the above appendix Figure 1As can be seen from the example video stream object tracking and detection method, on the one hand, by acquiring the video stream and determining the semantic description of the target object, semantic-driven precise localization is achieved. Based on this, according to the preset scene adaptive configuration, the combination of detection methods and corresponding parameters are flexibly selected, enabling the system to automatically adapt to the optimal detection strategy for different video scenes (e.g., indoor / outdoor, small / large target, high-speed motion / stillness, etc.), thus significantly improving the method's versatility and scene adaptability. On the other hand, by inputting the semantic description into a multimodal large language model to generate initial bounding boxes, and using these as prompts to guide the open-domain visual detection and segmentation model for pixel-level segmentation, the semantic understanding capability of the large language model and the open-domain model are fully utilized. The first aspect is the zero-shot segmentation capability, which enables accurate segmentation of any target without pre-training for specific categories, reducing development costs. Secondly, by utilizing the depth map obtained from the depth estimation model and combining it with multi-dimensional constraints such as area threshold, confidence threshold, and depth range in the rule model, the first mask sequence is filtered, effectively eliminating false masks caused by background interference, depth ambiguity, or segmentation errors. This significantly improves the accuracy and stability of target localization. When the automated processing result falls below the quality threshold, manual correction is introduced in a timely manner, ensuring the system's reliability and fault tolerance in extreme cases. The final output second mask sequence balances segmentation accuracy and noise suppression, providing high-quality basic data for subsequent video analysis and decision-making. In summary, the technical solution of this application, while maintaining a high level of automation, improves the accuracy and robustness of video stream object tracking and detection and expands the scope of applicable scenarios through multi-source information fusion and adaptive configuration.

[0065] Please see Figure 2 As shown, in one embodiment, a video stream object tracking and detection device is provided. This device may include an acquisition module 201, a selection module 202, a first generation module 203, a second generation module 204, a receiving module 205, and a correction module 206, as detailed below: The acquisition module 201 is used to acquire the video stream and determine the semantic description of the target object to be tracked and detected; The selection module 202 is used to select a combination of detection methods and corresponding parameters according to a preset scene adaptive configuration. The combination of detection methods includes a combination based on a multimodal large language model and an open-domain visual detection and segmentation model. The first generation module 203 is used to input the semantic description into the multimodal large language model according to the selected detection method combination, generate the initial bounding box of the target object in the video frame, and use the initial bounding box as a prompt to input the open domain visual detection and segmentation model to perform pixel-level segmentation of the video frame to obtain the first mask sequence corresponding to the target object. The second generation module 204 is used to perform pixel-level depth estimation on video frames using the depth estimation model when the detection method combination also includes a combination based on a depth estimation model and a rule model, to obtain a depth map, and to filter the first mask sequence using the depth information in the depth map and at least one of the area threshold, confidence threshold and depth range in the rule model to obtain a second mask sequence. The receiving module 205 is used to receive at least one point annotation input by the user on a single frame of video when the confidence level of the automated processing result is lower than the quality threshold. The correction module 206 is used to take at least one point annotation input by the user as a prompt to re-enter the open domain visual detection and segmentation model, and to perform pixel-level segmentation and filter the first mask sequence again to generate a corrected second mask sequence.

[0066] From the above appendix Figure 2 As illustrated by the example video stream object tracking and detection device, on the one hand, semantic-driven precise localization is achieved by acquiring the video stream and determining the semantic description of the target object. Based on this, the system flexibly selects a combination of detection methods and corresponding parameters according to a preset scene adaptive configuration. This allows the system to automatically adapt to the optimal detection strategy for different video scenes (e.g., indoor / outdoor, small / large target, high-speed motion / stillness, etc.), significantly improving the method's versatility and scene adaptability. On the other hand, by inputting the semantic description into a multimodal large language model to generate initial bounding boxes, and using these as prompts to guide the open-domain visual detection and segmentation model for pixel-level segmentation, the system fully utilizes the semantic understanding capabilities of the large language model and the open-domain model. The first aspect is the zero-shot segmentation capability, which enables accurate segmentation of any target without pre-training for specific categories, reducing development costs. Secondly, by utilizing the depth map obtained from the depth estimation model and combining it with multi-dimensional constraints such as area threshold, confidence threshold, and depth range in the rule model, the first mask sequence is filtered, effectively eliminating false masks caused by background interference, depth ambiguity, or segmentation errors. This significantly improves the accuracy and stability of target localization. When the automated processing result falls below the quality threshold, manual correction is introduced in a timely manner, ensuring the system's reliability and fault tolerance in extreme cases. The final output second mask sequence balances segmentation accuracy and noise suppression, providing high-quality basic data for subsequent video analysis and decision-making. In summary, the technical solution of this application, while maintaining a high level of automation, improves the accuracy and robustness of video stream object tracking and detection and expands the scope of applicable scenarios through multi-source information fusion and adaptive configuration.

[0067] In one embodiment, a computer device is provided, the internal structure of which can be shown as follows: Figure 3As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a video stream object tracking and detection method.

[0068] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, performs the following steps: Acquire the video stream and determine the semantic description of the target object to be tracked and detected; Based on the preset scene adaptive configuration, select the combination of detection methods and corresponding parameters. The combination of detection methods includes a combination based on multimodal large language model and open domain visual detection and segmentation model. Based on the selected combination of detection methods, the semantic description is input into the multimodal large language model to generate the initial bounding box of the target object in the video frame. The initial bounding box is used as a prompt to input the open domain visual detection and segmentation model to perform pixel-level segmentation of the video frame. The segmentation result is propagated forward and backward along the video time sequence using a bidirectional propagation mechanism to obtain the first mask sequence corresponding to the target object. When the detection method combination also includes a combination of depth estimation model and rule model, the depth estimation model is used to perform pixel-level depth estimation on video frames to obtain a depth map, and the depth information in the depth map and at least one of the area threshold, confidence threshold and depth range in the rule model are used to filter the first mask sequence to obtain the second mask sequence. When the confidence level of the automated processing result is lower than the quality threshold, at least one point annotation input by the user is received on a single frame of video. The user input of at least one point is used as a prompt to re-enter the open-domain visual detection and segmentation model, and pixel-level segmentation and filtering of the first mask sequence are performed again to generate a corrected second mask sequence.

[0069] The aforementioned computer program significantly improves the accuracy and robustness of video object tracking through multi-source information fusion and adaptive configuration.

[0070] In one embodiment, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, performs the following steps: Acquire the video stream and determine the semantic description of the target object to be tracked and detected; Based on the preset scene adaptive configuration, select the combination of detection methods and corresponding parameters. The combination of detection methods includes a combination based on multimodal large language model and open domain visual detection and segmentation model. Based on the selected combination of detection methods, the semantic description is input into the multimodal large language model to generate the initial bounding box of the target object in the video frame. The initial bounding box is used as a prompt to input the open domain visual detection and segmentation model to perform pixel-level segmentation of the video frame. The segmentation result is propagated forward and backward along the video time sequence using a bidirectional propagation mechanism to obtain the first mask sequence corresponding to the target object. When the detection method combination also includes a combination of depth estimation model and rule model, the depth estimation model is used to perform pixel-level depth estimation on video frames to obtain a depth map, and the depth information in the depth map and at least one of the area threshold, confidence threshold and depth range in the rule model are used to filter the first mask sequence to obtain the second mask sequence. When the confidence level of the automated processing result is lower than the quality threshold, at least one point annotation input by the user is received on a single frame of video. The user input of at least one point is used as a prompt to re-enter the open-domain visual detection and segmentation model, and pixel-level segmentation and filtering of the first mask sequence are performed again to generate a corrected second mask sequence.

[0071] The steps described above, implemented by the computer program when executed by the processor, significantly improve the accuracy and robustness of video object tracking through multi-source information fusion and adaptive configuration.

[0072] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0073] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0074] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0075] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for tracking and detecting objects in a video stream, characterized in that, The method includes: Acquire the video stream and determine the semantic description of the target object to be tracked and detected; Based on the preset scene adaptive configuration, select the combination of detection methods and corresponding parameters. The combination of detection methods includes a combination based on a multimodal large language model and an open-domain visual detection and segmentation model. Based on the selected combination of detection methods, the semantic description is input into a multimodal large language model to generate an initial bounding box of the target object in the video frame. The initial bounding box is then used as a prompt to input an open-domain visual detection and segmentation model to perform pixel-level segmentation on the video frame. A bidirectional propagation mechanism is used to propagate the segmentation results forward and backward along the video time sequence to obtain a first mask sequence corresponding to the target object. When the detection method combination further includes a combination of a depth estimation model and a rule model, the depth estimation model is used to perform pixel-level depth estimation on the video frame to obtain a depth map, and the depth information in the depth map and at least one of the area threshold, confidence threshold and depth range in the rule model are used to filter the first mask sequence to obtain a second mask sequence. When the confidence level of the automated processing result is lower than the quality threshold, at least one point annotation input by the user is received on a single frame of video. The point annotations are used as prompts to re-enter the open-domain visual detection and segmentation model, and pixel-level segmentation and filtering of the first mask sequence are performed again to generate a corrected second mask sequence.

2. The video stream object tracking and detection method according to claim 1, characterized in that, The semantic description is input into a multimodal large language model based on the selected combination of detection methods to generate an initial bounding box of the target object in the video frame. This initial bounding box is then used as a cue input to an open-domain visual detection and segmentation model to perform pixel-level segmentation of the video frame. A bidirectional propagation mechanism is used to propagate the segmentation results forward and backward along the video time sequence, resulting in a first mask sequence corresponding to the target object, including: The semantic description and the current video frame are input into the multimodal large language model, and contextual cues are added to indicate the generation of the target object bounding box to generate the initial bounding box. The initial bounding box is used as a rectangular cue input to the open-domain visual detection and segmentation model, and the segmentation results are propagated forward and backward along the video time sequence using a bidirectional propagation mechanism to generate the first mask sequence.

3. The video stream object tracking and detection method according to claim 2, characterized in that, The process of propagating the segmentation results forward and backward along the video time sequence using a bidirectional propagation mechanism to generate the first mask sequence includes: Propagating from the current frame forward along the video sequence to the end of the video, we obtain the first forward mask sequence; Propagate forward from the current frame along the video time sequence to the beginning of the video to obtain the first inverse mask sequence; The first forward mask sequence and the first reverse mask sequence are combined frame by frame using the cross-interference-union ratio (CIN) to remove duplicate and conflicting mask regions, thus obtaining the first mask sequence.

4. The video stream object tracking and detection method according to claim 1, characterized in that, The process of generating the corrected second mask sequence further includes: By utilizing the temporal propagation mechanism of the open-domain visual detection and segmentation model, the point annotations of a single frame are propagated to all frames of the entire video sequence, generating a globally consistent corrected mask sequence.

5. The video stream object tracking and detection method according to claim 1, characterized in that, The rule model also performs filtering based on temporal consistency, including: Calculate the intersection-union ratio (CIRR) of corresponding masks in adjacent video frames; If the cross-union ratio is lower than the preset timing consistency threshold, the mask is determined to be an abnormal result and is removed.

6. The video stream object tracking and detection method according to claim 1, characterized in that, The step of performing pixel-level depth estimation on the video frame using a depth estimation model to obtain a depth map, and then filtering the first mask sequence using the depth information in the depth map and at least one of the area threshold, confidence threshold, and depth range from the rule model to obtain the second mask sequence includes: Analyze the gradient of the depth map; If the depth gradient of the target region is greater than the preset gradient threshold, the region is determined to be occluded, and the depth filtering step is skipped, relying only on the rule model for filtering.

7. A video stream object tracking and detection apparatus, the apparatus comprising: The acquisition module is used to acquire the video stream and determine the semantic description of the target object to be tracked and detected; The selection module is used to adaptively configure the detection method combination and corresponding parameters according to the preset scene. The detection method combination includes a combination based on a multimodal large language model and an open-domain visual detection and segmentation model. The first generation module is used to input the semantic description into a multimodal large language model according to the selected combination of detection methods, generate the initial bounding box of the target object in the video frame, and use the initial bounding box as a prompt to input the open domain visual detection and segmentation model. The segmentation result is propagated forward and backward along the video time sequence using a bidirectional propagation mechanism to perform pixel-level segmentation on the video frame to obtain the first mask sequence corresponding to the target object. The second generation module is used to perform pixel-level depth estimation on the video frame using the depth estimation model when the detection method combination further includes a combination based on a depth estimation model and a rule model, to obtain a depth map, and to filter the first mask sequence using the depth information in the depth map and at least one of the area threshold, confidence threshold and depth range in the rule model to obtain a second mask sequence. The receiving module is used to receive at least one point annotation input by the user on a single frame of video when the confidence level of the automated processing result is lower than the quality threshold. The correction module is used to re-input the point annotations as prompts into the open domain visual detection and segmentation model, and to perform pixel-level segmentation and filter the first mask sequence again to generate a corrected second mask sequence.

8. A computer device, the computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the video stream object tracking and detection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the video stream object tracking and detection method as described in any one of claims 1 to 6.

Images