Statistical post-processing video object detection method based on multi-modal large model

Abstract

The application discloses a statistical post-processing video target detection method based on a multimodal large model, and relates to the technical field of open vocabulary video target detection. The application comprises the following steps: reading a video frame and establishing an index, constructing a strict output constraint prompt to drive multimodal large model inference; reading logits of model output tokens, aggregating pre-defined binary classification token scores, deriving stable frame-by-frame positive class probability, and avoiding dependence on generated text; extracting model internal vector representation and aligning with probability sequence by frame, screening significant dimensions based on differences between positive and negative sample groups, reducing dimensions, and weightedly fusing to construct a multivariate time sequence; locating target state change boundaries; calculating statistical quantities such as mean and peak values of probabilities in each segment, generating segment-level labels through rule judgment, and backtracking to assign frame-level results. The application solves the problems of low detection accuracy and poor interval structure consistency caused by unstable model output format and serious time sequence jitter in existing video target detection.

Description

Technical Field

[0001] This invention relates to the field of open vocabulary video object detection technology, and more specifically to a statistical post-processing video object detection method based on a multimodal large model. Background Technology

[0002] In real-world scenarios such as drone surveillance, security patrols, and traffic video, the common requirement is not target detection under traditional closed-category sets, but rather users temporarily specifying an object or concept (such as a type of object, a target with semantic description, or even a custom target not labeled in the dataset) and wanting to automatically locate the appearance, disappearance, or significant change of that target in a long sequence for alerting, retrieval, and event analysis. Existing technologies typically include: frame-by-frame bounding box output based on target detectors / trackers, or category probability output based on frame-by-frame classifiers, or using visual language models to perform natural language question answering on a single frame and determining the existence of the target based on the model's answer. These technical approaches often suffer from output instability or jitter under open vocabularies, custom targets, and complex shooting conditions (changes in viewpoint / scale, occlusion, motion blur, and lighting changes), and generate a large number of spurious changes when downstream "event localization / change point detection" is required. Furthermore, when directly using the "self-reported confidence score" of the model's natural language output as a continuous signal, the confidence score is often affected by prompt words, output style, and linguistic redundancy, lacking calibrability and repeatability, making it difficult to obtain stable input for statistical post-processing.

[0003] The current problems in the field of video object detection are as follows: Existing methods typically determine the existence of a target based directly on the natural language responses generated by the model or on the "yes / no" text results output by the model. This type of output is greatly affected by the design of prompt words, response style, linguistic redundancy, and generation randomness, making it difficult to form a stable, repeatable, and calibrable continuous statistical signal, thus hindering subsequent unified statistical analysis and robust discrimination.

[0004] In videos from drones, security systems, and other applications, common issues such as occlusion, scale changes, viewpoint changes, motion blur, and lighting variations can cause frame-by-frame output jitter, manifesting as the same target repeatedly appearing / disappearing between adjacent frames, detection box drift, or score fluctuations.

[0005] Existing methods lack statistical methods and processing, resulting in poor robustness and accuracy.

[0006] Currently, there is a lack of a solution that can target existence detection and event change point localization for open vocabularies / custom targets, utilize the internal representation of multimodal large models, and combine statistical learning methods. This would allow the system to maintain the semantic generalization ability of the large model while reducing jitter and improving the stability and interpretability of change point localization through statistical post-processing.

[0007] Therefore, proposing a statistical post-processing video target detection method based on a multimodal large model to address the difficulties of existing technologies is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0008] In view of this, the present invention provides a statistical post-processing video target detection method based on a multimodal large model, which solves the problems of low detection accuracy and poor interval structure consistency caused by unstable model output format and severe temporal jitter in existing video target detection.

[0009] To achieve the above objectives, the present invention provides the following technical solution: A statistical post-processing video object detection method based on a multimodal large model includes the following steps: S1. Read the video frame sequence directory or frame extraction output directory, compatible with any common image format filenames and sort them in natural order, then parse or generate... frame_idx ; S2, for each frame Construct a binary classification cue with the target description obj, input the image and cue into a multimodal large model M, and obtain frame-by-frame output; S3. During the model generation phase, read the logits of the first output token. Based on a predefined set of negative class tokens With positive class token set Calculate the positive class probability Symmetric confidence level ;use Thresholding yields frame-by-frame binary classification labels , Representing a frame The target object exists. This indicates that it does not exist; S4. Extract the vector for each frame from the hidden states of the multimodal large model M. , forming a matrix Where T is the total number of frames, D is the feature dimension, and the vector corresponding to each frame is also stored. frame_idx This forms a vector sequence; S5, with frame_idx As the key, align the vector sequence With binary sequences Probability sequences After alignment When there are frame drops, frame loss, or inconsistent frame sets between different modules, the frames are aligned using the intersection of indices and the aligned set of valid frames is output. S6, Press Will Divide into two groups and calculate the inter-group difference statistic dimension by dimension. Select the k dimensions with the greatest differences to form the index set. , to obtain the dimension-reduced vector ; S7. Reduce the dimension vector With continuous probability signals After frame alignment and fusion, a multivariate time series is constructed. ,in For adjustable weights, output the fused sequence file and record it in the filename or metadata. Key parameters; S8. Input the multivariate time series X into the change point detection algorithm and set an upper limit on the number of candidate change points. For each Calculate the optimal segmentation cost Obtain the cost curve and save it; For each Calculate the score Choose the one with the lowest score. and with The corresponding optimal set of variable points is denoted as the final output set of variable points. Or equivalent segmentation results; write the change point position, algorithm configuration and input sequence information into a structured result file to achieve traceability output for target appearance / disappearance event location; S9, Receive the set of variable points Index the aligned frame sequence By the set of change points Divided into interval sets If the change point is frame_idx This indicates that it is first mapped to the aligned sequence index or... frame_idx The space is directly segmented to ensure that the interval boundaries are consistent with the aligned data; S10, For each interval Extract frame-by-frame probability subsequences within the segment Calculate segment-level statistics At least the mean within the segment Peak value within the segment ; S11. Based on the discriminant function For each statistic The segment-level labels are obtained through discrimination. ; S12, divide each interval The output label for all frames is uniformly assigned to the segment-level label. Generate the final frame-level output sequence .

[0010] Optionally, the binary classification symbol set in S2 is {0,1} or {no,yes}.

[0011] Optional, the specific content in S3 is as follows: Predefined negative class token set With positive class token set ; Calculate the logarithmic scores for both types: ; Calculate the positive class probability :

[0012] in, It is the sigmoid function; Calculate symmetric confidence :

[0013] Frame-by-frame discrete discrimination is obtained through threshold comparison. , , Representing a frame The target object exists. This indicates that it does not exist.

[0014] Optional, vectors in S4 By analyzing the last hidden state of model M... attention mask The result is obtained by average pooling.

[0015] Optional, the specific content in S6 is as follows: according to Divided into two groups, for each dimension Calculate the difference statistic between groups ,use Cohen’s d :

[0016] in, Let be the mean of the positive and negative groups in the j-th dimension. To combine standard deviations; right Sort and select the index set of the top k dimensions The dimensionality-reduced vector is obtained. ,form .

[0017] Optional, the specific content in S8 is as follows: For each Calculate the optimal segmentation cost :

[0018] in, Representing an interval The intra-segment loss function; When using requires specifying When defining the algorithm, it is precisely at... The optimal cost under the constraint of each variable point is:

[0019] in, This indicates that the number of variables in the set of variables is . ,right Enumeration yields the cost curve ; For each Calculate the score Choose the one with the lowest score. and with The corresponding optimal set of variable points is denoted as the final output set of variable points. Or an equivalent segmented result; where,

[0020] Among them, the penalty item For the regularization terms relating to the number of variable points and sequence length, a BIC-like penalty is applied:

[0021] in, The value is ; The change point location, algorithm configuration, and input sequence information are written into a structured result file to achieve traceability output for target appearance / disappearance event location.

[0022] Optionally, S9 is based on Set of sequence indices Divide into m+1 consecutive intervals:

[0023] in, Representing an interval Number of intraframes.

[0024] Optionally, the specific content in S10 is as follows: For each interval Consider the frame-by-frame positive class probability subsequence within the interval. Define a set of segment-level statistics based on subsequences This includes: the mean within the segment, the peak value within the segment, the quantiles within the segment, and the proportion exceeding the threshold. The mean within a segment is defined as:

[0025] The peak value within a segment is defined as:

[0026] Intra-segment quantiles are defined as:

[0027] The proportion exceeding the threshold is defined as:

[0028] in, For threshold parameters, This is an indicator function.

[0029] Optional, the discriminant function in S11 Define a segment-level discriminant function as a rule-based discriminant function. Segment-level statistics Mapped to segment-level labels :

[0030] Rule-based discriminant functions include: 1) Mean threshold rule: If ,but ,otherwise ; 2) Joint rule of peak and mean: If ,but ,otherwise ; 3) Proportion rule: If ,but ,otherwise ; Among them, threshold and This is a preset value, or based on the entire sequence. The adaptive determined values ​​are obtained by estimating the quantiles, mean, and variance.

[0031] Optionally, each interval in S12 The output label for all frames is uniformly assigned to the segment-level label. Generate the final frame-level output sequence ,satisfy: .

[0032] As can be seen from the above technical solution, compared with the prior art, the present invention provides a statistical post-processing video target detection method based on a multimodal large model, which has the following beneficial effects: (1) This invention does not rely directly on the natural language answers generated by the multimodal large model as the final discrimination criterion, but extracts classification token logits related to the existence of the target from the model output and constructs frame-by-frame probability signals accordingly. In this way, the text output, which is originally greatly affected by the language expression form, can be transformed into a numerical, continuous, and repeatable discrimination signal, thereby solving the problems of unstable natural language output and difficulty in directly using self-reported confidence in statistical analysis in existing methods, and improving the consistency and computability of subsequent processing inputs. (2) In view of the problem that high-dimensional output vectors may contain a large number of changes that are irrelevant to the existence of the target, the present invention adopts a statistical dimension selection method of "grouping according to the result of the target existence judgment", calculates the inter-group difference statistics (preferably Cohen's d or equivalent index) on each dimension of the vector, and selects the top-k dimensions with the largest differences to form a task-related low-dimensional representation. This approach makes the features input to the sequence-level detection module more focused on the source of difference of "target existence / non-existence", which helps to reduce the false change points caused by irrelevant factors such as background and perspective changes. (3) This invention uses weight parameters Adjusting embedding and The relative contribution of the system allows for a trade-off between stability and sensitivity under different video conditions; (4) When using a change point detection algorithm that requires specifying the number of change points (e.g., dynamic programming), this invention provides a method to determine the optimal number of change points using a cost curve and a penalty criterion. Mechanism: Enumerate within the candidate range Obtain the optimal segmentation cost and introduce information about A penalty term based on sequence length is used to suppress excessive segmentation, thus achieving reproducibility. With the final set of change points; (5) In this invention, the output (set of changes) of the change point detection module is used to divide the frame sequence into intervals, and within each interval, the change point is... Calculate statistics such as mean, peak value, quantile, and over-threshold ratio within a segment; then, based on preset or adaptive rules, classify each interval as a segment-level label indicating "target presence / absence," thereby transforming the high-frequency jittery frame-level signal into a more robust output. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0034] Figure 1 The flowchart of the statistical post-processing video target detection method based on a multimodal large model provided by the present invention is shown below. Figure 2 A comparison diagram of the traditional frame-by-frame prediction method and the method of the present invention provided by the present invention; Figure 3 The image is the 145th frame of the sequence uav0000013_00000_v provided by this invention; Figure 4 Overall comparison chart of sequence uav0000013_00000_v provided by the present invention; Figure 5 The 34th frame image of the sequence uav0000013_01073_v provided by this invention; Figure 6 Overall comparison chart of sequence uav0000013_01073_v provided by the present invention; Figure 7 The 407th frame image of the sequence uav0000020_00406_v provided by this invention; Figure 8 Overall comparison chart of sequence uav0000020_00406_v provided by the present invention; Figure 9 The graph shows the changes in the five indicators provided for this invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] See Figure 1 As shown, a flowchart of a statistical post-processing video object detection method based on a multimodal large model is provided, where VLM represents the visual language model and M represents the multimodal large model. SEG-based This indicates that the output of this invention is based on segmentation followed by writing back.

[0037] This invention discloses a statistical post-processing video target detection method based on a multimodal large model, comprising the following steps: S1. Read the video frame sequence directory or frame extraction output directory, compatible with any common image format filenames and sort them in natural order, then parse or generate... frame_idx ; S2, for each frame Construct a binary classification cue with the target description obj, input the image and cue into a multimodal large model M, and obtain frame-by-frame output; S3. During the model generation phase, read the logits of the first output token. Based on a predefined set of negative class tokens With positive class token set Calculate the positive class probability Symmetric confidence level ;use Thresholding yields frame-by-frame binary classification labels , Representing a frame The target object exists. This indicates that it does not exist; S4. Extract the vector for each frame from the hidden states of the multimodal large model M. , forming a matrix Where T is the total number of frames, D is the feature dimension, and the vector corresponding to each frame is also stored. frame_idx This forms a vector sequence; S5, with frame_idx As the key, align the vector sequence With binary sequences Probability sequences After alignment When there are frame drops, frame loss, or inconsistent frame sets between different modules, the frames are aligned using the intersection of indices and the aligned set of valid frames is output. S6, Press Will Divide into two groups and calculate the inter-group difference statistic dimension by dimension. Select the k dimensions with the greatest differences to form the index set. , to obtain the dimension-reduced vector ; S7. Reduce the dimension vector With continuous probability signals After frame alignment and fusion, a multivariate time series is constructed. ,in For adjustable weights, output the fused sequence file and record it in the filename or metadata. Key parameters; S8. Input the multivariate time series X into the change point detection algorithm and set an upper limit on the number of candidate change points. For each Calculate the optimal segmentation cost Obtain the cost curve and save it; For each Calculate the score Choose the one with the lowest score. and with The corresponding optimal set of variable points is denoted as the final output set of variable points. Or equivalent segmentation results; write the change point position, algorithm configuration and input sequence information into a structured result file to achieve traceability output for target appearance / disappearance event location; S9, Receive the set of variable points Index the aligned frame sequence By the set of change points Divided into interval sets If the change point is frame_idx This indicates that it is first mapped to the aligned sequence index or... frame_idx The space is directly segmented to ensure that the interval boundaries are consistent with the aligned data; S10, For each interval Extract frame-by-frame probability subsequences within the segment Calculate segment-level statistics At least the mean within the segment Peak value within the segment ; S11. Based on the discriminant function For each statistic The segment-level labels are obtained through discrimination. ; S12, divide each interval The output label for all frames is uniformly assigned to the segment-level label. Generate the final frame-level output sequence .

[0038] Furthermore, the binary classification symbol set in S2 is either {0, 1} or {no, yes}.

[0039] Furthermore, the specific content of S3 is as follows: Predefined negative class token set With positive class token set ; Calculate the logarithmic scores for both types: ; Calculate the positive class probability :

[0040] in, It is the sigmoid function; Calculate symmetric confidence :

[0041] Frame-by-frame discrete discrimination is obtained through threshold comparison. , , Representing a frame The target object exists. This indicates that it does not exist.

[0042] Furthermore, in S4, vectors By analyzing the last hidden state of model M... attention mask The result is obtained by average pooling.

[0043] Furthermore, the specific content of S6 is as follows: according to Divided into two groups, for each dimension Calculate the difference statistic between groups ,use Cohen’s d :

[0044] in, Let be the mean of the positive and negative groups in the j-th dimension. To combine standard deviations; right Sort and select the index set of the top k dimensions The dimensionality-reduced vector is obtained. ,form .

[0045] Furthermore, the specific content of S8 is as follows: For each Calculate the optimal segmentation cost :

[0046] in, Representing an interval The intra-segment loss function; When using requires specifying When defining the algorithm, it is precisely at... The optimal cost under the constraint of each variable point is:

[0047] in, This indicates that the number of variables in the set of variables is . ,right Enumeration yields the cost curve ; For each Calculate the score Choose the one with the lowest score. and with The corresponding optimal set of variable points is denoted as the final output set of variable points. Or an equivalent segmented result; where,

[0048] Among them, the penalty item For the regularization terms relating to the number of variable points and sequence length, a BIC-like penalty is applied:

[0049] in, The value is ; The change point location, algorithm configuration, and input sequence information are written into a structured result file to achieve traceability output for target appearance / disappearance event location.

[0050] Furthermore, S9 is based on Set of sequence indices Divide into m+1 consecutive intervals:

[0051] in, Representing an interval Number of intraframes.

[0052] Furthermore, the specific content of S10 is as follows: For each interval Consider the frame-by-frame positive class probability subsequence within the interval. Define a set of segment-level statistics based on subsequences This includes: the mean within the segment, the peak value within the segment, the quantiles within the segment, and the proportion exceeding the threshold. The mean within a segment is defined as:

[0053] The peak value within a segment is defined as:

[0054] Intra-segment quantiles are defined as:

[0055] The proportion exceeding the threshold is defined as:

[0056] in, For threshold parameters, This is an indicator function.

[0057] Furthermore, the discriminant function in S11 Define a segment-level discriminant function as a rule-based discriminant function. Segment-level statistics Mapped to segment-level labels :

[0058] Rule-based discriminant functions include: 1) Mean threshold rule: If ,but ,otherwise ; 2) Joint rule of peak and mean: If ,but ,otherwise ; 3) Proportion rule: If ,but ,otherwise ; Among them, threshold and This is a preset value, or based on the entire sequence. The adaptive determined values ​​are obtained by estimating the quantiles, mean, and variance.

[0059] Furthermore, in S12, each interval The output label for all frames is uniformly assigned to the segment-level label. Generate the final frame-level output sequence ,satisfy: .

[0060] In one specific embodiment, the following is included: This invention first constructs a frame-by-frame probability signal by extracting classification token logits, which transforms the model output from an unstable natural language form into a continuous numerical signal, making it more suitable for subsequent statistical analysis and unified discrimination, thus improving the stability and repeatability of the output signal.

[0061] The output format changes from "frame-level jitter" to "interval-level existence result". The final output is no longer a frequent flipping of 0 / 1 frames, but rather several consecutive intervals, each uniformly given 0 or 1, and simultaneously outputting the interval's... Explanatory values ​​include mean and peak value.

[0062] like Figure 2 As shown in the drone dataset VisDrone The verification was performed using a multimodal large model. Qwen2.5vl-7B Model, prompt (Prompt word) is: f"Decide whether the target object '{object_name}' appears in the image"

[0063] f"Output exactly one character: {pos_label} = yes, {neg_label} = no"

[0064] f"Do not output any other text, spaces, or newlines"

[0065] in,{ object_name} represents the input detection target, { pos_label}and{ neg_label The values ​​were selected as 1 and 0.

[0066] The following is an example of the effect.

[0067] In the example sequence uav0000013_00000_v Above (frame 269), the detected target is black_dog This invention segments the frame sequence based on change point detection and performs segmentation within each interval. After calculating statistics (such as the mean), interval-level predictions are generated, and then these interval-level predictions are written back into a frame-level unified prediction sequence. Simultaneously, frame-level truth values ​​are generated using the same manually labeled real-world intervals. Compared with "frame-by-frame original prediction ( RAW ")" and "Segmented and then written back to predict () post The performance difference is shown in the example. The percentage of true positive frames is 0.77695.

[0068] In terms of correctness, the segmented write-back prediction of this invention ( post It outperforms frame-by-frame raw prediction in frame-level accuracy, F1 score, and IoU. RAW ): RAW (frame by frame) answer ):

[0069] This invention ( post According to the section (Mean rule generates segment-level predictions and writes them back):

[0070] In terms of robustness, RAW : This invention: ,

[0071] In the example sequence uav0000013_01073_v Above (58 frames), the detected target is dog In this example, the proportion of positive frames is 0.586207.

[0072] RAW (frame by frame) answer ):

[0073] This invention ( post According to the section (Mean rule generates segment-level predictions and writes them back):

[0074] In the example sequence uav0000020_00406_v Above (frame 501), the detected target is motor (Blurred and difficult to distinguish), the proportion of positive frames in this example is 0.740519.

[0075] RAW (frame by frame) answer ):

[0076] This invention ( post According to the section (Mean rule generates segment-level predictions and writes them back):

[0077] exist VisDrone-2019-train The metrics for the dataset, consisting of 32 available video frame sequences and a total of 13,191 frames, are calculated as follows: direction This indicates that higher is better or lower is better. raw_mean Represents the original frame by frame answer The average value of the indicator post_mean This invention represents answer The average value of the indicator delta_mean This represents the latter minus the former. The last row is the indicator of the decline in the reversal rate. mean The average value is represented by . As shown in Table 1, the performance metrics of the method of this invention on the VisDrone-2019-train dataset are compared as follows.

[0078] Table 1. Performance metrics comparison results of the method of the present invention on the VisDrone-2019-train dataset.

[0079] in, Figure 3 The image is the 145th frame of the sequence uav0000013_00000_v. The multimodal large model failed to identify the black dog because its body was hunched over, resulting in abnormal fluctuations in the answer and confidence. However, the method of this invention successfully classified this frame as a successful identification, improving accuracy while reducing the instability of the prediction.

[0080] Figure 4The top image shows the frame-by-frame probability signal constructed from token logits where the answer is "1" (representing "yes"), given the prompt and the sequence. The bottom image shows the ground truth (GT), raw (frame-by-frame answer), and post (intra-segment probability) of the sequence. The comparison chart of the mean rule-generated segment-level predictions and the predictions written back shows that it reduces the instability of the large model's frame-by-frame output.

[0081] Figure 5 The image is the 34th frame of the sequence uav0000013_01073_v. Due to the dog's actions and other reasons, the multimodal large model exhibited extremely high uncertainty and failed to identify the image, resulting in abnormal fluctuations in the answer and confidence level. However, the method of this invention successfully classified this frame as a successful identification, improving accuracy while reducing the instability of the prediction.

[0082] Figure 6 The top figure shows the frame-by-frame probability signal extracted from the multimodal large model under the given prompt and sequence conditions; the bottom figure is a comparison of the ground truth (GT), raw data, and post data of the sequence.

[0083] Figure 7 The image is the 407th frame of the sequence uav0000020_00406_v. Due to complex environment and other reasons, the multimodal large model experienced hallucinations and failed to identify the motor, showing frequent fluctuations in the answer and confidence. However, the method of this invention successfully classified this frame as a successful identification, improving accuracy while reducing the instability of prediction.

[0084] Figure 8 The top figure shows the frame-by-frame probability signal extracted from the multimodal large model under the given prompt and sequence conditions; the bottom figure is a comparison of the ground truth (GT), raw data, and post data of the sequence.

[0085] Figure 9 The results were obtained using the VisDrone-2019-train dataset, with 32 available video frame sequences, totaling 13,191 images. Five metrics were analyzed, with green representing improvement and red representing decrease.

[0086] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0087] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A statistical post-processing video target detection method based on a multimodal large model, characterized in that, Includes the following steps: S1. Read the video frame sequence directory or frame extraction output directory, compatible with any common image format filenames and sort them in natural order, then parse or generate... frame_idx ; S2, for each frame Construct a binary classification cue with the target description obj, input the image and cue into a multimodal large model M, and obtain frame-by-frame output; S3. During the model generation phase, read the logits of the first output token. Based on a predefined set of negative class tokens With positive class token set Calculate the positive class probability Symmetric confidence level ;use Thresholding yields frame-by-frame binary classification labels , Representing a frame The target object exists. This indicates that it does not exist; S4. Extract the vector for each frame from the hidden states of the multimodal large model M. , forming a matrix Where T is the total number of frames, D is the feature dimension, and the vector corresponding to each frame is also stored. frame_idx This forms a vector sequence; S5, with frame_idx As the key, align the vector sequence With binary sequences Probability sequences After alignment When there are frame drops, frame loss, or inconsistent frame sets between different modules, the frames are aligned using the intersection of indices and the aligned set of valid frames is output. S6, Press Will Divide into two groups and calculate the inter-group difference statistic dimension by dimension. Select the k dimensions with the greatest differences to form the index set. , to obtain the dimension-reduced vector ; S7. Reduce the dimension vector With continuous probability signals After frame alignment and fusion, a multivariate time series is constructed. ,in For adjustable weights, output the fused sequence file and record it in the filename or metadata. Key parameters; S8. Input the multivariate time series X into the change point detection algorithm and set an upper limit on the number of candidate change points. For each Calculate the optimal segmentation cost Obtain the cost curve and save it; For each Calculate the score Choose the one with the lowest score. and with The corresponding optimal set of variable points is denoted as the final output set of variable points. Or equivalent segmentation results; write the change point position, algorithm configuration and input sequence information into a structured result file to achieve traceability output for target appearance / disappearance event location; S9, Receive the set of variable points Index the aligned frame sequence By the set of change points Divided into interval sets If the change point is frame_idx This indicates that it is first mapped to the aligned sequence index or... frame_ idx The space is directly segmented to ensure that the interval boundaries are consistent with the aligned data; S10, For each interval Extract frame-by-frame probability subsequences within the segment Calculate segment-level statistics At least the mean within the segment Peak value within the segment ; S11. Based on the discriminant function For each statistic The segment-level labels are obtained through discrimination. ; S12, divide each interval The output label for all frames is uniformly assigned to the segment-level label. Generate the final frame-level output sequence .

2. The statistical post-processing video target detection method based on a multimodal large model according to claim 1, characterized in that, In S2, the binary classification symbol set is either {0, 1} or {no, yes}.

3. The statistical post-processing video target detection method based on a multimodal large model according to claim 1, characterized in that, The specific content in S3 is as follows: Predefined negative class token set With positive class token set ; Calculate the logarithmic scores for both types: ； Calculate the positive class probability : in, It is the sigmoid function; Calculate symmetric confidence : Frame-by-frame discrete discrimination is obtained through threshold comparison. , , Representing a frame The target object exists. This indicates that it does not exist.

4. The statistical post-processing video target detection method based on a multimodal large model according to claim 1, characterized in that, Vector in S4 By analyzing the last hidden state of model M... attention mask The result is obtained by average pooling.

5. The statistical post-processing video target detection method based on a multimodal large model according to claim 1, characterized in that, The specific content of S6 is as follows: according to Divided into two groups, for each dimension Calculate the difference statistic between groups ,use Cohen's d : in, Let be the mean of the positive and negative groups in the j-th dimension. To combine standard deviations; right Sort and select the index set of the top k dimensions The dimensionality-reduced vector is obtained. ,form .

6. The statistical post-processing video target detection method based on a multimodal large model according to claim 1, characterized in that, The specific content of S8 is as follows: For each Calculate the optimal segmentation cost : in, Representing an interval The intra-segment loss function; When using requires specifying When defining the algorithm, it is precisely at... The optimal cost under the constraint of each variable point is: in, This indicates that the number of variables in the set of variables is . ,right Enumeration yields the cost curve ; For each Calculate the score Choose the one with the lowest score. and with The corresponding optimal set of variable points is denoted as the final output set of variable points. Or an equivalent segmented result; where, Among them, the penalty item For the regularization terms relating to the number of variable points and sequence length, a BIC-like penalty is applied: in, The value is ; The change point location, algorithm configuration, and input sequence information are written into a structured result file to achieve traceability output for target appearance / disappearance event location.

7. The statistical post-processing video target detection method based on a multimodal large model according to claim 1, characterized in that, S9 based on Set of sequence indices Divide into m+1 consecutive intervals: in, Representing an interval Number of intraframes.

8. The statistical post-processing video target detection method based on a multimodal large model according to claim 1, characterized in that, The specific content of S10 is as follows: For each interval Consider the frame-by-frame positive class probability subsequence within the interval. Define a set of segment-level statistics based on subsequences This includes: the mean within the segment, the peak value within the segment, the quantiles within the segment, and the proportion exceeding the threshold. The mean within a segment is defined as: The peak value within a segment is defined as: Intra-segment quantiles are defined as: The proportion exceeding the threshold is defined as: in, For threshold parameters, This is an indicator function.

9. The statistical post-processing video target detection method based on a multimodal large model according to claim 1, characterized in that, Discriminant function in S11 Define a segment-level discriminant function as a rule-based discriminant function. Segment-level statistics Mapped to segment-level labels : Rule-based discriminant functions include: 1) Mean threshold rule: If ,but ,otherwise ; 2) Joint rule of peak and mean: If ,but ,otherwise ; 3) Proportion rule: If ,but ,otherwise ; Among them, threshold and This is a preset value, or based on the entire sequence. The adaptive determined values ​​are obtained by estimating the quantiles, mean, and variance.

10. The statistical post-processing video target detection method based on a multimodal large model according to claim 1, characterized in that, S12 will include each interval The output label for all frames is uniformly assigned to the segment-level label. Generate the final frame-level output sequence ,satisfy: 。

Images