An event time positioning method and device, computer equipment and storage medium

Abstract

The application discloses an event time positioning method and device, computer equipment and a storage medium, and the method comprises the following steps: acquiring historical event time positioning data; performing time alignment processing on a query instruction and video data and audio data to obtain alignment data; constructing triple training data based on the alignment data, training a triple cross-attention model by using the triple training data, and obtaining an initial time positioning model; acquiring loss data in the training process, and optimizing parameters of the initial time positioning model by using the loss data; performing structure fine-tuning on the initial time positioning model after parameter optimization by using compression technology to obtain a target time positioning model; and deploying the target time positioning model and performing inference prediction by using the target time positioning model. The application can improve the event time positioning efficiency and enhance the event time positioning effect.

Description

Technical Field

[0001] This invention relates to the field of computer software technology, and in particular to an event timing location method, apparatus, computer device, and storage medium. Background Technology

[0002] With the development of artificial intelligence and deep learning technologies, computer vision has achieved remarkable success in areas such as object detection and behavior recognition. However, these pure vision models still face many challenges in practical security applications. First, they typically rely on large amounts of labeled data to maintain high performance, but acquiring high-quality training data covering various abnormal events is both time-consuming and expensive. Taking property security scenarios as an example, it is difficult to collect enough samples for supervised training of low-frequency events such as vehicle scratches. Second, traditional vision models lack the ability to understand the semantic context of a scene, making it difficult to identify visually subtle but semantically important details—such as minor scratches or collision marks on a vehicle surface—which may be overlooked in the absence of additional information.

[0003] To address the aforementioned issues, multimodal large-scale models are becoming a research hotspot. By fusing information from multiple sources such as vision, audio, and text, these models promise to achieve a more comprehensive understanding of scenes. However, existing multimodal methods still have several shortcomings: First, most audio and video event detection studies use closed-set settings, meaning the models can only identify predefined event categories and struggle to generalize to new events not present in the training set. Even studies exploring open-set detection can only label unknown events as "unknown," failing to provide specific semantic categories and limiting their ability to handle novel or fine-grained events. Second, there is an inherent semantic gap between different modalities; the heterogeneity of audio and visual signals easily leads to inconsistencies in multimodal semantics. For example, based solely on scratch sounds or subtle changes in images, different modalities may make inconsistent judgments about events, requiring effective alignment mechanisms for fusion. Third, existing multimodal models often rely on large-scale, uniformly distributed training data. When facing special scenarios such as property security, differences in data distribution can lead to performance degradation and cross-domain data issues. Furthermore, large-scale multimodal models (such as cross-modal pre-trained models with billions of parameters) incur enormous inference costs, making them difficult to deploy in real-time on front-end devices. Research has found that even the most advanced multimodal models currently available still fall short in identifying visual anomalies and details, failing to accurately detect fine-grained anomalies in images. In summary, existing technologies have significant shortcomings in terms of training cost, semantic understanding, multimodal alignment, and lightweight model deployment.

[0004] Therefore, how to efficiently utilize existing multimodal data resources to integrate visual, audio, and textual information in property security scenarios to eliminate semantic differences between different modalities, thereby improving the model's ability to generalize and detect abnormal events, while simultaneously reducing the demand for training data and the size of the model, solving cross-domain data problems, and reducing deployment and maintenance costs to meet the needs of practical applications, is a problem that those skilled in the art need to solve. Summary of the Invention

[0005] This invention provides an event time localization method, apparatus, computer device, and storage medium, aiming to solve the problems of high training cost, insufficient semantic understanding, difficulty in multimodal alignment, large model size and difficulty in deployment, and cross-domain data in the prior art, thereby improving the efficiency and effectiveness of event time localization.

[0006] In a first aspect, embodiments of the present invention provide an event time location method, including:

[0007] Obtain historical event time location data; wherein, the historical event time location data includes query commands and corresponding video and audio data;

[0008] The query command and its corresponding video and audio data are time-aligned to obtain aligned data.

[0009] Based on the alignment data, triple training data is constructed, and the triple cross-attention model is trained using the triple training data to obtain the initial time localization model.

[0010] Obtain the loss data during the training process, and use the loss data to optimize the parameters of the initial time localization model;

[0011] The initial time-localization model after parameter optimization is fine-tuned using compression technology to obtain the target time-localization model.

[0012] The target time positioning model is deployed, and inference and prediction are performed on the specified event time query data using the target time positioning model.

[0013] Secondly, embodiments of the present invention provide an event time positioning device, comprising:

[0014] A data acquisition unit is used to acquire historical event time location data; wherein, the historical event time location data includes query instructions and corresponding video and audio data;

[0015] An alignment processing unit is used to perform time alignment processing on the query command and the corresponding video and audio data to obtain aligned data.

[0016] The model training unit is used to construct triple training data based on the alignment data, and to train the triple cross-attention model using the triple training data to obtain the initial time localization model.

[0017] The parameter optimization unit is used to acquire loss data during the training process and optimize the parameters of the initial time localization model using the loss data.

[0018] The structural fine-tuning unit is used to fine-tune the structure of the initial time-localization model after parameter optimization using compression technology to obtain the target time-localization model.

[0019] A deployment inference unit is used to deploy the target time positioning model and use the target time positioning model to perform inference and prediction on the specified event time query data.

[0020] Thirdly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the event timing method as described in the first aspect.

[0021] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the event timing location method as described in the first aspect.

[0022] This invention provides an event time localization method, apparatus, computer device, and storage medium. The method includes: acquiring historical event time localization data; wherein the historical event time localization data includes query instructions and corresponding video and audio data; performing time alignment processing on the query instructions and corresponding video and audio data to obtain aligned data; constructing triple training data based on the aligned data, and training a triple cross-attention model using the triple training data to obtain an initial time localization model; acquiring loss data during the training process, and optimizing the parameters of the initial time localization model using the loss data; fine-tuning the structure of the parameter-optimized initial time localization model using compression technology to obtain a target time localization model; deploying the target time localization model, and using the target time localization model to perform inference and prediction on specified event time query data. This invention effectively eliminates semantic differences between different modalities through cross-modal alignment and fusion processing, improving the model's generalization detection capability for abnormal events. Simultaneously, the precise selection of candidate time segments through boundary regression further enhances the accuracy of time localization. In addition, compression technology was used to fine-tune the structure of the model, reducing its size, solving the problem of cross-domain data, and reducing deployment and maintenance costs. This solved the problems of high training costs, insufficient semantic understanding, difficulty in multimodal alignment, large model size and difficulty in deployment, and cross-domain data in existing technologies, thus improving the efficiency of event time localization and enhancing the effect of event time localization. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating an event time location method provided in an embodiment of the present invention;

[0025] Figure 2 A principle block diagram of an event time positioning method provided in an embodiment of the present invention;

[0026] Figure 3 This is a schematic block diagram of an event time positioning device provided in an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0029] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0030] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0031] Please see below. Figure 1 The present invention provides an event time location method, which specifically includes steps S101 to S106.

[0032] Step S101: Obtain historical event time location data; wherein, the historical event time location data includes query instructions and corresponding video and audio data;

[0033] Step S102: Perform time alignment processing on the query command and the corresponding video and audio data to obtain aligned data;

[0034] Step S103: Construct triple training data based on the alignment data, and use the triple training data to train the triple cross-attention model to obtain the initial time localization model.

[0035] Step S104: Obtain the loss data during the training process, and use the loss data to optimize the parameters of the initial time localization model;

[0036] Step S105: Use compression technology to fine-tune the structure of the initial time positioning model after parameter optimization to obtain the target time positioning model;

[0037] Step S106: Deploy the target time positioning model and use the target time positioning model to perform inference and prediction on the specified event time query data.

[0038] In this embodiment, combined with Figure 2 First, historical event time-location data containing query commands, video, and audio is acquired. This data is then time-aligned to generate aligned data. Triple training data is constructed based on the aligned data, and a triple cross-attention model is trained using this data to obtain the initial time-location model. During training, the model parameters are optimized using loss data, and compression techniques are used to fine-tune the structure of the optimized model, forming the target time-location model. Finally, the target model is deployed and used for inference and prediction of specified event time query data.

[0039] This embodiment effectively eliminates semantic differences between different modalities through cross-modal alignment and fusion processing, improving the model's generalization ability to detect abnormal events. Simultaneously, it uses a boundary regression mechanism to accurately filter candidate time segments, further enhancing the accuracy of time localization. Furthermore, it utilizes compression technology to fine-tune the model structure, reducing its size, resolving the cross-domain data issue, and decreasing deployment and maintenance costs. This addresses existing problems such as high training costs, insufficient semantic understanding, difficulties in multimodal alignment, large model size and deployment challenges, and cross-domain data issues, thereby improving the efficiency and effectiveness of event time localization.

[0040] In particular, the event time localization method provided in this embodiment is especially applicable to abnormal event detection and decision-making in property security scenarios. This method effectively solves the problems faced by traditional technologies, such as high training costs, insufficient contextual semantic understanding capabilities, poor cross-domain adaptability, and large model size that makes deployment difficult. Based on the event time localization method provided in this embodiment, the occurrence time of abnormal events such as vehicle collisions can be automatically and accurately located in long-term videos, significantly improving the intelligence level and operational efficiency of security monitoring.

[0041] In practical applications, the problem is described first. Let the monitoring duration be T, and the video frame rate be f. v The audio sampling rate is f a Continuous time is discretized as follows:

[0042] ;

[0043] in, Where M is the video frame interval, and M is the floor function of the total number of frames. This is the timestamp of the m-th frame.

[0044] Video modality is denoted as: ,in, ;

[0045] The audio waveform is as follows: Its time spectrum is obtained through short-time Fourier transform (STFT):

[0046] ;

[0047] in, Here, h is the window function, h is the frame shift, and N is the number of FFT points; This represents the complex spectral coefficients at the k-th frequency point in the n-th frame. The text query (or command) is q, for example, q = "locate the time period during which the white vehicle was scratched". α is the discrete audio sequence. It is at the sampling rate f a The waveform value obtained below; This represents the amplitude of the τ-th sampling point, typically normalized to [-1, 1]. τ is the time sampling index (integer) used for summation, in units of "sampling points". Its relationship to physical time is as follows: n is the frame index of the STFT (the nth time window). The interval between the start points of adjacent frames is h sampling points, corresponding to the window function term. j is the imaginary unit. (The engineering notation often uses 'j' to distinguish it from the current symbol 'i'). It is a complex exponential basis function. represents the complex coefficients of the k-th frequency bin in the n-th frame; N is the number of FFT points. .

[0048] Divide the timeline into a set of candidate segments of length L and stride s:

[0049] ;

[0050] in, Let the u-th candidate time slice (window) be defined as an interval. Here, "[...)" indicates a left-closed, right-open interval. This notation is used to avoid double counting of adjacent windows at boundary times. Let u be the starting time of the u-th window, increasing by a step size s: u is the window index. Take discrete non-negative integers. U is the maximum window index. If the total duration is T, then... .

[0051] Given a trimodal input The model outputs a matching score for each candidate segment:

[0052] ;

[0053] Where X is the trimodal input space, and its elements are... V represents the video sequence (frame stream), A represents the audio representation (waveform or time spectrum), and q represents the text query / command. The real number field is the output range of the scoring function; in the implementation, it can be further mapped to [0,1] probability or confidence level. For the u-th candidate segment The matching score (scalar) given input X is used for segment ranking and alarm threshold determination. S is the segment scoring function / model, defined as follows: It receives candidate segments and trimodal inputs and outputs matching scores; in engineering, it consists of an encoder, a fusion layer, and a task header parameterized as follows: .

[0054] And return according to Descending order sorting is used for event matching and time period location. If the set of truthful event intervals is... Then the temporal intersection-union ratio (IoU) is defined as:

[0055] ;

[0056] Here, Abbreviations indicating the start and end times of a segment / event. Indicates the starting point, Indicates the endpoint; in local derivations, if written as... and Then all are with , Equivalent. S is the set of intervals of truth-valued events. `C` represents the number of event intervals (number of events) to be labeled. `C` represents the set of candidate segments (time windows). `c` is the notation (generic) for any candidate segment; its specific form is a time interval, such as... . Indicates the duration (measure) of an interval; for example . These represent the intersection and union operations of intervals on the time axis, respectively. Max represents the maximum value operator, which here takes the interval with the largest overlap ratio among all truth intervals in S.

[0057] The training goal is to make and Monotonic consistency, or making the score of positive sample segments (high tIoU) greater than that of negative sample segments.

[0058] In one embodiment, step S102 includes:

[0059] The video data and audio data are respectively encoded into time-series features of the same frequency;

[0060] The query command and the time series features are aligned using the optimal alignment offset estimated by the cross-correlation of energy / motion correlation to obtain the aligned data.

[0061] To unify the time steps, this embodiment encodes visual and audio data into temporal features of the same frequency:

[0062] ;

[0063] in, For the temporal feature sequence of video modalities, Each of them . This is a temporal feature sequence of audio modalities. Each of them . These are all feature vectors at the m-th time step, corresponding to video / audio respectively, with a corresponding dimension of d. v and d a m is the time step index, a discrete integer. The two modalities share the same time step after alignment. M is the maximum time step index (the unified length is M+1). For example, according to the video frame rate f... v During sampling, . A video encoder, used to map the original video sequence V into temporal features, denoted as... . An audio encoder is used to map audio input A (such as an STFT spectrum) into temporal features, denoted as... .

[0064] The text is encoded as follows: . The global embedding vector (fixed-length representation) for the text modality is then aligned and fused with video / audio features, with a dimension of [missing information]. . For text / language encoders (such as sentence vectors or LLM text towers), natural language queries are mapped to vectors: q is a string representing a text query / business instruction, such as "locate the time period during which the white vehicle was scratched". It is the text feature dimension (number of channels), which matches the projection dimension of the fusion layer.

[0065] Considering the audio-visual asynchrony, the optimal alignment offset is first estimated using the cross-correlation of energy / motion correlation. :

[0066] ;

[0067] in, This represents the optimal alignment offset (unit: shared time step / frame), which is the time shift that maximizes the correlation between the video energy sequence and the audio energy sequence. This refers to selecting the independent variable that maximizes the objective function; in this case, it means choosing the optimal value within a given search interval. . , indicates moving the search range. The maximum allowed time shift (in shared time steps). The correlation coefficient (usually Pearson correlation) is calculated over the overlapping intervals of all effective m, with a value range of [-1, 1]. This represents the sequence of video "energy / motion intensity" metrics (such as inter-frame optical flow amplitude, pixel difference energy, etc.) formed at the m-th time step. Indicates the audio at the 1st A sequence of energy metrics (such as band energy or spectral flux) at each time step; Perform a timeline shift. m is the unified time step index, a discrete integer. .

[0068] Subsequently ordered To achieve time alignment. The aligned audio features (already synchronized with the video timeline) are represented by a vector at the m-th time step, with dimensions of... . For the original audio feature sequence in the index The value at that point is equivalent to shifting the entire audio segment along the time axis. After one time step, sampling is performed at the m-th step. For the audio feature sequence items before alignment, . The optimal time-shift bias (obtained from maximum correlation estimation) is expressed in shared time steps per frame. The superscript 'a' indicates the "audio" modality, distinguishing it from video (v) and text (l).

[0069] In one embodiment, step S103 includes:

[0070] The duration corresponding to the video data is obtained, and the time axis of the video data is divided into multiple initial time segments in combination with a preset stride.

[0071] Define a segment-level feature set for each initial time segment, and set soft labels with smoothed boundaries for the initial time segments;

[0072] Based on the query command, the attribute probability of the historical event time location data is set, and the attribute probability is used to set a consistency weight for the initial time segment. Then, the soft tag is gating and enhanced in combination with the consistency weight.

[0073] Set segment difficulty scores and quality indicators for the initial time segments;

[0074] Candidate time segments are obtained by combining the soft tags, segment difficulty scores, and quality indicators from the initial time segments;

[0075] Based on the candidate time segments, the triplet training data is constructed by combining the query command and audio data.

[0076] This embodiment acquires the video data duration and divides it into initial time segments. For each segment, a feature set is defined, and soft labels with smoothed boundaries are applied, enabling a more detailed description of the segment content. Simultaneously, attribute probabilities of historical event time-location data are set based on query commands, and these probabilities are used to assign consistency weights to the initial time segments. Furthermore, gating is used to enhance the soft labels, improving label accuracy. In addition, setting segment difficulty scores and quality indicators for the initial time segments helps to filter out more representative candidate time segments. Finally, triplet training data is constructed by combining candidate time segments, query commands, and audio data, providing a high-quality data foundation for subsequent model training.

[0077] Specifically, for each initial time segment Define fragment-level feature aggregation:

[0078] ;

[0079] in, For the u-th candidate segment The visual aggregation features are obtained by temporal aggregation of video frame features within the window: . Let m be the feature vector of the video frame at the m-th time step, with dimension m. . The audio aggregation feature of the u-th candidate segment is obtained by temporal aggregation of the aligned audio features: . The aligned audio features at the m-th time step (see previous alignment offset) ), dimension . This is a fragment-level text representation, serving as a shared text vector for each fragment, used for cross-modal fusion with visual / audio input. . For global text embedding, by the text encoder The resulting fixed-length vector, . This represents a time aggregation operator, which can be mean pooling / weighted average / attention pooling; the output dimension is consistent with the input channel dimension. Take the time index m that falls within the window. Aggregate those features within.

[0080] It should be noted here that, combined with the above formula, the text modality does not have a time-varying sequence axis, and the encoding of query q... This is shared across all segments, so "segment-level text representation" is simply an alias for the same vector: The top horizontal line This only indicates the notation convention for "fragment-level aggregation representation" and does not change its value.

[0081] Fragment binary labels are given by thresholded temporal IoU:

[0082] ;

[0083] in, For the u-th candidate segment The binary label, when its time intersection with the set of truth intervals S is greater than the value of the binary label, is determined by the following formula: It takes the value 1 if the condition is met, and 0 otherwise. Therefore . This indicates an indicator function. It returns 1 if the condition within the parentheses is true, and 0 if it is false.

[0084] To reduce boundary blurring, soft labels with smoothed boundaries are used:

[0085] ;

[0086] in, For the u-th candidate segment The soft tag (value 0-1). Representing an interval The boundary set is equivalent to This is used to measure the minimum time distance from a fragment to an event boundary. ;like If the distance overlaps with the event interval, then the distance is [value]. The minimum time distance between the fragment and the event boundary. For boundary decay weights, Controls the smoothness level. d indicates the smoothness level. The boundary distance in is defined as Candidate fragments Minimum time distance to the boundary of the truth interval (The unit can be seconds or frames, depending on the timeline definition). A larger d indicates that the segment is farther from the boundary, and the weight... The smaller the value, the lower the weight; when it is close to or covers the event boundary, d=0 and the weight is 1.

[0087] For the query "white vehicle was scratched", estimate the probability of color attribute for each vehicle trajectory r. With vehicle mask / trajectory coverage (Indicates that trajectory r is in the segment) (Time percentage or IoU within the segment). Define the attribute consistency weights for the segment:

[0088] ;

[0089] Where r represents the index / ID of the vehicle trajectory (a continuous trajectory obtained by detection and multi-target tracking). In candidate fragments The set of trajectories that appear or overlap with the time window (i.e., all vehicle trajectories visible within the segment). Indicates taking all segments Each candidate trajectory r within is used to... The search is for the "best match" trajectory.

[0090] And enhance gating for soft tags:

[0091] ;

[0092] in, For fragments Attribute-gated enhanced soft tags (range 0-1). Based on the basic soft tag Above, if a "white vehicle" exists within the fragment and its trajectory is highly consistent ( (A larger value) increases the confidence of positive samples. Keep the upper bound at 1, where Control the magnitude of the increase. Here... Controlling attribute matching improves the confidence of positive samples. To bind a specific "scratched vehicle," the trajectory ID can be used. Consistent explicit injection: Let . This refers to the specified target vehicle trajectory ID (i.e., the trajectory of the vehicle that was "scratched"). Superscript These are marker symbols, indicating that from all trajectories The selected / bound entry. This is typically determined by manual annotation, license plate / appearance ReID binding, or rule-based checks. Once selected, the consistency weight changes from maximization to explicit injection.

[0093] ;

[0094] in, Let this be the probability that the trajectory belongs to a white vehicle. For the fragment The percentage of time spent within the period or the IoU.

[0095] Difficulty example mining and sampling distribution. Define segment difficulty scores (which can combine audio bursts and visual motion):

[0096] ;

[0097] in, The visual feature vector at the m-th time step (frame) is generated by the visual encoder. Extraction, dimension is . The visual features at the (m-1)th time step; Measure the magnitude of motion / appearance changes between adjacent frames. Candidate segments Difficulty score: a weighted sum of the frame average of visual "motion intensity" and the frame average of audio "spectral flux"; The number of time steps within the segment. For the two-modal weights. It is a norm 2. This represents the audio spectral flux.

[0098] Constructing a hierarchical sampling distribution:

[0099] ;

[0100] in, Controlling the ratio of positive to negative samples Control the intensity of excavation in difficult cases. Will Normalization to .

[0101] Data quality screening and cross-domain equilibrium. Let the quality index (such as AP or stability score obtained based on the prior model) of each original video i be denoted as . The probability of its selection is:

[0102] ;

[0103] in, q represents the quality score of video i. i The result of power-law amplification (or compression) of ∈[0,1]; β is an adjustable hyperparameter. When β=0, this factor is always 1 (does not distinguish between quality); when 0<β<1, the quality difference is weakened; when β>1, the preference for high-quality samples is strengthened. The domain / scene to which the video belongs (e.g., underground parking lot, ground level, rainy night, etc.). For empirical domain distribution estimation, To achieve inverse frequency reweighting, Control your preference for high-quality data. The sampling probability / weight for selecting video i to enter the training batch; This indicates that normalization is required across the entire set.

[0104] The final training set consists of the following formula:

[0105] ;

[0106] It is equipped with fragment-level sampling weights. Video-level selection weights .

[0107] Training triplets for ranking supervision and contrastive alignment. For subsequent ranking and alignment learning, a set of triplets is constructed based on the same video / query:

[0108] ;

[0109] in, It is an index for positive sample segments (with a high degree of matching with historical event time location data). The index is for negative sample segments (unrelated to historical event time location data / low matching degree). For fragments The soft label score. The score threshold for selecting positive samples (e.g.) ). For fragments The soft label score. The score threshold for selecting negative samples (e.g.) ).

[0110] And generate cross-modal contrast pairs:

[0111] ;

[0112] Where P is the cross-modal pair set (consistent with the query), and the subscript u + This represents the segment that matches the query semantics. N is the set of cross-modal negative pairs (inconsistent with the query), and the subscript u... - This indicates a segment that does not match the query semantics. For fragments Visual aggregation features (such as representations obtained by temporal averaging or attention convergence). For fragments Audio aggregation features. The encoded representation of the query text q (without a time dimension, therefore unrelated to u).

[0113] This is used to construct the subsequent ranking loss and cross-modal contrastive loss. (The above...) Positive and negative thresholds can be used respectively, and semi-hard cases in the temporal neighborhood can also be added. To improve generalization.

[0114] By constructing the above data, a unified time alignment and fragment annotation mechanism can be established. This includes boundary smoothing; and assigning fine-tunable weights to attribute constraints in the query (such as "white vehicle"). The system injects labels in a specific format; at the same time, it combines hard case mining and cross-domain balancing strategies to create a three-modal training set that closely meets business query needs and is robust to scenario changes, laying a solid data foundation for subsequent ternary cross-attention mechanisms and distillation training.

[0115] In one embodiment, step S104 includes:

[0116] Obtain the segment ranking loss and cross-modal contrast alignment loss of the ternary cross-attention model during the training process;

[0117] For the fragment sorting loss (pairwise / listwise), given a set of triples... Define an adaptive interval based on the tIoU difference:

[0118] ;

[0119] in, This is the adaptive interval, used to determine the interval size in the pairwise sorting loss. It is the basic interval constant (minimum interval), a lower bound independent of the data. The adjustment coefficient controls the additional interval magnitude caused by the difference in tIoU. , Each is a fragment (Positive sample) and (Negative sample) Time IoU with respect to the set of truth intervals S. To truncate the linear.

[0120] If listwise sorting is used, the candidate set under the same query q ,make ,but:

[0121] ;

[0122] in, The result of exponentially operating on the scores of candidate fragment u with the natural base e is used as the numerator of softmax. In other words, it transforms arbitrary real number scores into positive numbers for probability normalization. This represents the result of exponentially calculating the score of the v-th candidate time segment using the natural base e.

[0123] For tri-modal contrastive alignment loss, a modal projection head is defined to enhance tri-modal semantic alignment. And normalize to obtain the embedding It employs three pairs of InfoNCE and introduces adaptive temperature. (Adjusted according to difficulty or batch statistics):

[0124] ;

[0125] in, B represents the InfoNCE contrast loss of visual audio. S This represents the number of samples in a mini-batch (normalized average). u represents the u-th sample, which is the index of the anchor sample; j represents the j-th sample, which is the index of any sample within the same batch, where j=u is a positive sample and the rest are negative samples. Each by projection head Obtained and passed Normalized visual / audio embedding vectors. This is a temperature coefficient (positive number), which can be set as a constant or adaptively adjusted based on batch statistics.

[0126] ;

[0127] in, This represents the InfoNCE contrast loss of visual text. Visual / Textual Normalized embedding (text by projection head) get). Visual-text temperature coefficient;

[0128] ;

[0129] in, For audio Text InfoNCE contrast loss. For audio / text Normalized embedding (by) get). This represents the audio-text temperature coefficient. The other symbols are the same as the previous two labels, except that one modality is replaced with text.

[0130] The joint alignment objective of the three modes is:

[0131] ;

[0132] in, The goal is to achieve joint alignment of three modes; cross-modal consistency is improved by aggregating the losses of the three pairwise comparison paths. These are non-negative weighting coefficients, used to balance the contributions of the three paired pathways.

[0133] To highlight low-frequency anomalies, soft tags are used. Define anomaly weights The above sum is then averaged using weights.

[0134] The fusion features of the training data of the triplet are obtained through the triplet cross-attention model, and the distillation loss is obtained based on the fusion features.

[0135] In this step, for fragment scoring and fusion representation, the obtained fragment-level representation can be... Fused features are obtained through ternary cross attention (TMCA) And calculate the matching score:

[0136] ;

[0137] in, A scoring head is used to score the fragments and incorporate the fusion features. Mapped to scalar fractions s u w is the weight vector of the linear scoring head. B is the bias scalar of the linear scoring head. .

[0138] Perform linear projections on the visual, audio, and text separately, then fuse them:

[0139] ;

[0140] in, Represent the visual / audio / text branches of fragment u after their respective linear projections (and) (Same-dimensional or alignable embeddings). Linear fusion is a late-stage linear weighted fusion, where the weights reflect the proportion of each modality's contribution to the final fused representation.

[0141] For knowledge distillation (Logit / feature / attention multi-granularity), let the teacher model's ranking logits for fragment u be... (or single scalar fraction), the student model is... .temperature KL distillation:

[0142] ;

[0143] in, These are the logits (which can be a vector of length K or a single scalar fraction) for the teacher / student model on the u-th candidate time segment, respectively. They are used during distillation. The distribution of "soft labels" is obtained. B S This indicates the number of samples in the mini-batch, used to average the distillation loss.

[0144] Alignment fusion representation and branch representation (visual / audio / text) feature distillation ):

[0145] ;

[0146] in, These are the fragment-level fusion features of students / teachers after TMCA fusion (and...). same type). Students / teachers in modal Branching features (visual, audio, text). denoted as L2 norm squared; KD-feat is used to minimize the feature differences between students and teachers in the fusion layer and each branch layer.

[0147] KL distillation (layer / head summation) of attention distributions with cross-attention:

[0148] ;

[0149] in, The number of layers in the cross-attention module and the number of attention heads in each layer are respectively cross-referenced. These represent the attention distributions of the teacher / student at the l-th and h-th attention heads, respectively (e.g., the probability distributions after applying softmax to the rows of the attention matrix).

[0150] The overall distillation objective is:

[0151] ;

[0152] in, The non-negative weight coefficients of the three components in the total loss are distilled separately, corresponding to the relative importance of logits distillation, feature distillation, and attention distribution distillation, respectively; used to make a trade-off among the three.

[0153] The event boundary is set based on the query instruction, and the boundary regression loss is obtained based on the event boundary.

[0154] This step involves temporal boundary regression and the probability of segment existence. The event is treated as... Two boundaries. For the fragment. Predicted normalized offset With probability of existence , Separate segments The normalized start and end boundary offset pairs are used to obtain the boundary regression target:

[0155] ;

[0156] ;

[0157] ;

[0158] in, respectively with candidate fragments The matching truth value (or weighted aggregation truth value) event boundaries represent the start and end times, respectively. respectively with candidate fragments Matched predicted values.

[0159] For boundary regression loss; the predicted boundary and the true boundary on the positive sample set P are respectively... The loss is calculated and the average is taken to accurately determine the start and end times.

[0160] The binary cross-entropy loss is calculated based on whether the fragment "contains an event". For the model to fragment Probabilistic predictions containing historical event time location data (output by a classification head such as sigmoid).

[0161] Further enhancement by introducing time-based IoU regression:

[0162] ;

[0163] To suppress jitter, a timing smoothing regularization (TV regularization) is introduced:

[0164] ;

[0165] in, Time series output for frame-level (or fine-grained sub-fragment-level) boundary / existence probabilities.

[0166] A target loss function is constructed by combining the fragment sorting loss, cross-modal contrast alignment loss, distillation loss, and boundary regression loss, and the parameters are optimized using the target loss function.

[0167] The final training objective is a weighted multi-task approach combining sorting, alignment, distillation, and boundary decomposition:

[0168] ;

[0169] in, , It is an adjustable hyperparameter and can adaptively adjust weights according to modality / scenario (e.g., according to batch confidence or difficulty). (Perform dynamic scaling).

[0170] In one embodiment, step S105 includes:

[0171] The initial time positioning model is structurally reduced using a multidimensional reduction technique.

[0172] The LoRA low-rank adaptation algorithm is used to control the parameters of the initial time localization model.

[0173] The initial time positioning model is structurally pruned using structured pruning techniques.

[0174] The initial time positioning model is weighted using a quantization-sensory distillation technique.

[0175] This embodiment achieves efficient compression and performance optimization of the initial temporal localization model through the synergistic effect of multidimensional reduction, LoRA low-rank adaptation algorithm, structured pruning, and quantization-perceptual distillation. Specifically, multidimensional reduction refines the token and channel reduction of the model's visual and audio branches, effectively reducing computational complexity. The LoRA low-rank adaptation algorithm enhances the model's adaptability to specific tasks without significantly increasing the number of model parameters. Structured pruning further reduces model size and computational overhead by removing redundant structures. Quantization-perceptual distillation converts model parameters from high-precision floating-point numbers to low-precision representations through weight quantization, significantly reducing storage requirements and inference latency while maintaining model performance. The combined application of these techniques results in a compressed temporal localization model that maintains high localization accuracy while achieving faster inference speed and lower resource consumption.

[0176] Specifically, this embodiment performs a lightweight modification on the trimodal model, controlling the total parameter scale to within a certain range. (Right now The model achieves low-latency inference on long-duration video surveillance for property security, at the order of magnitude of 1000. It consists of three parts: (i) a visual encoder. (ii) Audio encoder (iii) Text / Language backbone It integrates with the ternary cross-attention fusion module TMCA. Through architecture pruning, low-rank adaptation, structured sparsity, quantization, and multi-granularity distillation, it collaboratively optimizes the number of parameters and computational cost.

[0177] Token / channel reduction and downsampling in the visual / audio branch. Assume the input resolution is... Patch size P, number of visual patch tokens:

[0178] ;

[0179] To reduce quadratic complexity terms (see Attention FLOPs), inter-layer asymptotic downsampling is introduced (e.g., hierarchical ViT or Conv-Former hybrid structures), reducing the number of tokens in the l-th layer to... ,in Furthermore, the stratification decreases monotonically. The layer-wise keep ratio of tokens retained for layer l. In this layer, the number of visual tokens is reduced proportionally from the number in the previous layer. This is used for progressive downsampling and reducing secondary complexity.

[0180] The audio end cropped the time-frequency graph to T a ×F a The result is obtained through bandwidth selection and time pooling. Each token is used, and non-collision-related frequency bands are filtered using a learnable bandpass mask to reduce redundancy. Among them, T... a ×F a This represents the size of the time-frequency plot at the audio end. T a For time frames (e.g., STFT frames), F a This represents the number of frequency bins (number of frequency bands) retained / selected. Overall, it indicates converting the audio waveform to a value of T. a ×F a The time-frequency characteristic diagram. The time dimension retention ratio (or downsampling ratio) for the audio. After bandwidth selection and time pooling, the number of audio tokens is determined by T. a Reduce to , Retain / downsample intensity.

[0181] Benchmark parameters and complexity of Transformer layers. Let the single-layer dimension be d, the number of attention heads be h, and the feedforward ratio be r (i.e., the hidden layer is rd). Ignoring bias terms and normalization layers, the single-layer parameters are approximately:

[0182] ;

[0183] Where r is the hidden expansion ratio of the FFN, i.e., the intermediate layer dimension in the feedforward network is rd. d is the hidden dimension (embedding / hidden size) of a single Transformer layer. O is the output projection.

[0184] Single-layer self-attention FLOPs (multiply-add once):

[0185] ;

[0186] FFN FLOPs:

[0187] ;

[0188] Therefore, the total FLOPs of a single layer are approximately:

[0189] ;

[0190] Where N is the number of tokens in this layer. It can be seen that when N is large, Since the N-axis is dominant, layered reduction of N is key.

[0191] Parameter calculation for target 0.3B. (This refers to the text backbone.) Layers, hidden dimensions Visual Layers, Dimensions Audio Layers, Dimensions TMCA Layers, Fusion Dimensions .

[0192] This embodiment employs a text backbone of L=10, d=1152, a visual backbone of L=10, d=640, an audio backbone of L=8, d=384, and a fusion layer of L=6, d=640. All layers have an FFN multiplier of 4, and the multi-head ratios are 16 / 10 / 6 / 10 respectively. The text vocabulary is 24k, with weighting. The four sub-modules contain approximately 252M parameters, and the superimposed embedding and small adapter heads add up to approximately 38M, resulting in a total parameter count of ≈0.29B. Furthermore, by controlling the secondary complexity through hierarchical token downsampling and Top-k selection, low-latency inference on edge devices or small to medium-sized GPUs is achieved.

[0193] LoRA low-rank fitting and the learnable part of parameters. For linear mappings Using LoRA:

[0194] ;

[0195] Where A and B represent the weight matrices in the low-rank update of LoRA, respectively. , The input feature dimension is the linear layer. α' represents the output feature dimension of the linear layer, and α' is the scaling factor.

[0196] The number of new parameters is:

[0197] ;

[0198] Enabling LoRA only on key layers (projected onto QKV, O, and FFN) can control the proportion of trainable parameters to a certain extent. It also supports incremental scenario adaptation (e.g., only replacing LoRA weights when switching scenarios). The LoRA layout adopts a head-wise / bottleneck-wise strategy to balance the heterogeneity of multi-head attention representation with caching efficiency.

[0199] Structured pruning (width / depth / attention head) and knowledge order preservation. In distillation teacher... Apply order-preserving constraints during the student process:

[0200] Prune the channels / intermediate dimensions using L1 norm / importance scoring;

[0201] Prune the attention heads using head importance scores (e.g., based on gradients or attention entropy);

[0202] Layer depth is achieved using distilled skip-layer training (layer drop + teacher hints). The channel retention rate is set to [value missing]. Head retention rate Layer retention rate The approximate compression ratio is as follows:

[0203] ;

[0204] Square factors originate from Dependency. To maintain sorting consistency, an order-preserving distillation loss is introduced:

[0205] ;

[0206] in, These represent the matching scores given by the teacher and student to segment u (the higher the score, the more relevant it is to the query), with the superscripts + and - corresponding to positive and negative segments, respectively.

[0207] Ensure that the students' segment sorting interval is not lower than a given threshold. At the same time, align the spacing between teachers This reduces the disruption to the order caused by pruning.

[0208] Quantization (weights / activations) combined with distillation. Symmetric / asymmetric linear quantization is used to perform quantization on the weights W and activations x. Bit quantization. Taking weighted quantization as an example:

[0209] ;

[0210] in, is the scaling factor. It is estimated by the maximum absolute value of the weights and is used to linearly map the real number field W to the integer field and facilitate inverse scaling back to the real number field. The number of bits for weight quantization; the integer range for symmetric uniform quantization is... . These are the weight tensors before and after quantization, respectively.

[0211] We introduce Quantization-Aware Distillation (QAT+KD), using a quantization operator in the forward pass and a pass-through estimator (STE) to approximate the gradient in the backward pass, aligning the teacher's floating-point distribution with the student's quantization distribution, and adding a loss:

[0212] ;

[0213] in, The mean and covariance of the hierarchical fusion representation (which can be approximated by batch statistics) represent the values ​​of the hierarchical fusion representation. This indicates the result after quantification. Combined with the aforementioned... In order to reduce the loss of accuracy caused by quantization.

[0214] For Teachers' Network The mean vector of the layer fusion features; To and The corresponding covariance matrix represents the second-order statistics of the characteristic distribution of teachers at this layer. For student network Mean vector estimation of layer fusion features. To and The corresponding covariance matrix represents the estimated or regularized / shrinkable version used for numerical stability.

[0215] Lightweight design of the three-modal fusion layer. TMCA's cross-modal attention is susceptible. Impact. To control complexity, multi-query low-rank cross-attention is employed: the cross-modal attention weight matrix... , Introducing low-rank decomposition ,rank Then the parameter is from Down to The cross-modal key values ​​are selected using Top-kToken (based on attention warm-up scores). The fusion layer parameters are approximately the same as those of FLOPs.

[0216] ;

[0217] ;

[0218] in, This represents the number of layers in the fusion layer (TMCA module); it is used to count the number of parameters and FLOPs in the fusion phase. This is the hidden dimension of the fusion layer (channel width / width represented by each token). All modalities are unified to this dimension after entering TMCA; the parameters are consistent with those in the FLOPs formula. This refers to this width.

[0219] and use (Only key tokens are retained), significantly reducing quadratic terms.

[0220] Latency and video memory usage estimation. Batch size setting. Number of tokens in the integration phase Given the number of layers L and the hidden dimension d, the single-path forward latency is approximately equal to that of the video memory (activation).

[0221] ;

[0222] ;

[0223] in, This is related to the framework implementation (e.g., attention caching / mixed precision). Using mixed precision (FP16 / BF16) and KV-Cache can... Significantly reduced. Quantified to , The weight / activation storage can be further reduced to FP16. .

[0224] Here, From Four projections and output projection (approximately) ) and the two linear layers of FFN (approximately ). Two matrix multiplications from attention ( and B represents the batch size. L represents the number of network layers (total number of layers used for fusion). This is the expansion factor of the FFN in the l-th layer (e.g., 4 for the classic Transformer, but this can be varied by layer in this embodiment). The number of tokens in the l-th layer (the length after downsampling and filtering). is the hidden dimension (channel width) of the l-th layer. It is a memory approximation of activations and intermediate values ​​during the forward (training and inference) process.

[0225] In summary, this embodiment, through the synergy of token / channel / layer multidimensional reduction, LoRA low-rank adaptation, structured pruning, and quantization-aware distillation, compresses the number of parameters to a minimum while maintaining trimodal alignment and sorting capabilities. and effectively control Its attention complexity allows it to run with low latency on edge devices or small to medium-sized GPUs, meeting the real-time event location requirements of long-term video surveillance in property security.

[0226] Based on the aforementioned lightweight student model (including TMCA fusion, pruning / quantization, and LoRA adaptation), a "nearest computing priority" deployment approach is adopted in production: edge-device integrated machine, small cluster in campus data center, or private / hybrid cloud. A unified inference interface is provided externally via HTTP / gRPC, connecting to RTSP / GB28181 video streams and synchronous audio, and supporting offline batch file processing. Different camera positions can be bound to different LoRA "slots," facilitating rapid scene-specific switching without changing container images.

[0227] The online data stream follows the same pipeline from "stream" to "alarm": the access layer completes decoding and basic preprocessing, segments long videos into fixed windows and reuses cross-window caches; the visual and audio encoders extract temporal features and then enter TMCA for cross-modal fusion and simplification (Top-k Token); the task header outputs segment matching scores, existence probabilities and start and end times, the system performs deduplication and time merging on adjacent high-confidence segments, and finally generates an "event time period list". When a threshold is hit, an alarm is triggered and a keyframe / short audio index is attached for easy manual review.

[0228] To maintain long-term effectiveness, manually verified true positive / false alarm samples and related metadata can be archived for periodic small-scale retraining and weekly LoRA fine-tuning to suppress concept drift. Models are registered in the order of "backbone version – TMCA version – LoRA slot – quantization configuration" to ensure traceability and one-click rollback. Data processing follows the principle of "controllable within the domain, minimal outbound transmission": raw audio and video are processed locally / within the campus as much as possible, with only event metadata and optional anonymized segments transmitted externally. In terms of business operations, alarm results automatically generate work orders and are stored in the database, supporting one-click playback and aggregation of multiple camera positions within the same vehicle.

[0229] Through the steps described above, the multimodal model constructed in this embodiment can automatically locate the occurrence time of specific abnormal events in surveillance videos lasting several hours. For example, when a query for "locating the time period when the white vehicle was scratched" is input along with the corresponding video and audio data, the model will comprehensively analyze the vehicle contact traces in the visual frames, the scratching sounds in the audio stream, and the semantic guidance of the text description to match and sort relevant events in the video content, ultimately accurately determining the start and end times of the scratch on the white vehicle. In particular, for subtle and unseen anomalies, the specialized semantic alignment and feature emphasis mechanism provided in this embodiment significantly improves the detection accuracy. Simultaneously, the construction of a small model and efficient training significantly reduce the system's deployment costs and hardware requirements, demonstrating good practical value and promising prospects for wider application.

[0230] Figure 3 This is a schematic block diagram of an event time positioning device 200 provided in an embodiment of the present invention. The device 200 includes:

[0231] The data acquisition unit 201 is used to acquire historical event time location data; wherein, the historical event time location data includes query instructions and corresponding video data and audio data;

[0232] Alignment processing unit 202 is used to perform time alignment processing on the query command and the corresponding video data and audio data to obtain aligned data;

[0233] The model training unit 203 is used to construct triple training data based on the alignment data, and to train the triple cross-attention model using the triple training data to obtain an initial time localization model.

[0234] The parameter optimization unit 204 is used to acquire loss data during the training process and use the loss data to optimize the parameters of the initial time localization model.

[0235] The structural fine-tuning unit 205 is used to fine-tune the structure of the initial time positioning model after parameter optimization using compression technology to obtain the target time positioning model.

[0236] The deployment inference unit 206 is used to deploy the target time positioning model and use the target time positioning model to perform inference and prediction on the specified event time query data.

[0237] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0238] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed, can perform the steps provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0239] This invention also provides a computer device, which may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps provided in the above embodiments. Of course, the computer device may also include various network interfaces, power supplies, and other components.

[0240] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0241] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for locating an event based on its time, characterized in that, include: Obtain historical event time location data; wherein, the historical event time location data includes query commands and corresponding video and audio data; The query command and its corresponding video and audio data are time-aligned to obtain aligned data. Based on the alignment data, triple training data is constructed, and the triple cross-attention model is trained using the triple training data to obtain the initial time localization model. Obtain the loss data during the training process, and use the loss data to optimize the parameters of the initial time localization model; The initial time-localization model after parameter optimization is fine-tuned using compression technology to obtain the target time-localization model. The target time positioning model is deployed, and the target time positioning model is used to perform inference and prediction on the specified event time query data; The process of constructing triplet training data based on the alignment data and training the triplet cross-attention model using the triplet training data to obtain the initial time localization model includes: The duration corresponding to the video data is obtained, and the time axis of the video data is divided into multiple initial time segments in combination with a preset stride. Define a segment-level feature set for each initial time segment, and set soft labels with smoothed boundaries for the initial time segments; Based on the query command, the attribute probability of the historical event time location data is set, and the attribute probability is used to set a consistency weight for the initial time segment. Then, the soft tag is gating and enhanced in combination with the consistency weight. Set segment difficulty scores and quality indicators for the initial time segments; Candidate time segments are obtained by combining the soft tags, segment difficulty scores, and quality indicators from the initial time segments; Based on the candidate time segments, the triplet training data is constructed by combining the query command and the audio data; The step of acquiring loss data during the training process and using the loss data to optimize the parameters of the initial time-localization model includes: Obtain the segment ranking loss and cross-modal contrast alignment loss of the ternary cross-attention model during the training process; The fusion features of the training data of the triplet are obtained through the triplet cross-attention model, and the distillation loss is obtained based on the fusion features. The event boundary is set based on the query instruction, and the boundary regression loss is obtained based on the event boundary. A target loss function is constructed by combining the fragment ranking loss, cross-modal contrast alignment loss, distillation loss, and boundary regression loss, and the parameters are optimized using the target loss function. The process of obtaining the segment ranking loss and cross-modal contrastive alignment loss of the ternary cross-attention model during training includes: The fragment sorting loss is calculated using the following formula: ； ； ； in, This represents the pairwise sorting loss. This represents the listwise sorting loss. This represents the loss for sorting fragments. Denotes the set of triples. For adaptive interval, This is the index of the candidate time segments for positive samples. Indexes of candidate time segments for negative samples. , The three-element cross-attention model is used to select candidate time segments. , Match score, Let q be the set of candidate time segments, and y be the query command. u The binary label representing the candidate time segment u. This represents the result of exponentially calculating the score of the u-th candidate time segment with respect to the natural base e. This represents the result of exponentially operating on the score of the v-th candidate time segment with the natural base e.

2. The event time location method according to claim 1, characterized in that, The step of performing time alignment processing on the query command and the corresponding video and audio data to obtain aligned data includes: The video data and audio data are respectively encoded into time-series features of the same frequency; The query command and the time series features are aligned using the optimal alignment offset estimated by the cross-correlation of energy / motion correlation to obtain the aligned data.

3. The event time location method according to claim 1, characterized in that, The process of fine-tuning the structure of the initial time-localization model after parameter optimization using compression technology to obtain the target time-localization model includes: The initial time positioning model is structurally reduced using a multidimensional reduction technique. The LoRA low-rank adaptation algorithm is used to control the parameters of the initial time localization model. The initial time positioning model is structurally pruned using structured pruning techniques. The initial time positioning model is weighted using a quantization-sensory distillation technique.

4. The event time location method according to claim 1, characterized in that, The step of obtaining the segment ranking loss and cross-modal contrast alignment loss of the ternary cross-attention model during training also includes: The cross-modal contrast alignment loss is calculated using the following formula: ； ； ； ； in, B represents the contrast loss between visual and audio. S Let represent the number of samples in a mini-batch, u represent the number of samples in the mini-batch, represent the index of the anchor sample, j represent the number of samples in the mini-batch, and represent any sample index within the same batch, where j=u represents a positive sample and the rest are negative samples. Each is composed of a projection head Obtained, and through Normalized visual embedding vectors and audio embedding vectors, Visual-audio temperature coefficient; This indicates the loss of contrast between visual and textual representations. For text Normalized embedding, Visual-text temperature coefficient; This indicates the contrast loss between audio and text. Audio and text respectively Normalized embedding, Audio-text temperature coefficient; For the joint alignment of the three modes, These are the non-negative weighting coefficients.

5. An event time positioning device applied to the event time positioning method as described in any one of claims 1-4, characterized in that, include: A data acquisition unit is used to acquire historical event time location data; wherein, the historical event time location data includes query instructions and corresponding video and audio data; An alignment processing unit is used to perform time alignment processing on the query command and the corresponding video and audio data to obtain aligned data. The model training unit is used to construct triple training data based on the alignment data, and to train the triple cross-attention model using the triple training data to obtain the initial time localization model. The parameter optimization unit is used to acquire loss data during the training process and optimize the parameters of the initial time localization model using the loss data. The structural fine-tuning unit is used to fine-tune the structure of the initial time-localization model after parameter optimization using compression technology to obtain the target time-localization model. A deployment inference unit is used to deploy the target time positioning model and use the target time positioning model to perform inference and prediction on the specified event time query data.

6. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the event timing method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the event timing location method as described in any one of claims 1 to 4.

Images