Video time sequence positioning method and system based on large model

Abstract

The invention belongs to the technical field of video time sequence positioning, and discloses a video time sequence positioning method and system based on a large model, and the method comprises the steps: obtaining original video data and a query statement, and compressing the original video data into a fixed number of frames; extracting image features of each frame, and mapping the image features to a semantic embedding space compatible with the large language model frame by frame to obtain continuous video representation; extracting text features of the query statement, and sequentially splicing the continuous video representation and the text features to obtain an image text pair; inserting a visual prompt at the front end of the image text pair to obtain a visual description sequence, inputting the visual description sequence into the large language model to perform time sequence positioning prediction on the visual description sequence, and outputting a video data positioning result; and defining a loss function, training the video time sequence positioning model, and performing video time sequence positioning by using the trained video time sequence positioning model. According to the method, the large model can be guided to be more accurate during semantic understanding and positioning of the video clip.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video timing positioning, in particular to a video timing positioning method and system based on a large model. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Today, with the rapid progress of Internet technology, video media is rapidly rising and gradually integrating into people's lives. The video timing positioning task aims to accurately identify and locate specific segments corresponding to natural language descriptions from video content, which not only involves deep understanding of algorithms for video and text, but also requires accurate association between multi-modal information. Traditional video timing positioning methods mainly include two-stage methods, end-to-end methods, reinforcement learning-based methods, and weakly supervised methods, etc. These traditional methods often use single-modal pre-training frameworks, which have limited performance in zero-shot scenarios. When faced with complex video scenarios, these methods also have problems such as low positioning efficiency, limited accuracy, and poor generalization ability.

[0004] In recent years, large model technology has developed rapidly, and its powerful cross-modal alignment capability provides a new technical path for building fine-grained video event positioning systems. Researchers have begun to explore the application of general large models to video timing positioning and have improved their performance through innovative architecture design. SUMMARY

[0005] To solve the above problems, the present application provides a video timing positioning method and system based on a large model, designs a new multi-modal alignment mechanism, provides strong technical support for video content analysis and understanding, builds a visual adapter based on a multi-layer perception, constructs a precise semantic alignment mechanism, proposes an evaluation algorithm based on fine-grained frame sampling, fully releases the potential video understanding ability and boundary perception ability of the model in the model inference stage, and designs a series of efficient prompt strategies to guide the large model to be more accurate in semantic understanding and positioning of video segments.

[0006] To achieve the above purpose, the present application adopts the following technical solutions: In a first aspect, the present application provides a video timing positioning method based on a large model, comprising the following steps: Obtain the original video data and the query statement, and compress the original video data of any length into a fixed number of frames; Extract the image features of each frame, map the image features frame by frame to a semantic embedding space compatible with a large language model, and obtain continuous video representations after dimension alignment; extracting text features of the query sentence, sequentially splicing the continuous video representation and the text features to obtain an image-text pair; inserting a learnable visual prompt at the front end of the image-text pair to obtain a visual description sequence, inputting the visual description sequence into a large language model to perform time sequence positioning prediction on the visual description sequence, and outputting a video data positioning result; defining a loss function, training the video time sequence positioning model, and using the trained video time sequence positioning model to perform video time sequence positioning.

[0007] As an optional implementation, the original video data of any length is compressed into a fixed number of frames by an adaptive time sequence uniform sampling method, and under the premise of retaining the integrity of the first and last frames, the spatiotemporal continuity of the motion evolution process is ensured by a linear interpolation algorithm.

[0008] As an optional implementation, the video time sequence positioning model is trained by a phased progressive method.

[0009] As an optional implementation, the first phase focuses on constructing a visual-linguistic joint representation space, and after encoding the visual information and the language information respectively, mapping them into a shared vector space through LoRA fine-tuning.

[0010] As an optional implementation, the second phase is a multi-round dialogue construction paradigm based on the LLaVA model, injects large-scale time-sensitive instruction fine-tuning data, and enhances the ability of the large model to perceive time boundaries by using a more fine-grained dataset, so that the large model outputs more accurate timestamps.

[0011] As an optional implementation, the image features are mapped frame by frame to a semantic embedding space compatible with the large language model, specifically: The image features are subjected to nonlinear space transformation by a perceptron composed of three fully connected networks, wherein the hidden layer adopts a GELU activation function and implements Dropout regularization.

[0012] In a second aspect, the present application provides a video time sequence positioning system based on a large model, comprising: A data acquisition module configured to acquire original video data and a query sentence, and compress the original video data of any length into a fixed number of frames; A semantic alignment module configured to extract image features of each frame, map the image features frame by frame to a semantic embedding space compatible with the large language model, and obtain continuous video representation after dimension alignment; A splicing module configured to extract text features of the query sentence, sequentially splice the continuous video representation and the text features to obtain an image-text pair; The video positioning module is configured to insert a learnable visual prompt at the front end of the image-text pair, obtain a visual description sequence, input the visual description sequence into a large language model, perform time sequence positioning prediction on the visual description sequence, and output a video data positioning result. The model training module is configured to define a loss function, train the video time sequence positioning model, and perform video time sequence positioning using the trained video time sequence positioning model.

[0013] In a third aspect, the present application provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions are executed by the processor to complete the method of the first aspect.

[0014] In a fourth aspect, the present application provides a computer readable storage medium for storing computer instructions, wherein the computer instructions are executed by a processor to complete the method of the first aspect.

[0015] In a fifth aspect, the present application provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to complete the method of the first aspect.

[0016] Compared with the prior art, the present application has the following beneficial effects: (1) The present application proposes a video time sequence positioning method based on a large model, which adopts a video time sequence positioning algorithm architecture based on LLaVA, and is composed of a visual encoder module, a visual adapter module and a large model module. The algorithm combines advanced natural language processing technology and a video encoder to design a new multi-modal alignment mechanism. Fine-grained features of video frames are extracted by the visual encoder, and cross-modal alignment is performed by the large language model, which greatly improves the analysis and understanding ability of video content and provides strong technical support for the analysis and understanding of video content.

[0017] (2) The present application proposes a video time sequence positioning method based on a large model, which constructs a visual adapter based on a multi-layer perception machine, and constructs a precise semantic alignment mechanism. The video features extracted by the advanced video encoder and the natural language are mapped into the semantic space of the large model, and the self-attention mechanism of the large language model is fully relied on for time sequence modeling without explicitly introducing a time modeling mechanism. This design enables the model to automatically discover time correlations in the time evolution of the video, further improving the alignment and understanding ability of cross-modal features, thereby laying a good foundation for the subsequent instruction fine-tuning process.

[0018] (3) The application provides a video timing positioning method based on a large model, and an evaluation algorithm based on fine-grained frame sampling is provided. The evaluation algorithm is proved to be capable of fully releasing the potential video understanding ability and boundary perception ability of the model in the model inference stage. In combination with a prompt learning strategy, a series of efficient prompt engineering is designed, the large model is guided to more accurately understand video content in the video timing positioning task through natural language prompts, and the performance of the model in the video timing positioning task is further improved.

[0019] Advantages of the additional aspects of the application will be partially given in the following description, partially become obvious from the following description, or be known by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which form a part of the specification, are included to provide a further understanding of the application and are incorporated herein by reference. The illustrations are shown for the purpose of enabling those skilled in the art to implement the application and are not intended to limit the scope of the application.

[0021] Fig. 1 A training process schematic diagram of the video timing positioning model provided for the embodiment 1 of the application is shown in the figure. Fig. 2 A flowchart of the video timing positioning method based on the large model provided for the embodiment 1 of the application is shown in the figure. DETAILED DESCRIPTION The application will be further described below in conjunction with the drawings and embodiments.

[0022] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as that understood by those skilled in the art.

[0023] It should be noted that the terms used herein are only for the purpose of describing the specific embodiments and are not intended to limit the exemplary embodiments of the application. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form, and in addition, it should be understood that the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to those steps or units clearly listed, but includes other steps or units not clearly listed or inherent to the process, method, product or device.

[0024] The embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0025] Embodiment 1 AsFigs. 1-2 As shown, the embodiment provides a large model-based video timing positioning method, comprising the following steps: Obtain the original video data and the query statement, and compress the original video data of any length into a fixed number of frames; Extract the image features of each frame, map the image features frame by frame to a semantic embedding space compatible with the large language model, align the dimensions, and obtain continuous video representation; Extract the text features of the query statement, sequentially splice the continuous video representation and the text features to obtain an image-text pair; Insert a learnable visual prompt at the front end of the image-text pair to obtain a visual description sequence, input the visual description sequence into the large language model to perform timing positioning prediction on the visual description sequence, and output the video data positioning result; Define a loss function to train the video timing positioning model, and use the trained video timing positioning model to perform video timing positioning.

[0026] The specific scheme of the application is as follows: Video timing positioning problem definition: The core of the video timing positioning task is to locate the time period matching the query statement from the video according to the input natural language. This task requires feature extraction, understanding, alignment and accurate matching of visual features of video content and semantic information of language description. This task naturally has multi-modal characteristics. The input data usually contains video stream, audio signal and natural language text. For the formal definition of the video timing positioning task, first consider an unpruned long video sequence containing Frame original video data And a natural language query statement , Describes the specific visual event or action that needs to be located. The goal of video timing positioning is to determine the continuous time interval that best matches the query semantics (satisfying ). This task is achieved by constructing a differentiable mapping function . This framework needs to overcome the core challenges of irrelevant fragments of unpruned videos, query semantic ambiguity and action boundary ambiguity to achieve precise alignment of fine-grained spatiotemporal semantics.

[0027] Video timing positioning large model architecture: The application adopts a multi-modal interaction architecture based on LLaVA. The architecture is mainly composed of three key components: visual encoder, visual adapter and large model, and has a complete "video understanding, semantic alignment, timing reasoning" technical closed loop. The three key components of the architecture are introduced as follows.

[0028] Visual encoder: In terms of visual encoder design, the application adopts a CLIP ViT-L / 14@336px pre-trained model with parameter freezing as the core visual feature extraction module. The model is built based on the CLIP dual-modal contrastive learning framework, and its training strategy of freezing parameters not only avoids the computational overhead of repeated training of large-scale video data, but also completely retains the advantages of the original model in open-domain visual semantic understanding and cross-modal alignment. Each frame of image is converted into a 768-dimensional feature vector containing spatial position encoding through the 16x16 image block embedding strategy of CLIP ViT-L / 14@336px. This design not only maintains the fine-grained expression of local visual features, but also maintains the temporal correlation between video frames through the position information of the embedding layer. It is particularly worth noting that the model loads the pre-trained weights of CLIP ViT-L / 14@336px, which makes the generated visual features naturally compatible with the text semantic space, laying a solid foundation for subsequent cross-modal alignment. For the video preprocessing link, the application adopts an adaptive temporal uniform sampling scheme. This strategy adaptively compresses an original video stream (containing frames) of any length into a compact representation sequence of frames. Under the premise of preserving the integrity of the first and last frames, the linear interpolation algorithm ensures the spatiotemporal continuity of the motion evolution process. This intelligent sampling mechanism not only reduces the computational complexity to a controllable range, but also effectively captures the key dynamic information in the video stream, providing high-quality input representations for subsequent temporal modeling. For a video , uniform sampling frames, and , using ViT-L / 14@336px to extract features, expressed as:

[0029] wherein represents the number of ViT blocks, represents the global feature of the video.

[0030] Visual adapter: The application selects the global semantic features of each frame extracted by ViT-L / 14@336px as the core representation carrier. The visual adapter maps these high-dimensional visual features frame by frame to a semantic embedding space compatible with large language models through the construction of a learnable projection layer based on a multi-layer perceptron. After dimension alignment, the feature sequence forms a continuous video representation input, whose dimension size matches the internal hidden layer of the large language model.

[0031] It is particularly necessary to point out that the module does not explicitly introduce a time modeling mechanism in the feature conversion process, but fully relies on the time parsing ability of the large language model on the serialized input - the inherent self-attention mechanism of the large language model can automatically mine the time correlation pattern between frames, thereby implicitly modeling the time evolution of the video at the semantic level.

[0032] The application projects the global image features of the video frames As the first The image features of the frames are projected into the same embedding space as the large model by a multi-layer perceptron. Specifically, the high-level semantic features of each frame image extracted by the visual encoder are subjected to nonlinear spatial transformation by a perceptron composed of three fully connected networks, wherein the hidden layer adopts a GELU activation function and implements Dropout regularization. The GELU activation function is selected instead of the traditional ReLU because of its smooth transition characteristics based on the Gaussian error function, which can better preserve the continuous distribution of visual details in the cross-modal feature alignment process, which is crucial for capturing subtle changes between video frames.

[0033] The network adopts a three-stage progressive dimension reduction architecture, and realizes fine conversion of visual features to language space through a "pot type" structure. The random regularization effect of GELU and the Dropout mechanism are complementary, and through the adaptive adjustment of neuron activation probability, the robustness of modal mapping is maintained in the feature compression process. GELU activation function and layer normalization operation are introduced after each transformation layer to ensure the stability of the nonlinear mapping process. Finally, the high-dimensional visual features are accurately mapped to a 768-dimensional semantic space matching the hidden layer of the large language model, which not only preserves the fine-grained expression of visual concepts, but also effectively bridges the representation gap between visual and textual modalities, establishing a compatible feature interaction foundation for subsequent cross-modal temporal reasoning.

[0034] The visual adapter projects the image features into the semantic space of the large model, which is expressed as:

[0035] Finally, the sequence is obtained as the image input of the large model. The large model input is expressed as:

[0036] The process of training the visual adapter is the process of aligning the visual features and natural language features. In the training process, the application trains the visual adapter and so that after the image (video frame) is extracted by CLIP, it is The mapped features can be understood and utilized by the language model as a "prefix" of natural language features, thereby correctly completing the text generation task. The present application refers to this process as the alignment of video frame feature sequences and text queries, and the specific training process will be described later.

[0037] The aligned video frame feature sequence and the embedding representation of the text query are input to the encoding layer of the large language model in a sequential splicing manner, intending to implicitly include the time sequence information contained in the video frames in the embedding in a sequential manner. This interaction mechanism enables the model to simultaneously model the time sequence evolution rule within the video and the semantic alignment relationship across modalities. Finally, the large language model can further encode the input embedding list to understand the video and the user query. Specifically, the base model adopted by the present application is Vicuna, which is based on a multi-layer Transformer decoder structure, and the Vicuna is trained on a large-scale multi-modal instruction dataset to adapt to the video time positioning task. This hybrid sequence is encoded as a long sequence. The self-attention mechanism models the relationship between the visual frames and the text, and all tokens are in the same sequence, which enables the language model to combine the context for cross-modal reasoning, which improves the ability of the large model to understand video information.

[0038] Large model: In terms of model architecture selection, the present application adopts Vicuna-7b-v1.5 as the core reasoning engine, and is deeply adapted to the video time positioning task based on large-scale multi-modal instruction data. The model improves the rotation position encoding mechanism of the LLaMA architecture, uses the position matrix transformation of the Fourier space to expand the context window to 4096 tokens, and its enhanced long sequence modeling capability can effectively process continuous input of up to hundreds of frames in video time positioning. In addition, Vicuna-7b-v1.5 has a lightweight architecture and an optimized KV cache mechanism, which maintains efficient computation of 7B parameters while achieving fast inference speed, fully meeting the experimental needs of the present application.

[0039] Video time positioning large model learning process: Step 1: The process of training the visual adapter can also be regarded as the process of aligning visual features and natural language features. In the training stage, is regarded as a prefix of long sequence encoding. At this time, the input of the large model can be expressed as follows:

[0040] The training goal of this stage is to maximize the probability of the model generating text tokens after inserting the visual frames, so the loss function is only calculated on the natural language tokens, and the visual embedding is skipped because there is no corresponding label. The loss function of this training process is expressed as follows: ; wherein denotes the prefix context at the generation of the th token.

[0041] In the cross-modal feature alignment stage, the application adopts the LCS-558K image-text dataset proposed by LLaVA to construct the training paradigm.

[0042] LCS-558K is a visual-linguistic alignment pre-training special dataset constructed by the LLaVA team, based on the LAION / CC / SBU open source dataset after concept balance screening to form 558K high-quality image-text pairs, and through the BLIP model to generate synthetic text description to enhance semantic richness. Specialized for the visual-linguistic feature alignment pre-training stage of the LLaVA model, it is very suitable for training the multi-perceptron visual adapter used in this paper.

[0043] The application uses image-text datasets to train the multi-perceptron-based visual adapter without using video-text datasets. The strategic exclusion of video-text pairing data is mainly based on the following two considerations: first, the existing large-scale video-text datasets generally have description noise problems, and the video and annotation in the dataset have problems such as time sequence misalignment and semantic ambiguity, which can easily lead to alignment deviation of visual-linguistic semantic space; second, there is a higher dimensional information collapse risk in the cross-modal mapping from video to text. Therefore, the application aims to only enhance the mapping ability of the mapping module to image information in the training of the visual adapter stage, and uses image-text datasets.

[0044] At the model training level, the application designs a visual prompt fusion mechanism: for each image-text pair, a learnable visual prompt is inserted at the front end of the text sequence.

[0045] Step 2: After completing the visual adapter parameter training, the algorithm applies a phased progressive instruction tuning framework: the first phase focuses on building a visual-linguistic joint representation space. Through LoRA fine-tuning, visual information (such as image frames, video clips) and language information (such as description sentences) are respectively encoded and then mapped into a shared vector space. In this phase, the invention fine-tunes the large model so that semantically similar image-text content is closer in space. Through boundary-aware contrastive learning loss, an explicit association between video clip spatiotemporal features and text description is established, specifically by strengthening the mapping accuracy of clip-level visual features and their corresponding timestamp labels; the second phase is a multi-round dialogue construction paradigm based on LLaVA, which injects large-scale time-sensitive instruction fine-tuning data. By using a more fine-grained dataset, the second phase of instruction tuning enhances the large model's ability to perceive temporal boundaries, aiming to make the large model output more accurate timestamps. The first and second phases of instruction tuning use the same loss function, which can be expressed as follows: ; Since annotating timestamps and semantics for video clips by humans requires a lot of effort, there is currently a lack of large-scale multi-event video-text datasets. The first phase uses a subset of the InternVid-10M-FLT dataset for instruction tuning under the premise of freezing the visual adapter parameters.

[0046] In the first phase of instruction tuning, the invention focuses on building a visual-linguistic cross-modal joint representation space, and achieves precise mapping of video clips and timestamp labels through boundary-aware contrastive learning. The algorithm converts existing dataset annotations into two types of question-answer data: single-round dense video description tasks and multi-round temporal positioning tasks, corresponding to video temporal positioning tasks and dense video detection tasks, respectively. In the annotation of single-round dense video description tasks, there is only one question about the activities in the video (such as "How many frames does the person in the video mow the lawn from to?"); in the annotation of multi-round dense video description tasks, there are multiple questions, and the time periods corresponding to the questions do not overlap (such as "How many frames does the person in the video mow the lawn from to? And how many frames does he water the flowers from to?").

[0047] In each instruction for a video, a suggestive statement is inserted: "This is a video that contains frames: ", where The number of frames sampled represents different scales. The reconstructed input sequence adopts a masked autoregressive training strategy, in which the cross-entropy loss function only calculates tokens in the answer text region labeled by the question and answer, and the question text and system prompt are used as conditional information that cannot be trained. To achieve efficient fine-tuning of parameters, the LoRA technology is introduced, which injects a low-rank adaptation matrix with a rank of 8 into each Transformer layer of the large model, while keeping the main parameters of the large model and the visual adapter parameters completely frozen.

[0048] After the first stage of instruction optimization, the model constructed by the application has the ability to understand all events in the video and match them with the specific frame interval obtained by sampling. However, there are still some problems in the first stage of instruction optimization: the automatic labeling dataset InternVid-10M-FLT has some timing noise and semantic deviation problems, and the action boundaries of some videos are confused, while the text describing the action is too fragmented. At the same time, the first stage of experimental data shows that the dialogue data generated by using a fixed template is prone to overfitting problems. For this reason, using a large language model to dynamically generate dialogue content to build a dataset shows a certain degree of advantage. In specific implementation, video event data can be used as input, and a large model can be guided to simulate an artificial intelligence assistant with video analysis capabilities through role setting prompts, thereby automatically generating natural dialogue data that meets the user's interaction with the visual assistant scenario.

[0049] The application introduces a hybrid instruction fine-tuning strategy in the second stage, first uses a large language model to assist in building an artificial fine-labeled multi-turn dialogue dataset, and through an expert labeling team, double-checks the key action segments, covering 8.2k video samples of ActivityNet Captions and DiDeMo. At the same time, a dynamic learning mechanism is used to mix and train automatic labeling data and artificial fine labeling data in a gradual proportion, effectively bridging the semantic gap between automatic labeling and artificial cognition.

[0050] In the design of the training strategy, the application integrates the optimization results of the first stage: after combining the trained LoRA module with the basic model parameters, an additional independent LoRA module is added, and it becomes the only trainable parameter in the entire algorithm. The training hyperparameter configuration, data organization form and optimization method of this stage continue the implementation scheme of the first stage of instruction fine-tuning, ensuring the continuity of the training process and the pertinence of the parameter adjustment.

[0051] Step 3: In order to realize fine-grained frame evaluation, the application trains a prompt and applies it to the timing positioning task.

[0052] The training of the prompt occurs after the end of the second stage instruction tuning. The present application adopts a similar loss function as the two-stage instruction tuning, which can be expressed as: ; wherein, represents the large model input after adding the prompt. For the dataset, the present application continues to use the ActivityNetCaptions and DiDeMo mixed dataset. The present application sets the default prompt length to 20 tokens, and forms a mixed sequence in the form of a prefix embedding or an overall embedding As the input of the large model, in the training stage, The definition of can be expressed as follows:

[0053] In the evaluation process, the model can better understand the video content according to the prompt obtained in the training, and accurately capture the key frames and event boundaries of the video in the temporal positioning process. In order to ensure that the prompt can be effectively integrated into the model inference, we use a forward splicing mode when inputting the video features, and embed the prompt obtained in the training into the front end of the video features. This approach ensures the deep integration of the prompt and the video content, thereby improving the model's perception of key moments. In specific implementation, we load the prompt from the weight file saved in the training process, and map its dimension to the same size as the video features through a projection layer.

[0054] Step 4: Based on the techniques of feature alignment and two-stage instruction tuning, the present application innovatively designs a multi-granularity temporal evaluation algorithm, aiming to improve the robustness of video temporal positioning in complex scenarios. In order to better fit the natural language scene, the input of the inference (evaluation stage) large model is changed, and the mixed sequence is defined as The definition can be expressed as follows:

[0055] In order to address the feature differences of different datasets, the present application constructs a differentiated frame segmentation strategy: for long videos (such as ActivityNet Captions, with an average duration of 180 seconds), uniform sampling is performed to divide them into 100 semantic segments (each segment contains 8-15 frames), mainly capturing the macro event evolution law; while for short videos (such as Charades-STA, with an average duration of 30 seconds), uniform sampling is performed to generate 120 fine-grained segments (each segment accurately corresponds to 1 second), in order to strengthen the perception of micro action boundaries.

[0056] The evaluation algorithm of the method innovatively constructs a generation and positioning dual-task cooperative verification framework, and realizes the joint verification of video semantic understanding and timing inference ability through a hierarchical evaluation protocol. The algorithm constructs a verification framework for video timing positioning and natural language description generation, aiming to generate frame positioning and action description according to the video content, and describe the queried action occurring between which two sampling frames in words (i.e. natural language). The start and end frame numbers of the model output are accurately extracted through instruction fine-tuning of a large model, and a time axis normalization-based evaluation system is constructed.

[0057] Considering that the evaluation algorithm processed videos of different lengths, the application adopted a video frame sampling mechanism of different granularities. Through experiments, we proved that the fine-grained sampling algorithm could obtain better performance in the evaluation stage. For each video in the test set, the predicted interval and the real label , the application projected them to interval to calculate the time intersection over union (IoU) as shown in the following formula. Wherein represents the number of frames sampled in the evaluation process, which is determined by the sampling granularity. represents the length of each video in the test set, and is expressed as follows:

[0058]

[0059] For the video timing positioning task, the module extracts the timestamp text from the model output by regular expression, calculates the IoU and regression rate (R@n, IoU=m) of the predicted interval and the real label as the core indicators.

[0060] Embodiment 2 The embodiment provides a video timing positioning system based on a large model, comprising: A data acquisition module configured to acquire original video data and query statements, and compress the original video data of any length into a fixed number of frames; A semantic alignment module configured to extract image features of each frame, map the image features to a semantic embedding space compatible with a large language model frame by frame, and obtain continuous video representations after dimension alignment; A splicing module configured to extract text features of the query statement, sequentially splice the continuous video representations and the text features to obtain an image-text pair; A video positioning module configured to insert a learnable visual prompt at the front end of the image-text pair to obtain a visual description sequence, input the visual description sequence into a large language model to perform timing positioning prediction on the visual description sequence, and output a video data positioning result; The model training module is configured to define a loss function, train the video temporal positioning model, and perform video temporal positioning by using the trained video temporal positioning model.

[0061] It should be noted that the above modules correspond to the steps in Embodiment 1, and have the same examples and application scenarios as the steps they correspond to, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules can be executed in a computer system as part of the system.

[0062] In more embodiments, there are also provided: An electronic device includes a memory and a processor, and computer instructions stored in the memory and running on the processor, which, when executed by the processor, complete the method in Embodiment 1. For brevity, this will not be described here.

[0063] It should be understood that in the embodiments, the processor can be a central processing unit CPU, and the processor can also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0064] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method in Embodiment 1.

[0065] The method in Embodiment 1 can be directly executed by a hardware processor, or executed by a combination of hardware and software modules in the processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. The storage medium is in the memory, and the processor reads the information in the memory to complete the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0066] A computer program product includes a computer program, which, when executed by a processor, implements the method in Embodiment 1.

[0067] The present application also provides at least one computer program product tangibly stored on a non-transitory computer readable storage medium. The computer program product includes computer executable instructions, for example, instructions embodied in program modules, executed by devices at the target real or virtual processor to perform the processes / methods as described above. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules can be combined or split between program modules as desired in various embodiments. Machine executable instructions for program modules can be executed within a local or distributed device. In a distributed device, program modules can be located in local and remote memory storage devices.

[0068] Computer program code for carrying out operations of the present application can be written in one or more programming languages. These computer program codes can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program codes, when executed by the computer or other programmable data processing apparatus, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program codes can be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0069] In the context of the present application, the computer program code or related data can be carried by any appropriate carrier to enable the device, apparatus or processor to perform the various processes and operations described above. Examples of the carrier include a signal, a computer readable medium, etc. Examples of the signal can include an electrical, optical, radio, sound or other forms of propagated signals, such as a carrier wave, an infrared signal, etc.

[0070] Those skilled in the art can realize that the units and algorithm steps of the examples described in conjunction with the present embodiments can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0071] The above describes the specific embodiments of the present application in conjunction with the accompanying drawings, but is not a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without inventive labor are still within the scope of protection of the present application.

Claims

1. A video temporal localization method based on a large model, characterized in that, Includes the following steps: Obtain raw video data and query statements, and compress raw video data of any duration into a fixed number of frames; Image features are extracted from each frame and mapped frame by frame to a semantic embedding space compatible with large language models. After dimensional alignment, a continuous video representation is obtained. Extract the text features of the query statement, and sequentially concatenate the continuous video representations with the text features to obtain image-text pairs; Learnable visual cues are inserted at the front of image-text pairs to obtain visual description sequences. These sequences are then input into a large language model to perform temporal localization prediction and output video data localization results. Define a loss function, train the video temporal localization model, and use the trained video temporal localization model to perform video temporal localization.

2. The video temporal localization method based on a large model as described in claim 1, characterized in that, An adaptive temporal uniform sampling method is used to compress raw video data of any duration into a fixed number of frames. While preserving the integrity of the first and last frames, a linear interpolation algorithm is used to ensure the spatiotemporal continuity of the action evolution process.

3. The video temporal localization method based on a large model as described in claim 1, characterized in that, A phased, progressive approach is used to train the video temporal localization model.

4. The video temporal localization method based on a large model as described in claim 3, characterized in that, The first stage focuses on constructing a visual-language joint representation space. Through LoRA fine-tuning, visual and linguistic information are encoded separately and mapped to a shared vector space.

5. The video temporal localization method based on a large model as described in claim 3, characterized in that, The second stage is to construct a multi-turn dialogue paradigm based on the LLaVA model, injecting large-scale time-sensitive instruction fine-tuning data. By using a finer-grained dataset, the ability of the large model to perceive time boundaries is enhanced, enabling the large model to output more accurate timestamps.

6. The video temporal localization method based on a large model as described in claim 1, characterized in that, Image features are mapped frame by frame to a semantic embedding space compatible with large language models, specifically as follows: Image features are transformed nonlinearly through a perceptron consisting of three fully connected layers. The hidden layers use the GELU activation function and implement Dropout regularization.

7. A video temporal localization system based on a large model, characterized in that, include: The data acquisition module is configured to: acquire raw video data and query statements, and compress raw video data of any duration into a fixed number of frames; The semantic alignment module is configured to: extract image features of each frame, map the image features frame by frame to a semantic embedding space compatible with a large language model, and obtain a continuous video representation through dimensional alignment. The splicing module is configured to: extract the text features of the query statement, and sequentially splice the continuous video representations with the text features to obtain image-text pairs; The video localization module is configured to: insert learnable visual cues at the front of image-text pairs to obtain a visual description sequence, input the visual description sequence into a large language model to perform temporal localization prediction on the visual description sequence, and output the video data localization result. The model training module is configured to: define a loss function, train the video temporal localization model, and use the trained video temporal localization model to perform video temporal localization.

8. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method described in any one of claims 1-6.

Images