Video positioning method and device based on multi-modal information prompt, equipment and medium

Abstract

The application provides a video positioning method and device based on multi-modal information prompt, equipment and medium, specifically relates to the technical field of video positioning, through text processing of the audio and video frames of the video to be positioned, the voice prompt text and the visual prompt text are obtained; determine the prompt text and the query text for executing the video positioning task on the video to be positioned; input the prompt text and the query text into the large language model, combine the time boundary perception prompt strategy, and output the positioning result. The above-mentioned video positioning method effectively integrates the voice information and visual information in the video, and combines the time boundary perception prompt strategy for time sequence reasoning, not only makes the output positioning result have explainability, but also improves the effect of video positioning. At the same time, the above-mentioned method uses the powerful reasoning ability of the large language model, without training and without processing the original features of the video, which is a light and generalizable method.

Description

Technical Field

[0001] This invention relates to the field of video positioning technology, and in particular to a video positioning method, apparatus, device, and medium based on multimodal information prompts. Background Technology

[0002] Currently, the main approach to improve the performance of video semantic localization tasks is through pre-training on different video language tasks. However, existing video segment semantic localization methods require specialized training on specific datasets, which incurs significant computational costs when processing long videos. Furthermore, they are highly susceptible to dataset bias, have poor cross-domain generalization capabilities, and cannot effectively transfer to new scenes with zero training.

[0003] Based on this, the present invention proposes a video localization method based on multimodal information prompts to solve the above problems. Summary of the Invention

[0004] This invention provides a video positioning method, apparatus, device, and medium based on multimodal information prompts to solve the problem of poor positioning effect of existing video positioning methods.

[0005] In a first aspect of the present invention, a video localization method based on multimodal information cues is proposed, the method comprising:

[0006] The audio and video frames of the video to be located are processed into text to obtain voice prompt text and visual prompt text. The voice prompt text is a formatted text of a speech transcription sentence and a corresponding time period, and the visual prompt text is a formatted text of an image annotation and a corresponding time point.

[0007] Determine the prompt text and query text for performing a video positioning task on the video to be located, wherein the prompt text includes task prompt text, voice prompt text, and visual prompt text;

[0008] The prompt text and the query text are input into a large language model, and combined with a time boundary-aware prompting strategy, the localization result is output, wherein the localization result is the start time and end time of the query text in the video to be located.

[0009] Optionally, the video frames of the video to be located are processed into text to obtain visual cue text, including:

[0010] Multiple video scenes in the video to be located are sampled to obtain multiple first video frames;

[0011] Image text annotation is performed on multiple first video frames using an image annotation model to obtain first image annotations and formatted text corresponding to a first time point as the visual text prompt, where the first time point is the moment of the first video frame in the video to be located.

[0012] Optionally, the video frames of the video to be located are processed into text to obtain visual cue text, including:

[0013] Based on the time period corresponding to the transcribed speech sentence, the midpoint of the time period is obtained as the second time point;

[0014] Based on the second time point, obtain the second video frame corresponding to the video to be located at the second time point;

[0015] The second video frame is annotated with image text using an image annotation model, resulting in second image annotations and formatted text at corresponding time points, which serve as the visual text prompts.

[0016] Optionally, the task prompt text includes a location task description text for the video to be located;

[0017] Based on the context learning capability of the large language model, the large language model determines the task to be completed through the localization task description text.

[0018] Optionally, the temporal boundary-aware cueing strategy enables the large language model to perform temporal reasoning in a noisy context and complete the video localization task. The temporal boundary-aware cueing strategy includes a multi-scale denoising thought chain, which constrains the large language model to perform temporal reasoning step by step and outputs the localization result. Specifically, it includes the following steps:

[0019] The first prompt text enables the large language model to globally understand the overall content of the video to be located, thereby obtaining a content summary of the video to be located.

[0020] The second cue text prompts the large language model to evaluate the effectiveness of the voice cue text and the visual cue text for the video localization task, so that the large language model can adaptively balance the information differences between the voice cue text and the visual cue text;

[0021] The third prompt text prompts the large language model to output the corresponding start time and end time for the query text, and the video to be located is segmented based on the start time and end time to obtain multiple segmented videos and corresponding time periods. The segmented videos are summarized and matched in combination with the query text to determine the segmented video that matches the query text.

[0022] The time period corresponding to the segmented video that matches the query text is used as the location result.

[0023] Optionally, the temporal boundary-aware cueing strategy further includes a validity criterion, which clarifies the text format of the input and output of the large language model during the execution of the video localization task, including:

[0024] Based on the first criterion, the large language model is made to reason according to the reasoning steps of the multi-scale noise reduction thinking chain, and the output content is in JSON format that can be automatically processed.

[0025] Based on the second criterion, the large language model is prompted to constrain each video localization task so that each video localization task outputs a unique localization result, and the start time in the localization result cannot exceed the predicted end time.

[0026] Based on the third criterion, the large language model is prompted to learn by following the context learning format and the reasoning steps of the multi-scale noise reduction thought chain.

[0027] Optionally, the temporal boundary-aware cueing strategy further includes single-sample context learning as an example, wherein the single-sample context learning is a fixed learning sample, so that the large language model follows the single-sample input-output format for temporal reasoning.

[0028] In a second aspect of the present invention, a video positioning device based on multimodal information prompts is provided, the device comprising:

[0029] The text processing module is used to perform text processing on the audio and video frames of the video to be located, to obtain voice prompt text and visual prompt text. The voice prompt text is a formatted text of a voice transcription sentence and a corresponding time period, and the visual prompt text is a formatted text of an image annotation and a corresponding time point.

[0030] The determination module is used to determine the prompt text and query text for performing a video positioning task on the video to be located, wherein the prompt text includes task prompt text, voice prompt text and visual prompt text;

[0031] The location result acquisition module is used to input the prompt text and the query text into the large language model, combine the time boundary awareness prompt strategy, and output the location result, wherein the location result is the start time and end time of the query text in the video to be located.

[0032] The text processing module further includes:

[0033] The sampling submodule is used to sample multiple video scenes in the video to be located to obtain multiple first video frames;

[0034] The first image annotation submodule is used to perform image text annotation on multiple first video frames through an image annotation model to obtain first image annotations and formatted text corresponding to a first time point as the visual text prompt, where the first time point is the moment of the first video frame in the video to be located.

[0035] The text processing module further includes:

[0036] The intermediate time acquisition submodule is used to acquire the intermediate time of the time period corresponding to the transcribed speech sentence as the second time point;

[0037] The second video frame acquisition submodule is used to acquire the second video frame corresponding to the video to be located at the second time point based on the second time point.

[0038] The second image annotation submodule is used to perform image text annotation on the second video frame through an image annotation model, and obtain the second image annotation and the formatted text of the corresponding time point as the visual text prompt.

[0039] The task prompt text includes a location task description text for the video to be located, and the determination module further includes:

[0040] The task determination submodule is used to determine the task to be completed based on the context learning capability of the large language model, which uses the localized task description text.

[0041] Specifically, the temporal boundary-aware cueing strategy enables the large language model to perform temporal reasoning in a noisy context and complete the video localization task. The temporal boundary-aware cueing strategy includes a multi-scale denoising thought chain, which constrains the large language model to perform temporal reasoning step-by-step and outputs the localization result. The localization result acquisition module further includes:

[0042] The first prompt submodule is used to enable the large language model to globally understand the overall content of the video to be located through the first prompt text, and to obtain a content summary of the video to be located.

[0043] The second prompt submodule is used to prompt the large language model to evaluate the effectiveness of the voice prompt text and the visual prompt text for the video localization task through the second prompt text, so that the large language model can adaptively balance the information differences between the voice prompt text and the visual prompt text;

[0044] The third prompt submodule is used to prompt the large language model to output the corresponding start time and end time for the query text through the third prompt text, and to segment the video to be located based on the start time and end time to obtain multiple segmented videos and corresponding time periods, and to summarize and match each segmented video in combination with the query text to determine the segmented video that matches the query text.

[0045] The location result acquisition submodule is used to take the time period corresponding to the segmented video that matches the query text as the location result.

[0046] The temporal boundary-aware prompting strategy further includes a validity criterion, which prompts the large language model on the input and output text format during the video localization task. The localization result acquisition module further includes:

[0047] The first criterion submodule is used to enable the large language model to reason according to the reasoning steps of the multi-scale noise reduction thinking chain based on the first criterion, and output the content in JSON format that can be automatically processed.

[0048] The second criterion submodule is used to prompt the large language model to constrain each video localization task based on the second criterion, so that each video localization task outputs a unique localization result, and the start time in the localization result cannot exceed the predicted end time.

[0049] The third criterion submodule is used to prompt the large language model to learn by following the context learning format and the reasoning steps of the multi-scale noise reduction thought chain based on the third criterion.

[0050] The time boundary-aware prompting strategy further includes single-sample context learning as an example, and the localization result acquisition module further includes:

[0051] A fixed-format submodule is used to treat the single-sample context learning as a fixed learning sample, so that the large language model follows the single-sample input-output format for temporal reasoning.

[0052] In a third aspect of the present invention, an electronic device is provided, comprising: a memory for storing one or more programs; a processor; and, when the one or more programs are executed by the processor, implementing the video positioning method based on multimodal information cues as described in any one of the first aspects above.

[0053] In a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the video positioning method based on multimodal information cues as described in any one of the first aspects above.

[0054] This invention offers the following advantages: Embodiments of this invention provide a video localization method, apparatus, device, and medium based on multimodal information prompts. By textualizing the audio and video frames of the video to be localized, voice prompt text and visual prompt text are obtained. The voice prompt text is formatted as a speech transcription sentence and corresponding time period, while the visual prompt text is formatted as an image annotation and corresponding time point. Prompt text and query text for performing a video localization task on the video to be localized are determined. The prompt text includes task prompt text, the voice prompt text, and the visual prompt text. The prompt text and query text are input into a large language model, and combined with a time boundary-aware prompting strategy, a localization result is output. The localization result is the start and end times of the query text in the video to be localized. This video localization method effectively integrates voice and visual information in the video and combines it with a time boundary-aware prompting strategy for temporal reasoning, not only making the output localization result interpretable but also improving the effectiveness of video localization. Meanwhile, the above method utilizes the powerful reasoning ability of large language models, requiring neither training nor processing of the original features of the video. It is a lightweight and generalizable method, revealing to some extent the potential of applying large language models to video temporal tasks. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart illustrating the steps of a video localization method based on multimodal information prompts provided in an embodiment of the present invention.

[0057] Figure 2 This is a flowchart illustrating a video localization method based on multimodal information prompts provided in an embodiment of the present invention;

[0058] Figure 3 This is an architecture diagram of a video positioning device based on multimodal information prompts provided in an embodiment of the present invention;

[0059] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] Most early research on video clip semantic localization addressed this task using supervised learning. However, these methods heavily relied on manually labeled data, resulting in high training costs. To mitigate this, many methods applied weakly supervised learning techniques, eliminating the need for temporal boundary annotations during training. However, these methods, trained on specific datasets, incur significant computational costs when applied to broader datasets and suffer from fitting bias leading to poor generalization. To address these issues, many studies have used different video language tasks for pre-training within a more unified framework to improve the performance of video clip semantic localization. However, these methods are generally more effective on short videos and have not been explored for long videos.

[0062] Existing traditional video segment semantic localization methods require specialized training on specific datasets, which incurs significant computational costs when processing long videos. Furthermore, they are highly susceptible to dataset bias, have poor cross-domain generalization capabilities, and cannot be effectively transferred to new scenarios with zero training.

[0063] Based on this, the present invention proposes a video localization method based on multimodal information prompts, which transcribes the audio and video information in the video into text, and uses a time boundary-aware prompting strategy to prompt a large language model to understand the requirements of the task and complete the prediction of the segment time period.

[0064] The first embodiment of this invention proposes a video localization method based on multimodal information prompts, see reference. Figure 1 , Figure 1 This is a flowchart illustrating the steps of a video localization method based on multimodal information prompts provided in an embodiment of the present invention. The video localization method includes the following steps:

[0065] Step 101: Perform text processing on the audio and video frames of the video to be located to obtain voice prompt text and visual prompt text. The voice prompt text is a formatted text of a voice transcription sentence and the corresponding time period, and the visual prompt text is a formatted text of an image annotation and the corresponding time point.

[0066] Step 102: Determine the prompt text and query text for performing a video positioning task on the video to be located, wherein the prompt text includes the task prompt text, the voice prompt text, and the visual prompt text;

[0067] Step 103: Input the prompt text and the query text into the large language model, combine the time boundary awareness prompt strategy, and output the positioning result, wherein the positioning result is the start time and end time of the query text in the video to be located.

[0068] To enable those skilled in the art to better understand the video localization method based on multimodal information prompts provided by this invention, Figure 2 This diagram illustrates a video localization method based on multimodal information cues provided by an embodiment of the present invention. The following section will discuss this method in conjunction with... Figure 2 The plan will be explained in detail:

[0069] In specific implementation step 101, firstly, any long video is acquired as the video to be located for video localization task. The difference from the localization task for short videos is that long videos contain complex and noisy long contexts, and long videos contain multimodal information such as voice and video frames. Therefore, the present invention converts the video localization task and its multimodal information of the video to be located into compressed data and uses it as prompt text of the language big model to improve the localization effect of the video localization task. Specifically, the audio of the video to be located is digitized, and the text information corresponding to the audio is processed through speech transcription to obtain a speech transcription sentence. This speech transcription process can be automatic speech recognition technology. The sentence is segmented, and the time period corresponding to the speech transcription sentence and the formatted text of the speech transcription sentence are used as the voice prompt text. For example, the format of the voice prompt text can be: "[start time]-[end time]:[speech transcription sentence]". The video frames of the video to be located are obtained, and each video frame is digitized. The image content of the video frame is labeled as the image label corresponding to the video frame. The image label of the video frame and the formatted text of the time point of the video frame in the video to be located are used as the visual prompt text. For example, the format of the visual prompt text can be: "[time point]:[image label]".

[0070] In an optional embodiment of the present invention, the video frames of the video to be located are processed into text to obtain visual prompt text. Specifically, firstly, multiple video scenes in the video to be located are sampled, where a video scene refers to the visual content in the video to be located. The sampling can be sparse sampling of video frames. For example, when the video to be located is movie A, movie A includes scenes of the protagonist a at home, at school, and participating in a sports meet, etc. Sparse sampling of the video scenes at home, school, and sports meet respectively can obtain multiple first video frames for the home scene, multiple first video frames for the school scene, and multiple first video frames for the sports meet scene. Then, image text annotation is performed on the multiple first video frames using an image annotation model to obtain multiple first image annotations. The image annotation model is a type of model that can obtain and annotate the corresponding content of video frames. The time of the first video frame in the video to be located is taken as the first time point, and the obtained first image annotations and the formatted text of the corresponding first time point are used as the visual prompt text. For example, sparse sampling is performed on the home scene to obtain 10 first video frames. The content corresponding to the first first video frame is that the protagonist A is eating at home, and the time in movie A is 150". Then the visual text cues for the first first video frame are "[150"]: [Protagonist A is eating at home]". The visual cues corresponding to the remaining 9 first video frames are obtained in sequence according to the above method, as well as the visual cues for other video scenes, which will not be elaborated here.

[0071] In an optional embodiment of the present invention, the video frames of the video to be located are processed into text to obtain visual prompt text. Specifically, based on the time period corresponding to the transcribed speech sentence, the midpoint of the time period is obtained as a second time point. Based on the second time point, the second video frame corresponding to the second time point in the video to be located is obtained. Then, the second video frame is annotated with image text using an image annotation model to obtain a second image annotation and formatted text corresponding to the time point as the visual text prompt. For example, if the time period corresponding to the transcribed speech sentence is [Time 1]-[Time 2], then the midpoint is (Time 1+Time 2) / 2. Taking (Time 1+Time 2) / 2 as the second time point, the video frame corresponding to the second time point [(Time 1+Time 2) / 2] in the video to be located is obtained. The image content "Protagonist A is in class" corresponding to the video frame is annotated to obtain a second image annotation. Then, the above visual text prompt is "[(Time 1+Time 2) / 2]: [Protagonist A is in class]". The visual prompt texts of the remaining transcribed speech sentences are obtained in sequence according to the above method, which will not be elaborated here.

[0072] In specific implementation step 102, the prompt text and query text for performing the video localization task on the video to be located are determined. The prompt text includes task prompt text, voice prompt text, and visual prompt text. The task prompt text includes a localization task description text for the video to be located. Given a query text, the start and end times of the video segment matching the query text are located from the video to be located. The query text is the text content that the user wants to know in the video to be located. For example, if the video localization task for video A is: the user wants to know when protagonist a goes to school in movie A, then the query text would be: protagonist a goes to school, and the task prompt text would be: based on the query text, determine the start and end times of the video segment in movie A that corresponds to the query text. Based on the context learning capability of the large language model, the large language model determines the task to be completed through the localization task description text and outputs the start and end times of the video content matching the query text in the video to be located. The large language model can be a series of models such as GPT (Generative Pre-trained Transformer, a deep learning-based language model) or LLAMA (Large Language Model MetaAI, a pre-trained language model), specifically GPT3, ChatGPT, LLAMA, or ChatGLM, etc.

[0073] In specific implementation step 103, since the large language model has few pre-training tasks and involves temporal reasoning, its understanding of the video localization task is not strong under the above input. Moreover, the context obtained after multimodal content textification is long and complex, and contains a lot of noise. In order to better enable the large language model to perform temporal reasoning in complex and noisy long contexts and effectively complete the video localization task, after inputting the prompt text and the query text into the large language model, it is also necessary to combine relevant strategies to make the large language model output a localization result that matches the query text. The localization result is the start time and end time of the query text in the video to be localized. For example, the localization result can be [start time, end time].

[0074] In an optional embodiment of the present invention, the temporal boundary-aware cueing strategy enables the large language model to perform temporal reasoning in a noisy context and complete the video localization task. The temporal reasoning refers to enabling the large language model to understand temporal relationships and the relationship between the content occurring in the video scene and the time of occurrence. The temporal boundary-aware cueing strategy includes a multi-scale noise reduction thought chain, which constrains the large language model to perform temporal reasoning step by step and outputs the localization result.

[0075] Specifically, the first prompt text enables the large language model to globally understand the overall content of the video to be located, thereby obtaining a content summary of the video to be located. The first prompt text then enables the large language model to summarize the entire content of the video to be located, so that the large language model ignores redundant details and some unnecessary repetitions in the video to be located, thereby strengthening the content of the video to be located.

[0076] Compared to speech-transcribed sentences, image annotations for video frames contain significant noise. Therefore, a second cue text is used to prompt the large language model to evaluate the effectiveness of the speech and visual cues for the video localization task, enabling the large language model to adaptively balance the information differences between the speech and visual cues. For example, at time 310 in movie A, the corresponding speech-transcribed sentence is "Protagonist A runs to school," while the corresponding image annotation might be "Protagonist A walks to school," "Protagonist A runs home," etc. It is evident that there is a significant information difference between the image annotation and the speech-transcribed sentence. When this occurs, the second cue text prompts the large language model to focus more on the content of the speech-transcribed sentence and uses it as the standard to adaptively balance the differences between the speech-transcribed sentence and the image text annotation.

[0077] Furthermore, to enable the large language model to simultaneously focus on the local details of both the matching and non-matching portions of the video to be located, a third prompt text is used to prompt the large language model to output the corresponding start and end times for the query text. Based on the start and end times, the video to be located is segmented to obtain multiple segmented videos and their corresponding time periods. Each segmented video is then summarized and matched in conjunction with the query text to determine the segmented video that matches the query text. For example, when the query is "The main character A is eating at home", and the large language model outputs a start time of 150" and an end time of 378" for the query text, the video to be located is segmented based on the start time 150" and the end time 378" to obtain time period 1: [start time of movie A - start time 150"], time period 2: [start time 150" - end time 378"], and time period 3: [end time 378" - end time of movie], as well as the segmented videos 1-a, 2-b, and 3-c corresponding to the above three time periods. The content of the above three segmented videos (1-a, 2-b, and 3-c) is summarized to obtain three corresponding summary texts. The query text is matched with the three summary texts. The summary text of the above segmented video 2-b matches the content of the query text, and segmented video 2-b is determined as the segmented video that matches the query text.

[0078] After identifying the video segment that matches the query text, the time period corresponding to that video segment is used as the localization result of the large language model for the video localization task. For example, when video segment 2-b is identified as the video segment that matches the query text, the localization result is [150, 378].

[0079] In an optional embodiment of the present invention, the time boundary-aware prompting strategy further includes a validity criterion. The validity criterion prompts the large language model regarding the input and output text format during the execution of the video localization task. Specifically, firstly, a first criterion guides the large language model to reason according to the inference steps of the multi-scale denoising thought chain, ensuring that the output content of the large language model is in JSON format that can be automatically processed. In this embodiment, the key-value pairs of the JSON format are set to the steps of the aforementioned multi-scale denoising thought chain. Then, based on a second criterion, the large language model is prompted to constrain each video localization task, ensuring that each video localization task outputs a unique localization result, and that the start time in the localization result cannot exceed the predicted end time; otherwise, the localization result is considered invalid. Further, a third criterion prompts the large language model to learn according to the context learning format and the inference steps of the multi-scale denoising thought chain. The third criterion can constrain the large language model to learn according to the reasoning steps of the multi-scale noise reduction thought chain. In the process of temporal reasoning, the format used by the large language model is learned and output according to the format of context learning, which prevents the large language model from directly copying the answer of context learning. This ensures that the large language model can only learn and output the localization result through the temporal boundary awareness prompt strategy.

[0080] In an optional embodiment of the present invention, the time boundary-aware prompting strategy further includes single-sample context learning as a case, wherein the single-sample context learning is a fixed learning sample, and the case is added to the prompting text of the large language model so that the large language model follows the input-output format of the single sample for temporal reasoning.

[0081] For example, the above learning sample includes the following: <Input> When performing a video localization task, the following prompt text and query text need to be provided: video name and localization task text, query text, voice prompt text (time period + voice transcription sentence) and visual prompt text (time point + image annotation), and it needs to be determined that there is a segment in the video that matches the query text, and the time period of the matching segment [ <x> ， <y>As the output localization result, X represents the start time, and Y represents the end time, with X not exceeding Y. Then, the first step is to summarize the video content; the second step is to balance the differences between the transcribed speech and the image text annotations; finally, based on the obtained... <x>and <y>The input video is segmented and summarized to determine the matching video segment. The start and end times of the matching video segment are used as the final localization result and output. The voice prompts and visual prompts input in the learning samples are not shown in detail; this part is input according to the text format described in step 101 above.

[0082] The output section includes the following: "Summary": This section summarizes the entire input video content as understood by the large language model; "Evaluation": This section evaluates the voice prompts and visual prompts; "Query Text": This section contains the input query text; "Before 150": This section summarizes the content in the input video before time 150; "Between 150 and 378": This section summarizes the content in the input video between time 150 and 378; "After 378": This section summarizes the content in the input video after time 378; "Result": This section shows the start and end times of the video segment corresponding to the input query text in the input video, which is [150, 378].

[0083] After replacing the input video with any other video, the large language model performs temporal reasoning according to the format of the above case, and performs input and output according to the above format until the final localization result is obtained. For all the operations performed by the large language model, the large language model will output the corresponding text content.

[0084] This invention provides a video localization method based on multimodal information prompts. The method involves textualizing the audio and video frames of the video to be localized, obtaining voice prompt text and visual prompt text. The voice prompt text is formatted as a transcript of a speech sentence and its corresponding time period, while the visual prompt text is formatted as an image annotation and its corresponding time point. The method then determines prompt text and query text for performing a video localization task on the video to be localized. The prompt text includes task prompt text, the voice prompt text, and the visual prompt text. These prompt text and query text are input into a large language model, and combined with a time boundary-aware prompting strategy, the localization result is output. The localization result is the start and end times of the query text within the video to be localized. This video localization method effectively integrates voice and visual information from the video and combines it with a time boundary-aware prompting strategy for temporal reasoning. This not only makes the output localization result interpretable but also improves the effectiveness of video localization. Furthermore, this method utilizes the powerful reasoning capabilities of a large language model, requiring neither training nor processing of the video's original features. It is a lightweight and generalizable method, revealing to some extent the potential of applying large language models to video temporal tasks.

[0085] In a second aspect of the present invention, a video positioning device based on multimodal information prompts is provided, see [reference]. Figure 3 , Figure 3 This is an architectural diagram of a video positioning device based on multimodal information prompts proposed in an embodiment of the present invention. The device includes:

[0086] The text processing module 301 is used to perform text processing on the audio and video frames of the video to be located, to obtain voice prompt text and visual prompt text. The voice prompt text is a formatted text of a voice transcription sentence and a corresponding time period, and the visual prompt text is a formatted text of an image annotation and a corresponding time point.

[0087] The determining module 302 is used to determine the prompt text and query text for performing a video positioning task on the video to be located, wherein the prompt text includes the task prompt text, the voice prompt text, and the visual prompt text;

[0088] The positioning result acquisition module 303 is used to input the prompt text and the query text into the large language model, combine the time boundary awareness prompt strategy, and output the positioning result, wherein the positioning result is the start time and end time of the query text in the video to be located.

[0089] The text processing module further includes:

[0090] The sampling submodule is used to sample multiple video scenes in the video to be located to obtain multiple first video frames;

[0091] The first image annotation submodule is used to perform image text annotation on multiple first video frames through an image annotation model to obtain first image annotations and formatted text corresponding to a first time point as the visual text prompt, where the first time point is the moment of the first video frame in the video to be located.

[0092] The text processing module further includes:

[0093] The intermediate time acquisition submodule is used to acquire the intermediate time of the time period corresponding to the transcribed speech sentence as the second time point;

[0094] The second video frame acquisition submodule is used to acquire the second video frame corresponding to the video to be located at the second time point based on the second time point.

[0095] The second image annotation submodule is used to perform image text annotation on the second video frame through an image annotation model, and obtain the second image annotation and the formatted text of the corresponding time point as the visual text prompt.

[0096] The task prompt text includes a location task description text for the video to be located, and the determination module further includes:

[0097] The task determination submodule is used to determine the task to be completed based on the context learning capability of the large language model, which uses the localized task description text.

[0098] Specifically, the temporal boundary-aware cueing strategy enables the large language model to perform temporal reasoning in a noisy context and complete the video localization task. The temporal boundary-aware cueing strategy includes a multi-scale denoising thought chain, which constrains the large language model to perform temporal reasoning step-by-step and outputs the localization result. The localization result acquisition module further includes:

[0099] The first prompt submodule is used to enable the large language model to globally understand the overall content of the video to be located through the first prompt text, and to obtain a content summary of the video to be located.

[0100] The second prompt submodule is used to prompt the large language model to evaluate the effectiveness of the voice prompt text and the visual prompt text for the video localization task through the second prompt text, so that the large language model can adaptively balance the information differences between the voice prompt text and the visual prompt text;

[0101] The third prompt submodule is used to prompt the large language model to output the corresponding start time and end time for the query text through the third prompt text, and to segment the video to be located based on the start time and end time to obtain multiple segmented videos and corresponding time periods, and to summarize and match each segmented video in combination with the query text to determine the segmented video that matches the query text.

[0102] The location result acquisition submodule is used to take the time period corresponding to the segmented video that matches the query text as the location result.

[0103] The temporal boundary-aware prompting strategy further includes a validity criterion, which prompts the large language model on the input and output text format during the video localization task. The localization result acquisition module further includes:

[0104] The first criterion submodule is used to enable the large language model to reason according to the reasoning steps of the multi-scale noise reduction thinking chain based on the first criterion, and output the content in JSON format that can be automatically processed.

[0105] The second criterion submodule is used to prompt the large language model to constrain each video localization task based on the second criterion, so that each video localization task outputs a unique localization result, and the start time in the localization result cannot exceed the predicted end time.

[0106] The third criterion submodule is used to prompt the large language model to learn by following the context learning format and the reasoning steps of the multi-scale noise reduction thought chain based on the third criterion.

[0107] The time boundary-aware prompting strategy further includes single-sample context learning as an example, and the localization result acquisition module further includes:

[0108] A fixed-format submodule is used to treat the single-sample context learning as a fixed learning sample, so that the large language model follows the single-sample input-output format for temporal reasoning.

[0109] Based on the same inventive concept, this invention discloses an electronic device. Figure 4 A schematic diagram of an electronic device disclosed in an embodiment of the present invention is shown, such as... Figure 4 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory of the electronic device is not less than 12G, and the main frequency of the processor is not less than 2.4GHz. The memory 110 and the processor 120 are connected by a bus communication. The memory 110 stores a computer program, which can run on the processor 120 to implement a video positioning method based on multimodal information prompts disclosed in the embodiments of the present invention.

[0110] Based on the same inventive concept, embodiments of the present invention disclose a computer-readable storage medium storing a computer program / instruction thereon, which, when executed by a processor, implements a video positioning method based on multimodal information prompts disclosed in embodiments of the present invention.

[0111] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0112] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0113] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0115] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0116] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0117] The above provides a detailed description of a video positioning method, apparatus, device, and medium based on multimodal information prompts provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.< / y> < / x> < / y> < / x>

Claims

1. A video localization method based on multimodal information cues, characterized in that, The method includes: The audio and video frames of the video to be located are processed into text to obtain voice prompt text and visual prompt text. The voice prompt text is a formatted text of a speech transcription sentence and a corresponding time period, and the visual prompt text is a formatted text of an image annotation and a corresponding time point. Determine the prompt text and query text for performing a video positioning task on the video to be located, wherein the prompt text includes task prompt text, voice prompt text, and visual prompt text; The prompt text and the query text are input into the large language model, and combined with the time boundary-aware prompt strategy, the localization result is output, wherein the localization result is the start time and end time of the query text in the video to be located; The time boundary-aware cueing strategy enables the large language model to perform temporal reasoning in noisy contexts and complete the video localization task. The time boundary-aware cueing strategy includes a multi-scale denoising thought chain, which constrains the large language model to perform temporal reasoning step-by-step and output the localization result. Specifically, it includes the following steps: The first prompt text enables the large language model to globally understand the overall content of the video to be located, thereby obtaining a content summary of the video to be located. The second cue text prompts the large language model to evaluate the effectiveness of the voice cue text and the visual cue text for the video localization task, so that the large language model can adaptively balance the information differences between the voice cue text and the visual cue text; The third prompt text prompts the large language model to output the corresponding start time and end time for the query text, and the video to be located is segmented based on the start time and end time to obtain multiple segmented videos and corresponding time periods. The segmented videos are summarized and matched in combination with the query text to determine the segmented video that matches the query text. The time period corresponding to the segmented video that matches the query text is used as the location result; The temporal boundary-aware cueing strategy also includes a validity criterion, which clarifies the text format of the input and output of the large language model during the execution of the video localization task, including: Based on the first criterion, the large language model is made to reason according to the reasoning steps of the multi-scale noise reduction thinking chain, and the output content is in JSON format that can be automatically processed. Based on the second criterion, the large language model is prompted to constrain each video localization task so that each video localization task outputs a unique localization result, and the start time in the localization result cannot exceed the predicted end time. Based on the third criterion, the large language model is prompted to learn by following the format of context learning and the reasoning steps of the multi-scale noise reduction thought chain; The time boundary-aware cueing strategy also includes single-sample context learning as an example. The single-sample context learning is a fixed learning sample, so that the large language model follows the single-sample input-output format for temporal reasoning.

2. The video localization method based on multimodal information cues according to claim 1, characterized in that, The video frames of the video to be located are processed into text to obtain visual cues, including: Multiple video scenes in the video to be located are sampled to obtain multiple first video frames; Image text annotation is performed on multiple first video frames using an image annotation model to obtain first image annotations and formatted text corresponding to a first time point, which are used as the visual cue text. The first time point is the moment of the first video frame in the video to be located.

3. The video localization method based on multimodal information cues according to claim 1 or 2, characterized in that, The video frames of the video to be located are processed into text to obtain visual cues, including: Based on the time period corresponding to the transcribed speech sentence, the midpoint of the time period is obtained as the second time point; Based on the second time point, obtain the second video frame corresponding to the video to be located at the second time point; The second video frame is annotated with image text using an image annotation model, and the resulting second image annotation and the formatted text at the corresponding time point are used as the visual cue text.

4. The video localization method based on multimodal information cues according to claim 1, characterized in that, The task prompt text includes a description of the location task for the video to be located; Based on the context learning capability of the large language model, the large language model determines the task to be completed through the localization task description text.

5. A video positioning device based on multimodal information prompts, characterized in that, The device includes: The text processing module is used to perform text processing on the audio and video frames of the video to be located, to obtain voice prompt text and visual prompt text. The voice prompt text is a formatted text of a voice transcription sentence and a corresponding time period, and the visual prompt text is a formatted text of an image annotation and a corresponding time point. The determination module is used to determine the prompt text and query text for performing a video positioning task on the video to be located, wherein the prompt text includes task prompt text, voice prompt text and visual prompt text; The positioning result acquisition module is used to input the prompt text and the query text into the large language model, combine the time boundary awareness prompt strategy, and output the positioning result, wherein the positioning result is the start time and end time of the query text in the video to be located; Specifically, the temporal boundary-aware cueing strategy enables the large language model to perform temporal reasoning in a noisy context and complete the video localization task. The temporal boundary-aware cueing strategy includes a multi-scale denoising thought chain, which constrains the large language model to perform temporal reasoning step-by-step and outputs the localization result. The localization result acquisition module further includes: The first prompt submodule is used to enable the large language model to globally understand the overall content of the video to be located through the first prompt text, and to obtain a content summary of the video to be located. The second prompt submodule is used to prompt the large language model to evaluate the effectiveness of the voice prompt text and the visual prompt text for the video localization task through the second prompt text, so that the large language model can adaptively balance the information differences between the voice prompt text and the visual prompt text; The third prompt submodule is used to prompt the large language model to output the corresponding start time and end time for the query text through the third prompt text, and to segment the video to be located based on the start time and end time to obtain multiple segmented videos and corresponding time periods, and to summarize and match each segmented video in combination with the query text to determine the segmented video that matches the query text. The location result acquisition submodule is used to take the time period corresponding to the segmented video that matches the query text as the location result; The temporal boundary-aware prompting strategy further includes a validity criterion, which prompts the large language model on the input and output text format during the video localization task. The localization result acquisition module further includes: The first criterion submodule is used to enable the large language model to reason according to the reasoning steps of the multi-scale noise reduction thinking chain based on the first criterion, and output the content in JSON format that can be automatically processed. The second criterion submodule is used to prompt the large language model to constrain each video localization task based on the second criterion, so that each video localization task outputs a unique localization result, and the start time in the localization result cannot exceed the predicted end time. The third criterion submodule is used to prompt the large language model to learn by following the context learning format and the reasoning steps of the multi-scale noise reduction thought chain based on the third criterion. The time boundary-aware prompting strategy further includes single-sample context learning as an example, and the localization result acquisition module further includes: A fixed-format submodule is used to treat the single-sample context learning as a fixed learning sample, so that the large language model follows the single-sample input-output format for temporal reasoning.

6. An electronic device, characterized in that, include: Memory, used to store one or more programs; processor; When the processor executes the one or more programs, it implements the video localization method based on multimodal information cues as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the video localization method based on multimodal information cues as described in any one of claims 1-4.

Images