A visual model-based large language model video timing positioning method and product
By converting video frame images into visual description information and combining them with input text to generate visual description sequences, and using a large language model for temporal localization prediction, the problem of poor performance of visual models in video temporal localization tasks is solved, and the accuracy and generalization ability of video temporal localization are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TSINGHUA UNIVERSITY
- Filing Date
- 2023-12-20
- Publication Date
- 2026-07-21
Smart Images
Figure CN117851638B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a method and product for video temporal localization based on a large language model with a visual model. Background Technology
[0002] Large language models are artificial intelligence models trained on large amounts of text data that can perform a wide range of tasks, including text summarization, translation, reasoning, and sentiment analysis. With recent developments, large language models have been extended to many tasks beyond natural language processing, such as motion graph processing, visual question answering, and reinforcement learning, demonstrating considerable generalization ability. In particular, applying large language models to multimodal tasks such as video temporal localization has become a popular research direction.
[0003] However, existing large language models for video temporal localization tasks require the integration of a visual model to form a larger video model. While the visual model enables processing video and image content, it focuses on the overall description of events occurring within the video content, paying relatively little attention to the temporal information of specific events. This results in poor performance for large video models in video temporal localization tasks. Therefore, improving the performance of large language models in video temporal localization tasks has become a pressing issue in this field. Summary of the Invention
[0004] This application provides a method and product for video temporal localization based on a large language model with a visual model, aiming to solve the problem of how to improve the performance of large language models in processing video temporal localization tasks.
[0005] A first aspect of this application provides a video temporal localization method based on a large language model and a visual model, the method comprising:
[0006] Obtain the target video and the corresponding input text;
[0007] Based on the target video, multiple video frame images are acquired, each video frame image is input into a visual model, and visual description information corresponding to the multiple video frame images is acquired based on the visual model. The visual description information is text information that characterizes the image content of each video frame image.
[0008] Based on the target video and the input text, the visual description information corresponding to the multiple video frame images is integrated and processed to obtain a visual description sequence.
[0009] The visual description sequence is input into a large language model to perform temporal localization prediction on the visual description sequence and output the video data localization result.
[0010] In one optional implementation, the step of acquiring multiple video frame images based on the target video, inputting each video frame image into a visual model, and acquiring visual description information corresponding to the multiple video frame images based on the visual model includes:
[0011] Obtain the total duration of the target video;
[0012] Based on the total duration, a preset sampling duration is determined, and the target video is sampled frame by frame using the preset sampling duration as a reference to obtain the multiple video frame images;
[0013] Each video frame image is input into the visual model, and the word text sequence of each video frame image in response to the first question is determined from the corpus of the visual model as the first answer information. The first question is used to represent a query about the content of the image.
[0014] The first answer information is used as the visual description information.
[0015] In one optional implementation, after obtaining the first answer information, the method further includes:
[0016] The input text is input into the visual model, so that the visual model generates a second question based on the keywords in the input text. The second question is used to characterize whether the image has experienced an event corresponding to the input text.
[0017] Each video frame image is input into the visual model to determine the second answer information for each video frame image in relation to the second question;
[0018] The first answer information and the second answer information are combined to form the visual description information.
[0019] In one optional implementation, the step of integrating the visual description information corresponding to the plurality of video frame images based on the target video and input text to obtain a visual description sequence includes:
[0020] Based on the target video, determine the time node information corresponding to each of the multiple video frame images;
[0021] Add the time node information corresponding to each video frame image to the visual description information corresponding to each video frame image;
[0022] Based on the time node information, the visual description information corresponding to the multiple video frame images is sorted to obtain the text description sequence corresponding to the multiple video frame images;
[0023] A large language problem corpus is generated based on the input text, and the large language problem corpus is combined with the text description sequences corresponding to the multiple video frame images to form the visual description sequence.
[0024] In one optional implementation, the visual description sequence is input into a large language model to perform temporal localization prediction on the visual description sequence, and outputs video data localization results, including:
[0025] The input large language model is used to obtain temporal visual features from the visual description sequence and the input text. The temporal visual features are features that characterize the changes in visual description information corresponding to adjacent time nodes in the visual description sequence.
[0026] Based on the large language problem corpus, the visual description sequence is temporally located and predicted based on the temporal visual features, and the start time node information and end time node information are filtered out from the time node information.
[0027] The start time node information and the end time node information are combined to form the video data positioning result and then output.
[0028] A second aspect of this application provides a large language model video temporal localization system based on a visual model, the system comprising:
[0029] The video reading unit is used to acquire a target video and input text corresponding to the target video, and send the target video and the input text to the video description generation unit and the Prompt guidance generation unit;
[0030] The video description generation unit is used to receive the target video and the input text sent by the video reading unit, obtain multiple video frame images based on the target video, and obtain visual description information corresponding to the multiple video frame images according to the multiple video frame images and send it to the Prompt guidance generation unit. The visual description information is text information that characterizes the image content of each video frame image.
[0031] The Prompt generation unit is used to receive the visual description information sent by the video description unit, and the target video and input text sent by the video reading unit; and to integrate and process the visual description information corresponding to the multiple video frame images based on the target video and input text to obtain a visual description sequence which is then sent to the large language processing unit.
[0032] The large language processing unit is used to receive the visual description sequence sent by the Prompt guidance generation unit, perform temporal localization prediction on the visual description sequence, and output the video data localization result.
[0033] A third aspect of this application provides a video temporal localization device based on a large language model with a visual model, the device comprising:
[0034] The acquisition module is used to acquire the target video and the input text corresponding to the target video;
[0035] The visual description module is used to acquire multiple video frame images based on the target video, input each video frame image into a visual model, and acquire visual description information corresponding to the multiple video frame images based on the visual model. The visual description information is text information that characterizes the image content of each video frame image.
[0036] An integration module is used to integrate and process the visual description information corresponding to the multiple video frame images based on the target video and the input text to obtain a visual description sequence;
[0037] The prediction module is used to input the visual description sequence into a large language model to perform temporal localization prediction on the visual description sequence and output the video data localization result.
[0038] In one optional implementation, the video description module includes:
[0039] The total duration submodule is used to obtain the total duration of the target video;
[0040] The sampling duration submodule is used to determine a preset sampling duration based on the total duration, and to perform frame sampling on the target video based on the preset sampling duration to obtain the multiple video frame images;
[0041] The first answer information submodule is used to input each video frame image into the visual model, and determine the word text sequence of each video frame image in response to the first question from the corpus of the visual model as the first answer information. The first question is used to represent a query on the image content.
[0042] The visual description submodule is used to use the first answer information as the visual description information.
[0043] In one optional implementation, the video description module further includes:
[0044] The second question submodule is used to input the input text into the visual model, so that the visual model generates a second question based on the keywords in the input text. The second question is used to characterize whether the image has experienced an event corresponding to the input text.
[0045] The second answer information submodule is used to input each video frame image into the visual model and determine the second answer information for each video frame image for the second question.
[0046] The answer combination submodule is used to combine the first answer information and the second answer information into the visual description information.
[0047] In one optional implementation, the integration module includes:
[0048] The time node submodule is used to determine the time node information corresponding to the multiple video frame images based on the target video;
[0049] The time node information addition submodule is used to add the time node information corresponding to each video frame image to the visual description information corresponding to each video frame image;
[0050] The sorting submodule is used to sort the visual description information corresponding to the multiple video frame images based on the time node information to obtain the text description sequence corresponding to the multiple video frame images.
[0051] The integration submodule is used to generate a large language problem corpus based on the input text, and to combine the large language problem corpus with the text description sequences corresponding to the multiple video frame images to form the visual description sequence.
[0052] In one optional implementation, the prediction module includes:
[0053] The temporal visual feature submodule is used to obtain temporal visual features from the visual description sequence, the input text, and the input large language model. The temporal visual features are features that characterize the changes in visual description information corresponding to adjacent time nodes in the visual description sequence.
[0054] The prediction submodule is used to perform temporal localization prediction on the visual description sequence based on the temporal visual features, using the large language problem corpus as a benchmark, and to filter out the start time node information and the end time node information from the time node information.
[0055] The output submodule is used to combine the start time node information and the end time node information into the video data positioning result for output.
[0056] A fourth aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the large language model video temporal localization method based on a visual model as described in any of the first aspects.
[0057] The fifth aspect of this application provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps in the large language model video temporal localization method based on a visual model as described in any of the first aspects.
[0058] The sixth aspect of this application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the large language model video temporal localization method based on a visual model as described in the first aspect.
[0059] Beneficial effects:
[0060] This application provides a video temporal localization method and product based on a visual model and a large language model. The method includes: acquiring a target video and corresponding input text; acquiring multiple video frame images based on the target video, inputting each video frame image into a visual model, acquiring visual description information corresponding to the multiple video frame images based on the visual model, wherein the visual description information is text information representing the image content of each video frame image; integrating and processing the visual description information corresponding to the multiple video frame images based on the target video and the input text to obtain a visual description sequence; inputting the visual description sequence into a large language model to perform temporal localization prediction on the visual description sequence, and outputting video data localization results. This application converts video information into visual description information of text attributes and locates the temporal content corresponding to each frame in the target video based on a visual model, thereby amplifying the finer-grained description of things in the video temporal localization task, ensuring that the large language model can obtain sufficient temporal information features from the text information, effectively improving the generalization ability of the large language model and the processing performance of the video temporal localization task. Attached Figure Description
[0061] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0062] Figure 1 This is a schematic flowchart of a video temporal localization method based on a visual model and a large language model proposed in an embodiment of this application;
[0063] Figure 2 This is an example diagram of a video temporal localization task based on a large language model with a visual model, proposed in an embodiment of this application.
[0064] Figure 3This is a schematic diagram of a video temporal localization system architecture based on a visual model and a large language model, proposed in an embodiment of this application.
[0065] Figure 4 This is a schematic diagram of a video temporal localization device based on a visual model and an embodiment of this application.
[0066] Figure 5 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation
[0067] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0068] Large language models are artificial intelligence models trained on large amounts of text data that can perform a wide range of tasks, including text summarization, translation, reasoning, and sentiment analysis. With recent developments, large language models have been extended to many tasks beyond natural language processing, such as motion graph processing, visual question answering, and reinforcement learning, demonstrating considerable generalization ability. In particular, applying large language models to multimodal tasks such as video temporal localization has become a popular research direction.
[0069] However, in related technologies, existing large language models need to be combined with visual models to form a large video model to perform video temporal localization tasks. The visual model enables it to process video and image content, but the visual model focuses on the comprehensive description of things happening in the video content, while paying relatively little attention to the temporal information of specific things in the video content. This makes the performance of large video models in processing video temporal localization tasks poor.
[0070] In view of this, embodiments of this application propose a video temporal localization method based on a large language model with a visual model. Figure 1 This paper illustrates a flowchart of a video temporal localization method based on a large language model and a visual model, according to an embodiment of this application. Figure 1 As shown, the method includes the following steps:
[0071] S101. Obtain the target video and the input text corresponding to the target video.
[0072] In specific implementation step S101, the target video and the corresponding input text are first acquired. In this embodiment, the target video and the input text are the input data for the video temporal localization task. The video temporal localization task is a downstream task that determines the video occurrence time node corresponding to the input text from the target video based on the information in the input text. The target video is video data composed of multiple consecutive video frame images arranged in chronological order. At least one frame in the target video contains image content corresponding to the input text, and the input text is a text description of the event to be located.
[0073] For example, the target video is video data composed of multiple consecutive video frames arranged in chronological order. This video data as a whole depicts the interaction between a diver and marine life. The input text is "a man reaches out to touch a fish." The video temporal localization task then determines the start and end time points of the corresponding input text "a man reaches out to touch a fish" from this overall video depicting the interaction between the diver and marine life (i.e., multiple consecutive video frames). It should be noted that the above example is merely an optional approach provided to help those skilled in the art better understand the solution of this application. The specific target video and input text can be determined according to the actual situation, and this application does not impose any limitations here.
[0074] S102. Based on the target video, acquire multiple video frame images, input each video frame image into a visual model, and acquire visual description information corresponding to the multiple video frame images based on the visual model.
[0075] Based on the target video, multiple video frame images are acquired, and each video frame image is input into a visual model. Based on the visual model, visual description information corresponding to the multiple video frame images is acquired. The visual description information is text information that characterizes the image content of each video frame image.
[0076] In specific implementation step S102, multiple consecutive video frame images from the target video are first acquired. Since the total video duration varies for different target videos when performing different video temporal localization tasks, using the same frame sampling method for target videos of different total durations will result in a difference in the number of video frame images sampled. Target videos with longer total durations will have more video frame images sampled, resulting in more feature information in subsequent processing; conversely, target videos with shorter total durations will have fewer video frame images sampled, resulting in less feature information in subsequent processing and negatively impacting the accuracy of the video data localization result.
[0077] Therefore, to ensure that the method provided in this application can guarantee high performance in video timing localization for target videos of different total durations, this application first needs to determine a preset sampling duration suitable for different target videos. Specifically, the total duration of the target video is first obtained; then, a preset sampling duration is determined based on the total duration, and frame sampling is performed on the target video using the preset sampling duration as a reference to obtain the multiple video frame images. Optionally, the preset sampling duration can be a duration determined according to a preset ratio of the total duration of the corresponding target video, thereby ensuring that the time interval between multiple video frame images obtained for target videos with a longer total duration is relatively longer, and the time interval between multiple video frame images obtained for target videos with a shorter total duration is relatively shorter. For example, for a target video with a total duration of 20 seconds, the preset sampling duration is determined to be 1 second (the time interval between two adjacent frames is 1 second), and frame sampling is performed on the target video to obtain 20 video frame images. It should be noted that the above example is only an optional approach given to enable those skilled in the art to better understand the solution of this application. The specific total duration of the target video and the preset sampling duration can be determined according to the actual situation, and this application does not impose any restrictions here.
[0078] After multiple video frame images corresponding to the target video, each video frame image is input into the visual model. The visual model includes an image description model and is configured to answer a first question and generate first answer information. The first question is used to represent a query about the image content. Specifically, the visual model includes a corpus. The visual model acquires each input video frame image, processes the video frame image, and sequentially determines the word text sequence of each video frame image in response to the first question from the corpus of the visual model to form the first answer information. The first answer information is text information representing the image content of each video frame image. Finally, the first answer information is used as the visual description information. In this embodiment, the visual model realizes the conversion of the image content of the video frame image into the text information of the first answer information, thereby providing information guidance for subsequent video temporal localization tasks. The information carrier to be processed by the large language model is fixed on the text carrier, thereby expanding the processing scenario of the large language model for video temporal localization tasks and effectively making up for the shortcomings of using a poor-performing large video model (input is video or image information) to process video temporal localization tasks.
[0079] In one optional implementation, during the process of obtaining the first answer information based on the visual model, since the process of obtaining the first answer information based on the corpus is independent of the input text, when the keywords in the input text appear infrequently or have a frequency of 0 in the corpus, the first answer information corresponding to the video frame image may use other words different from the keywords in the input text. This reduces the correlation between the first answer information and the input text when the visual description information is used as the visual description information for subsequent video temporal localization tasks, further making it difficult for the visual description information corresponding to the first answer information to provide effective assistance for video temporal localization tasks in this specific situation, i.e., reducing the accuracy of video data localization results. Therefore, in this embodiment, after obtaining the first answer information based on the visual model, it is necessary to add feature information that is related to the input text. Specifically, the visual model also includes a visual question-and-answer model (such as a VQA model). First, the input text is input into the visual question-and-answer model in the visual model. The visual question-and-answer model is configured to generate a second question based on the keywords in the input text. The second question is used to characterize whether the image has experienced an event corresponding to the input text. Subsequently, each video frame image is input into the visual question-answering model in the visual model. The visual question-answering model processes the video frame images and sequentially determines the binary answer (a yes / no answer) of each video frame image to the second question. This binary answer is used as the second answer information for each video frame image to the second question. The second answer information serves as additional retrieval information to expand the feature text information of the input text in the visual description information. Specifically, when the visual question-answering question determines that the event corresponding to the input text has occurred in each video frame image, "yes" is used as the second answer information; when the visual question-answering question determines that the event corresponding to the input text has not occurred in each video frame image, "no" is used as the second answer information. Finally, the first answer information and the second answer information are combined into the visual description information. At this time, the visual description information includes the text description features corresponding to each video frame image and the features indicating whether there is a correlation with the input text, thereby expanding the information width of the visual description information corresponding to each video frame image and effectively improving the performance of performing video temporal localization tasks.
[0080] For example, the input text "A boy is holding a pillow" contains the keyword "pillow". After inputting the video frame image into the image description model in the visual model, the word text sequence of the video frame image corresponding to the first question is sequentially determined from the corpus to form the first answer information. The generated first answer information uses "bedding" for text description. At this time, the first answer information corresponds to the keyword in the input text, and this frame should be identified as a frame within the duration of the event corresponding to the input text. However, because the word text in the first answer information is not the same as the keyword in the input text, the first answer information is not correctly predicted in subsequent processing based on the large language model. This causes a decrease in the accuracy of the video data localization result. Therefore, after obtaining the first answer information, this application generates a second question based on a visual question-and-answer model. This second question represents a query about whether the event "A boy is holding a pillow" has occurred in the image. The video frame image is input into the visual question-and-answer model, yielding the second answer information "yes." The first and second answer information are combined to form the visual description information corresponding to the video frame image. This visual description information, enriched with additional retrieval information, is input into a large language model, enabling the large language model to correctly predict the image frame based on more feature information. It should be noted that the above example is merely an optional approach provided to help those skilled in the art better understand the solution of this application. The specific first answer information, input text, etc., can be determined according to the actual situation, and this application does not impose any limitations here.
[0081] S103. Based on the target video and the input text, the visual description information corresponding to the multiple video frame images is integrated and processed to obtain a visual description sequence.
[0082] In specific implementation step S103, after obtaining the visual description information corresponding to the multiple video frame images, the visual description information needs to be preprocessed and adjusted to the format of the corresponding large language model. Specifically, firstly, based on the target video, the time node information corresponding to each of the multiple video frame images is determined. The time node information is used to characterize the specific time node in which each video frame image is arranged in chronological order in the target video. Subsequently, the time node information corresponding to each video frame image is added to the visual description information corresponding to each video frame image. For example, if the time node information corresponding to a video frame image is determined to be the 12.0s occurrence of the video frame image in the target video, "12.0s" is added to the beginning or end of the visual description information corresponding to that video frame image. After adding the time node information to the visual description information of all video frame images, the visual description information corresponding to the multiple video frame images is sorted based on the time node information to obtain the text description sequence corresponding to the multiple video frame images. In the text description sequence, the multiple video frame images are arranged in chronological order. This application arranges each video frame image in chronological order in the text description sequence, so that when the content represented by the image changes, the time node of the change can be determined, thereby enabling the model to capture the start and end time nodes of each event in the video.
[0083] In an optional implementation, after obtaining the text description sequence, since the input text only represents the event to be located and does not represent the specific problem that the large language model needs to process, it is also necessary to generate a large language problem corpus corresponding to the input text. This large language problem corpus is configured to represent the specific problem text that the large language model needs to execute. Finally, the large language problem corpus is combined with the text description sequences corresponding to the multiple video frame images to form the visual description sequence. For example, if the input text is A and the target video is B, the large language problem corpus determined based on the input text could be "find the start and end times of text A from video B". It should be noted that the above example is merely an optional approach provided to help those skilled in the art better understand the solution of this application. The specific form of the large language problem corpus can be determined according to the actual situation, and this application does not impose any limitations on it.
[0084] S104. Input the visual description sequence into the large language model to perform temporal localization prediction on the visual description sequence and output the video data localization result.
[0085] In specific implementation step S104, the visual description sequence is input into the large language model. The large language model generates the answer corresponding to the large language question corpus based on the text description sequence, which serves as the video data localization result. Optionally, the large language model can be a GPT (Generative Pre-Trained Transformer) model (such as GPT3.5), a Vicuna LongChat model, etc. Specifically, firstly, the visual description sequence and the input text are input into the large language model to obtain temporal visual features. The temporal visual features are features that characterize the changes in visual description information corresponding to adjacent time nodes in the visual description sequence. Subsequently, based on the large language question corpus, temporal localization prediction is performed on the visual description sequence based on the temporal visual features. The start time node information and the end time node information are filtered from the time node information, and the start time node information and the end time node information are combined to form the video data localization result for output.
[0086] Figure 2 This illustration shows an example diagram of a video temporal localization task based on a large language model with a visual model, according to an embodiment of this application. Figure 2 Taking a visual model as an example, this application provides a detailed explanation of the proposed video temporal localization method based on a large language model. Figure 2 As shown, the target video and input text are obtained. The target video is a 14-second video of a diver interacting with marine life. The input text ( Figure 2 The search result was "a man reaching out to touch a fish". Subsequently, the target video and input text were input into a visual model (…). Figure 2 The visual description generation module divides the target video into 14 video frames according to a preset sampling duration of 1 second. Each video frame is input into the image description model within the visual model to generate first answer information corresponding to each video frame, which serves as the visual description information. For example, the visual description information for frame 1 is "a person is swimming underwater," and the visual description information for frame 14 is "a man underwater extends his hand." Subsequently, based on the target video, the time node information corresponding to each of the multiple video frames is determined, and the time node information corresponding to each video frame is added to the visual description information corresponding to each video frame to obtain a text description sequence. Figure 2 The problem (Prompt) is generated, for example, the text corresponding to the first frame in the text description sequence is "1.0s: A person is swimming underwater", and the text corresponding to the 14th frame is "14.0s: The man underwater stretches out his hand"; at the same time, a large language problem corpus is generated to "find the input text from the video ( Figure 2The start and end times of the search content are determined by combining the large language problem corpus and text description sequences into a visual description sequence, which is then input into the large language model. The large language model processes the visual description sequence, filters out the start time node information "xs" and the end time node information "ys", and combines the start and end time node information to output the video data localization result.
[0087] This application provides a video temporal localization method and product based on a visual model and a large language model. The method includes: acquiring a target video and corresponding input text; acquiring multiple video frame images based on the target video, inputting each video frame image into a visual model, acquiring visual description information corresponding to the multiple video frame images based on the visual model, wherein the visual description information is text information representing the image content of each video frame image; integrating and processing the visual description information corresponding to the multiple video frame images based on the target video and the input text to obtain a visual description sequence; inputting the visual description sequence into a large language model to perform temporal localization prediction on the visual description sequence, and outputting video data localization results. This application converts video information into visual description information of text attributes and locates the temporal content corresponding to each frame in the target video based on a visual model, thereby amplifying the finer-grained description of things in the video temporal localization task, ensuring that the large language model can obtain sufficient temporal information features from the text information, effectively improving the generalization ability of the large language model and the processing performance of the video temporal localization task.
[0088] Based on the same inventive concept, embodiments of this application disclose a video temporal localization system based on a large language model and a visual model. Figure 3 This illustration shows a schematic diagram of a video temporal localization system architecture based on a large language model and a visual model, according to an embodiment of this application. Figure 3 As shown, the system includes:
[0089] The video reading unit is used to acquire a target video and input text corresponding to the target video, and send the target video and the input text to the video description generation unit and the Prompt guidance generation unit;
[0090] The video description generation unit is used to receive the target video and the input text sent by the video reading unit, obtain multiple video frame images based on the target video, and obtain visual description information corresponding to the multiple video frame images according to the multiple video frame images and send it to the Prompt guidance generation unit. The visual description information is text information that characterizes the image content of each video frame image.
[0091] The Prompt generation unit is used to receive the visual description information sent by the video description unit, and the target video and input text sent by the video reading unit; and to integrate and process the visual description information corresponding to the multiple video frame images based on the target video and input text to obtain a visual description sequence which is then sent to the large language processing unit.
[0092] The large language processing unit is used to receive the visual description sequence sent by the Prompt guidance generation unit, perform temporal localization prediction on the visual description sequence, and output the video data localization result.
[0093] Based on the same inventive concept, embodiments of this application disclose a video temporal localization device based on a large language model and a visual model. Figure 4 A schematic diagram of a video temporal localization device based on a visual model and a large language model, according to an embodiment of this application, is shown. Figure 4 As shown, the device includes:
[0094] The acquisition module is used to acquire the target video and the input text corresponding to the target video;
[0095] The visual description module is used to acquire multiple video frame images based on the target video, input each video frame image into a visual model, and acquire visual description information corresponding to the multiple video frame images based on the visual model. The visual description information is text information that characterizes the image content of each video frame image.
[0096] An integration module is used to integrate and process the visual description information corresponding to the multiple video frame images based on the target video and the input text to obtain a visual description sequence;
[0097] The prediction module is used to input the visual description sequence into a large language model to perform temporal localization prediction on the visual description sequence and output the video data localization result.
[0098] In one optional implementation, the video description module includes:
[0099] The total duration submodule is used to obtain the total duration of the target video;
[0100] The sampling duration submodule is used to determine a preset sampling duration based on the total duration, and to perform frame sampling on the target video based on the preset sampling duration to obtain the multiple video frame images;
[0101] The first answer information submodule is used to input each video frame image into the visual model, and determine the word text sequence of each video frame image in response to the first question from the corpus of the visual model as the first answer information. The first question is used to represent a query on the image content.
[0102] The visual description submodule is used to use the first answer information as the visual description information.
[0103] In one optional implementation, the video description module further includes:
[0104] The second question submodule is used to input the input text into the visual model, so that the visual model generates a second question based on the keywords in the input text. The second question is used to characterize whether the image has experienced an event corresponding to the input text.
[0105] The second answer information submodule is used to input each video frame image into the visual model and determine the second answer information for each video frame image for the second question.
[0106] The answer combination submodule is used to combine the first answer information and the second answer information into the visual description information.
[0107] In one optional implementation, the integration module includes:
[0108] The time node submodule is used to determine the time node information corresponding to the multiple video frame images based on the target video;
[0109] The time node information addition submodule is used to add the time node information corresponding to each video frame image to the visual description information corresponding to each video frame image;
[0110] The sorting submodule is used to sort the visual description information corresponding to the multiple video frame images based on the time node information to obtain the text description sequence corresponding to the multiple video frame images.
[0111] The integration submodule is used to generate a large language problem corpus based on the input text, and to combine the large language problem corpus with the text description sequences corresponding to the multiple video frame images to form the visual description sequence.
[0112] In one optional implementation, the prediction module includes:
[0113] The temporal visual feature submodule is used to obtain temporal visual features from the visual description sequence, the input text, and the input large language model. The temporal visual features are features that characterize the changes in visual description information corresponding to adjacent time nodes in the visual description sequence.
[0114] The prediction submodule is used to perform temporal localization prediction on the visual description sequence based on the temporal visual features, using the large language problem corpus as a benchmark, and to filter out the start time node information and the end time node information from the time node information.
[0115] The output submodule is used to combine the start time node information and the end time node information into the video data positioning result for output.
[0116] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0117] Based on the same inventive concept, this application discloses an electronic device. Figure 5 A schematic diagram of an electronic device according to an embodiment of this application is shown, such as... Figure 5 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus. The memory 110 stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, the at least one program, the code set, or the instruction set can be loaded and run on the processor 120 to implement the steps in the large language model video temporal localization method based on visual model disclosed in the embodiments of this application.
[0118] Based on the same inventive concept, embodiments of this application disclose a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set thereon. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the steps in the video temporal localization method based on a visual model disclosed in embodiments of this application.
[0119] Based on the same inventive concept, embodiments of this application disclose a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the video temporal localization method based on a visual model and disclosed in embodiments of this application.
[0120] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0121] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0122] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0124] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.
[0125] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.
[0126] The foregoing has provided a detailed description of a video temporal localization method and product based on a large language model with a visual model. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A video temporal localization method based on a large language model using a visual model, characterized in that, The method includes: Obtain the target video and the corresponding input text; Based on the target video, multiple video frame images are acquired, each video frame image is input into a visual model, and visual description information corresponding to the multiple video frame images is acquired based on the visual model. The visual description information is text information that characterizes the image content of each video frame image. Based on the target video and the input text, the visual description information corresponding to the multiple video frame images is integrated and processed to obtain a visual description sequence. The visual description sequence is input into a large language model to perform temporal localization prediction on the visual description sequence and output the video data localization result. The process of acquiring multiple video frame images based on the target video, inputting each video frame image into a visual model, and acquiring visual description information corresponding to the multiple video frame images based on the visual model includes: Obtain the total duration of the target video; Based on the total duration, a preset sampling duration is determined, and the target video is sampled frame by frame using the preset sampling duration as a reference to obtain the multiple video frame images; Each video frame image is input into the visual model, and the word text sequence of each video frame image in response to the first question is determined from the corpus of the visual model as the first answer information. The first question is used to represent a query about the content of the image. The input text is input into the visual model, so that the visual model generates a second question based on the keywords in the input text. The second question is used to characterize whether the image has experienced an event corresponding to the input text. Each video frame image is input into the visual model to determine the second answer information for each video frame image in relation to the second question; The first answer information and the second answer information are combined to form the visual description information.
2. The video temporal localization method based on a large language model using a visual model according to claim 1, characterized in that, The process of integrating visual description information corresponding to the multiple video frame images based on the target video and input text to obtain a visual description sequence includes: Based on the target video, determine the time node information corresponding to each of the multiple video frame images; Add the time node information corresponding to each video frame image to the visual description information corresponding to each video frame image; Based on the time node information, the visual description information corresponding to the multiple video frame images is sorted to obtain the text description sequence corresponding to the multiple video frame images; A large language problem corpus is generated based on the input text, and the large language problem corpus is combined with the text description sequences corresponding to the multiple video frame images to form the visual description sequence.
3. The video temporal localization method based on a large language model with a visual model according to claim 2, characterized in that, The visual description sequence is input into a large language model to perform temporal localization prediction on the visual description sequence, and outputs video data localization results, including: The visual description sequence and the input text are input into the large language model to obtain temporal visual features, which are features that characterize the changes in visual description information corresponding to adjacent time nodes in the visual description sequence. Based on the large language problem corpus, the visual description sequence is temporally located and predicted based on the temporal visual features, and the start time node information and end time node information are filtered out from the time node information. The start time node information and the end time node information are combined to form the video data positioning result and then output.
4. A video temporal localization system based on a large language model using a visual model, characterized in that, The system includes: The video reading unit is used to acquire a target video and input text corresponding to the target video, and send the target video and the input text to the video description generation unit and the Prompt guidance generation unit; The video description generation unit is used to receive the target video and the input text sent by the video reading unit, obtain multiple video frame images based on the target video, and obtain visual description information corresponding to the multiple video frame images according to the multiple video frame images and send it to the Prompt guidance generation unit. The visual description information is text information that characterizes the image content of each video frame image. The Prompt generation unit is used to receive the visual description information sent by the video description unit, and the target video and input text sent by the video reading unit; and to integrate and process the visual description information corresponding to the multiple video frame images based on the target video and input text to obtain a visual description sequence which is then sent to the large language processing unit. The large language processing unit is used to receive the visual description sequence sent by the Prompt guidance generation unit, perform temporal localization prediction on the visual description sequence, and output the video data localization result. The step of acquiring multiple video frame images based on the target video, and acquiring visual description information corresponding to the multiple video frame images according to the multiple video frame images, includes: Obtain the total duration of the target video; Based on the total duration, a preset sampling duration is determined, and the target video is sampled frame by frame using the preset sampling duration as a reference to obtain the multiple video frame images; Each video frame image is input into the visual model, and the word text sequence of each video frame image in response to the first question is determined from the corpus of the visual model as the first answer information. The first question is used to represent a query about the content of the image. The input text is input into the visual model, so that the visual model generates a second question based on the keywords in the input text. The second question is used to characterize whether the image has experienced an event corresponding to the input text. Each video frame image is input into the visual model to determine the second answer information for each video frame image in relation to the second question; The first answer information and the second answer information are combined to form the visual description information.
5. A video temporal localization device based on a large language model using a visual model, characterized in that, The device includes: The acquisition module is used to acquire the target video and the input text corresponding to the target video; The visual description module is used to acquire multiple video frame images based on the target video, input each video frame image into a visual model, and acquire visual description information corresponding to the multiple video frame images based on the visual model. The visual description information is text information that characterizes the image content of each video frame image. An integration module is used to integrate and process the visual description information corresponding to the multiple video frame images based on the target video and the input text to obtain a visual description sequence; The prediction module is used to input the visual description sequence into a large language model to perform temporal localization prediction on the visual description sequence and output the video data localization result. The visual description module includes: The total duration submodule is used to obtain the total duration of the target video; The sampling duration submodule is used to determine a preset sampling duration based on the total duration, and to perform frame sampling on the target video based on the preset sampling duration to obtain the multiple video frame images; The first answer information submodule is used to input each video frame image into the visual model, and determine the word text sequence of each video frame image in response to the first question from the corpus of the visual model as the first answer information. The first question is used to represent a query on the image content. The second question submodule is used to input the input text into the visual model, so that the visual model generates a second question based on the keywords in the input text. The second question is used to characterize whether the image has experienced an event corresponding to the input text. The second answer information submodule is used to input each video frame image into the visual model and determine the second answer information for each video frame image for the second question. The answer combination submodule is used to combine the first answer information and the second answer information into the visual description information.
6. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the video temporal localization method based on a visual model according to any one of claims 1-3.
7. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps in the video temporal localization method based on a large language model of a visual model as described in any of claims 1-3.
8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps in the video temporal localization method based on a large language model based on a visual model as described in any one of claims 1-3.