Bionic long video understanding method and related device

Abstract

The invention provides a bionic long video understanding method and a related device, and the method comprises the steps: obtaining long video content data, carrying out the video preprocessing of the long video content data, obtaining a plurality of short video segments, and enabling the short video segments to at least comprise video segment sizes, video subtitles and open vocabularies; storing the plurality of short video clips in a vector database, and meanwhile, receiving a dynamic task problem sent by a user side by the vector database; and finally, according to the plurality of short video clips, calling a multi-modal large model to perform task evaluation, decomposition, planning and tool calling on the dynamic task problem so as to realize long video understanding. According to the technical scheme, the purposes of reducing the calculation amount, improving the calculation efficiency and quickly adapting to downstream tasks are achieved by observing the process of searching and understanding the long video by human beings.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a bionic long video understanding method and related devices. Background Art

[0002] With the widespread application of video data and the rapid development of streaming media platforms, the scale and complexity of long video content (such as movies, TV series, long surveillance videos, etc.) have increased significantly. Traditional video processing methods are difficult to cope with the analysis needs of these large-scale, long-duration videos. At the same time, due to the complexity and high dimensionality of spatiotemporal input, long video understanding leads to huge computing requirements and has become a very challenging topic in the field of computer vision.

[0003] In the related technologies, algorithms for understanding long videos need to find a balance between computational efficiency and performance, which can usually be divided into two categories: selective strategies and compressed sparse strategies. Among them, advanced algorithms in the direction of selective strategies, such as HierVL, propose a bottom-up hierarchical video language coding method to capture short-term and long-term video representations by performing comparative learning between short video clips and their subtitles, and then aggregating these representations to complete the long video understanding task; while VideoReCap introduces a progressive video subtitle generation method through a compressed sparse strategy: first generate subtitles at the short clip level, and then aggregate them into longer clips. However, these methods all cause significant computational and time overhead. Summary of the invention

[0004] In order to overcome the problems existing in the related technology, the present application provides a bionic long video understanding method and related devices, which observes the process of human retrieval and understanding of long videos to achieve the purpose of reducing the amount of calculation, improving computing efficiency, and quickly adapting to downstream tasks.

[0005] The first aspect of the present application provides a bionic long video understanding method, comprising:

[0006] Acquire long video content data, and perform video preprocessing on the long video content data to obtain a plurality of short video clips, wherein the short video clips at least include a video segment size, a video subtitle, and an open vocabulary;

[0007] The plurality of short video clips are stored in a vector database, and the vector database receives dynamic task questions sent by a user terminal;

[0008] Based on the multiple short video clips, a multimodal large model is called to perform task evaluation, decomposition, planning and tool calling on the dynamic task problem to achieve long video understanding.

[0009] The second aspect of the present application provides a bionic long video understanding system, comprising:

[0010] A first processing unit is used to obtain long video content data and perform video preprocessing on the long video content data to obtain a plurality of short video clips, wherein the short video clips at least include a video segment size, a video subtitle, and an open vocabulary;

[0011] A second processing unit is used to store the multiple short video clips into a vector database, and the vector database receives the dynamic task problem sent by the user terminal;

[0012] The third processing unit is used to call the multimodal large model to perform task evaluation, decomposition, planning and tool calling on the dynamic task problem based on the multiple short video clips, so as to achieve long video understanding.

[0013] A third aspect of the present application provides an electronic device, including:

[0014] Processor; and

[0015] A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the bionic long video understanding method as described above.

[0016] The fourth aspect of the present application provides a non-temporary machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor executes the bionic long video understanding method as described above.

[0017] The technical solution provided by this application may have the following beneficial effects:

[0018] The technical solution of the present application provides a bionic long video understanding method and related devices, which obtains long video content data and performs video preprocessing on the long video content data to obtain multiple short video clips, wherein the short video clips at least include the video segment size, video subtitles, and open vocabulary; then the multiple short video clips are stored in a vector database, and the vector database receives dynamic task problems sent by the user end; finally, based on the multiple short video clips, a multimodal large model is called to perform task evaluation, decomposition, planning, and tool calling on the dynamic task problems to achieve long video understanding. The technical solution of the present application achieves the purpose of reducing the amount of calculation, improving calculation efficiency, and quickly adapting to downstream tasks by observing the process of human retrieval and understanding of long videos.

[0019] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects, features and advantages of the present application will become more apparent through a more detailed description of exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0021] Figure 1 It is a flowchart of a bionic long video understanding method shown in an embodiment of the present application;

[0022] Figure 2 It is a schematic diagram of the ReAct mechanism execution process shown in an embodiment of the present application;

[0023] Figure 3 It is a specific example of the prompt template implementing the large model processing ReAct shown in the embodiment of the present application;

[0024] Figure 4 It is a structural schematic diagram of a bionic long video understanding system shown in an embodiment of the present application;

[0025] Figure 5 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The preferred embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0027] The applicant found in the research that algorithms that realize long video understanding through large models with video understanding, such as VTimeLLM, Chat-UniVi, VideoChat, VideoChatGPT, Video-llava, Longvlm and other methods, all explore how to effectively train multimodal projectors to connect visual encoders and large language models (LLM, Large Language Model) or utilize subtitle-centric information to realize long video understanding. The main methods are as follows: (1) Data alignment: Ensure that visual data (such as images or video frames) and text data (such as descriptions or labels) are aligned on the same timeline so that the model can learn the relationship between the two; (2) Shared feature space: Design a projector to map the features extracted by the visual encoder and the text embedding of the language model to the same feature space, and use contrastive learning or loss functions to optimize the similarity of the two modalities; (3) Joint training: Train the visual encoder and language model at the same time so that they promote each other in the learning process. The learning of different modalities can be balanced through a joint loss function; (4) Multi-level feature extraction: Use visual encoders to extract multi-level visual features to ensure that the projector can capture information at different semantic levels, thereby improving multimodal understanding capabilities; (5) Task-driven learning: Design multimodal training objectives for specific tasks (such as image annotation, video description, etc.), and strengthen the model's multimodal interaction capabilities through these tasks; (6) Incremental learning: Gradually introduce new modalities or data during the training process to help the model adapt to different types of information and improve its flexibility and robustness. However, these methods, due to the need to process multiple types of inputs and the need to generate dense subtitles for each video, will increase the complexity and processing burden of the processing process, and frequent calls to large language models for task decomposition, planning, and evaluation require a lot of computational costs. Therefore, these methods face high computational and economic costs in the process of understanding long videos, and these methods cannot be quickly fine-tuned and adapted in vertical scenarios.

[0028] In response to the above problems, the embodiments of the present application provide a bionic long video understanding method and related devices, which achieve the purpose of reducing the amount of calculation, improving computing efficiency, and quickly adapting to downstream tasks by observing the process of human retrieval and understanding of long videos.

[0029] It should be noted that if humans want to find a certain target in a long video or achieve a certain comprehension task, the following steps are usually required: (1) Clear goal: Before watching a video, people usually clarify their purpose, such as looking for specific information, entertainment, or learning new knowledge; (2) Quick browsing: People tend to quickly browse the content of the video first, checking information such as the cover, title, and description to determine whether the video meets their needs; (3) Select key clips: During the viewing process, people will choose to jump to key clips based on the rhythm and content of the video. Especially in long videos, they will pay attention to the title, thumbnail, and comments to identify important moments; (4) Information extraction: After watching key clips, people will select key clips based on the rhythm and content of the video. When watching a video clip, the audience will actively extract information, pay attention to details, context and plot development to understand the core content of the video; (5) Plot understanding: People will understand the overall plot of the video by associating different scenes, characters and events, which includes memory and reasoning to form a coherent narrative; (6) Background knowledge application: When understanding the content of the video, the audience will combine their own background knowledge and experience to enhance their understanding of the video information; (7) Emotional resonance: The audience may resonate with the emotions, actions and stories in the video, thereby understanding the meaning of the video at a deeper level; (8) Summary and review: After watching the video, the audience usually summarizes the content, reflects on the value and significance of the information, and even discusses or shares it with others. Therefore, this application abstracts the above process of human understanding of long videos and integrates it into a multimodal large model intelligent agent to achieve the purpose of reducing the amount of computation, improving computational efficiency, and quickly adapting to downstream tasks.

[0030] The technical solution of the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0031] Figure 1 This is a flow chart of the bionic long video understanding method shown in the embodiment of the present application. The method can be executed by the bionic long video understanding system provided in the embodiment of the present application, and can be integrated into various electronic devices such as computers and mobile phones. Note: When integrated into mobile electronic devices such as computers and mobile phones, the system can be called through the cloud REST API service.

[0032] See also Figure 1 The present application embodiment provides a bionic long video understanding method, which specifically includes the following steps:

[0033] S101: Acquire long video content data, and perform video preprocessing on the long video content data to obtain a plurality of short video clips, wherein the short video clips at least include a video segment size, video subtitles, and an open vocabulary.

[0034] In this application, video preprocessing plays a vital role in video analysis and understanding, mainly including: video segmentation, subtitle generation, and open vocabulary detection.

[0035] Specifically, the long video content data is subjected to video preprocessing to obtain a plurality of short video clips, including:

[0036] Segmenting the long video content data to obtain a plurality of short video frame segments;

[0037] Subtitle recognition and open vocabulary detection are performed on the multiple short video frame segments to obtain multiple short video segments containing subtitles and open vocabulary.

[0038] Furthermore, the long video content data is segmented to obtain a plurality of short video frame segments, including:

[0039] Extracting video frames from the long video content data at a set frequency and extracting picture features;

[0040] The dimensions of different frame images are unified through the fully connected layer to obtain video frame images of uniform dimensions;

[0041] The cosine similarity between different consecutive frames is calculated, and the consecutive frames with similarity greater than a preset threshold are clustered to obtain multiple short video frame segments.

[0042] In a specific embodiment of the present application, the video segmentation module extracts video frames at a frequency of 1FPS, extracts image features through a ResNet50 module, and passes through a fully connected layer (FC) to unify the image dimensions of different frames to 1*128, and then calculates the cosine similarity between different consecutive frames, and aggregates consecutive frames with a similarity greater than a threshold of 0.78 as a video segment for video segmentation; and ensures that the video segment size ranges from 2s to 10s.

[0043] It should be noted that the main functions of video segmentation include: (1) Content division: splitting a long video into multiple short clips for more detailed analysis and understanding. Each clip usually contains a specific scene or event, which helps to focus on the key content; (2) Improving processing efficiency: By reducing the amount of input data, segmentation can significantly reduce the consumption of computing resources, making subsequent processing and analysis more efficient; (3) Enhancing contextual understanding: segmentation helps the model capture the timing and contextual information in each clip, promoting the understanding of event development and plot structure; (4) Facilitating information retrieval: After segmentation, users or systems can more easily retrieve specific content or scenes, improving the convenience of information acquisition; (5) Supporting multimodal fusion: In video analysis, segmentation can help effectively align videos with other modalities (such as text, audio) to promote multimodal understanding.

[0044] Furthermore, the performing subtitle recognition on the multiple short video frame segments to obtain multiple short video segments containing subtitles includes:

[0045] When the voice in the long video content data is valid, using automatic voice recognition technology to perform subtitle recognition on the multiple short video frame segments to obtain multiple short video segments containing subtitles;

[0046] In the case where the voice in the long video content data is valid, a video content subtitle generation technology is used to perform subtitle recognition on the multiple short video frame segments to obtain multiple short video segments containing subtitles.

[0047] It should be noted that subtitle generation mainly includes two parts: automatic speech recognition (ASR) and video content subtitle generation, which is used to effectively convert the audio and visual information in the video into clear and coherent text, thereby improving the accessibility and comprehension of the video content.

[0048] In a specific embodiment of the present application, if the voice in the video is valid, the voice in the video is converted into text using automatic speech recognition (ASR) technology. The present invention uses the open source model Whisper to implement automatic speech recognition.

[0049] The video content subtitle generation module automatically generates corresponding text descriptions or subtitles by analyzing the visual information (such as images, characters, actions and scenes) in the video. The present invention uses VideoLlama2 to generate subtitle content.

[0050] Due to the needs of the application scenario, the language of the video segment subtitle generation of the present invention mainly supports English and Chinese. However, in essence, the present invention relies on the language support capability of the large model in terms of language support, that is, the voice direction uses Whisper to support different languages, and the image subtitle generation is achieved through VideoLlama2. Because a large amount of information in different languages ​​is used in the training data of Whisper and VideoLlama2, In-conextlearning capabilities have emerged. Therefore, it can support the expansion of multiple other languages, and the expansion method can be achieved through the In-contextlearning method.

[0051] It should be noted that the main roles of subtitle generation in video understanding include: (1) Information transmission: By converting the voice and visual information in the video into text, the subtitle generation module helps the audience quickly understand the video content, especially in scenes with a lot of dialogue or explanations; (2) Enhanced accessibility: Providing readable text for hearing-impaired people and non-native audiences, ensuring that more people can understand and enjoy the video content, and improving the accessibility of information; (3) Contextual cues: The generated subtitles not only provide the dialogue content, but also combine visual elements to enhance the understanding of the plot and character emotions, helping the audience grasp the core information of the video; (4) Search and indexing: Subtitle text can be used as an index of video content, making it easier for users to search for specific information or clips, thereby improving content retrieval efficiency; (5) Multimodal fusion: By combining visual, audio, and text information, the subtitle generation module promotes multimodal understanding, enabling the model to comprehensively analyze video content and improve overall comprehension capabilities; (6) Emotional expression: Subtitles can reflect the emotions and tone of characters, and enhance the emotional resonance and expressiveness of the video through appropriate language and punctuation; (7) Data-driven analysis: The generated subtitles can be used for subsequent data analysis and model training to help improve the performance of the video understanding system.

[0052] Furthermore, open vocabulary detection is a form of object detection, which is mainly used to identify and detect various objects in images or videos, without being limited to fixed class labels. It can flexibly adapt to emerging object categories and support open detection tasks. It is the main research direction in the field of object detection in recent years. We perform a detection inference on the first frame, middle frame, and last frame of each video after the video is segmented. The open vocabulary detection module is implemented using YOLO-World and Grounding-DINOv2 after fine-tuning with a large amount of data.

[0053] It should be noted that the main roles of open vocabulary detection in video understanding include: (1) Flexible target recognition: It can identify and detect new target categories appearing in the video without defining all possible labels in advance. This allows the system to adapt to changing environments and content; (2) Enhanced semantic understanding: By detecting a variety of objects and scenes, it helps the system understand the video content more comprehensively, thereby improving the overall semantic analysis ability; (3) Support for complex scene analysis: It can handle multiple targets in complex scenes and improve the performance of the system in crowded or dynamic environments; (4) Context association: Understand the meaning and role of the target based on contextual information, and help infer the cause and effect of events and the development of the plot; (5) Cross-modal fusion: In the case of combining text, visual and audio information, it enhances the comprehensive understanding of video content and promotes multimodal analysis.

[0054] S102: The multiple short video clips are stored in a vector database, and the vector database receives dynamic task questions sent by the user terminal.

[0055] S103: Based on the multiple short video clips, call the multimodal large model to perform task evaluation, decomposition, planning and tool calling on the dynamic task problem to achieve long video understanding.

[0056] In this application, large model processing is mainly responsible for understanding dynamic task problems, and requires task evaluation, decomposition, planning, and tool calling for the responsible problems.

[0057] In the specific embodiment of the present application, ReAct is used to implement a large model to handle the above tasks. ReAct (Reasoning and Action) is a framework. Its core idea is to guide the large language model to split complex problems through the way of thought chain, and to carry out reasoning (Reasoning) and action (Action) step by step. At the same time, it also introduces the observation (Observation) link. After each execution (Action), the current status will be observed (Observation) before the next step of reasoning (Reason). The execution process of the ReAct mechanism is as follows Figure 2 As shown in the figure, the prompt template design of the ReAct mechanism also mainly designs four key contents: thinking, reasoning, action and observation. It should be further explained that the core contents of the ReAct framework include: (1) Chain of Thought: breaking down a large and complex task into multiple thinking steps; (2) Reasoning: responsible for analyzing and processing input data to generate effective decisions; (3) Action: performing specific operations, such as retrieving and executing VAQ models, or other customized actions; (4) Observation: monitoring and collecting environmental feedback data to provide a basis for the next step of reasoning and action.

[0058] In the specific embodiment of the present application, the above ReAct specific example can be Figure 3 The prompt template in the code enables large model processing.

[0059] In this application, the tool set is mainly used to call the Action in the ReAct link of the large model module and execute related behaviors. The tool set mainly includes visual text retrieval tools, memory query tools, video question and answer tools, and video segment positioning tools.

[0060] The visual text retrieval tool is that LLM first initializes the retrieval of the 0th segment of the video preprocessing output according to the prompt template, and uses the self-reflection ability of the large model to analyze whether the caption of the segment can answer the question and plan the next action.

[0061] The memory query tool will store the relevant content generated by the agent at each step, and query whether the results of the past output can answer related questions. The information related to the video and text will be recorded in the vector database for text and video segment similarity calculation and visual text retrieval. The information related to the open vocabulary will be recorded in json format for users to quickly search. The json text includes video information, video segment start and end markers, video frame index, detection box and category, etc.

[0062] The video question-answering tool mainly uses the most advanced multimodal large model to realize the perception, reasoning and answering of user questions and videos. In order to reduce the amount of calculation and the number of cycles of the Agent process, the input of the video question-answering tool is the segmentation results and caption content of the video preprocessing module, and combined with the corresponding prompt to achieve accurate and detailed answers. It should be noted that the tool is also integrated into the Agent process, and the large model plans and evaluates when to call the tool. The video question-answering tool can be implemented using the VILA multimodal large model algorithm.

[0063] The video segment positioning tool obtains the video segment and the caption of the video segment based on the video preprocessing, and performs visual encoding and text encoding on the video segment and the caption of the video segment respectively. Then, the Agent parses the user's problem to obtain the task description or subtask description of the problem, and performs text encoding on the task description. After that, the text similarity between the task description and the video segment caption, and the similarity between the task description and the visual information description of the video segment are calculated respectively, and the video segment to be found for the user's problem is quickly located. Its function is to quickly locate several video segments that solve the problem, which are used for subsequent video content understanding and information aggregation, and to improve the accuracy and execution efficiency of video analysis and understanding. The text encoder and visual encoder of video segment positioning use the text and visual encoders of the CLIP series.

[0064] In summary, through system design and optimization of each step, the bionic multimodal long video understanding agent has better performance and efficiency, and can adapt to rapid fine-tuning of downstream tasks. The technical advantages of the system include: (1) by simulating the way humans understand long videos, the ability to retrieve, locate, perceive and understand long videos can be improved quickly and efficiently; (2) the understanding ability of multi-modal large models is improved, and the video question and answer of vertical scenes can be understood better than using only multi-modal large models such as GPT-4o or VideoLlava; (3) the possibility of hallucinations in multi-modal large models is reduced, that is, similar to videollava, the output of scene description and question and answer may be opposite, but the present invention can avoid the occurrence of hallucinations to the greatest extent; (4) the problem that most MLLMs cannot process long videos is solved. The multi-modal agent solves the problem of long video understanding through video segmentation, task decomposition, planning, memory and other methods; (5) the context length limitation of LLM is solved, the computational efficiency is low and it is difficult to expand to an hour-long video, that is, video subtitles are generated through other models, and video understanding is performed through LLM; (6) by simulating the process of human video understanding, the most relevant video segments are dynamically extracted through video segmentation retrieval, reducing the introduction of a large amount of information and noise, and further reducing tokens consumption. (The more frames the better is not the input for video understanding); (7) By modifying modules such as Tools, Prompt, multi-modal large model, and llm, it can flexibly adapt to different downstream tasks.

[0065] It should be further explained that the long video understanding solution and the intelligent agent solution for long video understanding provided in the specific embodiments of this application use many multi-modal large models, which can be replaced by other models. For example, LLMs, which are the large models of the entire system, can use paid GPT-4, or open source large models such as LlaMA / QWen.

[0066] The embodiment of the present application provides a bionic long video understanding method, which obtains long video content data and performs video preprocessing on the long video content data to obtain multiple short video clips, wherein the short video clips at least include video segment size, video subtitles, and open vocabulary; then the multiple short video clips are stored in a vector database, and the vector database receives dynamic task problems sent by the user end; finally, based on the multiple short video clips, a multimodal large model is called to perform task evaluation, decomposition, planning, and tool calls on the dynamic task problems to achieve long video understanding. The embodiment of the present application achieves the purpose of reducing the amount of calculation, improving calculation efficiency, and quickly adapting to downstream tasks by observing the process of human retrieval and understanding of long videos.

[0067] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides a bionic long video understanding system, electronic device and corresponding embodiments.

[0068] Figure 4 Schematic diagram of the structure of the bionic long video understanding system shown in the embodiment of the present application. Figure 4 , the embodiment of the present application provides a bionic long video understanding system, the system specifically comprising:

[0069] The first processing unit 401 is used to obtain long video content data and perform video preprocessing on the long video content data to obtain multiple short video clips, wherein the short video clips at least include video segment size, video subtitles, and open vocabulary;

[0070] The second processing unit 402 is used to store the multiple short video clips into a vector database, and the vector database receives the dynamic task problem sent by the user terminal;

[0071] The third processing unit 403 is used to call the multimodal large model to perform task evaluation, decomposition, planning and tool calling on the dynamic task problem based on the multiple short video clips, so as to achieve long video understanding.

[0072] Furthermore, the first processing unit 401 is specifically configured to:

[0073] Segmenting the long video content data to obtain a plurality of short video frame segments;

[0074] Subtitle recognition and open vocabulary detection are performed on the multiple short video frame segments to obtain multiple short video segments containing subtitles and open vocabulary.

[0075] Furthermore, the long video content data is segmented to obtain a plurality of short video frame segments, including:

[0076] Extracting video frames from the long video content data at a set frequency and extracting picture features;

[0077] The dimensions of different frame images are unified through the fully connected layer to obtain video frame images of uniform dimensions;

[0078] The cosine similarity between different consecutive frames is calculated, and the consecutive frames with similarity greater than a preset threshold are clustered to obtain multiple short video frame segments.

[0079] Furthermore, the performing subtitle recognition on the multiple short video frame segments to obtain multiple short video segments containing subtitles includes:

[0080] When the voice in the long video content data is valid, using automatic voice recognition technology to perform subtitle recognition on the multiple short video frame segments to obtain multiple short video segments containing subtitles;

[0081] In the case where the voice in the long video content data is valid, a video content subtitle generation technology is used to perform subtitle recognition on the multiple short video frame segments to obtain multiple short video segments containing subtitles.

[0082] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0083] Figure 5 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application.

[0084] See also Figure 5 , the electronic device 500 includes a memory 510 and a processor 520 .

[0085] The processor 520 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0086] The memory 510 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 520 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at runtime. In addition, the memory 510 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 510 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0087] The memory 510 stores executable codes, and when the executable codes are processed by the processor 520 , the processor 520 can execute part or all of the methods described above.

[0088] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0089] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing the following method steps:

[0090] Acquire long video content data, and perform video preprocessing on the long video content data to obtain a plurality of short video clips, wherein the short video clips at least include a video segment size, a video subtitle, and an open vocabulary;

[0091] The plurality of short video clips are stored in a vector database, and the vector database receives dynamic task questions sent by a user terminal;

[0092] Based on the multiple short video clips, a multimodal large model is called to perform task evaluation, decomposition, planning and tool calling on the dynamic task problem to achieve long video understanding.

[0093] Furthermore, the long video content data is subjected to video preprocessing to obtain a plurality of short video clips, including:

[0094] Segmenting the long video content data to obtain a plurality of short video frame segments;

[0095] Subtitle recognition and open vocabulary detection are performed on the multiple short video frame segments to obtain multiple short video segments containing subtitles and open vocabulary.

[0096] Furthermore, the long video content data is segmented to obtain a plurality of short video frame segments, including:

[0097] Extracting video frames from the long video content data at a set frequency and extracting picture features;

[0098] The dimensions of different frame images are unified through the fully connected layer to obtain video frame images of uniform dimensions;

[0099] The cosine similarity between different consecutive frames is calculated, and the consecutive frames with similarity greater than a preset threshold are clustered to obtain multiple short video frame segments.

[0100] Furthermore, the performing subtitle recognition on the multiple short video frame segments to obtain multiple short video segments containing subtitles includes:

[0101] When the voice in the long video content data is valid, using automatic voice recognition technology to perform subtitle recognition on the multiple short video frame segments to obtain multiple short video segments containing subtitles;

[0102] In the case where the voice in the long video content data is valid, a video content subtitle generation technology is used to perform subtitle recognition on the multiple short video frame segments to obtain multiple short video segments containing subtitles.

[0103] The scheme of the present application has been described in detail above with reference to the accompanying drawings. In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, refer to the relevant description of other embodiments. Those skilled in the art should also know that the actions and modules involved in the description are not necessarily required for the present application. In addition, it can be understood that the steps in the method of the embodiment of the present application can be adjusted in order, merged and deleted according to actual needs, and the modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0104] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0105] Alternatively, the present application can also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or electronic device, server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0106] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the application herein may be implemented as electronic hardware, computer software, or a combination of both.

[0107] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a part of a module, a program segment or a code, and the part of the module, the program segment or the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or the flow chart, and the combination of the square boxes in the block diagram and / or the flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0108] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0109] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0110] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A bionic long video understanding method, characterized in that: include: Acquire long video content data, and perform video preprocessing on the long video content data to obtain a plurality of short video clips, wherein the short video clips at least include a video segment size, a video subtitle, and an open vocabulary; The plurality of short video clips are stored in a vector database, and the vector database receives dynamic task questions sent by a user terminal; Based on the multiple short video clips, a multimodal large model is called to perform task evaluation, decomposition, planning and tool calling on the dynamic task problem to achieve long video understanding.

2. The method according to claim 1, characterized in that The video preprocessing is performed on the long video content data to obtain a plurality of short video clips, including: Segmenting the long video content data to obtain a plurality of short video frame segments; Subtitle recognition and open vocabulary detection are performed on the multiple short video frame segments to obtain multiple short video segments containing subtitles and open vocabulary.

3. The method according to claim 2, characterized in that The step of segmenting the long video content data to obtain a plurality of short video frame segments includes: Extracting video frames from the long video content data at a set frequency and extracting picture features; The dimensions of different frame images are unified through the fully connected layer to obtain video frame images of uniform dimensions; The cosine similarity between different consecutive frames is calculated, and the consecutive frames with similarity greater than a preset threshold are clustered to obtain multiple short video frame segments.

4. The method according to claim 2, characterized in that: The step of performing subtitle recognition on the plurality of short video frame segments to obtain a plurality of short video segments containing subtitles includes: When the voice in the long video content data is valid, using automatic voice recognition technology to perform subtitle recognition on the multiple short video frame segments to obtain multiple short video segments containing subtitles; In the case where the voice in the long video content data is valid, a video content subtitle generation technology is used to perform subtitle recognition on the multiple short video frame segments to obtain multiple short video segments containing subtitles.

5. A bionic long video understanding system, characterized in that: include: A first processing unit is used to obtain long video content data and perform video preprocessing on the long video content data to obtain a plurality of short video clips, wherein the short video clips at least include a video segment size, a video subtitle, and an open vocabulary; A second processing unit is used to store the multiple short video clips into a vector database, and the vector database receives the dynamic task problem sent by the user terminal; The third processing unit is used to call the multimodal large model to perform task evaluation, decomposition, planning and tool calling on the dynamic task problem based on the multiple short video clips, so as to achieve long video understanding.

6. The system according to claim 5, characterized in that The first processing unit is specifically configured to: Segmenting the long video content data to obtain a plurality of short video frame segments; Subtitle recognition and open vocabulary detection are performed on the multiple short video frame segments to obtain multiple short video segments containing subtitles and open vocabulary.

7. The system according to claim 6, characterized in that The step of segmenting the long video content data to obtain a plurality of short video frame segments includes: Extracting video frames from the long video content data at a set frequency and extracting picture features; The dimensions of different frame images are unified through the fully connected layer to obtain video frame images of uniform dimensions; The cosine similarity between different consecutive frames is calculated, and the consecutive frames with similarity greater than a preset threshold are clustered to obtain multiple short video frame segments.

8. The system according to claim 6, characterized in that The step of performing subtitle recognition on the plurality of short video frame segments to obtain a plurality of short video segments containing subtitles includes: When the voice in the long video content data is valid, using automatic voice recognition technology to perform subtitle recognition on the multiple short video frame segments to obtain multiple short video segments containing subtitles; In the case where the voice in the long video content data is valid, a video content subtitle generation technology is used to perform subtitle recognition on the multiple short video frame segments to obtain multiple short video segments containing subtitles.

9. An electronic device, characterized in that: include: processor; and a memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the bionic long video understanding method as described in any one of claims 1-4.

10. A non-transitory machine-readable storage medium, characterized in that: An executable code is stored thereon, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the bionic long video understanding method as described in any one of claims 1-4.

Images