Video understanding method and related device

Abstract

Embodiments of the present application provide a video understanding method and related equipment, and relate to the field of artificial intelligence; in the method, a first video is processed to obtain first information of the first video. Video understanding processing is performed based on first input information to obtain first understanding information of the first video. The first information is mapped with the content of the first video. The first input information includes the first information, a query on the first video, and first thinking chain prompt information. In the embodiments of the present application, the video understanding is not directly based on the first video, but is based on the first information to understand the first video, the amount of calculation required for video understanding is smaller, and the speed of video understanding can be improved. In addition, the thinking chain technology is applied to video understanding, and the performance of video understanding is effectively improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a video understanding method and related equipment. Background Technology

[0002] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0003] Because of their powerful analytical and reasoning capabilities, Large Language Models (LLMs) have been used in video understanding. Scenarios for video understanding include reviewing and monitoring long videos on video terminal platforms, and detecting abnormal events in security videos. The video understanding process involves compressing the entire video into a fixed number of tokens, which are then used to read the entire video at once and extract features. The LLM then performs video understanding processing based on the tokens, ultimately yielding the understanding result. However, as the number of tokens increases, the computational burden on the LLM increases, thus affecting video understanding performance. Summary of the Invention

[0004] This application provides a video understanding method and related equipment, which can effectively improve video understanding performance.

[0005] Firstly, a video understanding method is provided, which can be executed by a video understanding device or by a chip in the video understanding device.

[0006] The video understanding method described above includes the following steps: processing a first video to obtain first information; performing video understanding processing based on first input information to obtain first understood information about the first video; wherein the first information maps to the content of the first video; and the first input information includes the aforementioned first information, a question about the first video, and first thought chain prompts.

[0007] Optionally, the aforementioned first video can be any type of video, such as a short video created by a user on social media, a movie video, or a long video captured by a security camera.

[0008] As you can understand, the video understanding processing described above refers to the automated understanding of the content of the first video; in layman's terms, it means "understanding" the first video. The aforementioned first understanding information is the automated understanding result obtained in response to the above query.

[0009] Optionally, the first input information can be processed using a first model to obtain first understanding information about the first video. The first model can be implemented using a large language model or other AI models.

[0010] In this embodiment, after obtaining the first information of the first video, video understanding processing is performed based on the first information, queries about the first video, and first thought chain prompts to obtain the first understanding information of the first video. It is evident that this embodiment does not directly perform video understanding based on the first video, but rather understands the first video based on the first information. This reduces the computational load required for video understanding, thus improving its speed. Furthermore, applying thought chain technology to video understanding based on a large language model effectively enhances the reasoning ability of the large language model, thereby improving video understanding performance.

[0011] In one possible implementation of the first aspect, the first understanding information includes a first understanding result of the first video and a confidence level of the first understanding result. The video understanding method further includes the following steps: if the confidence level of the first understanding result is less than a confidence threshold, invoking a video processing tool to process the first video to obtain first auxiliary information; and performing video understanding processing based on the first input information and the first auxiliary information to obtain second understanding information of the first video.

[0012] If the confidence level of the first understanding result is greater than or equal to the confidence level threshold, the first understanding information is output or the first operation is performed.

[0013] Optionally, the aforementioned first auxiliary information includes the processing result obtained by processing the first video using a video processing tool and the confidence level of the processing result. The aforementioned first operation can be an operation related to responding to the first understood information, such as outputting an alarm.

[0014] In this embodiment, when the confidence level of the first understanding result is less than the confidence threshold, meaning the first understanding result does not meet the output conditions for the understanding information, auxiliary information for understanding the first video needs to be obtained to help obtain a new understanding result for the first video, such as second understanding information. Conversely, when the confidence level of the first understanding result is greater than or equal to the aforementioned confidence threshold, the first understanding information can be output as the final understanding information of the above query or the first operation can be performed. By introducing a confidence level assessment, the noise in video understanding is effectively reduced, and the accuracy of video understanding is improved.

[0015] In one possible implementation of the first aspect, the above-mentioned invoking a video processing tool to process the first video to obtain first auxiliary information specifically includes the following steps: obtaining an auxiliary information acquisition scheme based on second input information; invoking the video processing tool to process the first video based on the auxiliary information acquisition scheme to obtain the processing result and the confidence level of the processing result. The second input information includes the first information and the second thought chain prompt information.

[0016] In this embodiment, a supplementary information acquisition plan is formulated based on the thought chain prompts, which can fully unleash the global perception and planning capabilities of this method; then, based on the supplementary information acquisition plan, video processing tools are invoked to obtain supplementary information. This enables the rapid and accurate acquisition of supplementary information, thereby improving video understanding efficiency.

[0017] In one possible implementation of the first aspect, the second understanding information includes a second understanding result of the first video and a confidence level of the second understanding result. The video understanding method further includes the following steps: if the confidence level of the second understanding result is less than a confidence threshold, adjusting the auxiliary information acquisition scheme based on the first auxiliary information and the third thought chain prompt information to obtain second auxiliary information; and performing video understanding processing based on the first input information and the second auxiliary information to obtain third understanding information of the first video.

[0018] In this embodiment, when the second understanding information obtained based on the first auxiliary information and the first input information does not meet the output conditions of the understanding information (such as the confidence level being less than the confidence level threshold), the third thinking chain prompt information and the first auxiliary information are used to obtain new auxiliary information, namely the second auxiliary information. In this embodiment, the second auxiliary information is obtained based on the first auxiliary information and the third thinking chain prompt information, which can improve the accuracy of the second auxiliary information.

[0019] In one possible implementation of the first aspect, the above-described processing of the first video to obtain first information specifically includes the following steps: obtaining multiple descriptions of the first video through a video description model; and processing the multiple descriptions and fourth thought chain prompts to obtain the first information.

[0020] In this embodiment, each of the multiple descriptions of the first video can be a description of all or part of the content of the first video. By using the mind chain technique to summarize the multiple descriptions to obtain the aforementioned first information, the accuracy of the first information of the first video can be improved, thereby improving the accuracy of video understanding.

[0021] In one possible implementation of the first aspect, the video understanding method further includes the following steps: acquiring image frames related to the query based on multiple descriptions, and processing the image frames using a video processing tool to obtain third auxiliary information. Based on the first input information and the third auxiliary information, video understanding processing is performed to obtain fourth understanding information of the first video.

[0022] In this embodiment of the application, when the understanding information of the first video cannot be obtained using the first information described above, the understanding information of the first video can be obtained by using multiple descriptions of the first video, thereby improving the robustness of video understanding.

[0023] In one possible implementation of the first aspect, the video processing tool includes at least one of the following: a video frame retrieval tool, a semantic object tracking tool, or a target detection tool for a specified region. In this embodiment, auxiliary information can be obtained by invoking the video processing tool, which effectively expands the application scenarios and performance of video understanding.

[0024] Secondly, this application also provides a video understanding device, including a unit or module for performing the video understanding method described in the first aspect.

[0025] Thirdly, this application also provides a video understanding device, including a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute the video understanding method as described in the first aspect.

[0026] Fourthly, this application also provides a computer-readable storage medium storing a computer program that is executed by a processor to implement the video understanding method as described in the first aspect.

[0027] Fifthly, this application also provides a computer program product containing instructions that, when the computer program product is run on a computer, cause the computer to perform the video understanding method described in the first aspect.

[0028] Sixthly, this application also provides a chip, the chip including a processor and a data interface, the processor reading instructions stored in a memory through the data interface to execute the video understanding method described in the first aspect.

[0029] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is configured to execute the instructions stored in the memory. When the instructions are executed, the processor is configured to perform the video understanding method described in the first aspect. Attached Figure Description

[0030] The accompanying drawings used in the embodiments of this application are described below.

[0031] Figure 1A A schematic diagram of a video understanding system based on a large language model provided for an embodiment of this application;

[0032] Figure 1B A schematic diagram of a system architecture provided for an embodiment of this application;

[0033] Figure 2 A flowchart illustrating a video understanding method provided for an embodiment of this application;

[0034] Figure 3A A detailed flowchart of a video understanding method provided for an embodiment of this application;

[0035] Figure 3B A detailed flowchart for generating first information is provided for an embodiment of the application;

[0036] Figure 3C A specific flowchart for generating an understanding result is provided for an embodiment of the application;

[0037] Figure 3D A detailed flowchart of a scheme for obtaining auxiliary information is provided for an embodiment of the application;

[0038] Figure 3E A detailed flowchart of an adjustment auxiliary information acquisition scheme provided for an embodiment of the application;

[0039] Figure 4A A schematic diagram illustrating the workflow of a semantic object tracking tool provided for an embodiment of the application;

[0040] Figure 4B A schematic diagram illustrating the workflow of a target detection tool for a specified area, provided for an embodiment of the application;

[0041] Figure 5 A schematic diagram of the structure of a video understanding device provided for an embodiment of this application;

[0042] Figure 6 This is a schematic diagram of the structure of another video understanding device provided in an embodiment of this application. Detailed Implementation

[0043] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0044] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0045] In this application, "at least one" in the embodiments refers to one or more items, and "more than one" refers to two or more items. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, (a and b), (a and c), (b and c), or (a and b and c), where a, b, and c can be single or multiple. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The step numbers in the embodiments of this application (such as step S1, step S21, etc.) are only for distinguishing different steps and do not limit the order of execution between steps.

[0046] Furthermore, unless otherwise stated, the use of ordinal numbers such as "first" and "second" in the embodiments of this application is for distinguishing multiple objects and is not for limiting the order, sequence, priority, or importance of multiple objects. For example, "first device" and "second device" are only for ease of description and do not indicate that the first device and the second device are different in structure, importance, etc. In some embodiments, the first device and the second device may also be the same device.

[0047] In the above embodiments, the term "when..." can be interpreted, depending on the context, as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". The above descriptions are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the concept and principles of this application should be included within the protection scope of this application.

[0048] To facilitate understanding, the relevant terms and concepts involved in the embodiments of this application will be introduced below.

[0049] (1) Large Language Model

[0050] Large language models refer to deep learning models trained on massive amounts of text data, enabling them to generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on a wide range of topics through training on huge datasets. The core idea is to learn the patterns and structures of natural language through large-scale unsupervised training, simulating, to some extent, the human language cognition and generation process. By continuously "reading" massive amounts of text data, large language models gradually learn the rules of language, word associations, and contextual relationships, thereby enabling them to generate natural and fluent language text or understand the meaning of language text.

[0051] A multimodal large language model (MLLM) is a large language model that combines information from multiple modalities (such as text, images, audio, video, etc.).

[0052] (2) Token

[0053] A token is the smallest unit or basic element that a model uses to process and understand text; it can be called a lexical unit. It can be a word, subword, character, punctuation mark, or special marker (such as a sentence beginning / ending character). The specific form of a token depends on the tokenizer and segmentation strategy used by the model.

[0054] (3) Agent

[0055] An intelligent agent is a proxy or entity capable of autonomously achieving pre-defined goals. An intelligent agent can be software, hardware, or a comprehensive system, possessing autonomy, adaptability, and interactive capabilities. By perceiving changes in the environment (e.g., through sensors or data input), an intelligent agent makes judgments and decisions based on its learned knowledge and algorithms, and then executes actions to influence the environment or achieve predetermined goals. Intelligent agents are widely used in the field of artificial intelligence, commonly found in automated systems, robots, virtual assistants, and game characters. Their core lies in their ability to learn autonomously and continuously evolve to better complete tasks and adapt to complex environments. In summary, an intelligent agent is an intelligent entity capable of perception, decision-making, execution, learning, and interaction, possessing the ability to autonomously complete tasks, much like a human.

[0056] For example, in an LLM-based video understanding agent system, the core idea is to leverage the human-like reasoning and planning capabilities of LLM in understanding complex data and scenarios, and to enhance these capabilities through tools (or plugins) to enable interaction with the real world. This system can simulate human cognitive processes, extracting key information from videos through interactive reasoning and planning, and using this information to answer questions or perform related tasks.

[0057] (4) Mind Chain

[0058] Chain-of-thought (CoT) refers to breaking down logically complex problems and forming a complete thought process through a series of logically related considerations. In the field of artificial intelligence, CoT is a technique that allows large models to explicitly output intermediate steps during reasoning, thereby improving their reasoning capabilities. Specifically, the goal of CoT is to enable large models to explicitly output the progressively deeper reasoning steps before providing the final answer, thus enhancing their conceptual understanding and reasoning abilities. Through CoT, large models can more closely resemble human problem-solving approaches, thereby improving their intelligence level.

[0059] (5) Panoptic Segmentation

[0060] Panoptic segmentation is an advanced task in computer vision that aims to assign each pixel in an image to a semantic category (such as "road" or "vehicle") and an instance identifier (such as "first car" or "second car"). The semantic category refers to the class of objects, while the instance identifier corresponds to different numbers for objects of the same class.

[0061] Semantic segmentation refers to distinguishing different object categories in an image (such as "person" or "tree"). Instance segmentation refers to distinguishing different individuals of the same category (such as "person 1" or "person 2"). Panoramic segmentation combines semantic and instance segmentation, outputting semantic and instance information for each pixel. Applications of panoramic segmentation tools include autonomous driving (identifying roads, pedestrians, vehicles, etc., and distinguishing different vehicle or pedestrian instances to assist decision-making systems), robot navigation (building environmental maps and distinguishing between moving and static objects), medical image analysis (distinguishing different organs and lesion areas to assist diagnosis), augmented reality (accurately segmenting objects in real-world scenes to achieve seamless integration of virtual objects with the real environment), and remote sensing image analysis (distinguishing land types and building instances to support urban planning), among others.

[0062] (6) Bounding Box (Bbox)

[0063] A bounding box (Bbox) is a rectangular frame used in computer vision, image processing, and geographic information systems to represent the position of an object, specifically the area within which the object is contained. For example, the parameters of a Bbox can be represented in the following ways: Based on the coordinates of the top-left corner: consisting of the coordinates of the top-left corner (x, y), width (Width), and height (Height), denoted as (x, y, w, h). Based on the coordinates of the center point: consisting of the coordinates of the center point (cx, cy), width, and height, denoted as (cx, cy, w, h). Two-point coordinates: the coordinates of the top-left corner (x1, y1) and the bottom-right corner (x2, y2) of the bounding box, denoted as (x1, y1, x2, y2).

[0064] (7) Video Comprehension

[0065] Video understanding refers to the analysis, parsing, and reasoning of video content using deep learning or machine learning models to extract semantic information, target behaviors, scene features, etc., thereby achieving automated understanding of video content. Video understanding aims to enable computers to "understand" video content, including but not limited to:

[0066] Target detection and tracking: Identify objects and people in videos and track their movement trajectories.

[0067] Behavior recognition: Analyzing the actions and behavioral patterns of people or objects (such as running, jumping).

[0068] Scene understanding: Identify the scene in which the video takes place (such as street, indoor, natural environment).

[0069] Event detection: Determines whether a specific event (such as a traffic accident or a celebration) has occurred in the video.

[0070] Sentiment analysis: Inferring a person's emotional state through voice, facial expressions, and actions.

[0071] refer to Figure 1A , Figure 1A A schematic diagram of a video understanding system based on a large language model is provided for embodiments of this application; Figure 1A In this architecture, an additional connector is trained to transform the features of frame images in the video into tokens that the LLM can understand. These tokens, along with text prompts (such as questions), are then input into the LLM for analysis and reasoning to achieve video understanding, such as obtaining the corresponding answer to a question. For example, the connector includes an image encoder and a projection processing module. The image encoder encodes each frame image in the video frame sequence to obtain frame image features. The projection processing module then projects these frame image features to obtain video tokens. It can be understood that... Figure 1A The LLM shown is a Multimodal Large Language Model (MLLM). MLLM leverages the powerful reasoning capabilities of LLM to enhance the understanding of spatiotemporal patterns in videos.

[0072] but Figure 1A Mid-length video understanding systems face two challenges:

[0073] (1) When the number of tokens in an image frame reaches a certain number, it will bring a serious computational burden to the LLM with the Transformer model as its core architecture; the video understanding scheme based on the video multimodal large language model faces the problem of balancing computational burden and model performance.

[0074] (2) Fixed image encoders are often used in video multimodal large language models, which makes the model unable to capture information of arbitrary granularity, resulting in large performance differences when answering questions that require different granularities.

[0075] In addition, some video understanding solutions compress the entire video into a fixed number of tokens to optimize the video sequence length. This may cause the loss of key information in the compressed frame image information, resulting in poor video understanding performance.

[0076] LLM-based video understanding solutions heavily rely on the inference capabilities of LLM, exhibiting performance limitations in some complex spatiotemporal patterns. Furthermore, when calling tools, the inaccurate processing results returned by these tools can introduce new noise and illusions.

[0077] In conclusion, the performance of LLM-based video understanding solutions urgently needs improvement.

[0078] Therefore, this application provides a video understanding method that can effectively improve video understanding performance.

[0079] The video understanding method in this application embodiment can be applied to various video understanding scenarios, such as question answering for camera videos, reviewing and supervising videos on video terminal platforms, automatically analyzing and learning autonomous driving videos, detecting abnormal events in security videos (such as vehicle-mounted videos), realizing automated warehouse product management or security in unmanned places through cameras, video recommendation systems and video summaries for user interaction, and traffic flow analysis in autonomous driving and intelligent transportation scenarios.

[0080] The video understanding method described above can be executed by a video understanding device or by a chip within the video understanding device. In some implementations, the video understanding method can be implemented by an intelligent agent (or intelligent agent system), but it is not limited to this.

[0081] The following describes a system architecture provided by an embodiment of this application.

[0082] See Figure 1B , Figure 1B This is a schematic diagram of a system architecture provided by an embodiment of this application; an embodiment of this application provides a system architecture 100. As shown in the system architecture 100, the data acquisition device 160 is used to acquire training data of the first model. In this embodiment, the training data includes text data, voice data, and video data, etc. The data acquisition device 160 stores the training data in the database 130.

[0083] The training device 120 can obtain the first model 101 based on the training data maintained in the database 130. For example, in this embodiment, after model training is completed, the first model 101 can acquire the ability to perform video understanding processing based on input information. The training device 120 can be a server or cloud service device, or it can be a mobile terminal, desktop computer, tablet computer, laptop computer, or other similar device.

[0084] It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, it should be noted that the training device 120 may not necessarily train the model entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0085] The first model 101 obtained after processing by the training device 120 can be applied to a system or device with the same hardware configuration as the training device 120, such as when applied to... Figure 1B The processing device 110 shown can perform video understanding tasks and other tasks according to the embodiments of this application based on the first model 101. The processing device 110 can be a terminal, such as a mobile terminal, desktop computer, tablet computer, laptop computer, etc., or it can be a server or cloud, etc. The training device 120 and the processing device 110 can be the same device.

[0086] In the appendix Figure 1B In the process, the processing device 110 is equipped with an I / O interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. In this embodiment, the input data can be questions entered by the user, and the input data can also be obtained from the database 130.

[0087] Optionally, the preprocessing module 113 is used to preprocess the input data received by the I / O interface 112. In this embodiment, the preprocessing module 113 is used to preprocess the input data received by the I / O interface 112, and the preprocessed data enters the calculation module 111.

[0088] During the preprocessing of input data by the processing device 110, or during the calculation and other related processing processes performed by the calculation module 111 of the processing device 110, the processing device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0089] Finally, exemplarily, the I / O interface 112 returns the processing result of the processing device 110 (e.g., the model result of the first model 101) to the client device 140, thereby providing it to the user. In this case, the client device 140 can be a display. In this embodiment, the processing device 110 executes the video understanding method of this embodiment.

[0090] In the appendix Figure 1B In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the processing device 110 on the client device 140, which can be presented in various ways such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting data such as... Figure 1B The input data and output results of the input I / O interface 112 are used as new sample data and stored in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140. Figure 1B The input data and output results of the input I / O interface 112 shown are stored in the database 130 as new sample data.

[0091] It is worth noting that, attached Figure 1B This is merely a schematic diagram of a system architecture provided in an embodiment of this application. Figure 1B The positional relationships between the devices, components, modules, etc., shown do not constitute any limitation. For example, in the attached... Figure 1B In this case, the data storage system 150 is an external storage device relative to the processing device 110. In other cases, the data storage system 150 may also be placed in the processing device 110.

[0092] The video description model and other models involved in the embodiments of this application can also be adopted. Figure 1B The architecture shown is used for training. In some implementations, the processing device 110 can utilize models such as the first model 101 and the video description model to implement the video understanding method of this application embodiment, as detailed in the following description.

[0093] The video understanding method in the embodiments of this application will be described in detail below.

[0094] In this embodiment of the application, the execution subject of the video understanding method is a video understanding device, as an example. Figure 2 , Figure 2 A flowchart of a video understanding method provided for an embodiment of this application, the video understanding method including the following steps:

[0095] 201. The video understanding device processes the first video to obtain the first information.

[0096] Specifically, the aforementioned "first video" can be any type of video, including short videos created by users on social media platforms, movies, and videos captured by security cameras. The first video must have a certain duration; however, there is no specific limitation on its length or format.

[0097] It is understandable that the first piece of information maps to the content of the first video. There can be a bidirectional mapping between the first piece of information and the content of the first video.

[0098] Optionally, the first information includes at least one of the following: first text information or first voice information, etc. It is understood that the first information carries the content of the first video in a non-video data form. The data volume of the first information is less than the data volume of the first video. For example, the first text information or first voice information can carry the context information of the first video. The context information can include at least one of the following: the temporal context of the first video, the spatial context of the first video, the content context of the first video, or the interaction context of the first video. The temporal context refers to the time information in the first video. The spatial context refers to the relative positions and spatial relationships between objects or scenes in the first video. The content context refers to the metadata included in the first video (such as titles, tags, user comments, etc.). The interaction context refers to the characters, objects, environment, and their interaction relationships in the first video.

[0099] 202. The video understanding device performs video understanding processing based on the first input information to obtain the first understanding information of the first video.

[0100] Specifically, the aforementioned first input information includes first information, a query about the first video, and first thought chain prompts. The aforementioned video understanding processing refers to the automated understanding of the content of the first video; in layman's terms, it's about "understanding" the first video. The aforementioned first understanding information is the automated understanding result obtained in response to the aforementioned query.

[0101] The queries to the first video can be fixed or variable, and can be built into the system or entered by the user in real time. For example, in a scenario involving question-and-answer based on camera video, the queries to the first video can be user-inputted questions. However, in a scenario involving anomaly detection in security video, the queries to the first video can be fixed questions, such as whether there are any abnormal events in the video.

[0102] The first thought chain prompts are used to guide the large language model to generate intermediate inference steps when acquiring the understanding information of the first video, so as to improve the performance of the large language model in video understanding tasks.

[0103] In this embodiment, after obtaining the first information of the first video, the video understanding device performs video understanding processing based on the first information, queries about the first video, and first thought chain prompts to obtain the first understanding information of the first video. It is evident that this embodiment does not directly perform video understanding based on the first video, but rather understands the first video based on the first information. This reduces the computational load required for video understanding, thus improving its speed and offering advantages in video understanding scenarios (especially long video understanding scenarios). Furthermore, applying thought chain technology to video understanding effectively improves performance and can mitigate noise and illusions introduced by calling tools.

[0104] Optionally, in step 202, the video understanding device can use a first model to process the first input information to obtain first understanding information of the first video. The first model can be implemented using a large language model or other AI models. In this embodiment, a large language model is used as an example for specific explanation.

[0105] As one implementation method, step 201 above specifically includes the following steps:

[0106] 211. The video understanding device obtains multiple descriptions of the first video through the video description model.

[0107] Specifically, the video description model described above can also be a visual language model or a video captioning model. It is used to convert visual content in a video into descriptions (or annotations or tags) to facilitate subsequent processing and understanding. Optionally, the description can be textual or audio descriptions, etc.

[0108] Each of the multiple descriptions of the first video can be a description obtained for all or part of the first video. For example, when the multiple descriptions of the first video are descriptions of a portion of the first video, the first video can be divided into multiple video segments, and then one or more descriptions can be generated for each video segment. When generating one description based on a video segment, the multiple descriptions of multiple video segments can be used as the multiple descriptions of the first video. When generating multiple descriptions based on a video segment, the multiple descriptions of multiple video segments can be directly used as the multiple descriptions of the first video. In this embodiment, the uncertainty of a single description is reduced by analyzing the multiple descriptions of each segment. Alternatively, one description of a video segment can be obtained first from the multiple descriptions of the video segment, and then the multiple descriptions of multiple video segments can be used as the multiple descriptions of the first video.

[0109] 212. The video understanding device obtains the first information based on multiple descriptions and fourth-order thought chain prompts.

[0110] Specifically, the fourth thought chain prompt is used to guide the large language model to generate intermediate reasoning steps when generating the first information, so as to improve the performance of the large language model in the first information generation task.

[0111] Optionally, the video understanding device can use the first model described above to process multiple descriptions and fourth thought chain prompts to obtain the first information.

[0112] Optionally, the video understanding device may obtain the aforementioned first text information and / or first voice information using multiple text descriptions and fourth thought chain prompts.

[0113] Optionally, the video understanding device may obtain the aforementioned first text information and / or first voice information using multiple voice descriptions and fourth thought chain prompts.

[0114] Optionally, the video understanding device can obtain the aforementioned first text information and / or first voice information using multiple text descriptions, multiple voice descriptions, and fourth thought chain prompts.

[0115] In this embodiment, the video understanding device uses thought chain technology to summarize multiple descriptions to obtain the aforementioned first information, which can improve the accuracy of the first information in the generated first video, thereby improving the accuracy of video understanding.

[0116] refer to Figure 3A , Figure 3AThis document provides a flowchart of a video understanding method for an embodiment of this application. In this embodiment, the first video uses a Large Language Model (LLM) as an example. The description is text-based. The first video is segmented into segments, and the video description model generates n descriptions for each segment, where n is greater than or equal to 2. The length of each segment can be set according to actual conditions; for example, a segment length of four seconds. For example, assuming there are segments 1, 2, and 3 occurring in chronological order, when analyzing segment 2 to generate its n descriptions, segments 1 and 3 can be simultaneously input into the video description model to improve the spatiotemporal coherence between the text descriptions. A video captioning pool (Caption Bank) for the first video is obtained based on the text descriptions of all segments. A video summary of the first video is generated based on the video captioning pool and the fourth thought chain prompt information as the aforementioned first information. Here, the first information can be first text information and / or first audio information.

[0117] refer to Figure 3B , Figure 3B This application provides a flowchart for generating first information in an embodiment. In this embodiment, the first model uses a Large Language Model (LLM) as an example, taking the generation of a text description for each video segment as an example, and using the fourth thought chain prompt 302 as the prompt text "Please think step by step, consider the spatial and temporal coherence of events in each segment, and summarize the entire video based on the following content" as an example. The LLM uses the video annotation pool 301 of the first video and the fourth thought chain prompt 302 to generate a video summary of the first video as the aforementioned first information. In this embodiment, the first information is first text information. The LLM analyzes the spatiotemporal coherence of each video segment's description, considering the descriptions of adjacent segments to obtain comprehensive and reliable first information and deepen the understanding of details. Furthermore, this embodiment alleviates the problem of excessively long input in subsequent steps by generating a concise video summary. Applying thought chain technology can improve the accuracy of the first information, thereby enhancing video comprehension capabilities.

[0118] As one implementation method, in step 202, the video understanding device uses a large language model to perform video understanding processing based on the first input information of the first video, and generates understanding information with uncertainty (confidence) perception, that is, the first understanding information includes the first understanding result of the first video and the confidence of the first understanding result.

[0119] refer to Figure 3C , Figure 3C A specific flowchart for generating understanding results is provided in the embodiments of the application; in this embodiment, the first model is taken as a Large Language Model (LLM), the first information is taken as first text information, and the first text information corresponds to Figure 3CThe "contextual information" in the text. Taking the first thought chain prompt 303 as the prompt text "Please provide your answer based on all the information you have, and demonstrate your confidence in generating this answer by providing a confidence level range of 0-5", LLM is used to perform video understanding processing on the first information of the first video and the first thought chain prompt information to obtain the understanding information of the first video (the answer to the question in this embodiment) and the corresponding confidence level.

[0120] As one implementation method, refer to Figure 3A The above video understanding method also includes the following steps:

[0121] 203. If the confidence level of the first understanding result is less than the confidence level threshold, the video understanding device calls the video processing tool to process the first video to obtain the first auxiliary information; and uses a large language model to perform video understanding processing based on the first input information and the first auxiliary information to obtain the second understanding information of the first video.

[0122] Specifically, the aforementioned video processing tool is used to extract and transform video information. The first auxiliary information includes the processing result obtained by processing the first video using the video processing tool, and the confidence level of the processing result. The auxiliary information (such as the first auxiliary information and the second auxiliary information mentioned below) can take the form of text and / or voice data. Introducing a confidence level into the auxiliary information helps to assess the credibility of the auxiliary information when making decisions based on it, thereby mitigating noise and illusions introduced by the tool.

[0123] Optionally, the video understanding device uses a first model to call a video processing tool to process the first video to obtain first auxiliary information, and uses the first model to perform video understanding processing based on the first input information and the first auxiliary information to obtain second understanding information of the first video.

[0124] 204. If the confidence level of the first understanding result is greater than or equal to the confidence level threshold, the video understanding device outputs the first understanding information or performs the first operation.

[0125] Specifically, the first operation mentioned above can be an operation related to responding to the first understood information, such as outputting an alarm.

[0126] Optionally, when the confidence level of the first understanding result is greater than or equal to the confidence level threshold, the video understanding device may output the second understanding result or perform the first operation.

[0127] refer to Figure 3C The process of acquiring auxiliary information is guided by the confidence level of the understood information. The operation of the video understanding system is controlled by the confidence level of the understood information. The acquisition of auxiliary information stops only when there is understood information with a confidence level greater than or equal to the confidence threshold.

[0128] In this embodiment, when the confidence level of the first understanding result is less than the confidence threshold, i.e., the first understanding result does not meet the output conditions of the understanding information, auxiliary information for understanding the first video needs to be obtained to help obtain a new understanding result for the first video, such as second understanding information. Conversely, when the confidence level of the first understanding result is greater than or equal to the aforementioned confidence threshold, the first understanding information can be output as the final understanding information of the above query or the first operation can be performed. By introducing a confidence level assessment, noise and illusions in video understanding are effectively reduced, and the accuracy and robustness of video understanding are improved. In addition, different video processing tools can be used to obtain auxiliary information of different granularities to efficiently assist in achieving video understanding.

[0129] As one implementation method, refer to Figure 3A In step 203 above, the video understanding device calls a video processing tool to process the first video to obtain first auxiliary information, specifically including the following steps:

[0130] 231. A video understanding device obtains an auxiliary information acquisition scheme based on a second input information.

[0131] Specifically, the second input information includes the first information and the second thought chain prompt information. The second thought chain prompt information is used to guide the large language model to generate intermediate inference steps when acquiring auxiliary information acquisition schemes, thereby improving the performance of the large language model in the task of acquiring auxiliary information acquisition schemes.

[0132] Optionally, the video understanding device utilizes the first model to obtain an auxiliary information acquisition scheme based on the second input information.

[0133] refer to Figure 3D , Figure 3D A detailed flowchart of an auxiliary information acquisition scheme is provided for an embodiment of the application; in this embodiment, the first model is taken as a Large Language Model (LLM), the first information is taken as first text information, and the first text information corresponds to... Figure 3C The "contextual information" in this context. Taking the second thought chain prompt 304 as the prompt text "Please think step by step, develop a plan, and explain how to use the available tools in sequence to obtain auxiliary information to answer the question. Provide input parameters for each tool, such as frame range," the large language model obtains auxiliary information acquisition schemes based on the second input information. For example, auxiliary information acquisition scheme 305 is "By analyzing the current information and the question, we can first use tool 1 to process frames 0-10, then we can use tool 2 to process frames 5-8, and finally, we can use tool 3 to process frames 4-9."

[0134] 232. The video understanding device calls a video processing tool to process the first video based on the auxiliary information acquisition scheme, and obtains the processing result and the confidence level of the processing result.

[0135] Optionally, the video understanding device uses the first model to call a video processing tool based on the auxiliary information acquisition scheme to process the first video and obtain the processing result and the confidence level of the processing result.

[0136] In this embodiment, the video understanding device, guided by thought chain prompts, formulates an auxiliary information acquisition plan, which can fully unleash its global perception and planning capabilities. Then, based on the auxiliary information acquisition plan, it calls upon video processing tools to obtain auxiliary information. This enables the rapid and accurate acquisition of auxiliary information, thereby improving video understanding efficiency.

[0137] refer to Figure 3A In this embodiment, the video understanding device performs video understanding processing again based on the first input information and the first auxiliary information to obtain the second understanding information. See details... Figure 3C LLM regenerates the answer and its confidence level based on the first input information and the first auxiliary information, and then compares the confidence level of the newly generated answer with the confidence level threshold to determine whether the newly generated answer meets the confidence level threshold requirement.

[0138] As one implementation, the aforementioned second understanding information includes the second understanding result of the first video and the confidence level of the second understanding result. The aforementioned video understanding method further includes the following steps:

[0139] 205. If the confidence level of the second understanding result is less than the confidence level threshold, the video understanding device adjusts the auxiliary information acquisition scheme based on the third thinking chain prompt information and the first auxiliary information to obtain the second auxiliary information; and performs video understanding processing based on the first input information and the second auxiliary information to obtain the third understanding information of the first video.

[0140] Optionally, the video understanding device uses a first model to adjust the auxiliary information acquisition scheme based on the third thought chain prompt information and the first auxiliary information to obtain the second auxiliary information; and uses a large language model to perform video understanding processing based on the first input information and the second auxiliary information to obtain the third understanding information of the first video.

[0141] refer to Figure 3A When the newly generated second understanding information does not meet the confidence threshold requirement, the video understanding device adjusts its auxiliary information acquisition scheme based on the third thought chain prompts and the first auxiliary information to obtain the second auxiliary information. Then, based on the first input information and the second auxiliary information, it performs video understanding processing to obtain the third understanding information. The video understanding device analyzes and adjusts its plan according to the newly acquired auxiliary information to adaptively acquire auxiliary information of arbitrary granularity, gradually approaching the target.

[0142] refer to Figure 3E , Figure 3E This document provides a flowchart illustrating an adjustment scheme for acquiring auxiliary information, as part of an embodiment of the application. In this embodiment, the first model is a Large Language Model (LLM), and the first auxiliary information is in text form. Taking the third thought chain prompt 306 as the prompt text "Please think step by step, consider your initial plan and the acquired auxiliary information, and adjust your plan for the next step," the LLM adjusts the auxiliary information acquisition scheme based on the first auxiliary information and the third thought chain prompt. The LLM then calls a video processing tool to process the first video according to the adjusted auxiliary information acquisition scheme to obtain the second auxiliary information.

[0143] 206. If the confidence level of the second understanding result is greater than or equal to the confidence level threshold, the video understanding device outputs the above-mentioned second understanding information or performs the first operation.

[0144] Optionally, when the confidence level of the second understanding result is greater than or equal to the confidence level threshold, the video understanding device may output the second understanding result or perform the first operation.

[0145] In this embodiment of the application, when the second understanding information obtained based on the first auxiliary information and the first input information does not meet the output conditions of the understanding information, the third thinking chain prompt information and the first auxiliary information are used to obtain new auxiliary information, namely the second auxiliary information. In this embodiment of the application, the second auxiliary information is obtained based on the first auxiliary information and the third thinking chain prompt information, which can improve the accuracy of the second auxiliary information.

[0146] As one implementation method, the confidence levels of different video processing tools are designed as follows:

[0147] (1) Image Caption Tool: This tool generates corresponding annotations based on the image, and generates a confidence score for each sentence / sentence in the annotation. For example: "The image shows a person sewing fabric (confidence score = 0.9), using a needle and thread (confidence score = 0.95). There is a green cutting mat on the table (confidence score = 0.85), and a pair of scissors next to it (confidence score = 0.9)."

[0148] (2) Image panorama segmentation tool: This tool can segment several individuals from an image. Each pixel in the segmented target individual has a confidence level. The average confidence level of all pixels in the target individual can be used as the final confidence level of the target individual.

[0149] (3) Target tracking tool: Uses the confidence level of the target to be tracked in each frame as the final confidence level. For example, when the target is tracked, the confidence level of the target is used as the confidence level when the tool returns information; if the target is not tracked, the tool returns tracking failure.

[0150] (4) Image magnification and annotation tools: Same as in (1).

[0151] (5) Image magnification and panoramic segmentation tools: designed the same as in (2).

[0152] As one implementation method, the video processing tool includes at least one of the following: a video frame retrieval tool, a semantic object tracking tool, or a target detection tool for a specified region. In the embodiments of this application, the large language model can call the video processing tool to obtain auxiliary information, which can effectively expand the application scenarios and performance of video understanding.

[0153] For example, the input parameters of the semantic tracking object tool are: target name and frame range, and the return value of the tool is the Bbox information of the target in each frame within that frame range.

[0154] refer to Figure 4A , Figure 4A This is a schematic diagram illustrating the workflow of a semantic object tracking tool provided in an embodiment of the application. In this embodiment, the semantic object tracking tool can be implemented using a target tracking model (e.g., Segment AnythingModel 2, abbreviated as SAM2) and a target detection model (e.g., YOLOv11), and a target detection filtering threshold is set to ensure the correctness of the target tracking model initialization. Taking a target named "phone" and a frame range of 50-60 as an example, YOLOv11 is first used for target detection until the target's confidence level is greater than the threshold. Then, the information of the high-confidence target is used to initialize SAM2 and bidirectional tracking is performed to extract complete information. Assuming that the confidence level of the phone in frame 53 is greater than the threshold, the information of the phone in frame 53 is used to initialize SAM2 and forward and backward tracking are performed to obtain complete information about the phone.

[0155] For example, the input parameters of the specified area target detection tool are: the bounding box of the specified area and the frame range. The return value of the tool is the name and bounding box of all targets that are contained in or belong to the specified area within the frame range.

[0156] refer to Figure 4B , Figure 4BThis is a schematic diagram illustrating the workflow of a designated area target detection tool provided in an embodiment of the application. In this embodiment, the designated area target detection tool can be implemented using a target tracking model (e.g., SAM2) and a target detection model (e.g., YOLOv11), and high-confidence frames are used to initialize the target tracking model, effectively avoiding false detection / missed detection problems. Taking a designated area bounding box of [100, 123, 400, 490] and a frame range of 60 as an example, first, five frames are extended before and after frame 60, and YOLOv11 is used to perform video target detection on these ten frames. Assuming the target confidence threshold is 0.85, when the target confidence is greater than or equal to 0.85, SAM2 initialization is triggered and bidirectional tracking is performed. The bounding boxes of high-confidence targets are compared with the bounding boxes of the designated area, and targets located within the designated area are taken as the final targets returned by the tool. Figure 4B In the process, the targets that trigger SAM2 to perform bidirectional tracking are a chair, a mobile phone, a laptop, and a bed, in order to obtain complete information about these targets. Finally, these multiple targets are compared with a designated area. Since the bed is not in the designated area's bounding box, the bounding boxes of the mobile phone, chair, and laptop are ultimately returned.

[0157] By utilizing semantic object tracking tools and / or specified region target detection tools, the application scenarios and performance of video understanding methods are effectively expanded, enabling the completion of complex tasks such as occlusion-based object detection.

[0158] As one implementation method, refer to Figure 3A The large language model can refer to multiple descriptions from the first video to obtain understanding results when needed; the above video understanding method also includes the following steps:

[0159] 207. The video understanding device acquires image frames related to the query based on multiple descriptions, and calls video processing tools to process the image frames to obtain third auxiliary information.

[0160] For example, when the video understanding device is unable to generate an understanding result of the first video based on the first information (and auxiliary information) mentioned above, it can re-search multiple descriptions of the first video to find image frames related to the query, and then call a video processing tool to process the image frames to obtain the third auxiliary information.

[0161] For another example, when multiple descriptions of the first video are multiple descriptions of multiple video segments, the video segments related to the query (the video segments consist of multiple image frames) can be retrieved again based on the above query, and then the video segments can be processed using video processing tools to obtain third auxiliary information.

[0162] Optionally, the video understanding device uses a first model to obtain image frames related to the query based on multiple descriptions, and calls a video processing tool to process the image frames to obtain third auxiliary information.

[0163] 208. The video understanding device performs video understanding processing based on the first input information and the third auxiliary information to obtain the fourth understanding information of the first video.

[0164] Optionally, the video understanding device uses a first model to perform video understanding processing based on the first input information and the third auxiliary information to obtain the fourth understanding information of the first video.

[0165] refer to Figure 3C Similarly, by using a large language model, a fourth understanding information can be generated based on the first input information (including the first information and the first thought chain prompt information) and the third auxiliary information.

[0166] In this embodiment of the application, when the understanding information of the first video cannot be obtained, multiple descriptions of the first video can be used to obtain the understanding information of the first video, which can effectively improve the robustness of video understanding.

[0167] As one implementation, the video understanding device can perform video understanding processing based on a first video and first input information to obtain understood information about the first video. For example, the video understanding device can perform video understanding processing based on the first video, the first input information, and auxiliary information obtained using video processing tools to obtain understood information about the first video. Both of these methods can effectively ensure the accuracy of the understood information.

[0168] The methods of the embodiments of this application have been described in detail above. The following is a combination of... Figure 5 and Figure 6 The apparatus provided in the embodiments of this application is described in detail. It should be understood that the description of the apparatus embodiments corresponds to the description of the method embodiments. Therefore, for content not described in detail, please refer to the method embodiments above. For the sake of brevity, it will not be repeated here.

[0169] refer to Figure 5 , Figure 5 The present application provides a schematic diagram of the structure of a video understanding device, wherein... Figure 5 The video understanding device shown can be used to achieve the above. Figure 2 The video understanding method embodiments shown herein can achieve the beneficial effects of the aforementioned video understanding method embodiments. In the embodiments of this application, the video understanding device can be an electronic device or a module (such as a chip) applied in an electronic device.

[0170] like Figure 5As shown, the video understanding device 500 includes a processing module 510 and an understanding module 520. The video understanding device 500 is used to implement the functions of the above-described video understanding method embodiments. Alternatively, the video understanding device 500 may include modules for implementing any function or operation of the above-described video understanding method embodiments, and these modules may be implemented wholly or partially through software, hardware, firmware, or any combination thereof.

[0171] When the video understanding device 500 is used to implement the functions of the above-described video understanding method embodiment, the processing module 510 is used to process the first video to obtain first information of the first video. The understanding module 520 is used to perform video understanding processing based on the first input information using a large language model to obtain first understanding information of the first video. The first information is mapped to the content of the first video. The first input information includes first information, a query about the first video, and first thought chain prompts.

[0172] In this embodiment, after obtaining the first information of the first video, the video understanding device 500 performs video understanding processing based on the first information, queries about the first video, and first thought chain prompts to obtain the first understanding information of the first video. It is evident that this embodiment does not directly perform video understanding based on the first video, but rather understands the first video based on the first information. This reduces the computational load required for video understanding and can improve its speed. Furthermore, applying thought chain technology to video understanding effectively enhances its performance.

[0173] As one implementation, the aforementioned first understanding information includes the first understanding result of the first video and the confidence level of the first understanding result. The processing module 510 is further configured to, when the confidence level of the first understanding result is less than a confidence threshold, invoke a video processing tool to process the first video to obtain first auxiliary information; the understanding module 520 is further configured to perform video understanding processing based on the first input information and the first auxiliary information to obtain second understanding information of the first video. The processing module 510 is also configured to, when the confidence level of the first understanding result is greater than or equal to the confidence threshold, output the first understanding information of the first video or perform a first operation. The aforementioned first auxiliary information includes the processing result obtained by processing the first video using the video processing tool and the confidence level of the processing result. The aforementioned first operation can be a related operation in response to the first understanding information.

[0174] As one implementation, the processing module 510, in calling a video processing tool to process the first video to obtain the first auxiliary information, is specifically used for:

[0175] A scheme for acquiring auxiliary information is obtained based on the second input information. The second input information includes the first information and the second thought chain prompt information.

[0176] Based on the auxiliary information acquisition scheme, a video processing tool is called to process the first video, and the processing result and the confidence level of the processing result are obtained.

[0177] As one implementation, the aforementioned second understanding information includes the second understanding result of the first video and the confidence level of the second understanding result. The processing module 510 is further configured to adjust the auxiliary information acquisition scheme based on the first auxiliary information and the third thought chain prompt information to obtain the second auxiliary information when the confidence level of the second understanding result is less than a confidence level threshold; the understanding module 520 is further configured to perform video understanding processing based on the first input information and the second auxiliary information to obtain the third understanding information of the first video.

[0178] As one implementation, the processing module 510, in processing the first video to obtain the first information of the first video, specifically performs the following steps: obtaining multiple descriptions of the first video through a video description model; and processing the multiple descriptions and fourth thought chain prompts to obtain the first information.

[0179] In one implementation, the processing module 510 is further configured to acquire image frames related to the query based on multiple descriptions, and to call a video processing tool to process the image frames to obtain third auxiliary information. The understanding module 520 is further configured to perform video understanding processing based on the first input information and the third auxiliary information to obtain fourth understanding information of the first video.

[0180] In this embodiment of the application, when the understanding information of the first video cannot be obtained using the first information described above, the understanding information of the first video can be obtained by using multiple descriptions of the first video, thereby improving the robustness of video understanding.

[0181] As one implementation, the aforementioned video processing tool includes at least one of the following: a video frame retrieval tool, a semantic object tracking tool, or a target detection tool for a specified region. In this embodiment, video processing tools can be invoked to obtain auxiliary information, effectively expanding the application scenarios and performance of video understanding.

[0182] refer to Figure 6 , Figure 6 This is a schematic diagram of the structure of a video understanding device provided in an embodiment of this application; the video understanding device 600 includes a memory 601, a processor 602, a communication interface 604, and a bus 603. The memory 601, processor 602, and communication interface 604 are interconnected via the bus 603. It should be understood that this application does not limit the number of processors and memories in the video understanding device 600.

[0183] Optionally, the video understanding device 600 further includes a display screen (not shown), which is connected to the memory 601, processor 602, and communication interface 604 via a bus 603. The display screen is used to output information and interact with the user, such as through voice output or display output.

[0184] The memory 601 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 601 may store a program. When the program stored in the memory 601 is executed by the processor 602, the processor 602 and the communication interface 604 are used to execute the various steps of the video understanding method of any embodiment of this application. For example, the processor 602 is used to implement the functions of the processing module 510 or the understanding module 520 described above.

[0185] The processor 602 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, for executing related programs to implement the functions required by the units in the video understanding device of any embodiment of this application, or to execute the video understanding method of any embodiment of this application.

[0186] The processor 602 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the video understanding method in any embodiment of this application can be completed by the integrated logic circuitry in the hardware of the processor 602 or by instructions in software form. The processor 602 described above can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the video understanding method, steps, and logic block diagram disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the video understanding method in any embodiment of this application can be directly embodied in the execution of the hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 601. The processor 602 reads the information in the memory 601 and, in conjunction with its hardware, performs the functions required by the units included in the video understanding device of any embodiment of this application, or executes the video understanding method of any embodiment of this application.

[0187] The communication interface 604 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the video understanding device 600 and other devices or communication networks. For example, the first video can be acquired through the communication interface 604.

[0188] Bus 603 may include a pathway for transmitting information between various components of video understanding device 600 (e.g., memory 601, processor 602, communication interface 604).

[0189] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0190] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0191] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0192] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be read-only memory (ROM), random access memory (RAM), or magnetic media, such as floppy disks, hard disks, magnetic tapes, magnetic disks, or optical media, such as digital versatile discs (DVDs), or semiconductor media, such as solid state disks (SSDs).

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A video understanding method, characterized in that, The method includes the following steps: The first video is processed to obtain first information, which is mapped to the content of the first video; Based on the first input information, video understanding processing is performed to obtain the first understanding information of the first video. The first input information includes the first information, the question about the first video, and the first thought chain prompt information.

2. The method according to claim 1, characterized in that, The first understanding information includes the first understanding result of the first video and the confidence level of the first understanding result; the method further includes: If the confidence level of the first understanding result is less than the confidence level threshold, a video processing tool is invoked to process the first video to obtain first auxiliary information; and video understanding processing is performed based on the first input information and the first auxiliary information to obtain second understanding information of the first video; the first auxiliary information includes the processing result obtained by processing the first video using the video processing tool and the confidence level of the processing result; If the confidence level of the first understanding result is greater than or equal to the confidence level threshold, the first understanding information is output or the first operation is performed.

3. The method according to claim 2, characterized in that, The step of calling a video processing tool to process the first video to obtain first auxiliary information includes: A scheme for obtaining auxiliary information is based on the second input information, wherein the second input information includes the first information and the second thought chain prompt information; Based on the auxiliary information acquisition scheme, the video processing tool is invoked to process the first video, and the processing result and the confidence level of the processing result are obtained.

4. The method according to claim 3, characterized in that, The second understanding information includes the second understanding result of the first video and the confidence level of the second understanding result; the method further includes: If the confidence level of the second understanding result is less than the confidence level threshold, the auxiliary information acquisition scheme is adjusted based on the first auxiliary information and the third thought chain prompt information to obtain the second auxiliary information; and video understanding processing is performed based on the first input information and the second auxiliary information to obtain the third understanding information of the first video.

5. The method according to any one of claims 2-4, characterized in that, The video processing tool includes at least one of the following: a video frame retrieval tool, a semantic tracking object tool, or a target detection tool for a specified area.

6. The method according to any one of claims 1-5, characterized in that, The process of processing the first video to obtain the first information includes: Multiple descriptions of the first video are obtained using a video description model; The first information is obtained by processing the multiple descriptions and the fourth thought chain prompt information.

7. The method according to claim 6, characterized in that, The method further includes: Based on the multiple descriptions, obtain image frames related to the query, and call a video processing tool to process the image frames to obtain third auxiliary information; Based on the first input information and the third auxiliary information, video understanding processing is performed to obtain the fourth understanding information of the first video.

8. A video understanding device, characterized in that, The device includes a unit or module for performing the video understanding method according to any one of claims 1-7.

9. A video understanding device, characterized in that, The device includes a processor and a memory, wherein the processor and the memory are connected together, wherein the memory is used to store program code, and the processor is used to call the program code to execute the video understanding method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the video understanding method as described in any one of claims 1-7.

11. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the video understanding method as described in any one of claims 1-7.

Images