Video understanding method and device of multi-modal large model based on multi-branch visual representation

Abstract

The invention provides a video understanding method and device of a multi-modal large model based on multi-branch visual representation, relates to the technical field of vision, and aims to realize universal, generalized and accurate fine-grained video understanding. The method comprises the steps of obtaining text feature representation corresponding to text question information; performing time sequence information coding on the to-be-understood video to obtain time sequence characteristic representation; fine-grained information extraction is carried out on the to-be-understood video to obtain visual feature representation, and the visual feature representation comprises visual-text alignment feature representation and fine-grained visual feature representation; at least performing spatial compression on the visual feature representation to obtain a compressed visual feature representation; and inputting the compressed visual feature representation, the time sequence feature representation and the text feature representation into a large language model, and generating a video understanding result corresponding to the text question information.

Description

Technical Field

[0001] The present application relates to the field of visual technology, and in particular to a video understanding method and device based on a multi-modal large model of multi-branch visual representation. Background Art

[0002] Fine-grained video understanding tasks require a deep understanding of the temporal and spatial relationships of the video content, and provide effective answers to text questions in different forms. Fine-grained video understanding can be applied to many video content-related applications, products, and platforms, such as short video streaming platforms and video playback platforms, to meet the needs of operators and users for video analysis.

[0003] However, most existing video understanding methods are based on coarse-grained summary answers to video content, which have the problems of poor universality and low accuracy. Therefore, how to achieve universal, generalized and accurate fine-grained video understanding is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of the above problems, the embodiments of the present application provide a video understanding method and device based on a multi-modal large model of multi-branch visual representation, so as to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect of an embodiment of the present application, a video understanding method of a multimodal large model based on multi-branch visual representation is disclosed, the method comprising: Obtaining text feature representation corresponding to text question information; Encode the temporal information of the video to be understood to obtain a temporal feature representation; Extracting fine-grained information from the video to be understood to obtain a visual feature representation, wherein the visual feature representation includes: a visual-text alignment feature representation and a fine-grained visual feature representation; performing at least spatial compression on the visual feature representation to obtain a compressed visual feature representation; The compressed visual feature representation, the temporal feature representation and the text feature representation are input into a large language model to generate a video understanding result corresponding to the text question information.

[0006] Optionally, the time sequence information of the video to be understood is encoded to obtain a time sequence feature representation, including: The timestamp text information of each video frame of the video to be understood is encoded to obtain a temporal feature representation; wherein the timestamp text information is the video frame of the i-th second, i is an integer between 1 and N, and N is the total number of video frames of the video to be understood.

[0007] Optionally, fine-grained information extraction is performed on the video to be understood to obtain visual feature representation, including: Inputting each video frame of the video to be understood into a first visual encoder for feature extraction to obtain a visual-text alignment feature representation corresponding to each video frame; the first visual encoder is trained to generate description information corresponding to an image; Each video frame of the video to be understood is input into a second visual encoder for feature extraction to obtain a fine-grained visual feature representation corresponding to each video frame; the second visual encoder is trained to perceive the detail differences between different images.

[0008] Optionally, performing at least spatial compression on the visual feature representation to obtain a compressed visual feature representation includes: Performing spatial compression on the visual feature representation to obtain a low-dimensional visual feature representation; Determining whether the total amount of the low-dimensional visual feature representation exceeds an upper limit value of the representation processing of the large language model; If the number does not exceed the value, the low-dimensional visual feature representation is used as the compressed visual feature representation; In case of exceeding, the low-dimensional visual feature representation is time-compressed according to a time compression multiple to obtain a compressed visual feature representation; the time compression multiple is determined based on the total amount of the low-dimensional visual feature representation and the upper limit of the representation processing of the large language model.

[0009] Optionally, in the case of exceeding, time compression is performed on the low-dimensional visual feature representation according to the time compression multiple to obtain a compressed visual feature representation, including: According to the sequence of each video frame in the video to be understood, sequentially selecting a plurality of the low-dimensional visual feature representations corresponding to the video frames of the time compression multiple; Compressing the plurality of low-dimensional visual feature representations into a target visual feature representation, wherein the time proportion and format of the target visual feature representation are the same as the time proportion and format of the visual representation of a video frame; According to all target visual feature representations, a compressed visual feature representation is obtained.

[0010] Optionally, the method further comprises: Arrange and reorganize the compressed visual feature representation and the temporal feature representation corresponding to each video frame to obtain a feature representation queue corresponding to each video frame; Inputting the compressed visual feature representation, the temporal feature representation, and the text feature representation into a large language model to generate a video understanding result corresponding to the text question information, including: The feature representation queues corresponding to all video frames and the text feature representations are input into the large language model to generate a video understanding result corresponding to the text question information.

[0011] In a second aspect of the embodiments of the present application, a video understanding device based on a multi-modal large model of multi-branch visual representation is disclosed, the device comprising: A text information representation module is used to obtain text feature representations corresponding to text question information; The temporal information representation module is used to encode the temporal information of the video to be understood and obtain the temporal feature representation. A visual information representation module is used to extract fine-grained information from the video to be understood to obtain a visual feature representation, wherein the visual feature representation includes: a visual-text alignment feature representation and a fine-grained visual feature representation; A sliding window compression module, configured to perform at least spatial compression on the visual feature representation to obtain a compressed visual feature representation; The result generation module is used to input the compressed visual feature representation, the temporal feature representation and the text feature representation into a large language model to generate a video understanding result corresponding to the text question information.

[0012] According to a third aspect of an embodiment of the present application, an electronic device is disclosed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the video understanding method of a multimodal large model based on multi-branch visual representation described in the first aspect of the embodiment of the present application are implemented.

[0013] The fourth aspect of the embodiments of the present application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the video understanding method of the multimodal large model based on multi-branch visual representation described in the first aspect of the embodiments of the present application are implemented.

[0014] The fifth aspect of the embodiments of the present application discloses a computer program product, including a computer program, which, when executed by a processor, implements the steps of the video understanding method of the multimodal large model based on multi-branch visual representation described in the first aspect of the embodiments of the present application.

[0015] The embodiments of the present application include the following advantages: In an embodiment of the present application, a text feature representation corresponding to the text question information is obtained; the time sequence information of the video to be understood is encoded to obtain a time sequence feature representation; fine-grained information is extracted from the video to be understood to obtain a visual feature representation, and the visual feature representation includes: a visual-text alignment feature representation and a fine-grained visual feature representation; the visual feature representation is at least spatially compressed to obtain a compressed visual feature representation; the compressed visual feature representation, the time sequence feature representation and the text feature representation are input into the large language model to generate a video understanding result corresponding to the text question information. In this way, the understanding of the fine-grained video task is strengthened from three perspectives. By encoding the time sequence information of the video to be understood, the time perception ability of the large language model for continuous visual information is strengthened, so as to more effectively distinguish the time sequence changes of the content; by extracting fine-grained information from the video to be understood, the visual detail perception ability of the large language model is strengthened, and the visual difference between pictures is more effectively perceived; by at least spatially compressing the visual feature representation, the long time sequence content information processing ability of the large language model is strengthened. In this way, the fine-grained video understanding task can be completed with higher accuracy and universality. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] Figure 1 It is a flowchart of the steps of a video understanding method based on a multi-modal large model of multi-branch visual representation provided by an embodiment of the present application; Figure 2 It is a flowchart of the steps of another video understanding method based on a multi-modal large model of multi-branch visual representation provided by an embodiment of the present application; Figure 3 is a schematic diagram of a multimodal large model based on multi-branch visual representation provided in an embodiment of the present application; Figure 4 It is a structural schematic diagram of a video understanding device based on a multi-modal large model of multi-branch visual representation provided in an embodiment of the present application; Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] Existing video understanding methods mainly include: 1) Methods that use deep learning models to focus on a single video understanding task: This method focuses on a single video understanding task by constructing a deep learning network to fit the prediction result error of a specific video understanding task (for example, video temporal positioning). This method cannot be directly generalized to other fine-grained video understanding tasks, thus limiting the practical application scenarios of this type of method.

[0020] 2) Method of using multimodal large models to generalize and complete video understanding tasks: This method combines the text-visual alignment representation method with a large language model to construct a multimodal large model, so that the model can give answers to input video content and multiple text questions, and solve general video understanding tasks. However, this method still remains at the stage of coarse-grained generalization and answering of video content. The visual representation method used has poor ability to distinguish changes in content between video frames, and it is difficult to handle more fine-grained analysis tasks in video content understanding, which has certain limitations. Specifically, since its visual representation method is trained based on the visual-text alignment task, the multimodal large model has poor overall performance in many visually distinguishable image features (for example, direction, viewing angle, color appearance, etc.), and cannot accurately complete downstream tasks such as visual question answering and judgment. In addition, visual understanding tasks usually contain multiple frames of continuous image content, and the temporal perception ability of video frames is also limited by the lack of visual representation modules, resulting in the inability of multimodal large models to understand the changing video content in a fine-grained manner. In addition, compared with large multimodal models that focus on understanding image content, large multimodal models that focus on fine-grained video understanding tasks often need to process hundreds of video frames at a time, which puts higher requirements on compressing visual information as losslessly as possible and completing the understanding.

[0021] In order to overcome the limitations of related technologies, an embodiment of the present application provides a video understanding method based on a multimodal large model with multi-branch visual representation. A multimodal large model is used to generalize different video understanding tasks, and the performance of the multimodal large model for video understanding tasks is enhanced from three perspectives, namely, the visual detail perception ability of the large language model is enhanced from the fine-grained visual representation branch, the temporal information representation branch enhances the temporal perception ability of the large language model for continuous visual information, and the sliding window compression mechanism enhances the long-term content information processing ability of the large language model. In this way, fine-grained video understanding tasks can be completed with higher accuracy and in a universal manner.

[0022] The following is a detailed description of the video understanding method of the multimodal large model based on multi-branch visual representation in the embodiment of the present application in conjunction with the accompanying drawings.

[0023] Reference Figure 1 As shown, Figure 1 1 is a flowchart of the steps of a video understanding method based on a multi-modal large model of multi-branch visual representation provided by an embodiment of the present application. Figure 1 As shown, the video understanding method based on the multi-modal large model of multi-branch visual representation may include steps S110 to S150: Step S110: Obtain text feature representation corresponding to the text question information.

[0024] In the embodiment of the present application, the text question information refers to the text input information of the multimodal large model, that is, the text question information refers to the question description information for the video content. For example, the text question information can be "locating the start and end time of event A in the video content", "summarizing the video content from the i-th second to the j-th second", and other question description information for the video content.

[0025] The text feature representation refers to a text information token. In some implementations, obtaining the text feature representation corresponding to the text question information includes: encoding the text question information into the text feature representation through a text encoder; wherein the text encoder is an encoder inside the multimodal large model.

[0026] Step S120: Encode the time sequence information of the video to be understood to obtain a time sequence feature representation.

[0027] The video to be understood refers to the visual information input into the multimodal large model for video understanding; the temporal feature representation refers to the temporal feature token, which is used to represent the temporal information of each video frame.

[0028] In the embodiments of the present application, considering that the existing multimodal large model rarely models timing information, but guides the multimodal large model to judge the time corresponding to each frame of visual feature representation (i.e., visual feature token) by informing the multimodal large model of the video duration when processing timing-related video tasks, this type of method lengthens the reasoning steps of the large model for the video content understanding task, and from the evaluation results, it is also difficult for the multimodal large model to associate important video content with its specific timestamp. Therefore, in order to solve this problem, the embodiments of the present application introduce additional encoding of timing information to establish a strong binding with the visual feature representation of the corresponding timestamp (i.e., encoding the timing information for the video to be understood), so as to jointly optimize the large model's ability to understand and judge the visual feature representation and the timing feature representation, so that the multimodal large model can more effectively distinguish the timing changes of the content in the video modality.

[0029] Step S130: extracting fine-grained information from the video to be understood to obtain a visual feature representation, wherein the visual feature representation includes a visual-text alignment feature representation and a fine-grained visual feature representation.

[0030] Among them, the visual feature representation refers to the visual feature token, and each video frame in the video to be understood corresponds to a visual feature representation; among them, the visual-text alignment feature representation contains the content description information corresponding to a video frame; the fine-grained visual feature representation contains the fine-grained information of the video frame, especially the difference information between the video frame and other video frames.

[0031] In the embodiments of the present application, considering that the existing multimodal large models mostly use a single visual-text aligned visual representation model, this helps the large model to understand the input multimodal content such as images and videos and output text for summarization, but lacks fine-grained analysis of the input visual information, resulting in the inability to effectively distinguish the input visual content with obvious visual differences but similar semantic information. Therefore, in order to solve this problem, the embodiments of the present application perform fine-grained information extraction on the video to be understood, and perform more comprehensive input modeling on the video information, so that the visual representation link can extract the visual information of the input content in a more fine-grained manner, which can more effectively support the large language model's perception of the visual differences between pictures.

[0032] Step S140: performing at least spatial compression on the visual feature representation to obtain a compressed visual feature representation.

[0033] In the embodiment of the present application, the compressed visual feature representation is a visual feature representation of a lower dimension compared to the visual feature representation. Considering that unlike the multimodal large model that focuses on the image modality, the complete image can be selected to directly align with the input modality of the large model for content understanding, the large model of video understanding needs to process hundreds of video frame image content at a time, and therefore puts forward higher requirements for compressing and understanding the visual information as losslessly as possible. Therefore, the visual feature representation obtained in step S130 is at least spatially compressed to compress the visual feature representation into a low-dimensional visual representation feature (i.e., a compressed visual feature representation), thereby achieving compression of input video content of non-uniform length into information content of constant length or variable length within a certain range to meet the upper limit of the large language model processing token information at a time.

[0034] Step S150: inputting the compressed visual feature representation, the temporal feature representation and the text feature representation into a large language model to generate a video understanding result corresponding to the text question information.

[0035] In the embodiment of the present application, the large language model generates a video understanding result corresponding to the text question information based on the input compressed visual feature representation, temporal feature representation and text feature representation. Among them, the input form of the video understanding result is flexibly determined according to the training data set and task requirements; the video understanding result mainly depends on the text question information; for example, if the text question information is "locate the start and end time of event A in the video content", the large language model generates a temporal positioning answer for the video content; if the text question information is "summarize the video content from the i-th second to the j-th second", the large language model generates a summary of the video content for the corresponding time period.

[0036] Through the above process, the understanding of fine-grained video tasks is strengthened from three perspectives. By encoding the temporal information of the video to be understood, the large language model's temporal perception of continuous visual information is strengthened, so that the temporal changes of the content can be more effectively distinguished; by extracting fine-grained information from the video to be understood, the large language model's visual detail perception ability is strengthened, and the visual differences between pictures are more effectively perceived; by at least spatially compressing the visual feature representation, the large language model's ability to process long-term content information is strengthened. In this way, fine-grained video understanding tasks can be completed with higher accuracy and in a universal manner.

[0037] In combination with the above embodiments, in one implementation, the present application also provides a video understanding method based on a multi-modal large model of multi-branch visual representation. In this method, the "encoding the time sequence information of the video to be understood to obtain the time sequence feature representation" in the above step S120 may specifically include: The timestamp text information of each video frame of the video to be understood is encoded to obtain a temporal feature representation; wherein the timestamp text information is the video frame of the i-th second, i is an integer between 1 and N, and N is the total number of video frames of the video to be understood.

[0038] In the embodiment of the present application, the timestamp text information of "the video frame of the ith second" is encoded into a temporal feature representation by a text encoder. For example, if the duration of the video to be understood is 10 seconds, the timestamp text information of "the video frame of the 1st second", the timestamp text information of "the video frame of the 2nd second", ..., the timestamp text information of "the video frame of the 9th second", and the timestamp text information of "the video frame of the 10th second" are encoded respectively to obtain the corresponding temporal feature representation.

[0039] In this way, the relationship between the visual feature representation and the timestamp can be established based on the temporal feature representation, so that the understanding and judgment ability of the large model in the visual feature representation and the temporal feature representation can be jointly optimized, so that the large model can more effectively distinguish the temporal changes of the content in the video modality.

[0040] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a video understanding method based on a multi-modal large model of multi-branch visual representation. In this method, the "extracting fine-grained information from the video to be understood to obtain a visual feature representation" in the above step S130 may specifically include sub-steps S130-1 to S130-2: Step S130-1: Input each video frame of the video to be understood into a first visual encoder for feature extraction to obtain a visual-text alignment feature representation corresponding to each video frame; the first visual encoder is trained to generate description information corresponding to the image.

[0041] Step S130-2: Input each video frame of the video to be understood into a second visual encoder for feature extraction to obtain a fine-grained visual feature representation corresponding to each video frame; the second visual encoder is trained to perceive detail differences between different images.

[0042] In an embodiment of the present application, the first visual encoder and the second visual encoder are both end-to-end visual detection models. The first visual encoder takes the generation of descriptive information corresponding to an image as a training goal. During the training process, a picture is matched with the corresponding descriptive information, so that the trained first visual encoder can convert the video frame into the features of the corresponding description, that is, the visual-text alignment feature representation. In some embodiments, the first visual encoder is a CLIP encoder.

[0043] The second visual encoder is trained to perceive the detail differences between different images. During the training process, the second visual encoder is trained to distinguish the features of the visual differences between different images, so that the trained second visual encoder can obtain fine-grained visual feature representations of video frames. In some embodiments, the second visual encoder is a DINO encoder.

[0044] Thus, each video frame of the video to be understood is input into the first visual encoder and the second visual encoder respectively to obtain the visual-text alignment feature representation and fine-grained visual feature representation corresponding to each video frame. In this way, a more comprehensive input modeling of the video information in the video to be understood is achieved, so that the visual representation link can extract the visual information of the video to be understood in a more fine-grained manner, and can more effectively support the perception of the visual differences between images by the large language model.

[0045] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a video understanding method based on a multi-modal large model of multi-branch visual representation. In this method, the "at least spatially compressing the visual feature representation to obtain a compressed visual feature representation" in the above step S140 may specifically include sub-steps S140-1 to S140-4: Step S140 - 1 : spatially compressing the visual feature representation to obtain a low-dimensional visual feature representation.

[0046] Step S140 - 2 : Determine whether the total amount of the low-dimensional visual feature representation exceeds the representation processing upper limit of the large language model.

[0047] Step S140 - 3 : If it does not exceed, use the low-dimensional visual feature representation as the compressed visual feature representation.

[0048] Step S140-4: In case of exceeding, the low-dimensional visual feature representation is time-compressed according to the time compression multiple to obtain a compressed visual feature representation; the time compression multiple is determined based on the total amount of the low-dimensional visual feature representation and the upper limit of the representation processing of the large language model.

[0049] In an embodiment of the present application, in order to solve the situation where the total amount of visual feature representation (i.e., the total number of tokens) caused by the number of video frames exceeds the upper limit of the representation processing of the large language model, a sliding window compression mechanism is adopted to compress the input video content of non-uniform length into information content of a constant length or a variable length within a certain range to meet the upper limit of the large language model's single processing of token information.

[0050] Specifically, the sliding window compression mechanism is used to determine whether the low-dimensional visual feature representation obtained through spatial compression exceeds the upper limit of the representation processing of the large language model; if it does not exceed, step S140-3 is executed; if it exceeds, step S140-4 is executed, and the low-dimensional visual feature representation is temporally compressed according to the time compression multiple. In some embodiments, the time compression multiple is determined based on the total amount of the low-dimensional visual feature representation and the upper limit of the representation processing of the large language model, which means: the minimum integer value determined based on the ratio of the total amount of the low-dimensional visual feature representation to the upper limit of the representation processing of the large language model; for example, if the total amount of the low-dimensional visual feature representation is 100 and the upper limit of the representation processing of the large language model is 30, then the time compression multiple can be determined as 4.

[0051] Specifically, in the case of exceeding, the low-dimensional visual feature representation is time-compressed according to the time compression multiple to obtain a compressed visual feature representation, including: according to the order of each video frame in the video to be understood, sequentially selecting multiple low-dimensional visual feature representations corresponding to the video frames with the time compression multiple; compressing the multiple low-dimensional visual feature representations into a target visual feature representation, the time proportion and format of the target visual feature representation are the same as the time proportion and format of the visual representation of a video frame; based on all the target visual feature representations, a compressed visual feature representation is obtained.

[0052] For example, if the time compression factor is 4, multiple low-dimensional visual feature representations corresponding to 4 video frames are selected in turn from the video to be understood; the 4 low-dimensional visual feature representations are compressed into a target visual feature representation; multiple target visual feature representations are obtained through multiple compressions, and finally all the target visual feature representations are used as compressed visual feature representations.

[0053] In some embodiments, compressing the plurality of low-dimensional visual feature representations into a target visual feature representation comprises: compressing the plurality of low-dimensional visual feature representations into a target visual feature representation by an average pooling compression method.

[0054] In this way, a sliding window compressor is used to spatially compress the visual feature representation. When the total amount of the low-dimensional visual feature representation obtained after spatial compression exceeds the upper limit of the representation processing of the large language model, the low-dimensional visual feature representation is temporally compressed to compress the visual feature representation into token information that can be accepted and processed by the large language model.

[0055] In combination with the above embodiments, in one implementation, the present application embodiment further provides a video understanding method based on a multi-modal large model of multi-branch visual representation. In this method, in addition to the above steps, the following steps are also included: The compressed visual feature representation and the temporal feature representation corresponding to each video frame are arranged and reorganized to obtain a feature representation queue corresponding to each video frame.

[0056] In an embodiment of the present application, the feature representation queue is a token queue. After obtaining the compressed visual feature representation, the compressed visual feature representation and the timing feature representation corresponding to each video frame are arranged and reorganized, that is, the timing feature representation, and the compressed visual-text alignment feature representation and the fine-grained visual feature representation corresponding to the description time are combined. For example, the feature representation queue corresponding to the i-th second video frame is: "the i-th second video frame" - "the visual-text alignment feature representation of the i-th second video frame" - "the fine-grained visual feature representation of the i-th second video frame".

[0057] Furthermore, in step S150, "inputting the compressed visual feature representation, the temporal feature representation and the text feature representation into a large language model to generate a video understanding result corresponding to the text question information" includes: inputting the feature representation queue corresponding to all video frames and the text feature representation into the large language model to generate a video understanding result corresponding to the text question information.

[0058] In the embodiment of the present application, the large language model generates a video understanding result corresponding to the text question information based on the input feature representation queue and text feature representation. Since each feature representation queue contains the timing information (timing feature guarantee) and visual information (visual feature representation) of the video frame, the large language model can better understand the video content and generate a video understanding result that is more consistent with the text question information.

[0059] It should be noted that the video understanding method of the multimodal large model based on multi-branch visual representation provided in the embodiment of the present application is also applicable to the understanding of the content of a single image. When understanding a single image, there is no need to encode the temporal information. The image to be understood is input into the first visual encoder (such as CLIP encoder) for feature extraction to obtain the visual-text alignment feature representation corresponding to the image to be understood; the image to be understood is input into the second visual encoder (DINO encoder) for feature extraction to obtain the fine-grained visual feature representation corresponding to the image to be understood. Afterwards, the visual-text alignment feature representation and the fine-grained visual feature representation are spatially compressed to obtain a compressed visual feature representation; the compressed visual feature representation and the text feature representation are input into the large language model to obtain the image understanding result.

[0060] Reference Figure 2 As shown, Figure 2 : is a flowchart of another method for video understanding based on a multi-modal large model of multi-branch visual representation provided by an embodiment of the present application, the method comprising steps S201 to S209: Step S201: Obtain text feature representation corresponding to text question information.

[0061] Step S202: Encode the timestamp text information of each video frame of the video to be understood to obtain a temporal feature representation; wherein the timestamp text information is the video frame of the i-th second, i is an integer between 1 and N, and N is the total number of video frames of the video to be understood.

[0062] Step S203: Input each video frame of the video to be understood into a first visual encoder for feature extraction to obtain a visual-text alignment feature representation corresponding to each video frame; the first visual encoder is trained to generate description information corresponding to the image.

[0063] Step S204: Input each video frame of the video to be understood into a second visual encoder for feature extraction to obtain a fine-grained visual feature representation corresponding to each video frame; the second visual encoder is trained to perceive detail differences between different images.

[0064] Step S205: spatially compressing the visual feature representation to obtain a low-dimensional visual feature representation, wherein the visual feature representation includes: a visual-text alignment feature representation and a fine-grained visual feature representation.

[0065] Step S206: Determine whether the total amount of the low-dimensional visual feature representation exceeds the representation processing upper limit of the large language model.

[0066] Step S207: If it does not exceed, the low-dimensional visual feature representation is used as the compressed visual feature representation; if it exceeds, the low-dimensional visual feature representation is time-compressed according to the time compression multiple to obtain a compressed visual feature representation; the time compression multiple is determined based on the total amount of the low-dimensional visual feature representation and the upper limit of the representation processing of the large language model.

[0067] Step S208: Arrange and reorganize the compressed visual feature representation and the temporal feature representation corresponding to each video frame to obtain a feature representation queue corresponding to each video frame.

[0068] Step S209: input the feature representation queues corresponding to all video frames and the text feature representations into the large language model to generate a video understanding result corresponding to the text question information.

[0069] In the embodiment of the present application, by encoding the temporal information of the video to be understood, the temporal perception ability of the large language model for continuous visual information is enhanced, so that the temporal changes of the content can be more effectively distinguished; by extracting fine-grained information from the video to be understood, the visual detail perception ability of the large language model is enhanced, and the visual differences between pictures are more effectively perceived; by at least spatially compressing the visual feature representation, the long-term content information processing ability of the large language model is enhanced. In this way, the fine-grained video understanding task can be completed with higher accuracy and universality.

[0070] For example, Figure 3 : is a schematic diagram of a multimodal large model based on multi-branch visual representation provided in an embodiment of the present application. The video understanding method of the multimodal large model based on multi-branch visual representation in the embodiment of the present application is realized by the multimodal large model based on multi-branch visual representation. Specifically, the multimodal large model based on multi-branch visual representation includes: a text information representation branch, a fine-grained visual representation branch, a temporal information representation branch, a sliding window compression mechanism and a large language model.

[0071] When understanding the video content, the text information representation branch encodes the text question information through a text encoder to obtain a text feature representation; the temporal information representation branch encodes the timestamp text information of each video frame of the video to be understood through a text encoder to obtain a temporal feature representation; and, each video frame of the video to be understood is input into the first visual encoder for feature extraction to obtain a visual-text alignment feature representation corresponding to each video frame; each video frame of the video to be understood is input into the second visual encoder for feature extraction to obtain a fine-grained visual feature representation corresponding to each video frame.

[0072] Next, the visual feature representation (visual-text alignment feature representation and fine-grained visual feature representation) is spatially compressed through a sliding window compression mechanism to obtain a low-dimensional visual feature representation; it is determined whether the total amount of the low-dimensional visual feature representation exceeds the upper limit of the representation processing of the large language model; if not, the low-dimensional visual feature representation is used as the compressed visual feature representation; if exceeded, the low-dimensional visual feature representation is temporally compressed according to the time compression multiple to obtain a compressed visual feature representation.

[0073] Finally, the sliding window compression mechanism arranges and reorganizes the compressed visual feature representation and temporal feature representation corresponding to each video frame to obtain the feature representation queue corresponding to each video frame. Then, the feature representation queues and text feature representations corresponding to all video frames are input into the large language model to generate video understanding results corresponding to the text question information.

[0074] The present application also provides a video understanding device based on a multi-modal large model of multi-branch visual representation, referring to Figure 4 As shown, Figure 4 : is a schematic diagram of the structure of a video understanding device based on a multi-modal large model of multi-branch visual representation provided in an embodiment of the present application, the device comprising: A text information representation module 410 is used to obtain text feature representations corresponding to text question information; A time sequence information representation module 420 is used to encode the time sequence information of the video to be understood to obtain a time sequence feature representation; A visual information representation module 430 is used to extract fine-grained information from the video to be understood to obtain a visual feature representation, wherein the visual feature representation includes: a visual-text alignment feature representation and a fine-grained visual feature representation; A sliding window compression module 440, configured to perform at least spatial compression on the visual feature representation to obtain a compressed visual feature representation; The result generation module 450 is used to input the compressed visual feature representation, the temporal feature representation and the text feature representation into a large language model to generate a video understanding result corresponding to the text question information.

[0075] In an optional embodiment, the timing information representation module is also used to encode the timestamp text information of each video frame of the video to be understood to obtain a timing feature representation; wherein the timestamp text information is the video frame of the i-th second, i is an integer between 1 and N, and N is the total number of video frames of the video to be understood.

[0076] In an optional embodiment, the visual information representation module includes: A first feature extraction module is used to input each video frame of the video to be understood into a first visual encoder for feature extraction to obtain a visual-text alignment feature representation corresponding to each video frame; the first visual encoder is trained to generate description information corresponding to an image; The second feature extraction module is used to input each video frame of the video to be understood into the second visual encoder for feature extraction to obtain a fine-grained visual feature representation corresponding to each video frame; the second visual encoder is trained to perceive the detail differences between different images.

[0077] In an optional embodiment, the sliding window compression module includes: A spatial compression module, used for spatially compressing the visual feature representation to obtain a low-dimensional visual feature representation; A judgment module, used to judge whether the total amount of the low-dimensional visual feature representation exceeds the upper limit of the representation processing of the large language model; A first visual feature representation module, configured to use the low-dimensional visual feature representation as a compressed visual feature representation if the number of pixels is not exceeded; A time compression module is used to compress the low-dimensional visual feature representation in time according to a time compression multiple to obtain a compressed visual feature representation when the time compression multiple is exceeded; the time compression multiple is determined based on the total amount of the low-dimensional visual feature representation and the upper limit of the representation processing of the large language model.

[0078] In an optional embodiment, the time compression module includes: A selection module, configured to sequentially select a plurality of low-dimensional visual feature representations corresponding to video frames of the time compression multiple according to the order of each video frame in the video to be understood; A compression submodule, used for compressing the plurality of low-dimensional visual feature representations into a target visual feature representation, wherein the time proportion and format of the target visual feature representation are the same as the time proportion and format of the visual representation of a video frame; The second visual feature representation module is used to obtain a compressed visual feature representation based on all target visual feature representations.

[0079] In an optional embodiment, the device further includes: An arrangement module, used for arranging and reorganizing the compressed visual feature representation and the temporal feature representation corresponding to each video frame to obtain a feature representation queue corresponding to each video frame; The result generation module is further used to input the feature representation queues corresponding to all video frames and the text feature representations into the large language model to generate a video understanding result corresponding to the text question information.

[0080] The present application also provides an electronic device, referring to Figure 5 , Figure 5 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 500 includes: a memory 510 and a processor 520. The memory 510 and the processor 520 are connected via a bus communication. A computer program is stored in the memory 510. The computer program can be executed on the processor 520 to implement the steps of the video understanding method based on a multi-modal large model of multi-branch visual representation described in the embodiment of the present application.

[0081] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the video understanding method based on a multi-modal large model of multi-branch visual representation described in the embodiment of the present application are implemented.

[0082] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the video understanding method based on a multi-modal large model of multi-branch visual representation described in the embodiment of the present application.

[0083] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0084] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods and devices according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0085] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0086] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0087] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0088] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0089] The above is a detailed introduction to the video understanding method and device of a multimodal large model based on multi-branch visual representation provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video understanding method based on a multi-modal large model with multi-branch visual representation, characterized in that: The method comprises: Obtaining text feature representation corresponding to text question information; Encode the temporal information of the video to be understood to obtain a temporal feature representation; Extracting fine-grained information from the video to be understood to obtain a visual feature representation, wherein the visual feature representation includes: a visual-text alignment feature representation and a fine-grained visual feature representation; performing at least spatial compression on the visual feature representation to obtain a compressed visual feature representation; The compressed visual feature representation, the temporal feature representation and the text feature representation are input into a large language model to generate a video understanding result corresponding to the text question information.

2. The method according to claim 1, characterized in that Encode the temporal information of the video to be understood and obtain temporal feature representation, including: The timestamp text information of each video frame of the video to be understood is encoded to obtain a temporal feature representation; wherein the timestamp text information is the video frame of the i-th second, i is an integer between 1 and N, and N is the total number of video frames of the video to be understood.

3. The method according to claim 1, characterized in that Fine-grained information is extracted from the video to be understood to obtain visual feature representation, including: Inputting each video frame of the video to be understood into a first visual encoder for feature extraction to obtain a visual-text alignment feature representation corresponding to each video frame; the first visual encoder is trained to generate description information corresponding to an image; Each video frame of the video to be understood is input into a second visual encoder for feature extraction to obtain a fine-grained visual feature representation corresponding to each video frame; the second visual encoder is trained to perceive the detail differences between different images.

4. The method according to claim 1, characterized in that: At least spatially compressing the visual feature representation to obtain a compressed visual feature representation includes: Performing spatial compression on the visual feature representation to obtain a low-dimensional visual feature representation; Determining whether the total amount of the low-dimensional visual feature representation exceeds an upper limit value of the representation processing of the large language model; If the number does not exceed the value, the low-dimensional visual feature representation is used as the compressed visual feature representation; In case of exceeding, the low-dimensional visual feature representation is time-compressed according to a time compression multiple to obtain a compressed visual feature representation; the time compression multiple is determined based on the total amount of the low-dimensional visual feature representation and the upper limit of the representation processing of the large language model.

5. The method according to claim 4, characterized in that If the time exceeds the limit, the low-dimensional visual feature representation is temporally compressed according to the time compression multiple to obtain a compressed visual feature representation, including: According to the sequence of each video frame in the video to be understood, sequentially selecting a plurality of the low-dimensional visual feature representations corresponding to the video frames of the time compression multiple; Compressing the plurality of low-dimensional visual feature representations into a target visual feature representation, wherein the time proportion and format of the target visual feature representation are the same as the time proportion and format of the visual representation of a video frame; According to all target visual feature representations, a compressed visual feature representation is obtained.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Arrange and reorganize the compressed visual feature representation and the temporal feature representation corresponding to each video frame to obtain a feature representation queue corresponding to each video frame; Inputting the compressed visual feature representation, the temporal feature representation, and the text feature representation into a large language model to generate a video understanding result corresponding to the text question information, including: The feature representation queues corresponding to all video frames and the text feature representations are input into the large language model to generate a video understanding result corresponding to the text question information.

7. A video understanding device based on a multi-modal large model with multi-branch visual representation, characterized in that: The system comprises: A text information representation module is used to obtain text feature representations corresponding to text question information; A temporal information representation module is used to encode the temporal information of the video to be understood and obtain a temporal feature representation; A visual information representation module is used to extract fine-grained information from the video to be understood to obtain a visual feature representation, wherein the visual feature representation includes: a visual-text alignment feature representation and a fine-grained visual feature representation; A sliding window compression module, configured to perform at least spatial compression on the visual feature representation to obtain a compressed visual feature representation; The result generation module is used to input the compressed visual feature representation, the temporal feature representation and the text feature representation into a large language model to generate a video understanding result corresponding to the text question information.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the video understanding method based on a multi-modal large model of multi-branch visual representation as described in any one of claims 1-6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the video understanding method based on a multi-modal large model of multi-branch visual representation as described in any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the video understanding method based on a multi-modal large model of multi-branch visual representation as described in any one of claims 1 to 6 are implemented.

Images