Video understanding method and device, model training method and device and electronic equipment

By introducing hypernetwork structures into the video understanding model, using multi-layer interactions of data encoding, memory and task neuron modules, the problem of poor multimodal video understanding in the existing technology is solved, and stronger generalization is achieved Good results for the ability and video comprehension tasks.

CN119068384AActive Publication Date: 2024-12-03SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202410988415.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-12-03
Estimated Expiration
2044-07-23

AI Technical Summary

Technical Problem

The existing video understanding methods are poor when processing videos with multimodal information, and lack deep multimodal interaction modeling and information encoding capabilities, resulting in weak generalization capabilities.

Method used

Using a video understanding model built on a hypernetwork, including data encoding neuron module, memory neuron module and task neuron module, the encoding and feature extraction of multimodal videos is carried out through the interaction of multi-layer neuron modules to achieve a deeper level of Information integration and task understanding.

Benefits of technology

It improves the generalization ability of the video understanding model, significantly improves the effectiveness of multimodal video understanding tasks, and can handle mainstream video understanding tasks and achieve good performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068384A_ABST
    Figure CN119068384A_ABST
Patent Text Reader

Abstract

The invention discloses a video understanding method and device, a model training method and device and electronic equipment, and the method is used for carrying out video task understanding through a trained video understanding model, so as to output a video understanding result. Wherein the video understanding model is constructed based on a super network, and comprises a data coding neuron module, a memory neuron module and a task neuron module; the method comprises the steps of obtaining a to-be-understood video; encoding each modal data in the to-be-understood video through a data encoding neuron module to obtain each modal encoded data; performing feature extraction on each modal coded data through a memory neuron module to obtain each modal feature data; and jointly inputting the modal coding data and the modal feature data into a task neuron module to output a video understanding result. The video understanding model constructed based on the super network is adopted to process the video understanding task, and the network structure can improve the model performance and has higher generalization ability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular relates to a video understanding method, a model training method, a device, and an electronic device. Background Art

[0002] Video is a natural data format that reflects the underlying dynamics of the physical world. Understanding what is happening in the video is crucial for world modeling. With the rapid development of deep learning, many AI-based deep learning video understanding networks have been proposed in recent years to handle video understanding tasks. Among them, general video understanding models are mainly designed specifically for single-modality video input. This single-modality approach limits the versatility of understanding real-world models and tasks, and has limited application value in multimodal tasks.

[0003] In recent years, large models obtained through large-scale multimodal pre-training have shown potential for processing video understanding tasks. However, these models simply align encoders of different modalities through contrastive learning, lacking the ability to model deeper multimodal interactions in videos, and the contrastive learning method is relatively simple. The interaction between multimodal models is limited, and there is only interaction between output vectors, and the generalization ability of the model is weak. The interaction between multimodal data of these models is poor, the information transfer between multimodal models is less, and they lack the ability to encode deep multimodal information.

[0004] Therefore, existing video understanding methods do not perform well when faced with videos with multimodal information. Summary of the invention

[0005] The present application provides a video understanding method, a model training method, a device and an electronic device, aiming to solve the problem that existing video understanding methods are not effective in processing video understanding tasks.

[0006] In a first aspect, the present application provides a video understanding method, the method is used to perform video task understanding through a trained video understanding model to output a video understanding result; wherein the video understanding model is constructed based on a hypernetwork, which includes a data encoding neuron module, a memory neuron module and a task neuron module; the method includes:

[0007] Acquire a video to be understood; the video to be understood includes at least one modality information;

[0008] Encoding each modality data in the to-be-understood video by means of the data encoding neuron module to obtain each modality encoding data;

[0009] Performing feature extraction on the coded data of each modality through the memory neuron module to obtain feature data of each modality;

[0010] The encoding data of each modality and the feature data of each modality are input into the task neuron module to output the video understanding result.

[0011] In a second aspect, the present application provides a model training method, the model training method comprising:

[0012] Acquire training samples; the training samples include acquiring video samples with multimodal information and understanding result labels corresponding to the video samples;

[0013] Inputting the video sample into a pre-built video understanding model to output a predicted video understanding result;

[0014] The video understanding model is trained by comparing the result label of the video sample with the predicted video understanding result.

[0015] In a third aspect, the present application provides a video understanding device, which is used to perform video task understanding through a trained video understanding model to output a video understanding result; wherein the video understanding model is constructed based on a hypernetwork, which includes a data encoding neuron module, a memory neuron module and a task neuron module; the device includes:

[0016] An acquisition module, used for acquiring a video to be understood; the video to be understood includes at least one modal information;

[0017] The processing module encodes each modality data in the video to be understood through the data encoding neuron module to obtain each modality encoding data; extracts features of the each modality encoding data through the memory neuron module to obtain each modality feature data; and inputs the each modality encoding data and the each modality feature data into the task neuron module to output the video understanding result.

[0018] In a fourth aspect, the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned video understanding method or the above-mentioned model training method when executing the computer program.

[0019] In a fifth aspect, a storage medium stores computer-readable instructions thereon, wherein the computer-readable instructions are executed by one or more processors to implement the video understanding method as described above or to implement the model training method as described above.

[0020] In a sixth aspect, a computer program product includes computer-readable instructions, the computer-readable instructions are stored in a storage medium, one or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the video understanding method as described above or implements the model training method as described above.

[0021] Compared with the prior art, the video understanding method of the present application uses a video understanding model built based on a hypernetwork to understand the video to be understood. The video understanding model includes a data encoding neuron module, a memory neuron module and a task neuron module. The encoding, feature extraction and task understanding of multimodal videos are realized through each neuron module. This hypernetwork structure can improve the performance of the model and has stronger generalization ability. The video understanding model has a good effect on mainstream video understanding tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A schematic diagram of the main process of a video understanding method shown in an embodiment of the present application;

[0024] Figure 2 for Figure 1 A schematic diagram of the sub-processes of the video understanding method shown;

[0025] Figure 3 for Figure 2 A schematic diagram of the sub-processes of the video understanding method shown;

[0026] Figure 4 A schematic diagram of a specific implementation of the video understanding method of the present application;

[0027] Figure 5 A schematic diagram of the main process of the video understanding model training method shown in an embodiment of the present application;

[0028] Figure 6 A structural block diagram of a video understanding device provided in an embodiment of the present application;

[0029] Figure 7 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0030] Figure 8A schematic diagram of the performance effect of the video understanding model in the video understanding method shown in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] Those skilled in the art will appreciate that, unless expressly stated otherwise, the singular forms "a", "an", "" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other elements, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" as used herein includes all or any unit and all combinations of one or more associated listed items.

[0033] Multimodal technology refers to the technology that uses multiple different perceptual modalities (such as vision, text, audio, etc.) for information processing and interaction. By combining multiple modal data, multimodal technology can obtain more comprehensive and accurate information, thereby improving the performance of human-computer interaction, data analysis, intelligent recognition, etc. One of the key challenges of multimodal technology is how to fuse and coordinate information from different perceptual modalities. This involves technologies such as data fusion, modal alignment, and feature extraction. In addition, multimodal technology also needs to solve the heterogeneity and uncertainty between different perceptual modalities, as well as how to effectively use multimodal information for analysis and decision-making.

[0034] See also Figures 1 to 4 , Figure 1 A main process diagram of the video understanding method provided in an embodiment of the present application is shown, wherein the method is used to perform video task understanding through a trained video understanding model to output a video understanding result; wherein the video understanding model is constructed based on a hypernetwork, which includes a data encoding neuron module, a memory neuron module and a task neuron module.

[0035] Specifically, a hypernetwork refers to a super-large network formed by stacking trained multimodal basic models. In a hypernetwork, there is a dedicated way for neurons to communicate with each other, and the input data is input to the neurons in each layer instead of just once.

[0036] In this embodiment, the neurons in the hypernetwork have three functions: data encoding, memory processing, and completing specific video understanding tasks. Specifically, Figure 4 As shown, the video understanding model constructed based on the hypernetwork includes a data encoding neuron module, a memory neuron module and a task neuron module, wherein the data encoding neuron module is connected to the memory neuron module and the task neuron module at the same time. The data encoding module includes one or more layers of data encoding neurons, the memory neuron module includes one or more layers of memory neurons, and the task neuron module includes one or more layers of task neurons; and each layer of neurons includes one or more neurons, and each multimodal basic model is a neuron of the hypernetwork, that is, each neuron corresponds to a basic model of a modality.

[0037] like Figure 1 As shown, an embodiment of the video understanding method of the present application includes the following steps S100 to S400:

[0038] S100, obtaining a video to be understood; the video to be understood includes at least one modality information.

[0039] A video to be used for a video understanding task is obtained, where the obtained video includes several video frames, each of which has information of one or more modalities, including but not limited to visual, audio, text, depth and other information.

[0040] S200, encoding each modality data in the video to be understood through the data encoding neuron module to obtain each modality encoding data.

[0041] First, extract the information of each modality in the video to be understood, such as extracting text information, audio information, etc., and then use the data encoding neuron module to convert each modality data in the input video to be processed into a data form that can be accepted by the memory neuron module, that is, perform separate data encoding on each modality data. Among them, the data encoding neuron module includes one or more data encoding neurons, each data encoding neuron is used to correspond to the information of a modality in the video to be understood, as shown in formula (1):

[0042] X m =ε m (D m ) (1)

[0043] Among them, X m Represents the mth modal data Dm By ε m The encoded data form, in general, allows the use of any data relevant to the target task of downstream video understanding. If the current data has no available modality, we can introduce additional data processing networks to obtain data of these modalities.

[0044] S300, performing feature extraction on each modal coding data through the memory neuron module to obtain feature data of each modality.

[0045] After obtaining the encoded data form of the input data, it is input into the memory neuron module to extract the features of each modality.

[0046] The video understanding model built based on the hypernetwork is used to process the video understanding task of the video files with multimodal information. The network includes a data encoding neuron module, a memory neuron module, and a task neuron module to realize the encoding, feature extraction, and task understanding of the multimodal video; and the pre-trained neural network is used as the neuron in the hypernetwork. Since the multimodal information has been obtained through large-scale pre-training, the multimodal knowledge has been stored inside the neuron. Therefore, this hypernetwork structure can improve the performance of the model, has stronger generalization ability, and achieves good results in mainstream video understanding tasks. For specific results, please refer to Figure 8 The video super network model shown in FIG. 1 is a video expert model that can be used as a task neuron module in this embodiment, so that the video understanding task in any task field can be solved.

[0047] The video understanding tasks in this embodiment include but are not limited to pixel-level video understanding, high-level video-level recognition, and video-to-text dialogue tasks, and text-to-video content editing tasks. Among them, pixel-level tracking refers to the continuous tracking of the position of the target object in the video sequence, usually representing the position and size of the target object in each frame with pixel-level accuracy. Video-level recognition aims to identify key information in the video, such as scenes, objects, actions, etc., and classify or label them. The video-to-text dialogue task involves converting video content into natural language descriptions to achieve interaction between video and text. The text-to-video content editing task involves generating or editing video content based on a given text description.

[0048] In an exemplary embodiment, the memory neuron module includes a memory neuron layer, the neuron layer includes at least one memory neuron, each neuron is used to correspond to the coded data of a modality, and the memory neuron layer is used to extract features of the coded data forms of each modality, as shown in formula (2):

[0049]

[0050] in, The mth modal data passes through the memory neuron layer The features obtained.

[0051] In this embodiment, the memory neuron can be a basic model encoder with frozen parameters (such as a multimodal encoder of an Imagebind model or a languagebind model). The multimodal encoder of the Imagebind and languagebind models is a pre-trained model, which has been well pre-trained through large-scale contrastive learning. Therefore, these features have recorded the aligned multimodal pre-trained knowledge in the memory neuron.

[0052] In an exemplary embodiment, the memory neuron module includes a first memory neuron layer and one or more cyclic memory association modules connected in sequence; the method for extracting features through the memory neuron module includes steps S310 to S320:

[0053] S310, inputting each modality encoding data into the first memory neuron layer for feature extraction to output each modality feature;

[0054] The first memory neuron layer is used for feature extraction to extract the corresponding feature information from the encoded data of each modality.

[0055] S320, inputting the features of each modality into at least one of the cyclic memory association modules to perform feature extraction again, so as to output feature data of each modality.

[0056] In this embodiment, the recurrent memory association module is used to perform feature extraction again on the feature information extracted by the first memory neuron layer to further obtain enhanced features of each modal information.

[0057] In an exemplary embodiment, the recurrent memory association module may include a fusion layer, a fully connected layer, a feature prompt layer, and a second memory neuron layer connected in sequence; the process of re-extracting features through the recurrent memory association module includes memory integration, memory conversion, and memory prompting; the integration of feature information is achieved through the above steps, specifically including steps S321 to S324:

[0058] S321, inputting the features of each modality into the fusion layer for feature fusion to output fusion features;

[0059] In the memory integration step, the output characteristics of the memory neurons in the i-th layer are obtained We first combine all the features of these modes to obtain the fusion feature F of all multimodal features of the i-th (i) , as shown in formula (3):

[0060]

[0061] in, Indicates a fusion method, such as maximum pooling, average pooling or splicing. In this embodiment, splicing is selected as a multi-modal feature fusion method through ablation experiments.

[0062] S322, inputting the fused feature data into the fully connected layer for feature mapping to obtain feature representations of each modality after mapping;

[0063] In the memory conversion step, the multimodal fusion feature F (i) It is regarded as the neurotransmitter to be transmitted to the next memory neuron layer, and the feature mapping is performed through the fully connected layer to obtain the feature representation of each modality after mapping. As shown in formula (4):

[0064]

[0065] Among them, Represents a fully connected layer that processes neurotransmitters that are input to the next layer of memory neurons.

[0066] S323, inputting the feature representations of each modality into the feature prompting layer and adding learning parameters to the feature representation of each modality to output prompting information of each modality.

[0067] In the memory cueing step, we convert neurotransmitters into cue information for each modality of the memory neuron in modality m of the i+1th layer As shown in formula (5):

[0068]

[0069] Among them, we introduced the learnable parameters To increase the adaptability of the recurrent memory association module.

[0070] S324, inputting the prompt information of each modality and the encoded data of each modality into the second memory neuron layer for feature extraction again to output feature data of each modality.

[0071] Finally, we transform all the input data X m And each modal prompt information All are input into the second memory neuron layer, and the second memory neuron can extract features of each modal information again according to the integrated prompt information to obtain enhanced features, which can then be input into other subsequent cyclic memory association modules again until they are input into the last cyclic memory association module. The enhanced feature data can be expressed by formula (6):

[0072]

[0073] in, is the memory neuron layer, It is the enhanced feature data after feature extraction by the i-th cyclic memory association module.

[0074] Understandably, the memory neuron module can include multiple memory neuron layers, and each memory neuron layer is connected through a communication unit; understandably, since a single memory neuron layer is not enough to encode the deeper interaction between the modalities, the model's ability in video understanding tasks is still limited. Therefore, a cyclic memory association idea is proposed, that is, the memory neuron module can include multiple memory neuron layers to cyclically extract data features and integrate multimodal information to achieve deep multimodal interaction.

[0075] It can be seen that by setting up communication units between each memory neuron layer, the amount of communication information between each memory neuron layer can be guaranteed, so as to integrate the information of multimodal video more deeply, thereby extracting deeper enhanced features. The encoded data will be input into the memory neurons of each layer, rather than just once.

[0076] It is important to know that the design of cyclic memory association is inspired by the multimodal association process of biological species. After receiving input stimulation of one modality, the corresponding brain area will produce neurotransmitters. Afterwards, the neurotransmitters of this modality will be transmitted to the processing part of the modality related to it, realizing multimodal association. This association is repeated many times, which can strengthen the association and information processing of various modalities. Based on this biological inspiration, cyclic memory association is completed and feature information integration is realized through three steps: memory integration, memory transformation and memory prompting.

[0077] On the basis of a memory neuron layer in the above embodiment, multiple memory neuron layers and multiple communication units are introduced as cyclic memory associative modules, that is, except for the original first memory neuron layer, each subsequent communication unit and each memory neuron layer can be regarded as a cyclic memory associative module. By stacking multiple cyclic memory associative modules, the feature information of the video with multimodal information can be integrated and converted multiple times, so that the super network model can gradually realize the integration of in-depth multimodal video information; thereby, a deeper understanding of the multimodal information can be achieved, and the understanding of the video with multimodal information can be improved. In addition, each neuron module of the video understanding model can be customized according to actual conditions, and all neurons are not fixed, but a freely customizable paradigm; for example, the input modality can be increased, the number of memory neuron layers can be adjusted, and the relevant model of the task neuron module can be changed, so that the model has better scalability.

[0078] S400, inputting the encoding data of each modality and the feature data of each modality into the task neuron module to output a video understanding result.

[0079] For any video understanding task, the expert model is used as the corresponding task neuron module. All modal feature data output by the recurrent memory association module and X of the video input data m They are sent to the task neuron module together. As shown in formula (7):

[0080]

[0081] Among them, Y represents the output video understanding result, Represents a task neuron module.

[0082] The video understanding method in each embodiment of the present application has achieved good results in mainstream video understanding tasks; among them, video understanding tasks include but are not limited to pixel-level video understanding, high-level video-level recognition, as well as video-to-text dialogue tasks, and text-to-video content editing tasks.

[0083] Based on the video understanding model in the above-mentioned video understanding method, the present application also provides a model training method, and the model training method includes the following steps S10 to S30:

[0084] S10, obtaining a training sample; the training sample includes obtaining a video sample with multimodal information and an understanding result label corresponding to the video sample;

[0085] S20, inputting the video sample into a pre-built video understanding model to output a predicted video understanding result;

[0086] S30: Training the video understanding model by comparing the result label of the video sample with the predicted video understanding result.

[0087] In an exemplary embodiment, the video understanding model includes a data encoding neuron module, a memory neuron module and a task neuron module; the memory neuron module includes a first memory neuron layer and at least one cyclic memory association module connected in sequence; step S30 includes:

[0088] Based on the comparison results of the result labels of the video samples and the predicted video understanding results, the network structures of the data encoding neuron module, the memory neuron module, the task neuron module and the recurrent memory association module are determined through ablation experiments.

[0089] In a specific embodiment, the recurrent memory association module includes a fusion layer, a fully connected layer, a feature prompt layer and a second memory neuron layer connected in sequence; the fully connected layer is used to perform feature mapping on the fusion feature data to obtain the processed feature representations of each modality; the feature prompt layer is used to add learning parameters to the feature representations of each modality to output the prompt feature information of each modality; the data encoding neuron module, the memory neuron module and the task neuron module all use a pre-trained network with frozen parameters; step S30 also includes:

[0090] Based on the comparison result between the result label of the video sample and the predicted video understanding result, the weight of the fully connected layer and the learning parameters of the feature prompt layer are optimized.

[0091] The hypernetwork is a concise and flexible network structure. The video understanding model based on the hypernetwork can use multimodal neuron association to solve any video task. And the hypernetwork only needs a small amount of parameter fine-tuning for any video task, because these well-pre-trained memory neurons are frozen during fine-tuning. Only the learnable parameters and the weight parameters of the fully connected layer need to be fine-tuned. Therefore, a large amount of pre-training is not required, which can save a lot of computing resources.

[0092] like Figure 3 As shown, an embodiment of the present application also provides a video understanding device 200, which is used to perform video task understanding through a trained video understanding model to output a video understanding result; wherein, the video understanding model is constructed based on a hypernetwork, and includes a data encoding neuron module, a memory neuron module and a task neuron module; the video understanding device 200 includes an acquisition module 201 and a processing module 202; wherein, the acquisition module 201 is used to acquire a video to be understood; the video to be understood includes at least one modal information; the processing module 202 is used to encode each modal data in the video to be understood through the data encoding neuron module to obtain each modal encoding data; perform feature extraction on the each modal encoding data through the memory neuron module to obtain each modal feature data; and input the each modal encoding data and the each modal feature data into the task neuron module to output a video understanding result.

[0093] In an exemplary embodiment, the processing module 202 also includes a first feature extraction unit and a second feature extraction unit; wherein the first feature extraction unit is used to input each modal coding data into the first memory neuron layer for feature extraction to output each modal feature; the second feature extraction unit is used to input each modal feature into at least one of the cyclic memory association modules for feature extraction again to output feature data of each modality.

[0094] In an exemplary embodiment, the integration unit further includes a fusion subunit, a mapping subunit, a prompt subunit and a feature extraction subunit; wherein the fusion subunit is used to input the features of each modality into the fusion layer for feature fusion to output fusion features; the mapping subunit is used to input the fusion feature data into the fully connected layer for feature mapping to obtain the mapped feature representations of each modality; the prompt subunit is used to input the feature representations of each modality into the feature prompt layer and add learning parameters to the feature representations of each modality to output prompt information of each modality; the feature extraction subunit is used to input the prompt information of each modality and the encoding data of each modality into the second memory neuron layer for feature extraction again to output feature data of each modality. .

[0095] In an exemplary embodiment, the processing module 202 also includes a model training module, which is used to obtain training samples; the training samples include obtaining video samples with multimodal information and understanding result labels corresponding to the video samples; inputting the video samples into a pre-built video understanding model to output predicted video understanding results; and using the comparison results of the result labels of the video samples and the predicted video understanding results to train the video understanding model.

[0096] The present application also provides an electronic device, such as Figure 8 As shown, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003 .

[0097] The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0098] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0099] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0100] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 400, but is not limited to this.

[0101] Computer-readable instructions are stored in the memory 4003 , and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .

[0102] The computer-readable instructions are executed by one or more processors 4001 to implement the video understanding method in the above-mentioned embodiments or the model training method in the above-mentioned embodiments.

[0103] In addition, a storage medium is provided in an embodiment of the present application, on which computer-readable instructions are stored, and the computer-readable instructions are executed by one or more processors to implement the video understanding method as described above or the model training method in the above embodiments.

[0104] A computer program product is provided in an embodiment of the present application. The computer program product includes computer-readable instructions. The computer-readable instructions are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the video understanding method as described above or the model training method in the above embodiments.

[0105] Compared with the related art, the video understanding method of the present application is to understand the video to be understood by adopting a video understanding model based on a hypernetwork, and the video understanding model includes a data encoding neuron module, a memory neuron module and a task neuron module, and each neuron module realizes the encoding of multimodal videos, feature extraction and then realizes task understanding; this hypernetwork structure can improve the performance of the model, has a stronger generalization ability, and the video understanding model has a good effect on the mainstream video understanding task. Moreover, since the pre-trained neural network with frozen parameters is creatively regarded as the neuron in the model, a large amount of pre-training is not required, which can save a lot of computing resources. In addition, by introducing a multi-memory neuron layer and a cyclic memory associative module composed of communication units established between each memory neuron layer, the feature information of the video can be integrated and converted multiple times, so that the multimodal information can be more deeply understood, thereby improving the understanding of the video with multimodal information; by using the video expert model as a task neuron module, the video understanding model can integrate information of any modality and solve the video understanding task of any task field. Moreover, the video understanding model of the present application has achieved good results in various video understanding tasks, such as bottom-level pixel-level understanding, high-level video-level recognition, video-to-text dialogue tasks, text-to-video content editing tasks, etc. Furthermore, each module of the video understanding model can be customized according to actual conditions, for example, the input modality can be increased, the number of memory neuron layers can be adjusted, and the related models of the task neuron modules can be changed, so that the model has better scalability.

[0106] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0107] The present application is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present application. In addition, under the teachings of the present application, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application are within the scope protected by the present application.

Claims

1. A video understanding method, characterized in that: The method is used to perform video task understanding through a trained video understanding model to output a video understanding result; wherein the video understanding model is constructed based on a hypernetwork, which includes a data encoding neuron module, a memory neuron module and a task neuron module; the method includes: Acquire a video to be understood; the video to be understood includes at least one modality information; Encoding each modality data in the to-be-understood video by means of the data encoding neuron module to obtain each modality encoding data; Performing feature extraction on the coded data of each modality through the memory neuron module to obtain feature data of each modality; The encoding data of each modality and the feature data of each modality are input into the task neuron module to output the video understanding result.

2. The method according to claim 1, characterized in that The memory neuron module comprises a first memory neuron layer and at least one cyclic memory associative module connected in sequence; The extracting features of the coded data of each modality by the memory neuron module to obtain feature data of each modality includes: Inputting each modality encoding data into the first memory neuron layer for feature extraction to output each modality feature; The features of each modality are input into at least one of the cyclic memory association modules for feature extraction again to output feature data of each modality.

3. The method according to claim 2, characterized in that The recurrent memory association module includes a fusion layer, a fully connected layer, a feature prompt layer and a second memory neuron layer connected in sequence; Inputting the features of each modality into the fusion layer for feature fusion to output fusion features; Inputting the fused feature data into the fully connected layer for feature mapping to obtain feature representations of each modality after mapping; Inputting the feature representations of each modality into the feature prompt layer and adding learning parameters to the feature representation of each modality to output prompt information of each modality; The prompt information of each modality and the encoded data of each modality are input into the second memory neuron layer for feature extraction again to output feature data of each modality.

4. The method according to claim 2, characterized in that: The memory neuron adopts a multimodal encoder of an Imagebind and / or languagebind model.

5. The method according to any one of claims 1 to 4, characterized in that: The video understanding model is used to process at least one of pixel-level video understanding, high-level video-level recognition, video-to-text dialogue tasks, and text-to-video content editing tasks.

6. A model training method, characterized in that: The model training method comprises: Acquire training samples; the training samples include acquiring video samples with multimodal information and understanding result labels corresponding to the video samples; Inputting the video sample into a pre-built video understanding model to output a predicted video understanding result; The video understanding model is trained by comparing the result label of the video sample with the predicted video understanding result.

7. The model training method according to claim 6, characterized in that: The video understanding model includes a data encoding neuron module, a memory neuron module and a task neuron module; the memory neuron module includes a first memory neuron layer and at least one cyclic memory association module connected in sequence; The using the comparison result of the result label of the video sample and the predicted video understanding result to train the video understanding model includes: Based on the comparison results of the result labels of the video samples and the predicted video understanding results, the network structures of the data encoding neuron module, the memory neuron module, the task neuron module and the recurrent memory association module are determined through ablation experiments.

8. The model training method according to claim 7, characterized in that: The recurrent memory association module includes a fusion layer, a fully connected layer, a feature prompt layer and a second memory neuron layer connected in sequence; the fully connected layer is used to perform feature mapping on the fusion feature data to obtain the processed feature representations of each modality; the prompt layer is used to add learning parameters to the feature representations of each modality to output the prompt feature information of each modality; the data encoding neuron module, the memory neuron module and the task neuron module all use a pre-trained network with frozen parameters; The comparing result of the result label of the video sample with the predicted video understanding result to train the video understanding model further includes: The weight of the fully connected layer and the learning parameters of the feature prompt layer are optimized based on the comparison result between the result label of the video sample and the predicted video understanding result.

9. A video understanding device, characterized in that: The device is used to perform video task understanding through a trained video understanding model to output a video understanding result; wherein the video understanding model is constructed based on a hypernetwork, which includes a data encoding neuron module, a memory neuron module and a task neuron module; the device includes: An acquisition module, used for acquiring a video to be understood; the video to be understood includes at least one modal information; The processing module encodes each modality data in the video to be understood through the data encoding neuron module to obtain each modality encoding data; extracts features of the each modality encoding data through the memory neuron module to obtain each modality feature data; and inputs the each modality encoding data and the each modality feature data into the task neuron module to output the video understanding result.

10. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the video understanding method as described in any one of claims 1 to 5 or the model training method as described in any one of claims 6 to 8 when executing the computer program.

Citation Information

Patent Citations

  • Video understanding method and device, equipment, storage medium and computer program product

    CN114913448A