A method, apparatus, device, and storage medium for long video understanding

Abstract

This invention discloses a long video understanding method, apparatus, device, and storage medium, comprising: acquiring a user's question instruction and a corresponding long video; outputting a semantic feature vector set based on the visual content of the long video and a pre-trained deep learning network model based on a multi-head self-attention mechanism; merging the semantic feature vectors in the semantic feature vector set to obtain the visual features of the long video; inputting the frame position of the long video and the semantic feature vector set into a pre-trained U-shaped neural network model to determine the relative position of the visual features with respect to preset frames of the long video; adding the relative position to the corresponding visual features to generate visual temporal features; and inputting the visual temporal features and the question instruction into a pre-trained long video understanding model to output the question answer corresponding to the question instruction. This method utilizes temporal localization to ensure the model finds the correct time position in the video, thereby enhancing the ability to understand long videos.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a long video understanding method, apparatus, device and storage medium. Background Technology

[0002] In recent years, multimodal video large language models have developed rapidly, demonstrating impressive instruction-following capabilities and helping people process video data faster and more accurately. However, existing technologies still fall short in their long-term understanding of minute-long videos. Minute-long videos may contain complex temporal logic relationships and a variety of chaotic actions, making it difficult for existing models to effectively locate the key moments for answering questions. When watching long videos, humans consciously focus their attention on prominent segments, while an event in a long video may only occur within a few seconds or tens of seconds.

[0003] Existing large-scale video language models struggle to effectively capture key moments and complex temporal relationships in long videos, resulting in poor performance on long video understanding tasks. Most existing large-scale video language models perform poorly on temporal localization tasks, with accuracy approaching randomness. While some studies have proposed methods to enhance temporal localization capabilities, these methods are often too proprietary, impairing the model's performance on routine tasks and sacrificing some generality. The "illusion phenomenon" in long video understanding tasks: Existing long video understanding models frequently focus on video segments irrelevant to the question when answering long video questions, leading to frequent "illusion phenomena." This indicates that current technologies lack effective training methods for long video understanding tasks, making it difficult to avoid information redundancy and improper handling of contextual relationships. Summary of the Invention

[0004] This invention provides a long video understanding method, apparatus, device, and storage medium, which utilizes time positioning to ensure that the model finds the correct time position in the video, thereby enhancing the ability to understand long videos.

[0005] In a first aspect, embodiments of the present invention provide a long video understanding method, including:

[0006] Obtain the user's question command and the corresponding long video;

[0007] Based on the visual content of the long video and a pre-trained deep learning network model based on a multi-head self-attention mechanism, a set of semantic feature vectors is output.

[0008] The semantic feature vectors in the set of semantic feature vectors are merged to obtain the visual features of the long video;

[0009] The frame positions of the long video and the semantic feature vector set are input into a pre-trained U-shaped neural network model to determine the relative position of the visual features with respect to the preset frames of the long video.

[0010] The relative position is added to the corresponding visual feature to generate a visual temporal feature;

[0011] The visual temporal features and the question instruction are input into a pre-trained long video understanding model, which outputs the question answer corresponding to the question instruction.

[0012] Secondly, embodiments of the present invention also provide a long video understanding device, the device comprising:

[0013] The instruction acquisition module is used to acquire the user's question instruction and the corresponding long video.

[0014] The feature vector set output module is used to output a semantic feature vector set based on the visual content of the long video and a pre-trained deep learning network model based on a multi-head self-attention mechanism.

[0015] The visual feature module is used to merge the semantic feature vectors in the semantic feature vector set to obtain the visual features of the long video.

[0016] The position determination module is used to input the frame position of the long video and the semantic feature vector set into a pre-trained U-shaped neural network model to determine the relative position of the visual features with respect to a preset frame of the long video.

[0017] The feature generation module is used to add the relative position to the corresponding visual feature to generate visual temporal features;

[0018] The answer output module is used to input the visual temporal features and the question instruction into a pre-trained long video understanding model and output the question answer corresponding to the question instruction.

[0019] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0020] One or more processors;

[0021] Storage device for storing one or more programs.

[0022] When the one or more programs are executed by the one or more processors, the one or more processors implement the long video understanding method provided in the embodiments of this disclosure.

[0023] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to implement the long video understanding method provided in embodiments of this disclosure.

[0024] Fifthly, this disclosure provides a computer program product, which includes a computer program that, when executed by a processor, implements the long video understanding method provided in the first aspect of the embodiment.

[0025] This invention discloses a method, apparatus, device, and storage medium for long video understanding, comprising: acquiring a user's question instruction and a corresponding long video; outputting a semantic feature vector set based on the visual content of the long video and a pre-trained deep learning network model based on a multi-head self-attention mechanism; merging the semantic feature vectors in the semantic feature vector set to obtain the visual features of the long video; inputting the frame position of the long video and the semantic feature vector set into a pre-trained U-shaped neural network model to determine the relative position of the visual features with respect to a preset frame of the long video; adding the relative position to the corresponding visual feature to generate visual temporal features; and inputting the visual temporal features and the question instruction into a pre-trained long video understanding model to output the question answer corresponding to the question instruction. This method utilizes temporal localization to ensure the model finds the correct time position in the video, thereby enhancing the ability to understand long videos. Attached Figure Description

[0026] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0027] Figure 1 A flowchart illustrating a long video understanding method provided in an embodiment of this disclosure;

[0028] Figure 2 This is a structural example diagram of a long video understanding method provided in an embodiment of the present disclosure;

[0029] Figure 3 This is a schematic diagram of the structure of a long video understanding device provided in an embodiment of the present disclosure;

[0030] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0031] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0032] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0033] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0034] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0035] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0036] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0037] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0038] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0039] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0040] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0041] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0042] Figure 1 This is a flowchart of a long video understanding method provided by an embodiment of the present disclosure. This embodiment of the disclosure is applicable to situations where it is difficult to accurately locate event-related sub-segments in a long video, resulting in poor performance of the long video understanding task. The method can be executed by a long video understanding device, which can be implemented in the form of software and / or hardware, and optionally, by an electronic device, such as a mobile terminal, a PC, or a server.

[0043] like Figure 1 As shown in the embodiments of this disclosure, a long video understanding method may specifically include the following steps:

[0044] S110. Obtain the user's problem command and the corresponding long video of the problem command.

[0045] In this embodiment, the question instruction can be a command issued by the user to the model to obtain a question. The long video is the video content corresponding to the question raised in the aforementioned question instruction.

[0046] In this embodiment, the invention further includes a new enhanced time-aware dataset, "Temporal Augmentation," constructed to enhance the model's fine-grained time-awareness capabilities. This dataset not only includes a series of refined open-source time-localization datasets but also incorporates released high-quality time-related task data. Traditional time-localization task datasets only contain a single ground truth, namely the start and end times of the target time period. While this type of data performs well in training expert models, it is difficult to unleash the potential of large language models. These datasets still lack in terms of data quantity, data quality, and task diversity. "Temporal Augmentation" contains nine task types strongly related to time-awareness, totaling approximately 349K high-quality time-aware data instances. Compared to previous time-aware datasets, "Temporal Augmentation" contains more data, has a wider data distribution, and greater task diversity, which helps multimodal large models learn more generalized time-series representations.

[0047] S120, based on visual content from long videos and a pre-trained deep learning network model based on multi-head self-attention mechanism, outputs a set of semantic feature vectors.

[0048] Figure 2 This is a structural example diagram of a long video understanding method provided in an embodiment of this disclosure. Figure 2 As shown, Figure 2 The video encoder in the model is a deep learning network model based on a multi-head self-attention mechanism.

[0049] Specifically, the long video is divided into video segments at equal intervals. Each video segment is processed into a semantic feature vector set by a deep learning network model based on a multi-head self-attention mechanism. Then, the semantic feature vectors in the semantic feature vector set are bound to the timestamp corresponding to that video segment by a quadrilateral attention visual transformer.

[0050] Based on the above embodiments, determining the location of the abnormal substrate for each failure mode from the substrate location coordinates according to the failure information may include the following steps:

[0051] a1) Segment the long video according to the set frame length.

[0052] b1) Linearly embed the visual content in the segmented long video into vectors of a set size, and use the set of vectors as a semantic feature vector set.

[0053] Specifically, the frame length can be preset and set according to the actual situation; no specific limitation is made in this embodiment. For example, the frame length can be 8 frames, 16 frames, or 32 frames.

[0054] Specifically, to segment long videos according to a set frame length, the long video can first be converted into structured visual content description text. The visual content in the segmented long video can then be linearly embedded into vectors of a set size, and the set of vectors can be used as a semantic feature vector set.

[0055] S130. Merge the semantic feature vectors in the semantic feature vector set to obtain the visual features of the long video.

[0056] Based on the above embodiments, merging semantic feature vectors in the semantic feature vector set to obtain the visual features of a long video may include the following steps: merging a set number of semantic feature vectors in the semantic feature vector set into a merged feature vector, and using the obtained merged feature vector set as the visual features; the merged feature vector set includes multiple merged feature vectors.

[0057] Specifically, multiple semantic feature vectors can be compressed to the channel dimension. Specifically, for each long video, we concatenate a set number of adjacent semantic feature vectors along the channel dimension to obtain compressed visual features. A linear projection layer is then applied to the merged visual features to generate the semantic feature vectors for the input large model. In this embodiment, a projector that reuses the base model is used. By copying the original linear layer parameters a set number of times along the channel dimension, we can obtain an initialization effect equivalent to mean pooling with a window length of a set number. This avoids introducing additional random initialization parameters that could perturb the original model and thus lose the original capabilities of the base model.

[0058] S140. Input the frame positions and semantic feature vector sets of the long video into a pre-trained U-shaped neural network model to determine the relative positions of the visual features with respect to the preset frames of the long video.

[0059] In this embodiment, the U-shaped neural network model includes depthwise separable convolutional layers, fully connected layers, and convolutional layers. The preset frame can be a frame that serves as a preset anchor point, and the relative position can be the position of the visual feature relative to the preset frame of the long video.

[0060] Specifically, through multi-level deep separable convolutions, the entire long visual feature is gradually shortened into short visual features. Then, a convolution with a sufficiently long window is applied to the shortest visual feature, using zero padding at the front end as an anchor point to encode and embed the temporal relative position of each visual feature in the sequence. Next, we upsample the temporal feature sequence from short to long to recover it step by step, and use residual connections to preserve visual features at different scales.

[0061] Based on the above embodiments, inputting the frame positions and semantic feature vector sets of a long video into a pre-trained U-shaped neural network model to determine the relative positions of visual features with respect to the start or end frames of the long video may include the following steps:

[0062] a2) Input the semantic feature vector set into a depthwise separable convolutional layer for dimensionality reduction, output a low-dimensional semantic feature vector set and input it into a fully connected layer.

[0063] b2) Zero-padding is performed on the front of the low-dimensional semantic feature vector set, and the front is used as the anchor point.

[0064] c2) Determine the relative position of the low-dimensional semantic feature vectors in the low-dimensional semantic feature vector set with respect to the anchor point according to the input frame position.

[0065] d2) Encode the relative positions and embed them into the corresponding low-dimensional semantic feature vectors.

[0066] e2) The embedded low-dimensional semantic feature vector is input into the convolutional layer for dimensionality increase, so as to obtain the relative position of the visual features with respect to the preset frames of the long video.

[0067] In this embodiment, encoding embedding is a process of converting data into a form that a computer can understand. In machine learning and natural language processing, encoding embedding is often used to convert textual or categorical data into numerical vectors so that computers can better understand and process the data. The low-dimensional semantic feature vector set can be a shortened set of semantic feature vectors after performing multiple layers of depthwise separable convolutions.

[0068] Specifically, the semantic feature vector set is input into a depthwise separable convolutional layer for dimensionality reduction, and the output low-dimensional semantic feature vector set is input into a fully connected layer. Zero padding is performed at the front end of the low-dimensional semantic feature vector set, and the front end is used as an anchor point. The relative position of the low-dimensional semantic feature vector in the low-dimensional semantic feature vector set with respect to the anchor point is determined according to the input frame position. The relative position is encoded and embedded into the corresponding low-dimensional semantic feature vector. The embedded low-dimensional semantic feature vector is input into a convolutional layer for dimensionality increase, and the relative position of the visual features with respect to the preset frame of the long video is obtained.

[0069] S150. Add the relative position to the corresponding visual feature to generate visual temporal features.

[0070] Specifically, the video frame corresponding to the visual feature is determined in the long video, and a relative position matching the visual feature is selected according to the frame position. The relative position is then spliced ​​into the corresponding visual feature to generate visual temporal features.

[0071] Based on the above embodiments, adding relative positions to the corresponding visual features to generate visual temporal features may include the following steps:

[0072] a3) Determine the frame position of the video frame corresponding to the visual feature in the long video.

[0073] b3) Select a relative position that matches the visual features according to the frame position.

[0074] c3) The relative positions are spliced ​​into the corresponding visual features to generate visual temporal features.

[0075] S160. Input the visual temporal features and question instructions into the pre-trained long video understanding model, and output the question answer corresponding to the question instruction.

[0076] Specifically, long-video understanding models can be large language models.

[0077] This invention has undergone multiple experiments and simulations, demonstrating its significant advantages in long video understanding and temporal localization tasks. Specific experimental results are as follows: The results show that this invention significantly improves the accuracy of long video understanding tasks. On different datasets, the accuracy improvement is 5.6% and 6.8%, respectively. These results indicate that this invention has a stronger capability when processing long videos.

[0078] Temporal localization task: The model significantly improves accuracy in temporal localization tasks and reduces the occurrence of illusions. Experimental results show that the average performance on the zero-shot temporal localization task reaches approximately 70% of the supervised optimal expert model, with relative improvements of approximately 50-100% across various metrics. On the short video understanding dataset, only a 0.5% loss in accuracy is observed, indicating that it significantly improves long video understanding and temporal localization capabilities while maintaining model generality. In summary, this invention, verified through experiments, simulations, and usage, demonstrates significant advantages in long video understanding and temporal localization tasks, proving its feasibility and effectiveness.

[0079] This method compresses visual tokens, enabling large language models to accept more frame inputs, thus effectively handling complex temporal logic relationships and multiple actions in long videos. Adaptive positional encoding enhances the model's temporal perception. Simultaneously, a temporal localization description method is proposed, requiring the model to output detailed descriptions when locating relevant time segments, ensuring the model finds the correct time position in the video and reducing illusions. By constructing an instruction fine-tuning dataset, training data helpful for long video understanding is provided, preventing overfitting to temporal localization tasks and improving the model's performance on long video understanding tasks.

[0080] This invention discloses a long video understanding method, comprising: acquiring a user's question instruction and a corresponding long video; outputting a semantic feature vector set based on the visual content of the long video and a pre-trained deep learning network model based on a multi-head self-attention mechanism; merging the semantic feature vectors in the semantic feature vector set to obtain the visual features of the long video; inputting the frame position of the long video and the semantic feature vector set into a pre-trained U-shaped neural network model to determine the relative position of the visual features with respect to preset frames of the long video; adding the relative position to the corresponding visual features to generate visual temporal features; and inputting the visual temporal features and the question instruction into a pre-trained long video understanding model to output the question answer corresponding to the question instruction. This method utilizes temporal localization to ensure the model finds the correct time position in the video, thereby enhancing the ability to understand long videos.

[0081] Figure 3 A schematic diagram of a long video understanding device is also provided as an embodiment of the present invention, such as... Figure 3 As shown, the device includes: an instruction acquisition module 210, a feature vector set output module 220, a visual feature module 230, a position determination module 240, a feature generation module 250, and an answer output module 260.

[0082] The instruction acquisition module 210 is used to acquire the user's question instruction and the long video corresponding to the question instruction;

[0083] The feature vector set output module 220 is used to output a semantic feature vector set based on the visual content of the long video and a pre-trained deep learning network model based on a multi-head self-attention mechanism.

[0084] The visual feature module 230 is used to merge the semantic feature vectors in the semantic feature vector set to obtain the visual features of the long video.

[0085] The position determination module 240 is used to input the frame position of the long video and the semantic feature vector set into a pre-trained U-shaped neural network model to determine the relative position of the visual features with respect to a preset frame of the long video.

[0086] The feature generation module 250 is used to add the relative position to the corresponding visual feature to generate visual temporal features;

[0087] The answer output module 260 is used to input the visual temporal features and the question instruction into a pre-trained long video understanding model and output the question answer corresponding to the question instruction.

[0088] The technical solution provided in this disclosure uses time positioning to ensure that the model finds the correct time position in the video, thereby enhancing the ability to understand long videos.

[0089] Furthermore, the feature vector set output module 220 can be used for:

[0090] The long video is segmented according to a set frame length;

[0091] The visual content in the segmented long video is linearly embedded into vectors of a set size, and the set of vectors is used as a semantic feature vector set.

[0092] Furthermore, the visual feature module 230 can be used for:

[0093] The set of semantic feature vectors is merged into a single merged feature vector, and the resulting merged feature vector set is used as the visual feature; the merged feature vector set includes multiple merged feature vectors.

[0094] Furthermore, the feature generation module 250 can be used for:

[0095] Determine the frame position of the video frame corresponding to the visual feature in the long video;

[0096] Select the relative position that matches the visual feature according to the frame position;

[0097] The relative positions are concatenated to the corresponding visual features to generate the visual temporal features.

[0098] Furthermore, the location determination module 240 can be used for:

[0099] The U-shaped neural network model includes depthwise separable convolutional layers, fully connected layers, and convolutional layers.

[0100] Furthermore, the location determination module 240 can be used for:

[0101] The semantic feature vector set is input into the depthwise separable convolutional layer for dimensionality reduction, and the low-dimensional semantic feature vector set is output and input into the fully connected layer.

[0102] Zero-padding is performed at the front end of the low-dimensional semantic feature vector set, and the front end is used as an anchor point;

[0103] The relative position of the low-dimensional semantic feature vector in the low-dimensional semantic feature vector set with respect to the anchor point is determined according to the input frame position.

[0104] The relative positions are encoded and embedded into the corresponding low-dimensional semantic feature vectors;

[0105] The embedded low-dimensional semantic feature vector is input into the convolutional layer for dimensionality upscaling to obtain the relative position of the visual feature with respect to the preset frame of the long video.

[0106] The above-described apparatus can execute the methods provided in all the foregoing embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the above methods. Technical details not described in detail in this embodiment can be found in the methods provided in all the foregoing embodiments of the present invention.

[0107] Figure 4 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is provided. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0108] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0109] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0110] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as long video understanding methods.

[0111] In some embodiments, the long video understanding method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the long video understanding method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the long video understanding method by any other suitable means (e.g., by means of firmware).

[0112] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0114] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0115] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0116] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0117] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0118] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0119] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for understanding long videos, characterized in that, include: Obtain the user's question command and the corresponding long video; Based on the visual content of the long video and a pre-trained deep learning network model based on a multi-head self-attention mechanism, a set of semantic feature vectors is output. The semantic feature vectors in the set of semantic feature vectors are merged to obtain the visual features of the long video; The frame positions of the long video and the semantic feature vector set are input into a pre-trained U-shaped neural network model to determine the relative position of the visual features with respect to the preset frames of the long video. The relative position is added to the corresponding visual feature to generate a visual temporal feature; The visual temporal features and the question instruction are input into a pre-trained long video understanding model, which outputs the question answer corresponding to the question instruction. The step of adding the relative position to the corresponding visual feature to generate visual temporal features includes: Determine the frame position of the video frame corresponding to the visual feature in the long video; Select the relative position that matches the visual feature according to the frame position; The relative positions are concatenated to the corresponding visual features to generate the visual temporal features; The U-shaped neural network model includes depthwise separable convolutional layers, fully connected layers, and convolutional layers; The step of inputting the frame positions of the long video and the semantic feature vector set into a pre-trained U-shaped neural network model to determine the relative position of the visual features with respect to the start or end frame of the long video includes: The semantic feature vector set is input into the depthwise separable convolutional layer for dimensionality reduction, and the low-dimensional semantic feature vector set is output and input into the fully connected layer. Zero-padding is performed at the front end of the low-dimensional semantic feature vector set, and the front end is used as an anchor point; The relative position of the low-dimensional semantic feature vector in the low-dimensional semantic feature vector set with respect to the anchor point is determined according to the input frame position. The relative positions are encoded and embedded into the corresponding low-dimensional semantic feature vectors; The embedded low-dimensional semantic feature vector is input into the convolutional layer for dimensionality upscaling to obtain the relative position of the visual feature with respect to the preset frame of the long video.

2. The method according to claim 1, characterized in that, The visual content based on the long video and the pre-trained deep learning network model based on a multi-head self-attention mechanism outputs a semantic feature vector set, including: The long video is segmented according to a set frame length; The visual content in the segmented long video is linearly embedded into vectors of a set size, and the set of vectors is used as a semantic feature vector set.

3. The method according to claim 1, characterized in that, The step of merging the semantic feature vectors in the semantic feature vector set to obtain the visual features of the long video includes: The set of semantic feature vectors is merged into a single merged feature vector, and the resulting merged feature vector set is used as the visual feature; the merged feature vector set includes multiple merged feature vectors.

4. A long video understanding device, characterized in that, include: The instruction acquisition module is used to acquire the user's question instruction and the corresponding long video. The feature vector set output module is used to output a semantic feature vector set based on the visual content of the long video and a pre-trained deep learning network model based on a multi-head self-attention mechanism. The visual feature module is used to merge the semantic feature vectors in the semantic feature vector set to obtain the visual features of the long video. The position determination module is used to input the frame position of the long video and the semantic feature vector set into a pre-trained U-shaped neural network model to determine the relative position of the visual features with respect to a preset frame of the long video. The feature generation module is used to add the relative position to the corresponding visual feature to generate visual temporal features; The answer output module is used to input the visual temporal features and the question instruction into a pre-trained long video understanding model and output the question answer corresponding to the question instruction; The feature generation module is used for: Determine the frame position of the video frame corresponding to the visual feature in the long video; Select the relative position that matches the visual feature according to the frame position; The relative positions are concatenated to the corresponding visual features to generate the visual temporal features; The location determination module is used for: The U-shaped neural network model includes depthwise separable convolutional layers, fully connected layers, and convolutional layers; The location determination module is used for: The semantic feature vector set is input into the depthwise separable convolutional layer for dimensionality reduction, and the low-dimensional semantic feature vector set is output and input into the fully connected layer. Zero-padding is performed at the front end of the low-dimensional semantic feature vector set, and the front end is used as an anchor point; The relative position of the low-dimensional semantic feature vector in the low-dimensional semantic feature vector set with respect to the anchor point is determined according to the input frame position. The relative positions are encoded and embedded into the corresponding low-dimensional semantic feature vectors; The embedded low-dimensional semantic feature vector is input into the convolutional layer for dimensionality upscaling to obtain the relative position of the visual feature with respect to the preset frame of the long video.

5. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the long video understanding method according to any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the long video understanding method according to any one of claims 1-3.

7. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the long video understanding method according to any one of claims 1-3.

Images