Long video understanding method and device, equipment and storage medium

Abstract

The invention discloses a long video understanding method, apparatus and device, and a storage medium. The method comprises the steps of obtaining a question instruction of a user and a long video corresponding to the question instruction; based on the visual content of the long video and a pre-trained deep learning network model based on a multi-head self-attention mechanism, outputting a semantic feature vector set; combining semantic feature vectors in the semantic feature vector set to obtain visual features of the long video; inputting the frame position of the long video and the semantic feature vector set into a pre-trained U-shaped neural network model, and determining a relative position of the visual feature relative to a preset frame of the long video; adding the relative position to the corresponding visual feature to generate a visual time sequence feature; and inputting the visual time sequence characteristics and the question instruction into a pre-trained long video understanding model, and outputting a question answer corresponding to the question instruction. According to the method, the model is ensured to find a correct time position in the video by using a time positioning mode, so that the ability of understanding the long video is enhanced.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of artificial intelligence technology, and in particular, to a long video understanding method, device, equipment and storage medium. Background Art

[0002] In recent years, multimodal video large language models have developed rapidly, demonstrating impressive instruction-following capabilities, helping people process video data faster and more accurately. However, existing technologies still have shortcomings in the long-term understanding of minute-level videos. Minute-level videos may contain complex temporal logic relationships and a variety of messy actions, making it difficult for existing models to effectively locate the key moments to answer questions. When humans watch long videos, their attention is always consciously focused on the prominent clips in the video, while an event in a long video may only occur within a few seconds or tens of seconds in the video.

[0003] Existing large language models for videos have difficulty effectively capturing key moments and complex temporal logical relationships when processing long videos, resulting in poor performance on long video understanding tasks. Most existing large language models for videos perform poorly on temporal localization tasks, with accuracy close to random results. Although some studies have proposed methods to enhance temporal localization capabilities, these methods are often too proprietary, resulting in impaired performance of the models on conventional tasks and a loss of certain versatility. Hallucination phenomenon in long video understanding tasks: Existing long video understanding models often focus on video clips that are irrelevant to the question when answering long video questions, resulting in frequent hallucination phenomena. This shows that existing technologies lack effective training methods for long video understanding tasks, and it is difficult to avoid problems with information redundancy and improper handling of contextual relationships. Summary of the invention

[0004] The embodiments of the present invention provide a long video understanding method, apparatus, device and storage medium, which use time positioning to ensure that the model finds the correct time position in the video, thereby enhancing the ability to understand long videos.

[0005] In a first aspect, an embodiment of the present invention provides a long video understanding method, including:

[0006] Obtaining a user's question instruction and a long video corresponding to the question instruction;

[0007] Based on the visual content of the long video and a pre-trained deep learning network model based on a multi-head self-attention mechanism, output a semantic feature vector set;

[0008] Merging the semantic feature vectors in the semantic feature vector set to obtain the visual features of the long video;

[0009] Inputting the frame position of the long video and the semantic feature vector set into a pre-trained U-shaped neural network model to determine the relative position of the visual feature with respect to a preset frame of the long video;

[0010] Adding the relative position to the corresponding visual feature to generate a visual temporal feature;

[0011] The visual temporal features and the question instructions are input into a pre-trained long video understanding model, and the answer to the question corresponding to the question instruction is output.

[0012] In a second aspect, an embodiment of the present invention further provides a long video understanding device, the device comprising:

[0013] An instruction acquisition module, used to acquire the user's question instruction and the long video corresponding to the question instruction;

[0014] A feature vector set output module, used for outputting a semantic feature vector set based on the visual content of the long video and a pre-trained deep learning network model based on a multi-head self-attention mechanism;

[0015] A visual feature module, used for merging the semantic feature vectors in the semantic feature vector set to obtain the visual features of the long video;

[0016] A position determination module, used for inputting the frame position of the long video and the semantic feature vector set into a pre-trained U-shaped neural network model to determine the relative position of the visual feature with respect to a preset frame of the long video;

[0017] A feature generation module, used for adding the relative position to the corresponding visual feature to generate a visual temporal feature;

[0018] The answer output module is used to input the visual temporal features and the question instructions into a pre-trained long video understanding model, and output the question answer corresponding to the question instruction.

[0019] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0020] one or more processors;

[0021] a storage device for storing one or more programs,

[0022] When the one or more programs are executed by the one or more processors, the one or more processors implement the long video understanding method provided by the embodiment of the present disclosure.

[0023] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to implement the long video understanding method provided by the embodiments of the present disclosure.

[0024] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the long video understanding method provided in the embodiment of the first aspect above.

[0025] The present invention discloses a long video understanding method, device, equipment and storage medium, including: obtaining a user's question instruction and a long video corresponding to the question instruction; outputting a semantic feature vector set based on the visual content of the long video and a pre-trained deep learning network model based on a multi-head self-attention mechanism; merging the semantic feature vectors in the semantic feature vector set to obtain the visual features of the long video; inputting the frame position of the long video and the semantic feature vector set into a pre-trained U-shaped neural network model to determine the relative position of the visual feature relative to the preset frame of the long video; adding the relative position to the corresponding visual feature to generate a visual timing feature; inputting the visual timing feature and the question instruction into a pre-trained long video understanding model, and outputting the question answer corresponding to the question instruction. Utilizing this method: using time positioning to ensure that the model finds the correct time position in the video, thereby enhancing the ability to understand long videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0027] Figure 1 A flowchart of a long video understanding method provided by an embodiment of the present disclosure;

[0028] Figure 2 A structural example diagram of a long video understanding method provided by an embodiment of the present disclosure;

[0029] Figure 3 A schematic diagram of the structure of a long video understanding device provided by an embodiment of the present disclosure;

[0030] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0032] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0033] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0034] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0035] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0036] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0037] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0038] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0039] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0040] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0041] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0042] Figure 1 A flowchart of long video understanding is provided in an embodiment of the present disclosure. The embodiment of the present disclosure is suitable for providing a solution to the problem that it is difficult to accurately locate event-related sub-segments in long videos, resulting in poor performance of long video understanding tasks. The method can be executed by a long video understanding device, which can be implemented in the form of software and / or hardware, and optionally, by an electronic device, which can be a mobile terminal, a PC or a server, etc.

[0043] like Figure 1 As shown, a long video understanding method provided by an embodiment of the present disclosure may specifically include the following steps:

[0044] S110: Obtain the user's question instruction and the long video corresponding to the question instruction.

[0045] In this embodiment, the question instruction may be an instruction issued by the user to the model in order to obtain the question raised. The long video is the video content corresponding to the question raised in the question instruction.

[0046] In this embodiment, it also includes: In order to enhance the fine-grained time perception ability of the model, the present invention constructs a new enhanced time perception dataset "Time Series Enhancement". This dataset not only includes a series of refined open source time positioning datasets, but also integrates the published high-quality time-related task data. Traditional time series positioning task datasets only contain a single ground truth, that is, the start and end time of the target period. Although this type of data performs well in the training of expert models, it is difficult to stimulate the potential of large language models. These datasets are still lacking in data quantity, data quality, task diversity, etc. "Time Series Enhancement" contains 9 task types that are strongly related to time series perception, and contains a total of about 349K high-quality time series perception data instances. Compared with previous time series perception datasets, "Time Series Enhancement" contains more data, its data distribution is wider, and its task diversity is stronger, which helps multimodal large models learn more generalized time series representations.

[0047] S120, based on the visual content of long videos and a pre-trained deep learning network model based on a multi-head self-attention mechanism, outputs a set of semantic feature vectors.

[0048] Figure 2 This is a structural example diagram of a long video understanding method provided by an embodiment of the present disclosure. Figure 2 As shown, Figure 2 The video encoder in is a deep learning network model based on multi-head self-attention mechanism.

[0049] Specifically, the long video is divided into video segments at equal intervals. Each video segment is processed into a semantic feature vector set through a deep learning network model based on the multi-head self-attention mechanism, and then the semantic feature vector in the semantic feature vector set of the quadrilateral attention visual transformer is bound to the timestamp corresponding to the video segment.

[0050] Based on the above embodiment, determining the abnormal substrate position of each failure mode from the substrate position coordinates according to the failure information may include the following steps:

[0051] a1) Split the long video into segments according to the set frame length.

[0052] b1) Linearly embed the visual contents in the segmented long video into vectors of set size respectively, and use the set of vectors as the semantic feature vector set.

[0053] Specifically, the set frame length may be pre-set and set according to actual conditions, and is not specifically limited in this embodiment. Exemplarily, the set frame length may be 8 frames, 16 frames, or 32 frames.

[0054] Specifically, the long video is segmented according to the set frame length. The long video can be first converted into a structured visual content description text, and the visual content in the segmented long video is linearly embedded into vectors of a set size, and the set of vectors is used as a semantic feature vector set.

[0055] S130, merging the semantic feature vectors in the semantic feature vector set to obtain visual features of the long video.

[0056] Based on the above embodiments, merging the semantic feature vectors in the semantic feature vector set to obtain the visual features of the long video may include the following steps: merging a set number of semantic feature vectors in the semantic feature vector set into a merged feature vector, and using the obtained merged feature vector set as the visual feature; the merged feature vector set includes multiple merged feature vectors.

[0057] Specifically, multiple semantic feature vectors can be compressed into the channel dimension. Specifically, for each long video, we link a set number of adjacent semantic feature vectors in the channel dimension to obtain compressed visual features. The linear projection layer will be applied to the merged visual features to generate a semantic feature vector for the input large model. In this embodiment, using a projector that reuses the base model, by copying the original linear layer parameters a set number of times in the channel dimension, we can obtain an initialization effect equivalent to mean pooling with a window length of a set number. This avoids the introduction of additional random initialization parameters to disturb the original model, thereby losing the original capabilities of the base model.

[0058] S140, inputting the frame position and semantic feature vector set of the long video into a pre-trained U-shaped neural network model to determine the relative position of the visual feature with respect to the preset frame of the long video.

[0059] In this embodiment, the U-shaped neural network model includes a depth-separable convolutional layer, a fully connected layer and a convolutional layer. The preset frame can be a frame preset as an anchor point, and the relative position can be a position of the visual feature relative to the preset frame of the long video.

[0060] Specifically, through multi-level depth-wise separable convolution, the entire long visual feature is gradually shortened to a short visual feature, and then a convolution with a window long enough is applied to the shortest visual feature, with the front zero padding as the anchor point, to encode the temporal relative position of each visual feature in the sequence. Then, we upsample the temporal feature sequence from short to long and restore it step by step, and use residual links to retain visual features at different scales.

[0061] Based on the above embodiment, the frame position and semantic feature vector set of the long video are input into a pre-trained U-shaped neural network model, and determining the relative position of the visual feature relative to the start frame or the end frame of the long video may include the following steps:

[0062] a2) Input the semantic feature vector set into the depthwise separable convolutional layer for dimensionality reduction, output the low-dimensional semantic feature vector set and input it into the fully connected layer.

[0063] b2) Perform zero padding on the front end of the low-dimensional semantic feature vector set and use the front end as an anchor point.

[0064] c2) determining the relative position of the low-dimensional semantic feature vector in the low-dimensional semantic feature vector set relative to the anchor point according to the input frame position.

[0065] d2) Encode the relative position and embed it into the corresponding low-dimensional semantic feature vector.

[0066] e2) The embedded low-dimensional semantic feature vector is input into the convolutional layer for dimensionality upgrading to obtain the relative position of the visual feature relative to the preset frame of the long video.

[0067] In this embodiment, encoding embedding is a process of converting data into a form that can be understood by a computer. In machine learning and natural language processing, encoding embedding is often used to convert text or category data into numerical vectors so that computers can better understand and process these data. The low-dimensional semantic feature vector set can be a shortened semantic feature vector set after performing multiple layers of deep separable convolution.

[0068] Specifically, the semantic feature vector set is input into a deep separable convolutional layer for dimensionality reduction, a low-dimensional semantic feature vector set is output and input into a fully connected layer, zero padding is performed on the front end of the low-dimensional semantic feature vector set, and the front end is used as an anchor point. The relative position of the low-dimensional semantic feature vector in the low-dimensional semantic feature vector set relative to the anchor point is determined according to the input frame position, the relative position is encoded and embedded into the corresponding low-dimensional semantic feature vector, and the embedded low-dimensional semantic feature vector is input into the convolutional layer for dimensionality increase to obtain the relative position of the visual feature relative to the preset frame of the long video.

[0069] S150: Add the relative position to the corresponding visual feature to generate a visual temporal feature

[0070] Specifically, the frame position of the video frame corresponding to the visual feature in the long video is determined, a relative position matching the visual feature is selected according to the frame position, and the relative position is spliced ​​into the corresponding visual feature to generate a visual temporal feature.

[0071] On the basis of the above embodiment, adding the relative position to the corresponding visual feature to generate the visual temporal feature may include the following steps:

[0072] a3) Determine the frame position of the video frame corresponding to the visual feature in the long video.

[0073] b3) Select the relative position that matches the visual feature according to the frame position.

[0074] c3) splicing the relative positions into the corresponding visual features to generate visual temporal features.

[0075] S160: Input the visual temporal features and question instructions into a pre-trained long video understanding model, and output the answer to the question corresponding to the question instruction.

[0076] Specifically, the long video understanding model can be a large language model.

[0077] The present invention has been verified by multiple experiments and simulations, and the results show that it has significant advantages in long video understanding and time positioning tasks. The specific experimental results are as follows: The experimental results show that the accuracy of the present invention in the long video understanding task is significantly improved. On different data sets, the accuracy is improved by 5.6% and 6.8% respectively. These results show that the present invention has stronger capabilities in processing long videos.

[0078] Temporal localization task: The accuracy of the model in the temporal localization task is significantly improved, reducing the occurrence of hallucinations. Experimental results show that the average performance on the zero-shot temporal localization task reached about 70% of the supervised optimal expert model, and each indicator was relatively improved by about 50-100%. On the short video understanding dataset, only 0.5% of the accuracy was lost, indicating that while maintaining the versatility of the model, it significantly improves the long video understanding and temporal localization capabilities. In summary, the present invention has been verified through experiments, simulations, and use, and has significant advantages in long video understanding and temporal localization tasks. The results show its feasibility and effectiveness.

[0079] This method compresses visual tokens so that large language models can accept more frame inputs, thereby effectively processing complex temporal logic relationships and multiple actions in long videos. By generating adaptive position encodings, the model's perception of time sequence is enhanced. At the same time, a temporal positioning description method is proposed, requiring the model to output a detailed description when locating the relevant time segment, ensuring that the model finds the correct time position in the video and reducing the generation of hallucinations. By constructing an instruction fine-tuning dataset, training data that is helpful for long video understanding is provided, preventing the model from overfitting the temporal positioning task and improving the model's performance on the long video understanding task.

[0080] The present invention discloses a long video understanding method, including: obtaining a user's question instruction and a long video corresponding to the question instruction; outputting a semantic feature vector set based on the visual content of the long video and a pre-trained deep learning network model based on a multi-head self-attention mechanism; merging the semantic feature vectors in the semantic feature vector set to obtain the visual features of the long video; inputting the frame position of the long video and the semantic feature vector set into a pre-trained U-shaped neural network model to determine the relative position of the visual feature relative to the preset frame of the long video; adding the relative position to the corresponding visual feature to generate a visual time sequence feature; inputting the visual time sequence feature and the question instruction into a pre-trained long video understanding model, and outputting the question answer corresponding to the question instruction. Utilizing this method: utilizing the time positioning method to ensure that the model finds the correct time position in the video, thereby enhancing the ability to understand long videos.

[0081] Figure 3 The present invention also provides a schematic diagram of the structure of a long video understanding device. Figure 3 As shown, the device includes: an instruction acquisition module 210, a feature vector set output module 220, a visual feature module 230, a position determination module 240, a feature generation module 250 and an answer output module 260.

[0082] The instruction acquisition module 210 is used to acquire the user's question instruction and the long video corresponding to the question instruction;

[0083] A feature vector set output module 220, configured to output a semantic feature vector set based on the visual content of the long video and a pre-trained deep learning network model based on a multi-head self-attention mechanism;

[0084] A visual feature module 230, configured to merge the semantic feature vectors in the semantic feature vector set to obtain the visual features of the long video;

[0085] A position determination module 240, for inputting the frame position of the long video and the semantic feature vector set into a pre-trained U-shaped neural network model to determine the relative position of the visual feature with respect to a preset frame of the long video;

[0086] A feature generation module 250, configured to add the relative position to the corresponding visual feature to generate a visual temporal feature;

[0087] The answer output module 260 is used to input the visual temporal features and the question instructions into a pre-trained long video understanding model, and output the question answer corresponding to the question instruction.

[0088] The technical solution provided by the embodiments of the present disclosure utilizes time positioning to ensure that the model finds the correct time position in the video, thereby enhancing the ability to understand long videos.

[0089] Furthermore, the feature vector set output module 220 may be used to:

[0090] Segmenting the long video according to set frame lengths;

[0091] The visual contents in the segmented long video are linearly embedded into vectors of a set size respectively, and a set of the vectors is used as a semantic feature vector set.

[0092] Furthermore, the visual feature module 230 may be used to:

[0093] A set number of the semantic feature vectors in the semantic feature vector set are merged into a merged feature vector, and the obtained merged feature vector set is used as the visual feature; the merged feature vector set includes a plurality of the merged feature vectors.

[0094] Furthermore, the feature generation module 250 may be used to:

[0095] Determine the frame position of the video frame corresponding to the visual feature in the long video;

[0096] selecting the relative position matching the visual feature according to the frame position;

[0097] The relative positions are spliced ​​into the corresponding visual features to generate the visual timing features.

[0098] Furthermore, the location determination module 240 may be used to:

[0099] The U-shaped neural network model includes a depthwise separable convolutional layer, a fully connected layer and a convolutional layer.

[0100] Furthermore, the location determination module 240 may be used to:

[0101] Input the semantic feature vector set into the depthwise separable convolutional layer for dimensionality reduction, output a low-dimensional semantic feature vector set and input it into the fully connected layer;

[0102] Perform zero padding on the front end of the low-dimensional semantic feature vector set and use the front end as an anchor point;

[0103] Determining a relative position of a low-dimensional semantic feature vector in the low-dimensional semantic feature vector set relative to the anchor point according to the input frame position;

[0104] Encoding the relative position and embedding it into the corresponding low-dimensional semantic feature vector;

[0105] The embedded low-dimensional semantic feature vector is input into the convolution layer for dimensionality upgrading to obtain the relative position of the visual feature with respect to the preset frame of the long video.

[0106] The above device can execute the methods provided by all the above embodiments of the present invention, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not described in detail in this embodiment, please refer to the methods provided by all the above embodiments of the present invention.

[0107] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is given. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0108] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0109] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0110] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a long video understanding method.

[0111] In some embodiments, the long video understanding method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the long video understanding method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the long video understanding method in any other appropriate manner (e.g., by means of firmware).

[0112] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0114] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0115] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0116] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0117] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0118] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0119] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A long video understanding method, characterized in that: include: Obtaining a user's question instruction and a long video corresponding to the question instruction; Based on the visual content of the long video and a pre-trained deep learning network model based on a multi-head self-attention mechanism, output a semantic feature vector set; Merging the semantic feature vectors in the semantic feature vector set to obtain the visual features of the long video; Inputting the frame position of the long video and the semantic feature vector set into a pre-trained U-shaped neural network model to determine the relative position of the visual feature with respect to a preset frame of the long video; Adding the relative position to the corresponding visual feature to generate a visual temporal feature; The visual temporal features and the question instructions are input into a pre-trained long video understanding model, and the answer to the question corresponding to the question instruction is output.

2. The method according to claim 1, characterized in that The deep learning network model based on the visual content of the long video and the pre-trained multi-head self-attention mechanism outputs a semantic feature vector set, including: Segmenting the long video according to set frame lengths; The visual contents in the segmented long video are linearly embedded into vectors of a set size respectively, and a set of the vectors is used as a semantic feature vector set.

3. The method according to claim 1, characterized in that The step of merging the semantic feature vectors in the semantic feature vector set to obtain the visual features of the long video includes: A set number of the semantic feature vectors in the semantic feature vector set are merged into a merged feature vector, and the obtained merged feature vector set is used as the visual feature; the merged feature vector set includes a plurality of the merged feature vectors.

4. The method according to claim 1, characterized in that The step of adding the relative position to the corresponding visual feature to generate a visual temporal feature includes: Determine the frame position of the video frame corresponding to the visual feature in the long video; selecting the relative position matching the visual feature according to the frame position; The relative positions are spliced ​​into the corresponding visual features to generate the visual timing features.

5. The method according to claim 1, characterized in that The U-shaped neural network model includes a depthwise separable convolutional layer, a fully connected layer and a convolutional layer.

6. The method according to claim 5, characterized in that The step of inputting the frame position of the long video and the semantic feature vector set into a pre-trained U-shaped neural network model to determine the relative position of the visual feature relative to the start frame or the end frame of the long video comprises: Input the semantic feature vector set into the depthwise separable convolutional layer for dimensionality reduction, output a low-dimensional semantic feature vector set and input it into the fully connected layer; Perform zero padding on the front end of the low-dimensional semantic feature vector set and use the front end as an anchor point; Determining a relative position of a low-dimensional semantic feature vector in the low-dimensional semantic feature vector set relative to the anchor point according to the input frame position; Encoding the relative position and embedding it into the corresponding low-dimensional semantic feature vector; The embedded low-dimensional semantic feature vector is input into the convolution layer for dimensionality upgrading to obtain the relative position of the visual feature with respect to the preset frame of the long video.

7. A long video understanding device, characterized in that: include: An instruction acquisition module, used to acquire the user's question instruction and the long video corresponding to the question instruction; A feature vector set output module, used for outputting a semantic feature vector set based on the visual content of the long video and a pre-trained deep learning network model based on a multi-head self-attention mechanism; A visual feature module, used for merging the semantic feature vectors in the semantic feature vector set to obtain the visual features of the long video; A position determination module, used for inputting the frame position of the long video and the semantic feature vector set into a pre-trained U-shaped neural network model to determine the relative position of the visual feature with respect to a preset frame of the long video; A feature generation module, used for adding the relative position to the corresponding visual feature to generate a visual temporal feature; The answer output module is used to input the visual temporal features and the question instructions into a pre-trained long video understanding model, and output the question answer corresponding to the question instruction.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the long video understanding method described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the long video understanding method described in any one of claims 1-6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the long video understanding method according to any one of claims 1 to 6.

Images