Video description method and device and electronic equipment

Abstract

The invention provides a video description method and device and electronic equipment, and the method comprises the steps: obtaining to-be-processed video data, wherein the video data comprises video frames at T moments; t is a positive integer; performing visual coding processing on the video data to obtain a visual feature vector sequence of the video data; the visual feature vector sequence comprises a plurality of visual feature vectors; obtaining a semantic description text matched with each video frame; performing text coding processing on the semantic description text to obtain a semantic feature vector sequence; the semantic feature vector sequences comprise semantic feature vector sequences in one-to-one correspondence with the video frames; and generating a video description text of the video data according to the visual feature vector sequence and the semantic feature vector sequence.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and more specifically, to a video description method, apparatus, and electronic device. Background Technology

[0002] Video captioning is a method that uses natural language to automatically describe video content, and it has attracted widespread attention in the field of computer vision in recent years. However, due to the complexity of video scenes and object interactions, video captioning tasks present certain challenges. These challenges include how to effectively utilize multiple feature vectors or other methods to represent video content, and how to combine visual and linguistic feature vectors to generate more detailed descriptions. Summary of the Invention

[0003] One object of this disclosure is to provide a new technical solution for generating video descriptions.

[0004] According to a first aspect of the present disclosure, a video description method is provided, comprising: Acquire the video data to be processed, which includes T video frames at different times; T is a positive integer. The video data is subjected to visual encoding processing to obtain a sequence of visual feature vectors of the video data; the sequence of visual feature vectors includes multiple visual feature vectors. Obtain the semantic description text that matches each video frame; The semantic description text is subjected to text encoding processing to obtain a semantic feature vector sequence; the semantic feature vector sequence includes a semantic feature vector sequence that corresponds one-to-one with each video frame. Based on the visual feature vector sequence and the semantic feature vector sequence, video description text for the video data is generated.

[0005] Optionally, the step of performing visual encoding processing on the video data to obtain a sequence of visual feature vectors for the video data includes: Each keyframe of the video data is subjected to a first visual encoding process to obtain a scene feature vector sequence. The scene feature vector sequence includes a scene feature vector that corresponds one-to-one with each keyframe. The scene feature vector represents the scene information of the corresponding keyframe. The scene feature vector sequence is used as the visual feature vector sequence.

[0006] Optionally, the step of performing visual encoding processing on the video data to obtain a sequence of visual feature vectors for the video data further includes: A second visual encoding process is performed on the video frames of each time window of the video data to obtain a motion feature vector sequence. The motion feature vector sequence includes a motion feature vector that corresponds one-to-one with each time window, and the motion feature vector represents the motion information of the corresponding time window. The motion feature vector sequence is also used as the visual feature vector sequence.

[0007] Optionally, the step of performing visual encoding processing on the video data to obtain a sequence of visual feature vectors for the video data further includes: The video frames at T time points of the video data are subjected to third-vision encoding processing to obtain a target feature vector sequence. The target feature vector sequence includes at least one target feature vector corresponding to each video frame. The target feature vector is used to represent the target information of the corresponding video frame. The visual feature vector sequence is determined based on the target feature vector sequence.

[0008] Optionally, determining the visual feature vector sequence based on the target feature vector sequence includes: Based on the scene feature vector sequence, the motion feature vector sequence, and the target feature vector sequence, a key target feature vector sequence is determined. The key target feature vector sequence includes a key target feature vector corresponding to each video frame. The key target feature vector is used to represent the information of key targets selected from the targets in the corresponding video frame. A relation feature vector sequence is obtained based on the key target feature vector sequence. The relation feature vector sequence includes a relation feature vector corresponding to each video frame. The relation feature vector is used to represent the relationship between key targets in the corresponding video frames. The relation feature vector sequence and the key target feature vector are used as the visual feature vector sequence.

[0009] Optionally, based on the visual feature vector sequence and the semantic feature vector sequence, a video description text for the video data is generated, including: The visual feature vector sequence is aligned with the semantic feature vector sequence to obtain the corresponding visual alignment feature vector sequence; wherein, the visual alignment feature vector sequence includes visual alignment feature vectors that correspond one-to-one with the video frames at the T time points; The semantic feature vector sequence and the visual alignment feature vector sequence are fused together to obtain a semantic-visual fusion feature vector sequence. The semantic-visual fusion feature vector sequence is decoded to obtain the video description text.

[0010] Optionally, aligning the visual feature vector sequence with the semantic feature vector sequence to obtain a corresponding visually aligned feature vector sequence includes: Obtain the semantic feature vector corresponding to the t-th time step of the semantic feature vector sequence; where t is a positive integer less than or equal to T; Based on each visual feature vector in the visual feature vector sequence and the semantic feature vector corresponding to the t-th time in the semantic feature vector sequence, the first weight of the corresponding visual feature vector for the t-th time is obtained; Based on each visual feature vector in the visual feature vector sequence and the first weight of the corresponding visual feature vector at time t, the visual alignment feature vector corresponding to time t in the visual alignment feature vector sequence is obtained.

[0011] Optionally, the step of fusing the semantic feature vector sequence and the visual alignment feature vector sequence to obtain a semantic-visual fusion feature vector sequence includes: The semantic visual splicing vector sequence is obtained by concatenating the semantic feature vector sequence and the visual alignment feature vector sequence at each time step in the semantic feature vector sequence. The semantic visual splicing vectors at T time points in the semantic visual splicing vector sequence are used to obtain the second weight for each time point; Based on the semantic visual splicing vector and the second weight at each time step, the semantic visual fusion feature vector at the corresponding time step in the semantic visual fusion feature vector sequence is obtained.

[0012] According to a second aspect of this disclosure, a video description apparatus is provided, comprising: The video acquisition module is used to acquire video data to be processed, the video data including T video frames at different times; T is a positive integer; A visual encoding module is used to perform visual encoding processing on the video data to obtain a sequence of visual feature vectors of the video data; the sequence of visual feature vectors includes multiple visual feature vectors. The text acquisition module is used to acquire semantic description text that matches each video frame; The text encoding module is used to perform text encoding processing on the semantic description text to obtain a semantic feature vector sequence; the semantic feature vector sequence includes a semantic feature vector sequence that corresponds one-to-one with each video frame. The text generation module is used to generate video description text for the video data based on the visual feature vector sequence and the semantic feature vector sequence.

[0013] According to a third aspect of this disclosure, an electronic device is provided, including a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the method as described in the first aspect of this disclosure under the control of the computer program.

[0014] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect of this disclosure.

[0015] Through the embodiments of this disclosure, video description text can be generated based on multimodal visual feature vectors and semantic feature vectors, which can improve the correlation between visual and textual features, and thus improve the accuracy of the generated video description text.

[0016] Other feature vectors and advantages of the present invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0018] Figure 1 This is a block diagram illustrating the hardware configuration of an electronic device that can implement embodiments of the present disclosure; Figure 2 This is a flowchart of a video description method according to an embodiment of the present disclosure; Figure 3 This is a schematic diagram of the structure of a video description model according to an embodiment of the present disclosure; Figure 4 This is a schematic diagram of the structure of a target perception module and a scene relationship reasoning module according to an embodiment of the present disclosure; Figure 5 This is a block diagram of a video description apparatus according to an embodiment of the present disclosure; Figure 6 This is a block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0019] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0020] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0021] Techniques, methods, and apparatus known to those skilled in the art in the relevant field may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

[0022] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0023] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0024] <Hardware Configuration> Figure 1 This is a block diagram illustrating the hardware configuration of an electronic device 1000 that can implement embodiments of the present disclosure.

[0025] Electronic device 1000 can be a portable computer, desktop computer, mobile phone, tablet computer, virtual display device, AR glasses, etc. For example... Figure 1 As shown, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, etc. The processor 1100 may be a CPU, a microprocessor (MCU), etc. The memory 1200 may include, for example, ROM (Read-Only Memory), RAM (Random Access Memory), or non-volatile memory such as a hard disk. The interface device 1300 may include, for example, a USB interface, a headphone jack, etc. The communication device 1400 may be capable of wired or wireless communication, specifically including Wi-Fi communication, Bluetooth communication, 2G / 3G / 4G / 5G communication, etc. The display device 1500 may be, for example, an LCD screen, a touch screen, etc. The input device 1600 may include, for example, a touch screen, a keyboard, motion input, etc. Users can input / output voice information through the speaker 1700 and the microphone 1800.

[0026] Figure 1 The electronic devices shown are merely illustrative and in no way intended to limit this disclosure, its application, or use. In embodiments applied to this disclosure, the memory 1200 of the electronic device 1000 is used to store instructions for controlling the processor 1100 to operate to perform any of the methods provided in the embodiments of this disclosure. Those skilled in the art will understand that, although... Figure 1The electronic device 1000 is shown with multiple devices shown; however, this disclosure may relate only to some of these devices. For example, electronic device 1000 may only relate to processor 1100 and memory 1200. Those skilled in the art can design instructions based on the schemes disclosed herein. How the instructions control the processor to operate is well known in the art and will not be described in detail here.

[0027] <Method Implementation> This disclosure provides a video description method, which can be implemented by an electronic device. Specifically, the electronic device may be, for example, as shown below. Figure 1 The electronic device shown is 1000.

[0028] Figure 2 This is a flowchart of a video description method according to an embodiment of the present disclosure.

[0029] In some embodiments, steps S2200 to S2500 of this embodiment may be performed based on a video description model.

[0030] like Figure 2 As shown, the video description method includes the following steps S2100 to S2500: Step S2100: Obtain the video data to be processed. The video data includes T video frames at different times; T is a positive integer.

[0031] In some embodiments, the number of video frames contained in the video data to be processed can be T.

[0032] In some embodiments, if the number of video frames contained in the video data to be processed is less than T, then the video data can be interpolated to obtain T video frames at different times.

[0033] In some embodiments, if the number of video frames in the video data to be processed is greater than T, then some video frames can be removed based on the similarity between video frames at two adjacent time points to obtain T video frames at those time points. Specifically, the similarity of each pair of video frames at adjacent time points can be sorted in descending order. If the number of video frames in the video data to be processed is S (S is greater than T), the pair of video frames at adjacent time points with the highest similarity (ST) can be obtained as target video frame pairs. Any one of the target video frame pairs can be removed to obtain T video frames at those time points.

[0034] Step S2200: Perform visual encoding processing on the video data to obtain a sequence of visual feature vectors for the video data; the sequence of visual feature vectors includes multiple visual feature vectors.

[0035] In this embodiment, the video frames at T time points can be encoded using a visual encoder to obtain a visual encoding vector. Specifically, the video frames at T time points of the video data to be processed can be input into the visual encoder, or the video data to be processed can be input into the visual encoder to obtain a sequence of visual feature vectors of the video data.

[0036] In some embodiments, visual encoding processing is performed on video data to obtain a sequence of visual feature vectors for the video data, including: performing a first visual encoding process on each keyframe of the video data to obtain a sequence of scene feature vectors, wherein the sequence of scene feature vectors includes a scene feature vector corresponding one-to-one with each keyframe, and the scene feature vector represents the scene information of the corresponding keyframe; and using the sequence of scene feature vectors as a sequence of visual feature vectors. For example, the scene feature vector can represent "in the wild".

[0037] In this embodiment, keyframes can be selected from T time-stamped video frames of the video data to be processed, and then the keyframes can be subjected to a first visual encoding process to obtain a scene feature vector sequence. The scene feature vector is a global semantic feature vector of the video data.

[0038] In this embodiment, the keyframes can be first visually encoded using 2DCNN (Two-Dimensional Convolutional Neural Network) to obtain a sequence of scene feature vectors.

[0039] Furthermore, the scene feature vector sequence is input into a bidirectional Bi-LSTM (Bidirectional Long Short-Term Memory) network, which can process sequential data to enhance the temporal nature of the feature vectors.

[0040] In some embodiments, visual encoding processing of video data to obtain visual feature vectors of video data further includes: performing a second visual encoding processing on video frames of each time window of the video data to obtain a motion feature vector sequence, wherein the motion feature vector sequence includes motion feature vectors corresponding one-to-one with each time window, and the motion feature vectors represent motion information of the corresponding time window; and the motion feature vector sequence is also used as a visual feature vector sequence.

[0041] In this embodiment, each time window may include K consecutive moments, where K is a positive integer less than T and greater than 1. K can be preset according to the application scenario or specific requirements. For example, K can be 5.

[0042] Furthermore, the first time window can include video frames from time 1 to time K, the second time window can include video frames from time 2 to time (K+1), the third time window can include video frames from time 3 to time (K+2), and so on, with the last time window including video frames from time (T-K+1) to time T. In other words, it can include T-K+1 time windows, and the motion feature vector sequence includes T-K+1 motion feature vectors.

[0043] In this embodiment, each motion feature vector can represent the motion information of video frames at K time points within a corresponding time window. The motion feature vector is a global semantic feature vector of the video data. For example, a motion feature vector can represent "cycling".

[0044] In this embodiment, the video frames at T time points of the video data to be processed can be subjected to second visual encoding processing based on 3DCNN (Three-Dimensional Convolutional Neural Network) to obtain a sequence of motion feature vectors.

[0045] Furthermore, the motion feature vector sequence is input into a bidirectional Bi-LSTM network, which can process sequential data to enhance the temporal nature of the feature vectors.

[0046] In some embodiments, visual encoding processing of video data to obtain visual feature vectors of video data further includes: performing third visual encoding processing on video frames at T time points of the video data to obtain a target feature vector sequence, wherein the target feature vector sequence includes at least one target feature vector corresponding to each video frame, and the target feature vector is used to represent the target information of the corresponding video frame; and determining a visual feature vector sequence based on the target feature vector sequence.

[0047] In this embodiment, the target is an object in the video frame, and the target feature vector is a local semantic feature vector of the video data.

[0048] Furthermore, a target feature vector can correspond to an object in a video frame, and the number of target feature vectors corresponding to video frames at T time points can be the same. For example, the number of target feature vectors corresponding to each video frame can be L.

[0049] Since frequently appearing targets in videos are often the basis for descriptions, target feature vectors play an indispensable role in generating accurate video description text. To explore the complementarity between global and local information, a third-vision encoding process can be performed on each of the T time-time video frames using R-CNN (Region-based Convolutional Neural Network) to extract the target feature vector for the corresponding video frame.

[0050] In some embodiments, determining the visual feature vector sequence based on the target feature vector sequence may include: using the target feature vector sequence as the visual feature vector sequence.

[0051] In some embodiments, determining the visual feature vector sequence based on the target feature vector sequence may include: determining a key target feature vector sequence based on the scene feature vector sequence, motion feature vector sequence, and target feature vector sequence, wherein the key target feature vector sequence includes a key target feature vector corresponding to each video frame, and the key target feature vector is used to represent information of key targets selected from the targets in the corresponding video frame; obtaining a relationship feature vector sequence based on the key target feature vector sequence, wherein the relationship feature vector sequence includes a relationship feature vector corresponding to each video frame, and the relationship feature vector is used to represent the relationship between key targets in the corresponding video frame; and using the relationship feature vector sequence and the key target feature vector as the visual feature vector sequence.

[0052] In this embodiment, to address the problem of target redundancy in videos causing video description models to struggle to accurately identify key targets, a target perception module can be designed. Specifically, the scene feature vector sequence, motion feature vector sequence, and target feature vector sequence can all be input into the target perception module to obtain the key target feature vector sequence.

[0053] This object perception module employs learnable query vectors to precisely filter the features of all objects in the video scene using attention. By using entity embedding vectors in the description as supervision signals, the video description model can more effectively handle key elements in the video. This not only enhances the video description model's ability to identify key objects but also improves the overall quality of the video description.

[0054] Specifically, in the multimodal feature extraction stage, in addition to the previously extracted global temporal scene feature vector sequence and motion feature vector sequence, a scene graph generation model is used to extract scene graph features from the video data. These scene graph features can include structured target features and relational features. The scene feature vector sequence and motion feature vector sequence are fed into a hierarchical multimodal attention module and fused with the semantic text feature vector generated at the decoder. Simultaneously, the target features of the video scene graph are input into the target perception module, where key targets are identified from the global temporal feature vector of the video through learnable query vectors.

[0055] Design details of the target perception module, such as Figure 4 As shown, the object perception module adopts a Transformer-based encoder-decoder structure. The encoder learns the contextual relationships between all object features in the scene graph, while the decoder uses query vectors embedded in the video temporal feature vectors to mine the latent semantic features of key objects. Inspired by DETR, the decoder's prediction head uses the Hungarian matching algorithm to establish an optimal one-to-one match between the predicted key object and entities in the sentence. Unlike DETR, the query of the decoder in the object perception module is enhanced by the temporal content of the video and supervised by entity embedding vectors in the video description. This allows the video description model to adaptively select key objects based on the video scene. Specifically, firstly, all object features extracted from the video scene graph are aggregated, and then the aggregated features are averaged to obtain initial object features. The encoder of the object perception module maps the input object features to a high-level semantic space, thereby obtaining context-relevant key object features. The output of the target perception module passes through a spatial attention mechanism layer and then enters the relation reasoning module, which consists of three GNNs (Graph Neural Networks): GAT, GATv2, and Graph Transformer. By using adjacent nodes between GNNs, features are better aggregated.

[0056] To address the challenge of multimodal visual-semantic fusion arising from complex dynamic relationships between video targets, a design is incorporated into the video description model as follows: Figure 4 The scene relationship reasoning module shown above uses a graph neural network to perform in-depth analysis of the complex relationship features in the entire video scene graph. Specifically, it can input the key target feature vector sequence into the scene relationship reasoning module to obtain a relationship feature vector sequence.

[0057] In this embodiment, not only is the model's ability to understand the interactive relationships in the video enhanced, but it can also provide a more refined visual representation, enabling the video description model to better understand and describe the video content.

[0058] The relational features of the video scene graph are fed into the relational reasoning module, which uses graph neural networks to capture the dynamic changes in video scene relations, enhances the semantic representation of the visual features of the hierarchical multimodal attention module and the scene target relational reasoning module, and finally generates descriptive statements.

[0059] The object perception module uses learnable query vectors to perform attention filtering on all object features in the scene, and uses entity embedding vectors in the description as supervision, enabling the video description model to generate descriptions of key objects. The scene relationship reasoning module utilizes the relationship parsing capabilities of graph neural networks to deeply analyze the complex relationship features under the entire video scene graph, enhancing the video description model's understanding of scene dynamics and interactive relationships. The object perception module and the scene relationship reasoning module work together to improve the accuracy of the video description model's understanding of the visual scene, thereby ensuring that the fused multimodal features contain the most relevant information within the video, which is beneficial for generating more accurate descriptive text.

[0060] In some embodiments, visual encoding processing is performed on video data to obtain a sequence of visual feature vectors for the video data, including: performing a first visual encoding process on each keyframe of the video data to obtain a sequence of scene feature vectors, wherein the sequence of scene feature vectors includes a scene feature vector corresponding to each keyframe, and the scene feature vector represents the scene information of the corresponding keyframe; performing a second visual encoding process on video frames of each time window of the video data to obtain a sequence of motion feature vectors, wherein the sequence of motion feature vectors includes a motion feature vector corresponding to each time window, and the motion feature vector represents the motion information of the corresponding time window; and using both the sequence of scene feature vectors and the sequence of motion feature vectors as a sequence of visual feature vectors.

[0061] In some embodiments, visual encoding processing is performed on video data to obtain a visual feature vector sequence of the video data, including: performing a first visual encoding process on each keyframe of the video data to obtain a scene feature vector sequence, the scene feature vector sequence including a scene feature vector corresponding one-to-one with each keyframe, the scene feature vector representing the scene information of the corresponding keyframe; performing a third visual encoding process on video frames at T time points of the video data to obtain a target feature vector sequence, the target feature vector sequence including at least one target feature vector corresponding to each video frame, the target feature vector being used to represent the target information of the corresponding video frame; and using both the scene feature vector sequence and the target feature vector sequence as visual feature vector sequences.

[0062] In some embodiments, visual encoding processing is performed on video data to obtain a visual feature vector sequence of the video data, including: performing a second visual encoding processing on video frames of each time window of the video data to obtain a motion feature vector sequence, the motion feature vector sequence including a motion feature vector corresponding to each time window, the motion feature vector representing the motion information of the corresponding time window; performing a third visual encoding processing on video frames of T time points of the video data to obtain a target feature vector sequence, the target feature vector sequence including at least one target feature vector corresponding to each video frame, the target feature vector being used to represent the target information of the corresponding video frame; and using both the motion feature vector sequence and the target feature vector sequence as visual feature vector sequences.

[0063] In some embodiments, visual encoding processing is performed on video data to obtain a visual feature vector sequence of the video data, including: performing a first visual encoding process on each keyframe of the video data to obtain a scene feature vector sequence, the scene feature vector sequence including a scene feature vector corresponding one-to-one with each keyframe, the scene feature vector representing the scene information of the corresponding keyframe; performing a second visual encoding process on video frames of each time window of the video data to obtain a motion feature vector sequence, the motion feature vector sequence including a motion feature vector corresponding one-to-one with each time window, the motion feature vector representing the motion information of the corresponding time window; performing a third visual encoding process on video frames of T time points of the video data to obtain a target feature vector sequence, the target feature vector sequence including at least one target feature vector corresponding to each video frame, the target feature vector being used to represent the target information of the corresponding video frame; and using the scene feature vector sequence, motion feature vector sequence, and target feature vector sequence as a visual feature vector sequence.

[0064] In this embodiment, as Figure 3 As shown, this can be a multi-granularity visual feature vector extraction module based on a video description model, which performs global and local representations of the video data. Specifically, the global representation of the video data includes: performing a first visual encoding process on each keyframe of the video data based on 2DCNN and Bi-LSTM to obtain a scene feature vector sequence. Based on 3DCNN and Bi-LSTM, a second visual encoding process is performed on video frames of each time window of the video data to obtain a sequence of motion feature vectors. Local representation of video data includes: performing third-person vision encoding on T time-stamped video frames to obtain a sequence of target feature vectors. .

[0065] In some embodiments, visual encoding processing is performed on video data to obtain a sequence of visual feature vectors for the video data, including: performing a first visual encoding process on each keyframe of the video data to obtain a sequence of scene feature vectors, wherein the sequence of scene feature vectors includes a scene feature vector corresponding to each keyframe, and the scene feature vector represents the scene information of the corresponding keyframe; performing a second visual encoding process on video frames of each time window of the video data to obtain a sequence of motion feature vectors, wherein the sequence of motion feature vectors includes a motion feature vector corresponding to each time window, and the motion feature vector represents the motion information of the corresponding time window; and performing a third visual encoding process on video frames at T time points of the video data to obtain a sequence of target feature vectors, wherein the target feature vector sequence includes a scene feature vector corresponding to each video frame. At least one target feature vector is obtained, which represents the target information of the corresponding video frame; a key target feature vector sequence is determined based on the scene feature vector sequence, motion feature vector sequence, and target feature vector sequence, which includes a key target feature vector corresponding to each video frame, and the key target feature vector is used to represent the information of key targets selected from the targets of the corresponding video frame; a relationship feature vector sequence is obtained based on the key target feature vector sequence, which includes a relationship feature vector corresponding to each video frame, and the relationship feature vector is used to represent the relationship between key targets in the corresponding video frame; the scene feature vector sequence, motion feature vector sequence, relationship feature vector sequence, and key target feature vector are all used as visual feature vector sequences.

[0066] Step S2300: Obtain the semantic description text that matches each video frame.

[0067] In some embodiments, obtaining the semantic description text of the video frame at time t (t is a positive integer less than or equal to T) may include: constructing the image feature vector of the video frame at time t; obtaining the text feature vector of each description text in a pre-constructed text library; and determining the description text with the highest similarity between the text feature vector and the image feature vector as the semantic description text of the video frame at time t.

[0068] In some embodiments, each video frame may be input into a pre-trained image description model to obtain semantic description text that matches each video frame.

[0069] In some embodiments, such as Figure 3 As shown, this can be a semantic text enhancement module based on a video description model, which processes the semantic description text matching each video frame to obtain a sequence of semantic feature vectors. .

[0070] Step S2400: Perform text encoding processing on the semantic description text to obtain a semantic feature vector sequence; the semantic feature vector sequence includes a semantic feature vector sequence corresponding to each video frame.

[0071] In one example, the text feature vector of the semantic description text of the video frame at time t can be used as the sequence of semantic feature vectors of the video frame at time t.

[0072] In another example, to effectively capture and learn the latent semantics of attention-driven text, a text semantic enhancement module can be set up to process the semantic description text of the input video frames at T time points through a multi-head attention mechanism, thereby obtaining a sequence of semantic feature vectors.

[0073] In this embodiment, the semantic enhancement module performs multi-head self-attention learning on the vector representations of words corresponding to the semantic description text of the video frame at each time moment, calculates the similarity of different semantic groups through attention scores, and filters out the context-related text semantic groups as the semantic feature vector sequence of the video frame at the corresponding time moment.

[0074] Step S2500: Generate video description text for the video data based on the visual feature vector sequence and the semantic feature vector sequence.

[0075] In some embodiments, the visual feature vector sequence includes a scene feature vector sequence and / or a motion feature vector sequence. The number of scene feature vectors contained in the scene feature vector sequence, the number of motion feature vectors contained in the motion feature vector sequence, and the number of semantic feature vectors contained in the semantic feature vector sequence are different. That is, the video frames corresponding to the scene feature vectors in the scene feature vector sequence, the motion feature vectors in the motion feature vector sequence, and the semantic feature vectors in the semantic feature vector sequence are different.

[0076] Therefore, based on the visual feature vector sequence and the semantic feature vector sequence, video description text is generated for the video data, including: aligning the visual feature vector sequence and the semantic feature vector sequence to obtain a corresponding visual alignment feature vector sequence; wherein, the visual alignment feature vector sequence includes visual alignment feature vectors that correspond one-to-one with video frames at T time points; fusing the semantic feature vector sequence and the visual alignment feature vector sequence to obtain a semantic-visual fusion feature vector sequence; and decoding the semantic-visual fusion feature vector sequence to obtain the video description text.

[0077] In the case where the visual feature vector sequence includes at least two of the following: scene feature vector sequence, motion feature vector sequence, and target feature vector sequence, i.e., when there are at least two visual feature vector sequences, or in the embodiment where the visual feature vector sequence includes scene feature vector sequence, motion feature vector sequence, key target feature vector, and relationship feature vector sequence, each visual feature vector sequence may be aligned with the semantic feature vector sequence to obtain a visually aligned feature vector sequence that corresponds one-to-one with each visual feature vector sequence; then the semantic feature vector sequence and the obtained at least two visually aligned feature vector sequences may be fused to obtain a semantic-visual fusion feature vector sequence.

[0078] In an embodiment where the visual feature vector sequence includes a scene feature vector sequence, the scene feature vector sequence and the semantic feature vector sequence may be aligned to obtain a scene-aligned feature vector sequence, which is then used as the visual alignment feature vector sequence; the semantic feature vector sequence and the scene-aligned feature vector sequence may then be fused to obtain a semantic visual vector.

[0079] In embodiments where the visual feature vector sequence includes a motion feature vector sequence, the motion feature vector sequence and the semantic feature vector sequence may be aligned to obtain a motion-aligned feature vector sequence, which is then used as the visual alignment feature vector sequence; the semantic feature vector sequence and the motion-aligned feature vector sequence may then be fused to obtain a semantic visual vector.

[0080] In an embodiment where the visual feature vector sequence includes the target feature vector sequence, the target feature vector sequence and the semantic feature vector sequence may be aligned to obtain a target-aligned feature vector sequence, which is then used as the visual alignment feature vector sequence; the semantic feature vector sequence and the target-aligned feature vector sequence may then be fused to obtain a semantic visual vector.

[0081] In an embodiment where the visual feature vector sequence includes a scene feature vector sequence and a motion alignment feature vector sequence, the scene feature vector sequence and the semantic feature vector sequence may be aligned to obtain a scene alignment feature vector sequence; the motion feature vector sequence and the semantic feature vector sequence may be aligned to obtain a motion alignment feature vector sequence; both the scene alignment feature vector sequence and the motion alignment feature vector sequence may be used as visual alignment feature vector sequences; and the semantic feature vector sequence, the scene alignment feature vector sequence, and the motion alignment feature vector sequence may be fused to obtain a semantic visual vector.

[0082] In an embodiment where the visual feature vector sequence includes a scene feature vector sequence and a target feature vector sequence, the scene feature vector sequence and the semantic feature vector sequence may be aligned to obtain a scene-aligned feature vector sequence; the target feature vector sequence and the semantic feature vector sequence may be aligned to obtain a target-aligned feature vector sequence; both the scene-aligned feature vector sequence and the target-aligned feature vector sequence may be used as visual alignment feature vector sequences; and then the semantic feature vector sequence, the scene-aligned feature vector sequence, and the target-aligned feature vector sequence may be fused to obtain a semantic visual vector.

[0083] In embodiments where the visual feature vector sequence includes a motion feature vector sequence and a target feature vector sequence, the motion feature vector sequence and the semantic feature vector sequence may be aligned to obtain a motion-aligned feature vector sequence; the target feature vector sequence and the semantic feature vector sequence may be aligned to obtain a target-aligned feature vector sequence; both the motion-aligned feature vector sequence and the target-aligned feature vector sequence may be used as visual alignment feature vector sequences; and the semantic feature vector sequence, the motion-aligned feature vector sequence, and the target-aligned feature vector sequence may be fused to obtain a semantic visual vector.

[0084] In an embodiment where the visual feature vector sequence includes a scene feature vector sequence, a motion feature vector sequence, and a target feature vector sequence, the scene feature vector sequence and the semantic feature vector sequence may be aligned to obtain a scene-aligned feature vector sequence; the motion feature vector sequence and the semantic feature vector sequence may be aligned to obtain a motion-aligned feature vector sequence; the target feature vector sequence and the semantic feature vector sequence may be aligned to obtain a target-aligned feature vector sequence; the scene-aligned feature vector sequence, the motion-aligned feature vector sequence, and the target-aligned feature vector sequence may all be used as visual alignment feature vector sequences; and the semantic feature vector sequence, the scene-aligned feature vector sequence, the motion-aligned feature vector sequence, and the target-aligned feature vector sequence may be fused to obtain a semantic visual vector.

[0085] In embodiments where the visual feature vector sequence includes a scene feature vector sequence, a motion feature vector sequence, a key target feature vector sequence, and a relationship feature vector sequence, the following steps can be taken: aligning the scene feature vector sequence and the semantic feature vector sequence to obtain a scene-aligned feature vector sequence; aligning the motion feature vector sequence and the semantic feature vector sequence to obtain a motion-aligned feature vector sequence; aligning the key target feature vector sequence and the semantic feature vector sequence to obtain a key target-aligned feature vector sequence; aligning the relationship feature vector sequence and the semantic feature vector sequence to obtain a relationship-aligned feature vector sequence; using the scene-aligned feature vector sequence, the motion-aligned feature vector sequence, the key target-aligned feature vector sequence, and the relationship-aligned feature vector sequence as the visual alignment feature vector sequence; and then fusing the semantic feature vector sequence, the scene-aligned feature vector sequence, the motion-aligned feature vector sequence, the key target-aligned feature vector sequence, and the relationship-aligned feature vector sequence to obtain a semantic visual vector.

[0086] In some embodiments, Global Semantic Attention (GSA) can be used to align scene feature vector sequences and semantic feature vector sequences, as well as motion feature vector sequences and semantic feature vector sequences. Specifically, GSA uses the semantic feature vector sequence as a query to guide the generation of scene-aligned feature vectors and motion-aligned feature vectors. For any given semantic feature vector sequence, the text-related scene-aligned feature vector is calculated by dynamically summarizing the temporal scene feature vectors.

[0087] In some embodiments, Local Semantic Attention (LSA) can be used to align the target feature vector sequence and the semantic feature vector sequence. Local spatial region feature vectors contain meaningful target information, which is often the basis of video description. Local Semantic Attention tends to model the underlying target semantics through fine-grained alignment.

[0088] In some embodiments, such as Figure 3 As shown, it can be a visual-text layer based on a hierarchical multimodal attention module of a video description model, which processes the scene feature vector sequence. and semantic feature vector sequence Alignment processing is performed to obtain a sequence of scene alignment feature vectors. The sequence of motion feature vectors and semantic feature vector sequence Alignment processing is performed to obtain a sequence of motion-aligned feature vectors. , target feature vector sequence and semantic feature vector sequence Alignment processing is performed to obtain the target alignment feature vector sequence. .

[0089] In some embodiments, the visual feature vector sequence is aligned with the semantic feature vector sequence to obtain the corresponding visual alignment feature vector sequence, including: obtaining the semantic feature vector corresponding to the t-th time step of the semantic feature vector sequence; where t is a positive integer less than or equal to T; obtaining the first weight of the corresponding visual feature vector for the t-th time step based on each visual feature vector of the visual feature vector sequence and the semantic feature vector corresponding to the t-th time step of the semantic feature vector sequence; and obtaining the visual alignment feature vector corresponding to the t-th time step of the visual alignment feature vector sequence based on each visual feature vector of the visual feature vector sequence and the first weight of the corresponding visual feature vector for the t-th time step.

[0090] In this embodiment, for any visual feature vector sequence among the scene feature vector sequence, motion feature vector sequence, and target feature vector sequence, the visual alignment feature vector at each of the T time points can be determined sequentially based on the semantic feature vector sequence and the visual feature vectors contained in the visual feature vector sequence. Then, the visual alignment feature vectors at the T time points are sorted according to the time order to obtain the visual alignment feature vector sequence.

[0091] In an embodiment where the visual feature vector sequence includes a scene feature vector sequence, the first weight of the corresponding scene feature vector at time t is obtained based on each scene feature vector in the scene feature vector sequence and the semantic feature vector corresponding to the semantic feature vector sequence at time t; and the scene alignment feature vector corresponding to the scene alignment feature vector sequence at time t is obtained based on each scene feature vector in the scene feature vector sequence and the first weight of the corresponding scene feature vector at time t.

[0092] Specifically, the generation process of the scene alignment feature vector at time t can be represented by the following formula:

[0093]

[0094]

[0095] in, , , , These are parameter values ​​obtained through pre-training. It is the feature vector of the i-th scene. For the first weight at time t, N is the number of scene feature vectors in the scene feature vector sequence. It is the scene alignment feature vector at time t. It is the sequence of semantic feature vectors at time t.

[0096] In an embodiment where the visual feature vector sequence includes a motion feature vector sequence, a first weight for the corresponding motion feature vector at time t is obtained based on each motion feature vector in the motion feature vector sequence and the semantic feature vector corresponding to time t in the semantic feature vector sequence; and a motion alignment feature vector corresponding to time t is obtained based on each motion feature vector in the motion feature vector sequence and the first weight for the corresponding motion feature vector at time t.

[0097] Specifically, the generation process of the motion alignment feature vector at time t can be represented by the following formula:

[0098]

[0099]

[0100] in, , , , These are parameter values ​​obtained through pre-training. It is the i-th motion feature vector For the first weight at time t, T-K+1 is the number of motion feature vectors in the motion feature vector sequence. It is the motion alignment feature vector at time t. It is the sequence of semantic feature vectors at time t.

[0101] In an embodiment where the visual feature vector sequence includes a target feature vector sequence, the first weight of the target feature vector sequence at time t is obtained based on the target feature vector at time t corresponding to the target feature vector sequence and the semantic feature vector at time t corresponding to the semantic feature vector sequence; and the target alignment feature vector at time t is obtained based on the target feature vector sequence and the first weight at time t.

[0102] Specifically, the generation process of the target alignment feature vector at time t can be represented by the following formula:

[0103]

[0104]

[0105] in, , , , These are parameter values ​​obtained through pre-training. It is the i-th target feature vector For the first weight at time t, T*L is the number of target feature vectors in the target feature vector sequence, and L is the number of target feature vectors corresponding to a video frame. It is the target alignment feature vector at time t. It is the sequence of semantic feature vectors at time t.

[0106] This embodiment enables the alignment of any visual feature vector sequence with a semantic feature vector sequence, resulting in a visually aligned feature vector sequence with the same length as the semantic feature vector sequence, facilitating subsequent fusion processing.

[0107] In an embodiment where the visual feature vector sequence includes a target feature vector sequence, since the target feature vector sequence includes at least one target feature vector corresponding to each video frame, and the semantic feature vector sequence includes a semantic feature vector sequence corresponding to each video frame, the target feature vector sequence can be used as a target alignment feature vector sequence.

[0108] In an embodiment where the visual feature vector sequence includes a key target feature vector sequence and a relation feature vector sequence, since the key target feature vector sequence includes key target feature vectors corresponding to each video frame, the relation feature vector sequence includes relation feature vectors corresponding to each video frame, and the semantic feature vector sequence includes semantic feature vector sequences that correspond one-to-one with each video frame, the key target feature vector sequence can be used as a key target alignment feature vector sequence, and the relation feature vector sequence can be used as a relation alignment feature vector sequence.

[0109] In this embodiment, by designing a text semantic enhancement module and a hierarchical multimodal attention module in the video description model, the relevance between vision and text can be improved.

[0110] The hierarchical multimodal attention module uses text semantic groups as guidance. First, it aligns visual feature vectors of different granularities to obtain a sequence of context-related visual alignment feature vectors. Then, it performs multimodal fusion of the rich sequence of visual alignment feature vectors and semantic feature vectors, which can reduce the semantic gap between vision and text.

[0111] In some embodiments, the semantic feature vector sequence and the visual alignment feature vector sequence are fused to obtain a semantic-visual fusion feature vector sequence, including: concatenating the semantic feature vector sequence at each time step in the semantic feature vector sequence and the visual alignment feature vector at the corresponding time step in the visual alignment feature vector sequence to obtain a semantic-visual concatenated vector sequence; obtaining a second weight for each time step from the semantic-visual concatenated vectors at T time steps in the semantic-visual concatenated vector sequence; and obtaining the semantic-visual fusion feature vector at the corresponding time step in the semantic-visual fusion feature vector sequence based on the semantic-visual concatenated vector at each time step and the second weight.

[0112] In this embodiment, semantic fusion attention can be used to jointly model the correlation between global-local multimodal visual alignment feature vector sequences. Since most video descriptions contain information about the scene, the target, and the target's motion, the semantic fusion attention module can take three visual alignment feature vector sequences related to textual semantics as input. To enhance the fusion of visual information at different granularities, the semantic feature vector sequence is also fused with these three visual alignment feature vector sequences.

[0113] In this embodiment, an LSTM (Long Short-Term Memory) network can be used as a language generation decoder to decode the visual semantic fusion feature vector sequence to obtain the video description text.

[0114] Specifically, the process of concatenating the semantic feature vector sequence at time t and the visual alignment feature vector to obtain the visual semantic concatenated feature vector at time t in the visual semantic concatenated feature vector sequence can be represented by the following formula:

[0115] in, This represents the visual semantic concatenation feature vector at time t. It is the scene alignment feature vector at time t. It is the motion alignment feature vector at time t. It is the target alignment feature vector at time t. It is the sequence of semantic feature vectors at time t. Indicates splicing.

[0116] Specifically, the generation process of the semantic-visual fusion feature vector at time t can be represented by the following formula:

[0117]

[0118]

[0119] in, It is the semantic fusion feature vector at time t. , , , These are parameter values ​​obtained through pre-training. It is the i-th visual semantic concatenation feature vector For the second weight at time t, T is the number of visual semantic concatenation feature vectors in the visual semantic concatenation feature vector sequence. This represents the historical hidden state of the LSTM network at time t-1.

[0120] In some embodiments, such as Figure 3 As shown, the vision-vision layer of the hierarchical multimodal attention module based on the video description model aligns the scene-to-visual feature vector sequence. Motion alignment feature vector sequence Target alignment feature vector sequence and semantic feature vector sequence The sequence of visual semantic concatenated feature vectors is obtained by concatenating the vectors. Then, the visual semantic concatenated feature vector at time t in the sequence is fused using the semantic fusion attention module at the visual-visual layer to generate the semantic-visual fusion feature vector at time t. .

[0121] In some embodiments, such as Figure 3 As shown, the decoder based on the video description model decodes the semantic-visual fusion feature vector sequence to obtain the video description text.

[0122] Through the embodiments of this disclosure, video description text can be generated based on multimodal visual feature vectors and semantic feature vectors, which can improve the correlation between visual and textual features, and thus improve the accuracy of the generated video description text.

[0123] In this embodiment, any feature vector may include one or more features, and no limitation is made here.

[0124] <Device Embodiment> This disclosure provides a video description device, such as... Figure 5 As shown, the video description device 5000 includes a video acquisition module 5100, a visual encoding module 5200, a text acquisition module 5300, a text encoding module 5400, and a text generation module 5500.

[0125] The video acquisition module 5100 is used to acquire video data to be processed, which includes T video frames at different times; T is a positive integer.

[0126] The visual encoding module 5200 is used to perform visual encoding processing on the video data to obtain a visual feature vector sequence of the video data; the visual feature vector sequence includes multiple visual feature vectors.

[0127] The text acquisition module 5300 is used to acquire semantic description text that matches each video frame.

[0128] The text encoding module 5400 is used to perform text encoding processing on the semantic description text to obtain a semantic feature vector sequence; the semantic feature vector sequence includes a semantic feature vector sequence that corresponds one-to-one with each video frame.

[0129] The text generation module 5500 is used to generate video description text for the video data based on the visual feature vector sequence and the semantic feature vector sequence.

[0130] In some embodiments, the step of performing visual encoding processing on the video data to obtain a sequence of visual feature vectors for the video data includes: Each keyframe of the video data is subjected to a first visual encoding process to obtain a scene feature vector sequence. The scene feature vector sequence includes a scene feature vector that corresponds one-to-one with each keyframe. The scene feature vector represents the scene information of the corresponding keyframe. The scene feature vector sequence is used as the visual feature vector sequence.

[0131] In some embodiments, the step of performing visual encoding processing on the video data to obtain a sequence of visual feature vectors for the video data further includes: A second visual encoding process is performed on the video frames of each time window of the video data to obtain a motion feature vector sequence. The motion feature vector sequence includes a motion feature vector that corresponds one-to-one with each time window, and the motion feature vector represents the motion information of the corresponding time window. The motion feature vector sequence is also used as the visual feature vector sequence.

[0132] In some embodiments, the step of performing visual encoding processing on the video data to obtain a sequence of visual feature vectors for the video data further includes: The video frames at T time points of the video data are subjected to third-vision encoding processing to obtain a target feature vector sequence. The target feature vector sequence includes at least one target feature vector corresponding to each video frame. The target feature vector is used to represent the target information of the corresponding video frame. The visual feature vector sequence is determined based on the target feature vector sequence.

[0133] In some embodiments, determining the visual feature vector sequence based on the target feature vector sequence includes: Based on the scene feature vector sequence, the motion feature vector sequence, and the target feature vector sequence, a key target feature vector sequence is determined. The key target feature vector sequence includes a key target feature vector corresponding to each video frame. The key target feature vector is used to represent the information of key targets selected from the targets in the corresponding video frame. A relation feature vector sequence is obtained based on the key target feature vector sequence. The relation feature vector sequence includes a relation feature vector corresponding to each video frame. The relation feature vector is used to represent the relationship between key targets in the corresponding video frames. The relation feature vector sequence and the key target feature vector are used as the visual feature vector sequence.

[0134] In some embodiments, generating video description text for the video data based on the visual feature vector sequence and the semantic feature vector sequence includes: The visual feature vector sequence is aligned with the semantic feature vector sequence to obtain the corresponding visual alignment feature vector sequence; wherein, the visual alignment feature vector sequence includes visual alignment feature vectors that correspond one-to-one with the video frames at the T time points; The semantic feature vector sequence and the visual alignment feature vector sequence are fused together to obtain a semantic-visual fusion feature vector sequence. The semantic-visual fusion feature vector sequence is decoded to obtain the video description text.

[0135] In some embodiments, aligning the visual feature vector sequence with the semantic feature vector sequence to obtain a corresponding visually aligned feature vector sequence includes: Obtain the semantic feature vector corresponding to the t-th time step of the semantic feature vector sequence; where t is a positive integer less than or equal to T; Based on each visual feature vector in the visual feature vector sequence and the semantic feature vector corresponding to the t-th time in the semantic feature vector sequence, the first weight of the corresponding visual feature vector for the t-th time is obtained; Based on each visual feature vector in the visual feature vector sequence and the first weight of the corresponding visual feature vector at time t, the visual alignment feature vector corresponding to time t in the visual alignment feature vector sequence is obtained.

[0136] In some embodiments, the fusion processing of the semantic feature vector sequence and the visual alignment feature vector sequence to obtain a semantic-visual fusion feature vector sequence includes: The semantic visual splicing vector sequence is obtained by concatenating the semantic feature vector sequence and the visual alignment feature vector sequence at each time step in the semantic feature vector sequence. The semantic visual splicing vectors at T time points in the semantic visual splicing vector sequence are used to obtain the second weight for each time point; Based on the semantic visual splicing vector and the second weight at each time step, the semantic visual fusion feature vector at the corresponding time step in the semantic visual fusion feature vector sequence is obtained.

[0137] <Electronic Device Examples> This embodiment provides an electronic device, which in one aspect may include the aforementioned video description device 5000.

[0138] On the other hand, such as Figure 6 As shown, the electronic device 6000 may include a processor 6100 and a memory 6200. The memory 6200 is used to store computer programs, and the processor 6100 is used to control the electronic device to execute the methods of any embodiment of this disclosure under the control of the computer programs.

[0139] <Example of a readable storage medium> This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the methods described in any of the method embodiments of this disclosure.

[0140] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0141] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0142] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0143] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of the present invention.

[0144] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0145] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0146] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0148] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A video description method, characterized in that, include: Acquire the video data to be processed, which includes T video frames at different times; T is a positive integer. The video data is subjected to visual encoding processing to obtain a sequence of visual feature vectors of the video data; the sequence of visual feature vectors includes multiple visual feature vectors. Obtain the semantic description text that matches each video frame; The semantic description text is subjected to text encoding processing to obtain a semantic feature vector sequence; the semantic feature vector sequence includes a semantic feature vector sequence that corresponds one-to-one with each video frame. Based on the visual feature vector sequence and the semantic feature vector sequence, video description text for the video data is generated.

2. The method according to claim 1, characterized in that, The step of performing visual encoding processing on the video data to obtain a sequence of visual feature vectors for the video data includes: Each keyframe of the video data is subjected to a first visual encoding process to obtain a scene feature vector sequence. The scene feature vector sequence includes a scene feature vector that corresponds one-to-one with each keyframe. The scene feature vector represents the scene information of the corresponding keyframe. The scene feature vector sequence is used as the visual feature vector sequence.

3. The method according to claim 2, characterized in that, The step of performing visual encoding processing on the video data to obtain a sequence of visual feature vectors for the video data further includes: A second visual encoding process is performed on the video frames of each time window of the video data to obtain a motion feature vector sequence. The motion feature vector sequence includes a motion feature vector that corresponds one-to-one with each time window, and the motion feature vector represents the motion information of the corresponding time window. The motion feature vector sequence is also used as the visual feature vector sequence.

4. The method according to claim 3, characterized in that, The step of performing visual encoding processing on the video data to obtain a sequence of visual feature vectors for the video data further includes: The video frames at T time points of the video data are subjected to third-vision encoding processing to obtain a target feature vector sequence. The target feature vector sequence includes a target feature vector corresponding to each video frame. The target feature vector is used to represent the target information of the corresponding video frame. The visual feature vector sequence is determined based on the target feature vector sequence.

5. The method according to claim 4, characterized in that, Determining the visual feature vector sequence based on the target feature vector sequence includes: Based on the scene feature vector sequence, the motion feature vector sequence, and the target feature vector sequence, a key target feature vector sequence is determined. The key target feature vector sequence includes a key target feature vector corresponding to each video frame. The key target feature vector is used to represent the information of key targets selected from the targets in the corresponding video frame. A relation feature vector sequence is obtained based on the key target feature vector sequence. The relation feature vector sequence includes a relation feature vector corresponding to each video frame. The relation feature vector is used to represent the relationship between key targets in the corresponding video frames. The relation feature vector sequence and the key target feature vector are used as the visual feature vector sequence.

6. The method according to claim 1, characterized in that, Based on the visual feature vector sequence and the semantic feature vector sequence, video description text for the video data is generated, including: The visual feature vector sequence is aligned with the semantic feature vector sequence to obtain the corresponding visual alignment feature vector sequence; wherein, the visual alignment feature vector sequence includes visual alignment feature vectors that correspond one-to-one with the video frames at the T time points; The semantic feature vector sequence and the visual alignment feature vector sequence are fused together to obtain a semantic-visual fusion feature vector sequence. The semantic-visual fusion feature vector sequence is decoded to obtain the video description text.

7. The method according to claim 6, characterized in that, The step of aligning the visual feature vector sequence with the semantic feature vector sequence to obtain the corresponding visually aligned feature vector sequence includes: Obtain the semantic feature vector corresponding to the t-th time step of the semantic feature vector sequence; where t is a positive integer less than or equal to T; Based on each visual feature vector in the visual feature vector sequence and the semantic feature vector corresponding to the t-th time in the semantic feature vector sequence, the first weight of the corresponding visual feature vector for the t-th time is obtained; Based on each visual feature vector in the visual feature vector sequence and the first weight of the corresponding visual feature vector at time t, the visual alignment feature vector corresponding to time t in the visual alignment feature vector sequence is obtained.

8. The method according to claim 6, characterized in that, The process of fusing the semantic feature vector sequence and the visual alignment feature vector sequence to obtain a semantic-visual fusion feature vector sequence includes: The semantic visual splicing vector sequence is obtained by concatenating the semantic feature vector sequence and the visual alignment feature vector sequence at each time step in the semantic feature vector sequence. The semantic visual splicing vectors at T time points in the semantic visual splicing vector sequence are used to obtain the second weight for each time point; Based on the semantic visual splicing vector and the second weight at each time step, the semantic visual fusion feature vector at the corresponding time step in the semantic visual fusion feature vector sequence is obtained.

9. A video description device, characterized in that, include: The video acquisition module is used to acquire video data to be processed, the video data including video frames at T time points; T is a positive integer; A visual encoding module is used to perform visual encoding processing on the video data to obtain a sequence of visual feature vectors of the video data; the sequence of visual feature vectors includes multiple visual feature vectors. The text acquisition module is used to acquire semantic description text that matches each video frame; The text encoding module is used to perform text encoding processing on the semantic description text to obtain a semantic feature vector sequence; the semantic feature vector sequence includes a semantic feature vector sequence that corresponds one-to-one with each video frame. The text generation module is used to generate video description text for the video data based on the visual feature vector sequence and the semantic feature vector sequence.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory being used to store a computer program, and the processor being used, under the control of the computer program, to execute the method as described in any one of claims 1 to 8.

Images