Efficient fine-grained video dialogue generation method based on cascaded spatio-temporal attention network

Abstract

The embodiment of the application relates to the technical field of video dialogue generation, and particularly relates to an efficient fine-grained video dialogue generation method based on a cascaded space-time attention network, which comprises the following steps: inputting a to-be-processed video into a visual Transformer model to obtain visual semantic features; inputting a dialogue context corresponding to the to-be-processed video into a BERT model to obtain dialogue context semantic features; utilizing a cascaded space-time attention network, obtaining a video segment set and a spatial region set related to the dialogue context based on the dialogue context semantic features and the visual semantic features, and obtaining video space-time features based on the two sets; utilizing an iterative cross-modal cross-attention network, obtaining cross-modal interaction semantic features based on the dialogue context and the video space-time features; and generating a dialogue reply based on the cross-modal interaction semantic features, so that the semantic association between the video and the dialogue context can be accurately captured, and a high-quality dialogue reply can be generated.

Description

Technical Field

[0001] The embodiments of this application relate to the field of video dialogue generation technology, and in particular to an efficient fine-grained video dialogue generation method based on cascaded spatiotemporal attention networks. Background Technology

[0002] Multimodal semantic understanding and generation is an important research direction in the field of artificial intelligence, and many cutting-edge applications have been developed, such as visual question answering systems and video summarizing systems. In recent years, research on video dialogue systems has received widespread attention. Based on given video content and multi-turn dialogue context, video dialogue systems need to generate dialogue responses that conform to both visual and textual semantic information, taking into account both the video content and the multi-turn dialogue context, according to the current dialogue input.

[0003] Research on video dialogue systems faces many challenges: First, dialogue questions are often associated with multiple concepts at different granularities in the video, such as spatial relationships between objects and spatiotemporal relationships between events. Such complex relationships require joint reasoning to capture accurately. Second, there is a huge semantic gap between video and text modalities. Comprehensive cross-modal semantic alignment is crucial for accurately understanding the semantic information of the video based on the dialogue context.

[0004] Researchers have explored video dialogue generation technology. Initial studies utilized pre-trained 3D convolutional neural networks, simple recurrent neural networks, or Transformer models to extract cross-modal features, but these methods failed to accurately capture the complex spatiotemporal information in videos. Subsequent research proposed graph-based video dialogue generation methods, modeling the spatiotemporal semantics of videos as graph structures and using graph neural networks for cross-modal semantic reasoning, or employing graph walking methods to capture semantic relationships between different modalities. Other researchers have combined pre-trained language models to represent video and text modalities as unified sequence representations, modeling video dialogue tasks as sequence prediction tasks to generate response content.

[0005] However, while the above methods have addressed some of the challenges faced by video dialogue systems, they still have some problems: First, these methods process all video content without considering the problem of semantic redundancy. A large number of unrelated video frames and spatial regions can cause significant interference to the accurate localization of visual semantics and cross-modal semantic associations. Second, these methods usually use standard cross-attention methods to capture semantic associations between different modalities, ignoring the complexity of semantic concepts between different modalities and making it difficult to accurately capture spatiotemporal associations at different granularities. Summary of the Invention

[0006] The main objective of the embodiments of this application is to propose an efficient fine-grained video dialogue generation method based on a cascaded spatiotemporal attention network, which aims to achieve accurate localization of video features, thereby accurately capturing the correlation between video semantics and dialogue context semantics, and generating high-quality dialogue responses.

[0007] To achieve the above objectives, embodiments of this application provide an efficient fine-grained video dialogue generation method based on a cascaded spatiotemporal attention network, comprising the following steps: S1, inputting the video to be processed into a pre-trained visual Transformer model to obtain the visual semantic features output by the visual Transformer model; wherein, the visual Transformer model divides the video to be processed into L equal-length video segments, each video segment is divided into M video frames, each video frame is divided into N spatial regions, and L, M, and N are all integers greater than 0; S2, inputting the dialogue context corresponding to the video to be processed into a pre-trained BERT model to obtain the output of the BERT model. S3, using a pre-trained cascaded spatiotemporal attention network, relevant matching is performed based on the dialogue context semantic features and the visual semantic features to locate the set of video segments most relevant to the dialogue context in the temporal dimension and the set of spatial regions most relevant to the dialogue context in the spatial dimension, and video spatiotemporal features are obtained based on the set of video segments and the set of spatial regions; S4, using a pre-trained iterative cross-modal cross-attention network, cross-modal interaction semantic features are obtained based on the dialogue context corresponding to the video to be processed and the video spatiotemporal features; S5, using a pre-trained cross-modal response decoder, a dialogue response corresponding to the video to be processed is generated based on the cross-modal interaction semantic features.

[0008] To achieve the above objectives, embodiments of this application also provide an efficient fine-grained video dialogue generation system based on a cascaded spatiotemporal attention network, comprising: an acquisition module, a visual Transformer model, a BERT model, a cascaded spatiotemporal attention network, an iterative cross-modal cross-attention network, and a cross-modal response decoder; the acquisition module is used to acquire the video to be processed and the dialogue context corresponding to the video to be processed, input the video to be processed into the visual Transformer model, and input the dialogue context into the BERT model and the iterative cross-modal cross-attention network; the visual Transformer model is used to extract the visual semantic features of the video to be processed; wherein, the visual Transformer model divides the video to be processed into L video segments of equal length, each video segment The video is divided into M frames, and each video frame is further divided into N spatial regions, where L, M, and N are all integers greater than 0. The BERT model is used to extract the dialogue context semantic features of the dialogue context. The cascaded spatiotemporal attention network is used to perform correlation matching based on the dialogue context semantic features and the visual semantic features, locating the set of video segments related to the dialogue context in the temporal dimension and the set of spatial regions related to the dialogue context in the spatial dimension, and obtaining video spatiotemporal features based on the set of video segments and the set of spatial regions. The iterative cross-modal cross-attention network is used to obtain cross-modal interaction semantic features based on the dialogue context corresponding to the video to be processed and the video spatiotemporal features. The cross-modal response decoder is used to generate a dialogue response corresponding to the video to be processed based on the cross-modal interaction semantic features.

[0009] To achieve the above objectives, embodiments of this application also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the efficient fine-grained video dialogue generation method based on cascaded spatiotemporal attention networks described above.

[0010] To achieve the above objectives, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the efficient fine-grained video dialogue generation method based on cascaded spatiotemporal attention networks described above.

[0011] The embodiments of this application propose an efficient fine-grained video dialogue generation method based on a cascaded spatiotemporal attention network. This method utilizes visual semantic features extracted by a visual Transformer model and dialogue context semantic features extracted by a BERT model. By employing a cascaded spatiotemporal attention network, it selects the video time segments and spatial regions most relevant to the dialogue context using prior knowledge of the dialogue context. This effectively ignores a large number of redundant video frames and spatial regions in the video that are irrelevant to the dialogue context, thereby achieving accurate localization of visual features and accurately answering video-related dialogue questions. Furthermore, considering the significant semantic gap between visual and textual modalities, this application utilizes an iterative cross-modal cross-attention network to perform multiple iterative cross-attention processes, thereby accurately capturing the correlation between video semantics and dialogue context semantics, further improving the quality of the generated dialogue responses and significantly enhancing the user experience.

[0012] In some optional embodiments, the step of utilizing a pre-trained cascaded spatiotemporal attention network to perform relevance matching based on the dialogue context semantic features and the visual semantic features, locating the set of video segments most relevant to the dialogue context in the temporal dimension, locating the set of spatial regions most relevant to the dialogue context in the spatial dimension, and obtaining video spatiotemporal features based on the set of video segments and the set of spatial regions includes: S301, using a first word pooling method to take the vector of the first word in the word sequence of the dialogue context semantic features as the dialogue context feature; S302, using the first word pooling method to take the first spatial region feature in each spatial region feature sequence in the visual semantic features as the video frame feature of the corresponding video frame, and taking the video frame features of each video frame belonging to the same video segment as the video frame feature of the visual semantic features. S303: Perform average pooling to obtain video segment features of the video segments; S304: Calculate the matching score between each video segment and the dialogue context based on the dialogue context features and the video segment features; S305: Utilize Gumbel-Softmax technology to perform discrete attention on each video segment based on the matching score between each video segment and the dialogue context to obtain a first one-hot vector; wherein, the first one-hot vector is used to represent the correlation probability value between each video segment and the dialogue context; S306: Select the video segment with the highest correlation probability value in the first one-hot vector as the video segment most relevant to the dialogue context, and combine it with the first one-hot vector to obtain the video segment features of the selected video segment; S307: Repeat steps S303 to S305 to execute TOP. c Next, the selected TOP c The video clips are combined into a set of video clips most relevant to the dialogue context, and the TOP obtained by combining the first one-hot vector is then used. cThe video segment features of each video clip are combined to form the video time features corresponding to the video clip set; TOP c S307: Based on the spatial region feature sequences in the visual semantic features, obtain the spatial region feature sequences corresponding to the video segment set; S308: Based on the dialogue context features and the spatial region features in the spatial region feature sequences corresponding to the video segment set, calculate the matching score between each spatial region and the dialogue context; S309: Using the Gumbel-Softmax technique, perform discrete attention on each spatial region based on the matching score between each spatial region and the dialogue context to obtain a second one-hot vector; wherein, the second one-hot vector is used to characterize the correlation probability value between each spatial region and the dialogue context; S310: Select the spatial region with the highest correlation probability value in the second one-hot vector as the spatial region most relevant to the dialogue context, and combine the second one-hot vector to obtain the spatial region features of the selected spatial region; S311: Repeat steps S308 to S310 to execute TOP r Next, the selected TOP r The spatial regions are combined to form a set of spatial regions most relevant to the dialogue context, and the TOP obtained by combining the second one-hot vector is used. r The spatial features of each spatial region are combined to form the video spatial features corresponding to the set of spatial regions; TOP r The integer is greater than 1; S312, the video temporal features and the video spatial features are concatenated to obtain the video spatiotemporal features corresponding to the video to be processed. The dialogue context semantic features and visual semantic features are processed by a cascaded spatiotemporal attention network, enabling the rapid, scientific, accurate, and low-cost extraction of the most relevant video segments and spatial regions from the original video to be processed. This reduces the workload and computation in both the temporal and spatial dimensions, thereby further improving the quality, speed, and efficiency of video dialogue generation.

[0013] In some optional embodiments, the video temporal characteristics are represented by the following formula:

[0014]

[0015]

[0016] idxc = argmax(g L )

[0017] g L =Gumbel_Softmax(S q,C )

[0018]

[0019]

[0020] Among them, c i Q represents the video segment feature of the i-th video segment. t K represents the query vector used to calculate the matching score between each video segment and the dialogue context. t This represents the key vector used to calculate the matching score between each video segment and the dialogue context, where q represents the dialogue context feature. and All are trainable weight matrices, D represents the dimension of the visual semantic features, Softmax(·) represents the Softmax function, and S q,C This represents the matching score between each video segment and the dialogue context, and Gumbel_Softmax(·) represents the Gumbel-Softmax technique. L Let g represent the first one-hot vector, argmax(·) represent the function to find the position of the maximum value, idxc represent the index of the video segment with the highest relevance probability value in the first one-hot vector, and g represent the index of the video segment with the highest relevance probability value in the first one-hot vector. l c represents the correlation probability value of the l-th term in the first one-hot vector. idxc This represents the video segment features obtained by combining the first one-hot vector with the selected video segment features, where V represents the set of indices of the video segments with the highest relevance probability values, and C represents the set of indices of the selected video segments. top This represents the video's temporal characteristics.

[0021] In some optional embodiments, the video spatial features are represented by the following formula:

[0022]

[0023]

[0024]

[0025] idxr = argmax(g N )

[0026] g N =Gumbel_Softmax(S q,r )

[0027]

[0028]

[0029] Where, r θThis represents the spatial region feature sequence corresponding to the θ-th video frame in the video clip set, where the video clip set contains Top... c ×M video frames, Q represents the spatial region feature of the nth spatial region in the spatial region feature sequence corresponding to the θth video frame in the video segment set. s K represents the query vector used to calculate the matching score between each spatial region and the dialogue context. s This represents the key vector used to calculate the matching score between each spatial region and the dialogue context, where q represents the dialogue context feature. and All are trainable weight matrices, D represents the dimension of the visual semantic features, Softmax(·) represents the Softmax function, and S q,r Gumbel_Softmax(·) represents the matching score between each spatial region in the spatial region feature sequence corresponding to the θ-th video frame and the dialogue context, and represents the Gumbel-Softmax technique. N Let g represent the second one-hot vector, argmax(·) represent the function to find the position of the maximum value, idxr represent the index of the spatial region with the highest correlation probability value in the second one-hot vector, and g n This represents the correlation probability value of the nth term in the second one-hot vector. R represents the spatial region features of the selected spatial region in the θ-th video frame obtained by combining the second one-hot vector, U represents the set of indices of the spatial regions with the highest correlation probability values, and R represents the spatial region features of the selected spatial region in the θ-th video frame obtained by combining the second one-hot vector. top This represents the video spatial features.

[0030] In some optional embodiments, the step of using a pre-trained iterative cross-modal cross-attention network to obtain cross-modal interaction semantic features based on the dialogue context corresponding to the video to be processed and the video spatiotemporal features includes: S401, using the embedding layer of the pre-trained iterative cross-modal cross-attention network to extract features from the dialogue context to obtain dialogue context features; S402, using the self-attention layer of the iterative cross-modal cross-attention network to perform self-attention extraction on the dialogue context features to obtain dialogue context semantic dependency features; S403, using the cross-attention layer of the iterative cross-modal cross-attention network to perform cross-attention extraction on the dialogue context semantic dependency features and the video spatiotemporal features to obtain spatiotemporal semantic association features; S404, repeating step S403 Y times, performing average pooling on the obtained Y spatiotemporal semantic association features to obtain cross-modal interaction semantic features; where Y is an integer greater than 1. In order to capture the semantic relationship between video modality and dialogue context modality in a deep and accurate manner, this application adopts a progressive fusion method, repeats the semantic fusion process Y times, and then performs information fusion through average pooling to finally obtain cross-modal interactive semantic features, which further improves the quality, speed and efficiency of video dialogue generation.

[0031] In some optional embodiments, the dialogue context semantic dependency features are expressed by the formula:

[0032]

[0033]

[0034] Among them, Q e K represents the query vector used for self-attention extraction of the dialogue context features. e V represents the key vector used for self-attention extraction of the dialogue context features. e K represents e The corresponding value, and All are trainable weight matrices, D represents the dimension of the visual semantic features, Softmax(·) represents the Softmax function, and X... con X represents the semantic dependency feature of the dialogue context. e This represents the dialogue context features;

[0035] The spatiotemporal semantic association features are expressed by the following formula:

[0036]

[0037]

[0038] Among them, Q c K represents the query vector used for cross-attention extraction of the dialogue context semantic dependency features and the video spatiotemporal features. c V represents the key vector used for cross-attention extraction of the dialogue context semantic dependency features and the video spatiotemporal features. c K represents c The corresponding value, and All are trainable weight matrices, F represents the video spatiotemporal features, and X... cro This represents the spatiotemporal semantic association features;

[0039] The cross-modal interaction semantic features are expressed by the following formula:

[0040]

[0041] in, This represents the spatiotemporal semantic association features obtained when executing step S403 for the yth time, and Mean_Pool(·) represents the average pooling operation. ite This represents the cross-modal interaction semantic features.

[0042] In some optional embodiments, the step of generating a dialogue response corresponding to the video to be processed based on the cross-modal interaction semantic features using a pre-trained cross-modal response decoder specifically involves: using the cross-modal interaction semantic features as an initial feature vector, capturing cross-modal features using the self-attention module and cross-attention module of the pre-trained cross-modal response decoder, and then generating a dialogue response corresponding to the video to be processed through the fully connected network and Softmax function layer of the cross-modal response decoder. Attached Figure Description

[0043] Figure 1 This is a flowchart of an efficient fine-grained video dialogue generation method based on a cascaded spatiotemporal attention network provided in one embodiment of this application;

[0044] Figure 2 This is a flowchart provided in one embodiment of the present application, which utilizes a pre-trained cascaded spatiotemporal attention network to perform relevant matching based on dialogue context semantic features and visual semantic features, locates the set of video segments most relevant to the dialogue context in the temporal dimension, locates the set of spatial regions most relevant to the dialogue context in the spatial dimension, and obtains the video spatiotemporal features based on the set of video segments and the set of spatial regions.

[0045] Figure 3This is a flowchart provided in one embodiment of the present application, which describes how a pre-trained iterative cross-modal cross-attention network is used to obtain cross-modal interaction semantic features based on the dialogue context and video spatiotemporal features corresponding to the video to be processed.

[0046] Figure 4 This is a schematic diagram of the principle of video dialogue generation based on a visual Transformer model, a BERT model, a cascaded spatiotemporal attention network, an iterative cross-modal cross-attention network, and a cross-modal response decoder, provided in one embodiment of this application.

[0047] Figure 5 This is a schematic diagram of the structure of an efficient fine-grained video dialogue generation system based on a cascaded spatiotemporal attention network provided in another embodiment of this application;

[0048] Figure 6 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0050] One embodiment of this application proposes an efficient fine-grained video dialogue generation method based on a cascaded spatiotemporal attention network, which is applied to an electronic device. The electronic device can be a terminal or a server. This embodiment and the following embodiments all use a server as an example for description. The implementation details of the efficient fine-grained video dialogue generation method based on a cascaded spatiotemporal attention network proposed in this embodiment are described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0051] The specific process of the efficient fine-grained video dialogue generation method based on cascaded spatiotemporal attention networks proposed in this embodiment can be described as follows: Figure 1 As shown, it includes:

[0052] Step S1: Input the video to be processed into the pre-trained visual Transformer model to obtain the visual semantic features output by the visual Transformer model.

[0053] Step S2: Input the dialogue context corresponding to the video to be processed into the pre-trained BERT model to obtain the dialogue context semantic features output by the BERT model.

[0054] In the specific implementation, after the server obtains the video to be processed for dialogue generation and the corresponding dialogue context, it needs to encode the inputs of different modalities into semantic vector representations. First, the server inputs the video to be processed into a pre-trained visual Transformer model and uses the visual Transformer model to perform semantic encoding on the video to obtain the visual semantic features output by the visual Transformer model. Then, the server inputs the dialogue context corresponding to the video to be processed into a pre-trained BERT model and uses the BERT model to perform semantic encoding on the dialogue context corresponding to the video to be processed and obtain the dialogue context semantic features output by the BERT model.

[0055] In one example, the visual Transformer model contains 12 Transformer layers, each with 12 attention heads. The visual Transformer model divides the video to be processed into L equal-length video segments, each video segment into M video frames, and each video frame into N spatial regions, where L, M, and N are all positive integers. After semantic encoding by the visual Transformer model, the video to be processed is encoded into visual semantic features P∈R. L×M×N×D D represents the dimension of visual semantic features.

[0056] In one example, the BERT model concatenates all the text content in the dialogue context corresponding to the video to be processed into a word sequence X. l x This represents the number of words in the word sequence X, which is the semantic feature of the dialogue context. Since the dimensions of the contextual semantic features of dialogue are the same as those of the visual semantic features, D in this paper is interpreted as the dimension of the visual semantic features.

[0057] Step S3: Using a pre-trained cascaded spatiotemporal attention network, relevant matching is performed based on the semantic features of the dialogue context and the visual semantic features. The set of video segments most relevant to the dialogue context is located in the time dimension, and the set of spatial regions most relevant to the dialogue context is located in the spatial dimension. Based on the set of video segments and the set of spatial regions, the spatiotemporal features of the video are obtained.

[0058] In the specific implementation, after obtaining the visual semantic features and the dialogue context semantic features, the server can input the visual semantic features and the dialogue context semantic features into a pre-trained cascaded spatiotemporal attention network. Using the cascaded spatiotemporal attention network, relevant matching is performed based on the dialogue context semantic features and the visual semantic features. In the time dimension, the set of video segments most relevant to the dialogue context is located, and in the spatial dimension, the set of spatial regions most relevant to the dialogue context is located. Finally, the video spatiotemporal features are obtained based on the set of video segments and the set of spatial regions.

[0059] In one example, the server utilizes a pre-trained cascaded spatiotemporal attention network to perform relevance matching based on dialogue context semantic features and visual semantic features. It locates the set of video segments most relevant to the dialogue context in the temporal dimension and the set of spatial regions most relevant to the dialogue context in the spatial dimension. Based on the set of video segments and the set of spatial regions, it obtains the video spatiotemporal features, which can be obtained through methods such as... Figure 2 The implementation of each sub-step shown includes:

[0060] Sub-step S301: Using the first word pooling method, the vector of the first word in the word sequence of the dialogue context semantic features is taken as the dialogue context feature.

[0061] In the specific implementation, the server uses the first word pooling method to take the vector of the first word in the word sequence X of the dialogue context semantic features as the dialogue context features. The dialogue context features can be represented as q, q = Ft_Pool(X), where Ft_Pool(·) represents the first word pooling operation.

[0062] Sub-step S302: Using the first word pooling method, the first spatial region feature in each spatial region feature sequence in the visual semantic features is taken as the video frame feature of the corresponding video frame, and the video frame features of each video frame belonging to the same video segment are subjected to average pooling to obtain the video segment feature of the video segment.

[0063] In the specific implementation, for the j-th video frame in the i-th video segment, the server can use Let f represent the spatial region feature of the nth spatial region. The spatial region features of N spatial regions form a spatial region feature sequence. The server uses the first word pooling method to take the first spatial region feature from each spatial region feature sequence in the visual semantic features as the video frame feature of the corresponding video frame. Then, for the i-th video segment containing M video frames, the video frame feature of its j-th video frame can be represented as f. i j , The server performs average pooling on the video frame features of all video frames belonging to the same video segment to obtain the video segment features of the video segment. Therefore, for the i-th video segment containing M video frames, its video segment features can be represented as c. i c i =Mean_Pool(f i 1 ,…,f i j ,…,f i M Finally, the video features of the video to be processed, containing L video segments, can be represented as C, where C = {c1, ..., c2}. i ,…,c L}

[0064] Sub-step S303: Based on the dialogue context features and the features of each video segment, calculate the matching score between each video segment and the dialogue context.

[0065] In the specific implementation, after obtaining the dialogue context features and the features of each video segment, the server can use the temporal attention mechanism of contextual attention to calculate the matching score between each video segment and the dialogue context.

[0066] In one example, the server calculates the matching score between each video segment and the dialogue context based on the features of the dialogue context and the features of each video segment. This can be achieved using the following formula:

[0067]

[0068]

[0069] In the formula, c i Q represents the video segment feature of the i-th video segment. t K represents the query vector used to calculate the matching score between each video segment and the dialogue context. t This represents the key vector used to calculate the matching score between each video segment and the dialogue context, where q represents the dialogue context feature. and All of these are trainable weight matrices. and The cascaded spatiotemporal attention network is iteratively updated during training, where D represents the dimension of the visual semantic features, Softmax(·) represents the Softmax function, and S... q,C This represents the matching score between each video clip and the dialogue context.

[0070] In sub-step S304, the Gumbel-Softmax technique is used to perform discrete attention on each video segment based on the matching score between each video segment and the dialogue context, to obtain the first one-hot vector.

[0071] In the specific implementation, after calculating the matching score between each video segment and the dialogue context, the server can use the Gumbel-Softmax technique to perform discrete attention on each video segment based on the matching score, thus obtaining the first one-hot vector. The first one-hot vector has a dimension of L, which represents the correlation probability value between each video segment and the dialogue context.

[0072] In one example, the first one-hot vector can be represented by the following formula:

[0073] g L =Gumbel_Softmax(S q,C )

[0074] In the formula, Gumbel_Softmax(·) represents the Gumbel-Softmax technique, g L This represents the first one-hot vector.

[0075] Sub-step S305: Select the video segment with the highest relevance probability value in the first one-hot vector as the video segment most relevant to the dialogue context, and combine it with the first one-hot vector to obtain the video segment features of the selected video segment.

[0076] Sub-step S306: Repeat sub-steps S303 to S305 to execute TOP. c Next, the selected TOP c The video clips are combined into a set of video clips most relevant to the dialogue context, and then combined with the TOP obtained from the first one-hot vector. c The video segment features of each video clip are combined to form the video time features corresponding to the video clip set.

[0077] In the specific implementation, after the server calculates the first one-hot vector, it can select the video segment with the highest relevance probability value in the first one-hot vector as the video segment most relevant to the dialogue context, and combine it with the first one-hot vector to obtain the video segment features of the selected video segment. Then, sub-steps S303 to S305 are repeated to execute TOP. c Next, the selected TOP c The video clips are combined into a set of video clips most relevant to the dialogue context, and then combined with the TOP obtained from the first one-hot vector. c The video segment features of each video clip are combined to form the video time feature C corresponding to the video clip set. top Among them, TOPc It is an integer greater than 1.

[0078] In one example, the video temporal features can be represented by the following formula:

[0079]

[0080]

[0081] idxc = argmax(g L )

[0082] In the formula, argmax(·) represents the function for finding the position of the maximum value, idxc represents the index of the video segment with the highest relevance probability in the first one-hot vector, and g l c represents the correlation probability value of the l-th term in the first one-hot vector. idxc This represents the video segment features obtained by combining the first one-hot vector, where V represents the set of indices of the video segments with the highest relevance probability values, and C represents the set of indices of the selected video segments. top This indicates the temporal characteristics of the video.

[0083] It is worth noting that the server uses soft attention when selecting video frames. Compared to hard attention, soft attention can avoid the situation where the backpropagation process becomes non-differentiable.

[0084] Sub-step S307: Based on the feature sequences of each spatial region in the visual semantic features, obtain the feature sequences of each spatial region corresponding to the video segment set.

[0085] In the specific implementation, after obtaining the video clip set, the server can obtain the spatial region feature sequences corresponding to each spatial region in the video clip set based on the spatial region feature sequences in the visual semantic features. The video clip set contains Top c ×M video frames, which includes Top c ×M spatial region feature sequences.

[0086] Sub-step S308: Based on the dialogue context features and the spatial region features in the spatial region feature sequences corresponding to the video segment set, calculate the matching score between each spatial region and the dialogue context.

[0087] In the specific implementation, after the server determines the feature sequences of each spatial region corresponding to the video segment set, it can calculate the matching score between each spatial region and the dialogue context based on the dialogue context features and the spatial region features in the feature sequences of each spatial region corresponding to the video segment set.

[0088] In one example, the server calculates the matching score between each spatial region and the dialogue context based on the spatial region features in the sequence of spatial region features corresponding to the video segment set. This can be achieved using the following formula:

[0089]

[0090]

[0091] In the formula, r θ This represents the spatial region feature sequence corresponding to the θ-th video frame in the video clip set. Q represents the spatial region feature of the nth spatial region in the spatial region feature sequence corresponding to the θth video frame in the video clip set. s K represents the query vector used to calculate the matching score between each spatial region and the dialogue context. s This represents the key vector used to calculate the matching score between each spatial region and the dialogue context, where q represents the dialogue context feature. and All of these are trainable weight matrices. and The cascaded spatiotemporal attention network is iteratively updated during training, where D represents the dimension of the visual semantic features, Softmax(·) represents the Softmax function, and S... q,r This represents the matching score between each spatial region in the spatial region feature sequence corresponding to the θ-th video frame and the dialogue context.

[0092] Sub-step S309 uses the Gumbel-Softmax technique to perform discrete attention on each spatial region based on the matching score between each spatial region and the dialogue context, to obtain the second one-hot vector.

[0093] In the specific implementation, after calculating the matching scores between each spatial region and the dialogue context, the server can use the Gumbel-Softmax technique to perform discrete attention on each spatial region based on the matching scores, thus obtaining a second one-hot vector. The second one-hot vector has a dimension of N, which is used to represent the correlation probability values ​​between each spatial region and the dialogue context.

[0094] In one example, the second one-hot vector can be expressed by the following formula:

[0095] g N =Gumbel_Softmax(S q,r )

[0096] In the formula, Gumbel_Softmax(·) represents the Gumbel-Softmax technique, gN This represents the second unique heat vector.

[0097] Sub-step S310: Select the spatial region with the highest correlation probability value in the second one-hot vector as the spatial region most relevant to the dialogue context, and combine the second one-hot vector to obtain the spatial region features of the selected spatial region.

[0098] Sub-step S311: Repeat sub-steps S308 to S310 to execute TOP. r Next, the selected TOP r The spatial regions are combined to form a set of spatial regions most relevant to the dialogue context, and the TOP obtained by combining the second one-hot vector is then used. r The spatial features of each spatial region are combined to form the video spatial features corresponding to the spatial region set.

[0099] In the specific implementation, after the server calculates the second one-hot vector, it can select the spatial region with the highest relevance probability value in the second one-hot vector as the spatial region most relevant to the dialogue context, and combine it with the second one-hot vector to obtain the spatial region features of the selected spatial region. Then, sub-steps S308 to S310 are repeated to execute TOP. r Next, the selected TOP r The spatial regions are combined to form a set of spatial regions most relevant to the dialogue context, and the TOP obtained by combining the second one-hot vector is then used. r The spatial features of each spatial region are combined to form the video spatial feature R corresponding to the spatial region set. top Among them, TOP r It is an integer greater than 1.

[0100] In one example, video spatial features can be represented by the following formula:

[0101]

[0102]

[0103]

[0104] idxr = argmax(g N )

[0105] In the formula, argmax(·) represents the function for finding the position of the maximum value, idxr represents the index of the spatial region with the highest correlation probability value in the second one-hot vector, and g n This represents the correlation probability value of the nth term in the second one-hot vector. R represents the spatial region features of the selected spatial region in the θ-th video frame obtained by combining the second one-hot vector, U represents the set of indices of the spatial regions with the highest relevance probability values, and R represents the spatial region features of the selected spatial region in the θ-th video frame. top Represents the spatial features of the video.

[0106] It is worth noting that the server uses soft attention when selecting spatial regions. Compared with hard attention, soft attention can avoid the situation where the backpropagation process becomes non-differentiable.

[0107] Sub-step S312 involves concatenating the video temporal features and video spatial features to obtain the video spatiotemporal features corresponding to the video to be processed.

[0108] In the specific implementation, the server obtains the video time feature C. top and video spatial features R top After that, C can be... top and R top By concatenating the components, we obtain the spatiotemporal features F corresponding to the video to be processed, where F = C. top +R top .

[0109] It is worth noting that after the dialogue context semantic features and visual semantic features are processed by the cascaded spatiotemporal attention network, the video segments and spatial regions most relevant to the dialogue context can be extracted from the original video to be processed quickly, scientifically, accurately, and at low cost. This reduces the workload and computation in both the temporal and spatial dimensions, thereby further improving the quality, speed, and efficiency of video dialogue generation.

[0110] Step S4: Using a pre-trained iterative cross-modal cross-attention network, cross-modal interaction semantic features are obtained based on the dialogue context and video spatiotemporal features corresponding to the video to be processed.

[0111] In the specific implementation, after obtaining the spatiotemporal features of the video, the server can input the dialogue context and video spatiotemporal features corresponding to the video to be processed into a pre-trained iterative cross-modal cross-attention network. Using the iterative cross-modal cross-attention network, cross-modal interactive semantic features are obtained based on the dialogue context and video spatiotemporal features corresponding to the video to be processed.

[0112] In one example, the server utilizes a pre-trained iterative cross-modal cross-attention network to obtain cross-modal interaction semantic features based on the dialogue context and spatiotemporal features of the video to be processed. These features can be obtained through methods such as... Figure 3 The implementation of each sub-step shown includes:

[0113] Sub-step S401: Use the embedding layer of a pre-trained iterative cross-modal cross-attention network to extract features from the dialogue context and obtain dialogue context features.

[0114] Sub-step S402 involves using the self-attention layer of an iterative cross-modal cross-attention network to extract dialogue context features through self-attention, thereby obtaining dialogue context semantic dependency features.

[0115] In the specific implementation, after obtaining the spatiotemporal features of the video, the server can input the dialogue context and video spatiotemporal features corresponding to the video to be processed into a pre-trained iterative cross-modal cross-attention network. The embedding layer of the pre-trained iterative cross-modal cross-attention network is used to extract features from the dialogue context, resulting in dialogue context features X. e Subsequently, the self-attention layer of an iterative cross-modal cross-attention network is used to process the dialogue context features X. e Perform self-attention extraction to obtain the semantic dependency features X of the dialogue context. con .

[0116] In one example, the semantic dependency features of the dialogue context can be expressed by the formula:

[0117]

[0118]

[0119] In the formula, Q e K represents the query vector used for self-attention extraction of dialogue context features. e V represents the key vector used for self-attention extraction of dialogue context features. e K represents e The corresponding value, and All of these are trainable weight matrices. and The iterative cross-modal attention network is continuously updated during training, where D represents the dimension of the visual semantic features, Softmax(·) represents the Softmax function, and X... con This represents the semantic dependency features of the dialogue context.

[0120] Sub-step S403 utilizes the cross-attention layer of the iterative cross-modal cross-attention network to extract cross-attention features of dialogue context semantic dependence and video spatiotemporal features, thereby obtaining spatiotemporal semantic association features.

[0121] In the specific implementation, the server obtains the semantic dependency features X of the dialogue context. conThen, the cross-attention layer of an iterative cross-modal cross-attention network can be used to process the semantically dependent features X of the dialogue context. con Cross-attention extraction is performed on the video spatiotemporal feature F to obtain the spatiotemporal semantic association feature X. cro .

[0122] In one example, spatiotemporal semantic association features can be expressed by the formula:

[0123]

[0124]

[0125] In the formula, Q c K represents the query vector used for cross-attention extraction of dialogue context semantic dependency features and video spatiotemporal features. c V represents the key vector used for cross-attention extraction of dialogue context semantic dependency features and video spatiotemporal features. c K represents c The corresponding value, and All of these are trainable weight matrices. and The iterative cross-modal cross-attention network is continuously updated during training, where F represents the video spatiotemporal features, and X represents the video spatiotemporal features. cro It represents the spatiotemporal semantic association features.

[0126] Sub-step S404 involves repeating sub-step S403 Y times to perform average pooling on the obtained Y spatiotemporal semantic association features, thereby obtaining cross-modal interaction semantic features.

[0127] In the specific implementation, in order to capture the semantic association between the video modality and the dialogue context modality in a deep and accurate manner, the server adopts a progressive fusion method, repeating sub-step S403 Y times, and performing average pooling operation on the obtained Y spatiotemporal semantic association features to obtain cross-modal interaction semantic features, which further improves the quality, speed and efficiency of video dialogue generation.

[0128] In one example, cross-modal interaction semantic features can be expressed by the formula:

[0129]

[0130] In the formula, X represents the spatiotemporal semantic association features obtained when executing sub-step S403 for the yth time, and Mean_Pool(·) represents the average pooling operation. ite This represents the semantic features of cross-modal interaction.

[0131] Step S5: Using a pre-trained cross-modal response decoder, a dialogue response corresponding to the video to be processed is generated based on the cross-modal interaction semantic features.

[0132] In the specific implementation, the server inputs the cross-modal interaction semantic features into the pre-trained cross-modal response decoder. The cross-modal response decoder uses the cross-modal interaction semantic features as the initial feature vector, uses its own self-attention module and cross-attention module to capture cross-modal features, and then generates and outputs the dialogue response corresponding to the video to be processed through its own fully connected network and Softmax function layer.

[0133] In one example, the principle of video dialogue generation based on the visual Transformer model, BERT model, cascaded spatiotemporal attention network, iterative cross-modal cross-attention network, and cross-modal response decoder can be described as follows: Figure 4 As shown.

[0134] In this embodiment, based on the visual semantic features extracted by the visual Transformer model and the dialogue context semantic features extracted by the BERT model, a cascaded spatiotemporal attention network is used to select the video time segments and spatial regions most relevant to the dialogue context by utilizing prior knowledge of the dialogue context. This effectively ignores a large number of redundant video frames and redundant spatial regions in the video that are irrelevant to the dialogue context, thereby achieving accurate localization of visual features and accurately answering video-related dialogue questions. At the same time, considering the huge semantic gap between visual and text modalities, this application uses an iterative cross-modal cross-attention network to perform multiple iterations of cross-attention processes, thereby accurately capturing the correlation between video semantics and dialogue context semantics, further improving the quality of the generated dialogue responses and significantly enhancing the user experience.

[0135] Another embodiment of this application proposes an efficient fine-grained video dialogue generation system based on a cascaded spatiotemporal attention network. The details of this efficient fine-grained video dialogue generation system based on a cascaded spatiotemporal attention network are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this example. Figure 5 This is a schematic diagram of an efficient fine-grained video dialogue generation system based on a cascaded spatiotemporal attention network proposed in this embodiment, including: an acquisition module 601, a visual Transformer model 602, a BERT model 603, a cascaded spatiotemporal attention network 604, an iterative cross-modal cross-attention network 605, and a cross-modal response decoder 606.

[0136] The acquisition module 601 is used to acquire the video to be processed and the dialogue context corresponding to the video to be processed, input the video to be processed into the visual Transformer model 602, and input the dialogue context into the BERT model 603 and the iterative cross-modal cross-attention network 605.

[0137] The visual Transformer model 602 is used to extract visual semantic features of the video to be processed. The visual Transformer model 602 divides the video to be processed into L video segments of equal length. Each video segment is divided into M video frames. Each video frame is divided into N spatial regions. L, M and N are all integers greater than 0.

[0138] BERT model 603 is used to extract the semantic features of the dialogue context.

[0139] The cascaded spatiotemporal attention network 604 is used to perform relevant matching based on dialogue context semantic features and visual semantic features. It locates a set of video segments related to the dialogue context in the temporal dimension and a set of spatial regions related to the dialogue context in the spatial dimension, and obtains video spatiotemporal features based on the set of video segments and the set of spatial regions.

[0140] The iterative cross-modal cross-attention network 605 is used to obtain cross-modal interactive semantic features based on the dialogue context and video spatiotemporal features corresponding to the video to be processed.

[0141] The cross-modal response decoder 606 is used to generate dialogue responses corresponding to the video to be processed based on cross-modal interaction semantic features.

[0142] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0143] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0144] Another embodiment of this application relates to an electronic device, such as... Figure 6As shown, it includes: at least one processor 701; and a memory 702 communicatively connected to the at least one processor 701; wherein the memory 702 stores instructions executable by the at least one processor 701, the instructions being executed by the at least one processor 701 to enable the at least one processor 701 to execute the efficient fine-grained video dialogue generation method based on cascaded spatiotemporal attention networks in the above embodiments.

[0145] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0146] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0147] Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0148] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, ROM (Read-Only Memory), RAM (Random Access Memory), a magnetic disk, or an optical disk.

[0149] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A highly efficient fine-grained video dialogue generation method based on a cascaded spatiotemporal attention network, characterized in that, The method includes: S1, the video to be processed is input into a pre-trained visual Transformer model to obtain the visual semantic features output by the visual Transformer model; wherein, the visual Transformer model segments the video to be processed into... Each video segment is divided into several equal-length video clips, and each video clip is further divided into... Each video frame is divided into [number] video frames. A spatial region, , and All are integers greater than 0; S2, input the dialogue context corresponding to the video to be processed into the pre-trained BERT model to obtain the dialogue context semantic features output by the BERT model; S3. Using a pre-trained cascaded spatiotemporal attention network, relevant matching is performed based on the semantic features of the dialogue context and the visual semantic features. The set of video segments most relevant to the dialogue context is located in the time dimension, and the set of spatial regions most relevant to the dialogue context is located in the spatial dimension. Video spatiotemporal features are obtained based on the set of video segments and the set of spatial regions. S4. Using a pre-trained iterative cross-modal cross-attention network, cross-modal interaction semantic features are obtained based on the dialogue context corresponding to the video to be processed and the video spatiotemporal features. S5, using a pre-trained cross-modal response decoder, generate a dialogue response corresponding to the video to be processed based on the cross-modal interaction semantic features; The method utilizes a pre-trained cascaded spatiotemporal attention network to perform relevance matching based on the semantic features of the dialogue context and the visual semantic features. This locates the set of video segments most relevant to the dialogue context in the temporal dimension and the set of spatial regions most relevant to the dialogue context in the spatial dimension. Based on the set of video segments and the set of spatial regions, video spatiotemporal features are obtained, including: S301, using the first word pooling method, the vector of the first word in the word sequence of the dialogue context semantic features is taken as the dialogue context feature; S302, using the first word pooling method, the first spatial region feature in each spatial region feature sequence in the visual semantic features is taken as the video frame feature of the corresponding video frame, and the video frame features of each video frame belonging to the same video segment are subjected to average pooling operation to obtain the video segment feature of the video segment. S303, based on the dialogue context features and the features of each video segment, calculate the matching score between each video segment and the dialogue context; S304, using the Gumbel-Softmax technique, based on the matching score between each video segment and the dialogue context, discrete attention is performed on each video segment to obtain a first one-hot vector; wherein, the first one-hot vector is used to characterize the correlation probability value between each video segment and the dialogue context; S305, select the video segment with the highest relevance probability value in the first one-hot vector as the video segment most relevant to the dialogue context, and combine the first one-hot vector to obtain the video segment feature of the selected video segment; S306, repeat steps S303 to S305. Next, the selected The video clips are combined into a set of video clips most relevant to the dialogue context, and then combined with the first one-hot vector to obtain... The video segment features of each video clip are combined to form the video time features corresponding to the set of video clips; It is an integer greater than 1; S307, Based on the spatial region feature sequences in the visual semantic features, obtain the spatial region feature sequences corresponding to the video segment set; S308, Based on the dialogue context features and the spatial region features in the spatial region feature sequences corresponding to the video segment set, calculate the matching score between each spatial region and the dialogue context; S309, using the Gumbel-Softmax technique, based on the matching score between each spatial region and the dialogue context, discrete attention is performed on each spatial region to obtain a second one-hot vector; wherein, the second one-hot vector is used to characterize the correlation probability value between each spatial region and the dialogue context; S310, Select the spatial region with the highest correlation probability value in the second one-hot vector as the spatial region most relevant to the dialogue context, and combine the second one-hot vector to obtain the spatial region features of the selected spatial region; S311, repeat steps S308 to S310. Next, the selected The spatial regions are combined to form a set of spatial regions most relevant to the dialogue context, and this set is combined with the second one-hot vector to obtain... The spatial features of each spatial region are combined to form the video spatial features corresponding to the set of spatial regions; It is an integer greater than 1; S312, the video temporal features and the video spatial features are concatenated to obtain the video spatiotemporal features corresponding to the video to be processed; The method utilizes a pre-trained iterative cross-modal cross-attention network to obtain cross-modal interaction semantic features based on the dialogue context corresponding to the video to be processed and the video's spatiotemporal features, including: S401, the dialogue context is extracted using the embedding layer of a pre-trained iterative cross-modal cross-attention network to obtain dialogue context features; S402, the self-attention layer of the iterative cross-modal cross-attention network is used to perform self-attention extraction on the dialogue context features to obtain the dialogue context semantic dependency features; S403, using the cross-attention layer of the iterative cross-modal cross-attention network, cross-attention is performed on the dialogue context semantic dependency features and the video spatiotemporal features to extract spatiotemporal semantic association features; S404, repeat step S403. Next, for the obtained Average pooling is performed on the spatiotemporal semantic association features to obtain cross-modal interaction semantic features; It is an integer greater than 1.

2. The efficient fine-grained video dialogue generation method based on cascaded spatiotemporal attention networks according to claim 1, characterized in that, The video temporal features are expressed by the following formula: ； ； ； ； ； ； in, Indicates the first Video segment features of a video clip This represents the query vector used to calculate the matching score between each video segment and the dialogue context. This represents the key vector used to calculate the matching score between each video segment and the dialogue context. This represents the dialogue context features. and All of these are trainable weight matrices. The dimension representing the visual semantic features, This represents the Softmax function. This represents the matching score between each video segment and the dialogue context. This refers to the Gumbel-Softmax technology. This represents the first one-hot vector. This represents a function that finds the location of the maximum value. This represents the index of the video segment with the highest relevance probability value in the first one-hot vector. Indicates the first unique heat vector. The relevant probability value of the item, This represents the video segment features obtained by combining the first one-hot vector with the selected video segment. The set of indices representing the video segments with the highest relevance probability values. This represents the video's temporal characteristics. Indicates inclusion The video features of the video to be processed from each video segment.

3. The efficient fine-grained video dialogue generation method based on cascaded spatiotemporal attention networks according to claim 2, characterized in that, The video spatial features are represented by the following formula: ； ； ； ； ； ； ； in, Indicates the first video segment in the set. The set of video segments contains spatial region feature sequences corresponding to each video frame. One video frame, Indicates the first video segment in the set. The spatial region feature sequence corresponding to the video frame is the first one Spatial characteristics of a spatial region This represents the query vector used to calculate the matching score between each spatial region and the dialogue context. This represents the key vector used when calculating the matching score between each spatial region and the dialogue context. This represents the dialogue context features. and All of these are trainable weight matrices. The dimension representing the visual semantic features, This represents the Softmax function. Indicates the first The matching score between each spatial region in the spatial region feature sequence corresponding to each video frame and the dialogue context. This refers to the Gumbel-Softmax technology. This represents the second one-hot vector. This represents a function that finds the location of the maximum value. This represents the index of the spatial region with the highest correlation probability value in the second one-hot vector. Indicates the second unique heat vector. The relevant probability value of the item, This indicates that the value obtained by combining the second unique heat vector at the 1st... Spatial region features of a selected spatial region in a video frame The set of indices representing the spatial regions with the highest relevance probability values. This represents the video spatial features.

4. The efficient fine-grained video dialogue generation method based on cascaded spatiotemporal attention networks according to claim 1, characterized in that, The semantic dependency features of the dialogue context are expressed by the formula: ； ； in, This represents the query vector used when performing self-attention extraction on the dialogue context features. This represents the key vector used when performing self-attention extraction on the dialogue context features. express The corresponding value, , and All of these are trainable weight matrices. The dimension representing the visual semantic features, This represents the Softmax function. This represents the semantic dependency features of the dialogue context. This represents the dialogue context features; The spatiotemporal semantic association features are expressed by the following formula: ； ； in, This represents the query vector used when performing cross-attention extraction on the semantic dependency features of the dialogue context and the spatiotemporal features of the video. This represents the key vector used when performing cross-attention extraction on the semantic dependency features of the dialogue context and the spatiotemporal features of the video. express The corresponding value, , and All of these are trainable weight matrices. This represents the spatiotemporal features of the video. This represents the spatiotemporal semantic association features; The cross-modal interaction semantic features are expressed by the following formula: in, Indicates the first The spatiotemporal semantic association features obtained during the execution of step S403. This refers to the average pooling operation. This represents the cross-modal interaction semantic features.

5. The efficient fine-grained video dialogue generation method based on cascaded spatiotemporal attention networks according to any one of claims 1 to 4, characterized in that, The step of generating a dialogue response corresponding to the video to be processed based on the cross-modal interaction semantic features using a pre-trained cross-modal response decoder specifically involves: Using the cross-modal interaction semantic features as the initial feature vector, the self-attention module and cross-attention module of the pre-trained cross-modal response decoder are used to capture the cross-modal features, and then the fully connected network and Softmax function layer of the cross-modal response decoder are used to generate the dialogue response corresponding to the video to be processed.

6. A highly efficient fine-grained video dialogue generation system based on a cascaded spatiotemporal attention network, characterized in that, include: Acquisition module, visual Transformer model, BERT model, cascaded spatiotemporal attention network, iterative cross-modal cross-attention network, and cross-modal response decoder; The acquisition module is used to acquire the video to be processed and the dialogue context corresponding to the video to be processed, input the video to be processed into the visual Transformer model, and input the dialogue context into the BERT model and the iterative cross-modal cross-attention network; The visual Transformer model is used to extract the visual semantic features of the video to be processed; wherein, the visual Transformer model segments the video to be processed into... Each video segment is divided into several equal-length video clips, and each video clip is further divided into... Each video frame is divided into [number] video frames. A spatial region, , and All are integers greater than 0; The BERT model is used to extract the semantic features of the dialogue context. The cascaded spatiotemporal attention network is used to perform relevant matching based on the semantic features of the dialogue context and the visual semantic features, locate the set of video segments related to the dialogue context in the time dimension, locate the set of spatial regions related to the dialogue context in the spatial dimension, and obtain video spatiotemporal features based on the set of video segments and the set of spatial regions. The iterative cross-modal cross-attention network is used to obtain cross-modal interaction semantic features based on the dialogue context corresponding to the video to be processed and the video spatiotemporal features; The cross-modal response decoder is used to generate a dialogue response corresponding to the video to be processed based on the cross-modal interaction semantic features; Using a pre-trained cascaded spatiotemporal attention network, relevance matching is performed based on the semantic features of the dialogue context and the visual semantic features. This locates the set of video segments most relevant to the dialogue context in the temporal dimension and the set of spatial regions most relevant to the dialogue context in the spatial dimension. Based on the set of video segments and the set of spatial regions, video spatiotemporal features are obtained, including: S301, using the first word pooling method, the vector of the first word in the word sequence of the dialogue context semantic features is taken as the dialogue context feature; S302, using the first word pooling method, the first spatial region feature in each spatial region feature sequence in the visual semantic features is taken as the video frame feature of the corresponding video frame, and the video frame features of each video frame belonging to the same video segment are subjected to average pooling operation to obtain the video segment feature of the video segment. S303, based on the dialogue context features and the features of each video segment, calculate the matching score between each video segment and the dialogue context; S304, using the Gumbel-Softmax technique, based on the matching score between each video segment and the dialogue context, discrete attention is performed on each video segment to obtain a first one-hot vector; wherein, the first one-hot vector is used to characterize the correlation probability value between each video segment and the dialogue context; S305, select the video segment with the highest relevance probability value in the first one-hot vector as the video segment most relevant to the dialogue context, and combine the first one-hot vector to obtain the video segment feature of the selected video segment; S306, repeat steps S303 to S305. Next, the selected The video clips are combined into a set of video clips most relevant to the dialogue context, and then combined with the first one-hot vector to obtain... The video segment features of each video clip are combined to form the video time features corresponding to the set of video clips; It is an integer greater than 1; S307, Based on the spatial region feature sequences in the visual semantic features, obtain the spatial region feature sequences corresponding to the video segment set; S308, Based on the dialogue context features and the spatial region features in the spatial region feature sequences corresponding to the video segment set, calculate the matching score between each spatial region and the dialogue context; S309, using the Gumbel-Softmax technique, based on the matching score between each spatial region and the dialogue context, discrete attention is performed on each spatial region to obtain a second one-hot vector; wherein, the second one-hot vector is used to characterize the correlation probability value between each spatial region and the dialogue context; S310, Select the spatial region with the highest correlation probability value in the second one-hot vector as the spatial region most relevant to the dialogue context, and combine the second one-hot vector to obtain the spatial region features of the selected spatial region; S311, repeat steps S308 to S310. Next, the selected The spatial regions are combined to form a set of spatial regions most relevant to the dialogue context, and this set is combined with the second one-hot vector to obtain... The spatial features of each spatial region are combined to form the video spatial features corresponding to the set of spatial regions; It is an integer greater than 1; S312, the video temporal features and the video spatial features are concatenated to obtain the video spatiotemporal features corresponding to the video to be processed; Using a pre-trained iterative cross-modal cross-attention network, cross-modal interaction semantic features are obtained based on the dialogue context corresponding to the video to be processed and the video's spatiotemporal features, including: S401, the dialogue context is extracted using the embedding layer of a pre-trained iterative cross-modal cross-attention network to obtain dialogue context features; S402, the self-attention layer of the iterative cross-modal cross-attention network is used to perform self-attention extraction on the dialogue context features to obtain the dialogue context semantic dependency features; S403, using the cross-attention layer of the iterative cross-modal cross-attention network, cross-attention is performed on the dialogue context semantic dependency features and the video spatiotemporal features to extract spatiotemporal semantic association features; S404, repeat step S403. Next, for the obtained Average pooling is performed on the spatiotemporal semantic association features to obtain cross-modal interaction semantic features; It is an integer greater than 1.

7. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the efficient fine-grained video dialogue generation method based on cascaded spatiotemporal attention networks as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the efficient fine-grained video dialogue generation method based on cascaded spatiotemporal attention networks as described in any one of claims 1 to 5.

Images