Dialogue generation method and device based on multi-modal clue aggregation and storage medium

Abstract

The invention discloses a dialogue generation method and device based on multi-modal clue aggregation and a storage medium, and relates to the technical field of cross-media generation, and the method comprises the steps: after encoding image blocks corresponding to video data, splicing embedded sequences obtained through encoding according to the time sequence of the video data, and obtaining visual features; extracting and transcribing audios in the video data to obtain text features corresponding to the video data; on the basis of a target multi-modal clue aggregator, the visual features and the text features are aligned in a semantic space, and multi-modal features are generated; and retrieving related semantic segments of the user question from the multi-modal features, and generating answer content corresponding to the user question based on semantic association between a retrieval result and a historical context. According to the method, the matching degree between the answer content and the user demand and the logic continuity between the answer content and the historical context are improved by fusing the multi-modal features.

Description

Technical Field

[0001] This application relates to the field of cross-media generation technology, and in particular to a dialogue generation method, device and storage medium based on multimodal cue aggregation. Background Technology

[0002] With the widespread application of deep learning technology in the field of video understanding, it has provided core technical support for the automated analysis of video content. However, the video understanding models in related technologies are still limited to the execution of specific tasks, lacking the ability to understand complex and dynamic content and generate natural language dialogues. This results in the output language descriptions being incoherent and inconsistent with the context, ultimately leading to the inability to stably generate high-quality natural language dialogue content.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this application is to provide a dialogue generation method, device, and storage medium based on multimodal cue aggregation, aiming to solve the technical problem of the inability to stably generate high-quality natural language dialogue content.

[0005] To achieve the above objectives, this application proposes a dialogue generation method based on multimodal cue aggregation, the method comprising: After encoding the image blocks corresponding to the video data, the encoded embedding sequence is concatenated according to the temporal order of the video data to obtain visual features; The audio is extracted and transcribed from the video data to obtain the text features corresponding to the video data; Based on the target multimodal cue aggregator, the visual features and the text features are aligned in the semantic space to generate multimodal features; Retrieve relevant semantic fragments of the user's question from the multimodal features, and generate the answer content corresponding to the user's question based on the semantic association between the retrieval results and the historical context.

[0006] In one embodiment, each image patch is linearly embedded into a fixed-dimensional vector, and an embedding vector corresponding to each image patch is output based on global context modeling. Arrange the embedding vectors corresponding to the image blocks in the same frame according to their spatial position to obtain the frame embedding sequence corresponding to the same frame. By concatenating the frame embedding sequence of all frames according to the temporal order of the video data, the visual features containing video spatiotemporal information are obtained.

[0007] In one embodiment, the audio track in the video data is extracted, and the original audio signal corresponding to the video data is divided into multiple audio segments of fixed length; The speech recognition process extracts the spectrogram of each audio segment and generates a text sequence and its embedded representation corresponding to each audio segment. The text sequence is concatenated according to the temporal order of the audio segments, and the embedded representation is concatenated according to the temporal order to generate the text features corresponding to the video data.

[0008] In one embodiment, based on the target multimodal cue aggregator, the visual features and the text features are mapped to a common semantic space, and the feature distribution is adjusted by calculating the semantic correlation between modalities to obtain aligned visual features and text features; Calculate the association weights corresponding to the aligned visual features and text features, and integrate key semantic information in a weighted manner to obtain preliminary fused features; The preliminary fusion features are modeled based on temporal dependencies, and the feature representation is optimized through nonlinear transformation and normalization to generate multimodal features.

[0009] In one embodiment, the user question is converted into a text embedding sequence consistent with the multimodal feature semantic space through a vector mapping operation; Using the text embedding sequence as the retrieval benchmark, the semantic similarity between the text embedding sequence and each semantic unit in the multimodal features is calculated, and semantic units with similarity higher than a preset threshold are selected to form a candidate semantic fragment set. Extract historical context information related to the user's question from the historical context semantic database, conduct a secondary evaluation of the logical coherence and demand matching degree of the candidate semantic fragment set, and retain the qualified fragments as target-related semantic fragments; Using the target-related semantic fragments as core semantic support, combined with historical context information, an autoregressive generation mode is adopted to gradually generate preliminary answer text; The initial response text is checked for semantic coherence and factual accuracy. Content with logical contradictions or information discrepancies is corrected, and the response content corresponding to the user's question is generated.

[0010] In one embodiment, the video data is extracted at fixed time intervals, and the continuous video stream is split into multiple independent image frames to obtain a set of image frames corresponding to the video data. The image frames are evenly divided according to a preset fixed size to obtain an initial image block group corresponding to each image frame; Based on the temporal order of the image frames in the video data and the spatial order of each image frame, the initial image block group corresponding to the image frames is summarized to generate the image block sequence corresponding to the video data.

[0011] In one embodiment, the basic framework of the multimodal cue aggregator is initialized, and the basic network pathway for feature transfer is built to obtain the initial framework of the aggregator; Based on the initial aggregator framework, cross-modal alignment components, dynamic weighted fusion components, and temporal modeling components are embedded to generate the aggregator framework. Initialize the similarity calculation threshold of the cross-modal alignment component, the attention mechanism scaling factor of the dynamic weighted fusion component, and the network layer parameters of the temporal modeling component, and output the aggregator model to be trained; The aggregator training model is trained based on labeled multimodal training data, and the parameter configuration of the aggregator training model is optimized through a loss function to generate the target multimodal cue aggregator.

[0012] In one embodiment, key issues and core needs are extracted by parsing the user's input opinions and follow-up questions to obtain user feedback information; Based on the user feedback information, the adjustable parameters of the associated model are analyzed, and the adjustment direction and magnitude of the adjustable parameters are analyzed to generate a parameter optimization scheme. Adjust the model parameters according to the parameter optimization scheme, verify the effect with corresponding dialogue scenario data, and output the optimized model parameters.

[0013] In addition, to achieve the above objectives, this application also proposes a dialogue generation device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the dialogue generation method based on multimodal cue aggregation as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the dialogue generation method based on multimodal cue aggregation as described above.

[0015] This application provides a dialogue generation method based on multimodal cue aggregation, including: performing encoding operations on image blocks corresponding to video data; concatenating the encoded embedding sequences according to the temporal order of the video data to generate visual features that can characterize the visual information of the video; extracting audio signals from the video data and transcribing them to convert the audio information into text form to obtain text features directly corresponding to the video data; calling a target multimodal cue aggregator and inputting the generated visual and text features into the aggregator; aligning the two modal features in a unified semantic space to generate multimodal features that integrate spatiotemporal and textual information; using the user question as the retrieval target; selecting semantic fragments semantically related to the user question from the generated multimodal features; and gradually generating answer content for the user question by combining the semantic relationship between the retrieval results and the historical dialogue context. This application overcomes the technical problems in traditional multimodal processing, such as the misalignment of visual and textual features in semantic space, insufficient cross-modal information fusion, low matching degree between user questions and multimodal content, and the generation of answers that are detached from contextual logic. It achieves accurate semantic alignment and efficient fusion of multimodal features, improves the retrieval accuracy of semantic fragments related to user questions, and ensures the matching degree between answer content and user needs as well as the logical coherence with historical context.

[0016] In summary, this application obtains visual features through video image block encoding and text features through audio transcription. These features are then aligned by an aggregator to generate multimodal features. Semantic fragments are retrieved and combined with the context to generate answers. This improves the semantic alignment accuracy of multimodal features and the retrieval accuracy of semantic fragments related to user questions. It achieves a high degree of matching between the answer content and user needs, and logical coherence with historical context, thereby enhancing the reliability and practicality of dialogue interaction based on video multimodal information. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the dialogue generation method based on multimodal cue aggregation in this application; Figure 2 This is a diagram of the algorithm architecture of this application; Figure 3This is a flowchart illustrating the fifth embodiment of the dialogue generation method based on multimodal cue aggregation in this application; Figure 4 This is a flowchart illustrating the seventh embodiment of the dialogue generation method based on multimodal cue aggregation in this application; Figure 5 This is a schematic diagram of the dialogue generation device of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] Video understanding models in related technologies are still limited to the execution of specific tasks, lacking the ability to understand complex dynamic content and generate natural language dialogues. This results in incoherent and inconsistent language descriptions in the output, ultimately leading to the inability to stably generate high-quality natural language dialogue content.

[0023] This application provides a solution: First, after encoding the image blocks corresponding to the video data, the encoded embedding sequences are concatenated according to the temporal order of the video data to obtain visual features. Then, the audio in the video data is extracted and transcribed to obtain the text features corresponding to the video data. Next, based on a target multimodal cue aggregator, the visual features and the text features are aligned in the semantic space to generate multimodal features. Finally, relevant semantic fragments of the user question are retrieved from the multimodal features. Based on the semantic association between the retrieval results and the historical context, the answer content corresponding to the user question is generated.

[0024] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or dialogue generation device capable of performing the above functions. The following description uses a dialogue generation device as an example to illustrate this embodiment and the subsequent embodiments.

[0025] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0026] This application provides a dialogue generation method based on multimodal cue aggregation, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the dialogue generation method based on multimodal cue aggregation in this application.

[0027] In this embodiment, the dialogue generation method based on multimodal cue aggregation includes steps S10 to S40: Step S10: After encoding the image blocks corresponding to the video data, the encoded embedding sequence is spliced ​​together according to the temporal order of the video data to obtain visual features.

[0028] In this embodiment, the image block corresponding to the video data refers to an image unit that carries local visual information of a single frame after the continuous video stream is split according to a preset rule. Encoding refers to the process of extracting visual information from the image block and converting it into a structured vector. The resulting embedding sequence is a set of sequences of vectors formed by each image block after encoding, arranged in a specific order. The temporal order of the video data refers to the chronological order of the video frames. Visual features refer to comprehensive features that integrate the encoded information and temporal relationships of the image blocks and can characterize the visual content of the video.

[0029] As an optional implementation, each image block extracted from the video data is first encoded. During encoding, the focus is on capturing local visual information such as texture and edges within the image block, converting each image block into a fixed-dimensional vector. Then, according to the spatial arrangement of the image blocks within the same frame, the encoded vectors of all image blocks in that frame are combined to form a single-frame embedding sequence. Subsequently, all video frames are traversed, and the single-frame embedding sequences are sequentially concatenated according to the temporal order of the video frames, maintaining the complete structure of each single-frame embedding sequence during concatenation to avoid disordering the intra-frame vector order. The temporal relationship of all single-frame embedding sequences is integrated with the image block encoding information to form visual features that fully reflect the spatial details and temporal continuity of a single frame. This accurately preserves the spatial details of image blocks within a single frame, avoids the loss of visual information in a single frame, and allows for more detailed capture of local features in the encoding process.

[0030] As an alternative implementation, each image block extracted from the video data is first encoded. During encoding, the focus is on capturing local visual information such as texture and edges within the image block, converting each image block into a fixed-dimensional vector. Then, according to the spatial arrangement of the image blocks within the same frame, the encoded vectors of all image blocks in that frame are combined to form a single-frame embedding sequence. Subsequently, all video frames are traversed, and the single-frame embedding sequences are sequentially concatenated according to the temporal order of the video frames, maintaining the complete structure of each single-frame embedding sequence during concatenation and avoiding disorder of the intra-frame vector order. Finally, the temporal relationship of all single-frame embedding sequences is integrated with the image block encoding information to form visual features that fully reflect the spatial details and temporal continuity of a single frame. This method captures cross-frame correlation information in advance through frame grouping, strengthening the temporal dimension feature correlation and reducing the fragmentation of inter-frame information during subsequent concatenation.

[0031] Step S20: Extract and transcribe the audio from the video data to obtain the text features corresponding to the video data.

[0032] In this embodiment, extraction refers to the processing operation of separating independent audio signals from the composite data stream of video data. Transcription refers to the process of converting the speech content in the audio signal into text form. The text features corresponding to the video data refer to the structured features that characterize the semantic information of the audio after processing the text content obtained by transcribing the audio.

[0033] As an optional implementation, independent audio signals are separated from video data. During the separation process, visual signals and other interference signals in the video data stream are filtered to ensure the purity of the acquired audio signal. Next, the separated audio signal is preprocessed to remove background noise and adjust the audio signal's volume equalization, thereby optimizing the audio signal's recognizability. Subsequently, a transcription operation is performed on the preprocessed audio signal, converting it into text content segment by segment according to the playback sequence of the audio signal, while simultaneously recording the temporal relationship between each text segment and its corresponding audio segment. Finally, the transcribed text content undergoes semantic analysis, removing redundant expressions, correcting transcription errors, and transforming the analyzed text content into a sequence that reflects the semantic logic of the audio, forming text features corresponding to the video data. The text features generated by this method have high semantic accuracy and can accurately reproduce the speech content in the audio, making it suitable for scenarios with high requirements for audio semantic accuracy.

[0034] As an alternative implementation, the video data is divided into multiple consecutive video segments at preset time intervals, ensuring that each video segment contains complete audio information units. The corresponding audio signal is then extracted from each video segment, and the temporal correspondence between the audio signal and its respective video segment is recorded simultaneously during extraction. Subsequently, a transcription operation is performed on the audio signal of each video segment. During transcription, the temporal position information of the video segment is incorporated to add temporal markers to the transcribed text content. Finally, all time-marked text content is integrated, and the semantic relationships of the text content are analyzed according to the temporal order of the video segments to construct a text sequence containing temporal information and semantic logic, forming the text features corresponding to the video data. The text features generated by this method have strong temporal correlation and can clearly reflect the correspondence between audio content and video temporal sequence, making it suitable for scenarios that require emphasis on audio-video temporal synchronization.

[0035] Step S30: Based on the target multimodal cue aggregator, align the visual features and the text features in the semantic space to generate multimodal features.

[0036] In this embodiment, the target multimodal cue aggregator refers to a dedicated processing component used to integrate different modal features such as visual and textual features to achieve semantic association and fusion between modalities. Semantic alignment refers to the process of mapping different modal features to a unified semantic dimension and adjusting the feature distribution to match semantic associations between modalities. Multimodal features refer to comprehensive features that, after alignment and fusion, simultaneously contain both visual and textual semantic information.

[0037] As an optional implementation, visual and textual features are input separately into a target multimodal cue aggregator, transforming both types of features into a unified semantic space dimension. Next, in the aggregator's semantic association calculation module, the semantic similarity between visual and textual features is calculated unit by unit. Based on the similarity results, the local weights of the two features are adjusted, giving higher weights to semantically similar feature units. Subsequently, the feature fusion module integrates corresponding units of visual and textual features according to the adjusted weights, preserving the core semantic information of both features. Finally, the integrated features are normalized to generate multimodal features that combine visual details and textual semantics. This method, through unified semantic space mapping and weight adjustment, can accurately match the core semantics of the two types of features, reducing semantic bias between modalities.

[0038] As an alternative implementation, a temporal correlation module is invoked within the target multimodal cue aggregator to extract temporal information from visual features and text features, establishing a temporal correspondence between the two types of features. Next, the visual features, text features, and temporal correspondence are input into the semantic matching module. Based on the temporal correlation, semantic units at the same time node in both features are located, and semantic consistency is verified for each semantic unit at each time node. Subsequently, the aggregator's dynamic fusion module fuses the verified semantic units segment by segment according to time nodes, while semantically correcting the units that fail verification before fusion. Finally, the fusion results of all time nodes are summarized to form a multimodal feature arranged chronologically and with semantic coherence. This method ensures the temporal logical coherence of the fused features and avoids temporal misalignment issues.

[0039] Step S40: Retrieve relevant semantic fragments of the user question from the multimodal features, and generate the answer content corresponding to the user question based on the semantic association between the retrieval results and the historical context.

[0040] In this embodiment, the relevant semantic fragments of a user question refer to the local feature units in the multimodal features that semantically match the core needs of the user question. Historical context refers to the dialogue information and associated semantics generated during the user's previous interactions with the system. Generating answer content refers to constructing a user question response text that conforms to natural language logic based on the retrieved semantic fragments and historical context.

[0041] As an optional implementation, user questions are transformed into structured semantic vectors. Using these vectors as a basis, all semantic units in the multimodal features are traversed, and the similarity between each semantic unit and the user question's semantic vector is calculated. Semantic units with sufficient similarity are selected as preliminary relevant semantic fragments. Next, semantic clustering is performed on these preliminary relevant semantic fragments, grouping fragments with consistent semantic themes together and extracting the core semantic information of each group. Then, historical context is retrieved, and the semantic relevance between the core semantic information of each group and the historical context is calculated. The group with the highest relevance is retained as the target-related semantic fragment. Finally, based on the core semantics of the target-related semantic fragments and combined with the dialogue logic of the historical context, natural language text is gradually constructed, maintaining logical coherence with the historical dialogue to form the answer content corresponding to the user question. This method can directly cover the core needs of the question and is suitable for scenarios where user question needs are clear and the requirements for targeted answers are high.

[0042] As an alternative implementation, historical context is retrieved and its core semantic themes and dialogue logic are extracted to determine the semantic understanding direction of the user's question. Based on this semantic understanding direction, multimodal features are then hierarchically filtered. First, irrelevant semantic units are excluded according to the semantic themes of the historical context, resulting in a candidate semantic range within the multimodal features. Next, within this candidate semantic range, semantic relevance is matched unit by unit for the core needs of the user's question, selecting semantic fragments that directly respond to the question's needs as target-related semantic fragments. Subsequently, these target-related semantic fragments are sorted according to their relevance to the user's question, prioritizing the most closely related fragments as the core of the answer. Finally, using the semantics of the core fragment as the core, combined with the dialogue logic of the historical context, auxiliary information from fragments with lower relevance is supplemented to construct a complete and logically coherent answer. This method maintains consistency in dialogue interaction and is suitable for multi-turn dialogue scenarios.

[0043] For example, refer to Figure 2 , Figure 2 This is a diagram of the algorithm architecture of this application. In the dialogue generation scenario, firstly, the visual encoder encodes the input video image blocks to obtain an embedding sequence, then extracts spatial features through spatial pooling and temporal features through temporal pooling. Simultaneously, the audio in the video is transcribed, and text features are generated using a text encoder. Next, a multimodal cue aggregator aligns the visual and text features in the semantic space to generate multimodal features. Then, the multimodal cue aggregator inputs the multimodal features and the user question, such as "Please summarize the main content of this video," into a large language model (LLM). The LLM retrieves relevant semantic fragments from the multimodal features and generates an answer by combining them with historical context, such as generating "This video is a recording of the opening ceremony of ××, a wonderful opening ceremony performance" in the example.

[0044] By encoding visual features from video image blocks and transcribing textual features from audio, aligning them with an aggregator to generate multimodal features, and then retrieving semantic fragments and combining them with context to generate answers, the semantic alignment accuracy of multimodal features and the retrieval accuracy of semantic fragments related to user questions are improved.

[0045] Based on any of the above embodiments, in Embodiment 2 of this application, step S10 includes steps A11 to A13: Step A11: Linearly embed each image patch into a fixed-dimensional vector, and model according to the global context to output the embedding vector corresponding to each image patch.

[0046] In this embodiment, linear embedding refers to the operation of converting visual information of image patches into vectors through linear transformation. A fixed-dimensional vector refers to a numerical sequence with a uniform length and a structured representation of image patch information. Global context modeling refers to the process of integrating information from all image patches to establish interrelationships; the embedded vector is a vector representing the semantics of the image patch after the embedding operation.

[0047] As an optional implementation, raw features representing local visual information are extracted from each image patch. A linear transformation is then performed on these raw features, mapping them to fixed-dimensional initial vectors. Subsequently, the initial vectors of all image patches are collected to construct a global feature matrix. This matrix is ​​used to calculate the association weights between each image patch and all other image patches. These association weights are then used to weight and aggregate the initial vectors to achieve global context modeling. Finally, the weighted aggregation result is used as the embedding vector for each image patch. This method directly performs global modeling on all image patches, comprehensively capturing the global relationships between them.

[0048] As an alternative implementation, all image patches are divided into multiple groups according to preset rules. Each group contains several adjacent image patches. Local features are extracted from the image patches within each group, and then a linear transformation is performed on the local features of each image patch within the group to obtain temporary vectors of fixed dimensions within the group. Next, the association between each temporary vector and other temporary vectors within the group is calculated to generate a context vector within the group, which is then fused with the corresponding temporary vector to obtain the intermediate vector of the image patch within the group. The intermediate vectors of all groups are collected, and cross-group global context modeling is performed. The association weights of each intermediate vector with intermediate vectors of other groups are calculated, and the intermediate vectors are weighted and aggregated again to obtain the embedding vector corresponding to each image patch. This method, by performing local modeling first and then global modeling through grouping, not only captures local adjacent relationships meticulously but also reduces computational complexity.

[0049] Step A12: Arrange the embedding vectors corresponding to the image blocks in the same frame according to their spatial position to obtain the frame embedding sequence corresponding to the same frame.

[0050] In this embodiment, "same frame" refers to a single complete image at the same time point in the video data stream. The embedding vector corresponding to an image patch is the vector used to represent the visual semantics of each image patch after linear embedding and context modeling. Spatial position order refers to the physical order in which image patches are arranged within a single frame. The frame embedding sequence is an ordered set of vectors formed by arranging the embedding vectors of all image patches within the same frame in spatial position order.

[0051] As an optional implementation, the spatial location identifiers of each image block within a single frame are first determined. Then, the embedding vector, modeled using global context modeling, is retrieved for each image block within that frame. Following the spatial location order of traversing each row horizontally from left to right and each column vertically from top to bottom, starting from the top left corner of the image, the embedding vectors of each image block are selected one by one. These vectors are then concatenated into a continuous vector sequence, ultimately yielding the frame embedding sequence corresponding to the same frame. The frame embedding sequence generated by this method can meticulously reflect the visual features and spatial arrangement of each local area within a single frame, making it suitable for scenarios requiring detailed analysis of local details within a single frame.

[0052] As an alternative implementation, the space of a single-frame image is divided into multiple preset logical sub-regions, each containing several adjacent image blocks. Then, for all image blocks within each sub-region, their embedding vectors are retrieved, and the fusion features of the vectors within the sub-region are calculated to obtain the feature vector of the sub-region. Next, according to the spatial order of the sub-regions within the single-frame image, the feature vectors of each sub-region are selected sequentially, and these sub-region feature vectors are concatenated in order to generate a frame embedding sequence. The frame embedding sequence generated by this method focuses more on region-level spatial semantic generalization and is suitable for scenarios that emphasize the overall spatial layout and semantic induction of a single frame.

[0053] Step A13: Concatenate the frame embedding sequence of all frames according to the temporal order of the video data to obtain the visual features containing video spatiotemporal information.

[0054] In this embodiment, the temporal order of video data refers to the sequential arrangement of video frames in the time dimension, that is, the order in which the frames are played.

[0055] As an optional implementation, the video data frames are divided into several consecutive time windows, each containing several consecutive frames. For all frames within each time window, their respective frame embedding sequences are retrieved, and feature aggregation is performed on the frame embedding sequences of each frame within the window to obtain the aggregated embedding sequence for that time window. Then, according to the temporal order of the time windows, the aggregated embedding sequences of each time window are concatenated sequentially, with the end of the aggregated embedding sequence of the previous window connected to the beginning of the aggregated embedding sequence of the next window. Finally, the aggregated embedding sequences of all time windows are concatenated to obtain visual features containing the spatiotemporal information of the video. This method focuses more on the temporal dimension segmented aggregation semantics and is suitable for scenarios that focus on the overall temporal stage of the video rather than frame-by-frame details.

[0056] For example, in the scenario of spatiotemporal feature generation, CLIP ViT-L / 14 is used as the visual encoder. This encoder is based on the Vision Transformer (ViT) architecture, and its pre-training process uses large-scale vision-language instruction tuning data from the LLaVa framework. The CLIP model aligns image and text representations through contrastive learning, possessing powerful cross-modal semantic understanding capabilities. The LLaVa pre-training further enhances CLIP's inference capabilities for tasks such as visual question answering and image caption generation. Although the visual encoder in LLaVa was initially designed for processing single-frame images, it has been adapted to effectively model spatiotemporal information in video data. Specifically, given a video sample... ,in, Indicates the number of frames contained in the video. These represent the height and width of each frame, respectively. This refers to the number of channels in the image. First, the video sequence is split into... The image is divided into individual image frames, and each frame is input into the CLIP ViT-L / 14 visual encoder for feature extraction. The CLIP ViT-L / 14 encoder divides the image into fixed-size image blocks. These image patches are linearly embedded into vectors for global context modeling. For each frame, the image patch size in ViT is... Therefore, each frame of the image is divided into ,in, ; Therefore, we define the number of visual tokens N corresponding to each frame of the image: ; After encoding, each frame will output a dimension of The embedded sequence, where The embedding dimension is defined for each visual token. After concatenating the embeddings of all frames, the final video-level spatiotemporal feature representation is obtained: .

[0057] By using a linear embedding-based model combined with global context modeling, the problem of spatiotemporal information fragmentation and insufficient local-to-global semantic correlation in traditional video feature extraction is solved, effectively improving the accuracy and spatiotemporal awareness of video multimodal question answering.

[0058] Based on any of the above embodiments, in Embodiment 3 of this application, step S20 includes steps B11 to B13: Step B11: Extract the audio track from the video data and divide the original audio signal corresponding to the video data into multiple audio segments of fixed length.

[0059] In this embodiment, the audio track in the video data refers to an independent data stream that is stored synchronously with the video screen and carries the video sound information, including audio content such as voice and sound effects; the original audio signal corresponding to the video data is the original sound electrical signal directly extracted from the audio track without subsequent processing; the fixed-length audio segment is the audio part with the same duration obtained after dividing the original audio signal into preset uniform lengths.

[0060] As an optional implementation, the original audio signal is monitored in real time and its energy changes are recorded while the audio track of the video data is being extracted. After the audio track extraction is complete, low-energy boundary points in the original audio signal are determined based on the recorded energy change data. According to a preset fixed-length standard, the segmentation position is adjusted using the low-energy boundary points as a reference: if the audio duration between two adjacent low-energy boundary points is close to the fixed length, a fixed-length segment is directly segmented using these two boundary points as the boundary. If the duration exceeds or falls short of the fixed length, the segmentation position is fine-tuned between adjacent low-energy boundary points to ensure that the duration of the segmented segment meets the fixed-length requirement. Each final fixed-length audio segment is obtained by using the low-energy boundary points as the segmentation boundary. This method generates audio segments with strong semantic integrity, suitable for scenarios requiring the preservation of complete audio units such as speech semantics and continuous sound effects.

[0061] Step B12: Speech recognition extracts the spectrogram of each audio segment, generating a text sequence and its embedded representation corresponding to each audio segment.

[0062] In this embodiment, the spectrogram extracted from the audio segment refers to a visualized frequency distribution chart obtained after frequency analysis of the audio segment, with time as the horizontal axis, frequency as the vertical axis, and amplitude as the feature value. The text sequence refers to the continuous text sequence generated after speech recognition that corresponds to the content of the audio segment. Embedding representation refers to transforming the text sequence into a fixed-dimensional vector with semantic information, used to represent the semantic features of the text.

[0063] As an optional implementation, the spectrograms extracted from each audio segment are first preprocessed. By adjusting the frequency range and time resolution of the spectrograms, low-frequency interference regions corresponding to background noise are removed, while retaining the high-frequency effective regions concentrated in the speech signal. Next, the preprocessed spectrograms are input into the speech recognition process. Frequency features are analyzed segment by segment according to the time axis of the spectrograms, matching the corresponding speech syllables and text to generate a text sequence consistent with the audio segment content. Subsequently, semantic cleaning is performed on the generated text sequence to correct typos and omissions caused by recognition errors, ensuring the semantic integrity of the text sequence. Finally, the cleaned text sequence is mapped to a fixed-dimensional vector. By integrating the semantic relationships of words in the text, an embedding representation that can characterize the core semantics of the text sequence is generated, completing the transformation of each audio segment from spectrogram to text sequence and embedding representation. This method, through spectrogram preprocessing and text cleaning, can improve the accuracy of text recognition for individual audio segments, and the embedding representation provides a more accurate representation of the semantics of single-segment text.

[0064] Step B13: Concatenate the text sequence according to the temporal order of the audio segments, and concatenate the embedded representation according to the temporal order to generate the text features corresponding to the video data.

[0065] As an optional implementation, the temporal identifiers of all audio segments are obtained, and the temporal position of each segment in the original audio is recorded. The segments are then sorted in ascending order of temporal identifiers to determine the splicing order. Next, the text sequence corresponding to each audio segment is retrieved segment by segment in this order. The text sequence of the next segment is directly appended to the end of the text sequence of the previous segment. Punctuation marks and semantic pauses in each segment are preserved during splicing to ensure overall textual logical coherence. Simultaneously, the embedding representations of each audio segment are retrieved in the same temporal order, and the embedding representation vector of the next segment is appended to the embedding representation vector of the previous segment, forming a continuous embedding vector sequence. Finally, the semantic integrity of the spliced ​​text sequence is checked to correct semantic breaks caused by segment splicing. The embedding vector sequence is normalized to eliminate vector numerical differences, ultimately generating text features corresponding to the video data. This method generates text features with rich temporal details, accurately reconstructing the semantics of each audio segment, making it suitable for scenarios requiring fine-grained tracing of audio temporal semantics.

[0066] For example, in the scenario of text feature generation, frame-level embedding First, average pooling is performed along the spatial dimension to obtain a video-level temporal representation. This operation is performed by analyzing the spatial dimensions of each frame. Visual tokens are averaged and aggregated, implicitly incorporating the learning of temporal information. Similarly, frame-level embeddings are also averaged along the temporal dimension to generate video-level spatial representations. This operation aggregates features from all frames at the same spatial location, resulting in a representation with global spatial awareness. Subsequently, the temporal representation is... Spatial representation By concatenating the features along the dimensional lines, a unified video-level feature representation can be obtained. ,Right now Next, a simple trainable linear layer g(•) projects the video-level features into the embedding space of the multimodal cue fusion engine, transforming them into the corresponding embedding sequence of the multimodal cue fusion engine. ,Right now ,in, This represents the size of the embedding dimension in the language model. The function `g(•)` acts as an adapter module here. Although it could be implemented using more complex structures (such as multilayer perceptrons or Transformer layers), it has demonstrated competitive performance in experiments with a concise design, showing no significant loss compared to more complex structures. First, the audio track is extracted from the input video as input for subsequent speech recognition. The original audio signal corresponding to the video is represented as: ; in, This represents the total number of audio sampling points. To adapt to the input format of the Whisper model, the original audio signal... Cut the audio into multiple 30-second segments, each segment being [length missing]. There are a total of A segment, therefore It can also be expressed as: ; For each audio segment Mel spectrograms are extracted and used as input features for the Whisper model. The extracted Mel spectrograms are represented as follows: ; in For time steps, This represents the number of Mel bands. The final features of the Whisper input are: ; The Whisper decoder is an autoregressive model that predicts the text token at the current time step based on the encoder output and historical text tokens. Let the text sequence output by the decoder be: ; in Indicates the first The length of the text generated from each audio segment, and the corresponding embedding representation of the text sequence are: ; Finally, the text of all the fragments was pieced together to form the complete audio transcription result: ; The semicolon ";" indicates a concatenation operation along the sequence dimension (time dimension). For the sum of all text: .

[0067] By extracting audio spectrograms and combining them with model-generated embedding representations, the problems of disordered temporal sequence, semantic fragmentation, and insufficient feature representation in traditional audio-to-text conversion are solved, effectively improving the accuracy of audio semantic understanding and question-answering matching.

[0068] Based on any of the above embodiments, in Embodiment 4 of this application, step S30 includes steps C11 to C13: Step C11: Based on the target multimodal cue aggregator, the visual features and the text features are mapped to a common semantic space, and the feature distribution is adjusted by calculating the semantic correlation between modalities to obtain aligned visual features and text features.

[0069] In this embodiment, the intermodal semantic correlation degree refers to a quantitative indicator that measures the degree of semantic matching between visual features and text features. Aligned visual features and text features refer to features that, after correlation adjustment, have a consistent semantic distribution in a common semantic space and can be used collaboratively.

[0070] As an optional implementation, visual and textual features are input into the feature mapping module of the target multimodal cue aggregator. Through the transformation rules of this module, the two types of features are mapped from their original dimensions to a preset common semantic space, ensuring complete dimensional consistency after mapping. Next, in the aggregator's correlation calculation module, the semantic attributes of visual and textual features are compared one by one for each feature unit, calculating the semantic correlation between each visual feature unit and its corresponding textual feature unit, generating a correlation matrix. Subsequently, the feature distribution is adjusted based on the correlation matrix. Feature units with high correlation retain their original feature strength, while feature units with low correlation have their feature strength reduced proportionally, while supplementing semantically complementary information from both types of features. Finally, the adjusted visual and textual features are validated for consistency, ensuring that their semantic trends in the common semantic space are consistent, and the aligned visual and textual features are output. The aligned features generated by this method are suitable for scenarios with high semantic detail requirements, accurately associating visual details with textual descriptions.

[0071] Step C12: Calculate the association weights corresponding to the aligned visual features and text features, and integrate the key semantic information in a weighted manner to obtain the preliminary fused features.

[0072] In this embodiment, the association weight refers to the numerical value calculated based on the semantic association degree of the two features, used to measure the contribution of each semantic unit in a single feature to the fusion. Key semantic information refers to the set of semantic units in the two features that can represent the core content of the video. The preliminary fusion feature refers to the intermediate feature formed after weighting and integrating the key semantic information through association weights, which combines visual and textual semantics.

[0073] As an optional implementation, a global visual theme is extracted from aligned visual features, and a global textual theme is extracted from textual features to determine the global semantic association direction between the two features. Next, the matching degree between the global visual theme and the global textual theme is calculated, and this matching degree is used as the basic association weight. Then, local weights are assigned to each semantic unit within a single feature based on the degree of association between the semantic unit and the global theme. The basic association weights are then multiplied by the local weights to obtain the final association weight for each semantic unit. Afterward, key semantic information from the two features is selected according to the final weights, and the visual key information and textual key information are hierarchically integrated according to their weight ratios. High-weight information retains core details first, while low-weight information retains only the summary, ultimately generating a preliminary fusion feature with a clear structure and highlighting the global theme. The preliminary fusion feature generated by this method focuses more on global semantic integration and is suitable for scenarios that require quickly grasping the overall content of the video (such as summarizing the video theme), highlighting the core semantic trends of the video.

[0074] Step C13: Model the preliminary fused features based on temporal dependencies, and optimize the feature expression through nonlinear transformation and normalization to generate multimodal features.

[0075] In this embodiment, temporal dependency refers to the sequential association and mutual influence of different parts of the initially fused features in the time dimension. Modeling refers to constructing a feature structure that reflects temporal dependency and strengthening the temporal correlation between features. Nonlinear transformation refers to adjusting the feature distribution through processing non-linear relationships to enhance the discriminative power of semantic representation. Normalization refers to bringing feature values ​​within a uniform range and eliminating the magnitude differences between different parts of the features.

[0076] As an optional implementation, the initially fused features are divided into multiple temporal units at fixed time intervals. Each unit contains multiple consecutive feature segments. The co-occurrence frequency and semantic coherence of each feature segment within each temporal unit are calculated to construct a local temporal dependency structure within the unit. Then, the semantic theme changes of adjacent temporal units are analyzed, and the transition correlation between units is calculated to construct a global temporal dependency structure between units. The local and global dependency structures are integrated to generate hierarchical temporal features. A nonlinear transformation is applied to the hierarchical temporal features to enhance the temporal stage discriminativeness of the features by highlighting the key semantics at the transitions between units. Finally, normalization is performed to balance the numerical proportions of features within and between units, generating multimodal features that emphasize temporal stage correlations. The multimodal features generated by this method are suitable for scenarios requiring an understanding of the overall temporal stages and can clearly reflect the temporal structure and stage semantics of the video.

[0077] For example, in a dialogue generation scenario, an efficient multimodal cue aggregator is constructed to deeply fuse visual features from videos with text features generated from audio transcription, thereby generating a unified multimodal feature representation with rich semantics. Specifically, the system first extracts visual features containing spatiotemporal information from the video, and simultaneously uses the Whisper model to perform speech recognition on the audio tracks in the video, generating corresponding text content. Then, the information from these two heterogeneous modalities is input into the multimodal cue aggregator for cross-modal semantic understanding and information integration. This aggregator consists of several functional components, including: a cross-modal alignment component for identifying and aligning semantic relationships between visual and textual features; an attention fusion component that dynamically fuses key information through an attention mechanism; a context modeling component that captures temporal dependencies to enhance semantic coherence; and a feature enhancement component that optimizes the expressive power of the fused features. In actual operation, the aggregator first performs semantic space alignment processing on visual and textual features, enabling information from different modalities to interact within a common semantic dimension. Subsequently, through an attention mechanism, key semantic information from visual and textual modalities is automatically weighted and fused according to different task requirements, achieving dynamic information complementarity. Next, a temporal modeling module based on Transformer or LSTM structures is employed to perform temporal context modeling on the fused features, effectively capturing the development process of events or actions in the video. Finally, a lightweight feature enhancement module performs nonlinear transformation and normalization on the fused representation, further improving its expressive power and generalization performance, resulting in multimodal features. This enhancement module not only achieves efficient collaboration and complementarity between visual and textual information but also significantly enhances the consistency of multimodal semantics. After the multimodal cue aggregator fuses visual and textual features, the system generates a unified multimodal feature representation with rich semantics. This feature not only includes visual information such as scenes, actions, and objects in the video but also incorporates speech semantic content extracted from the audio, possessing strong context awareness and semantic expression capabilities. Simultaneously, questions posed by users in natural language are encoded into corresponding text embeddings, serving as guiding information for model inference. Next, the Large Language Model (LLM) receives two key inputs: a multimodal feature sequence output from the multimodal cue aggregator and a question embedding sequence transformed from the user-input questions. These two sequences are jointly modeled within the language model. The model selectively focuses on key information in multimodal features through self-attention and cross-modal attention mechanisms, and combines the semantics of the question to make inferences and judgments.

[0078] By mapping visual and textual features to a common semantic space and combining the semantic correlation between modalities, the semantic misalignment problem caused by modal differences in traditional multimodal fusion is solved, effectively improving the semantic matching accuracy and answer accuracy of video question answering.

[0079] Based on any of the above embodiments, in Embodiment 5 of this application, referring to Figure 3 , Figure 3 This is a flowchart illustrating the fifth embodiment of the dialogue generation method based on multimodal cue aggregation in this application. Step S40 includes steps D11 to D15: Step D11: The user question is converted into a text embedding sequence consistent with the multimodal feature semantic space through vector mapping operations.

[0080] In this embodiment, vector mapping refers to the process of transforming the textual information of the user question into vectors with specific dimensions and semantic attributes. The multimodal feature semantic space refers to the dimensional space in which multimodal features reside, used to uniformly represent the semantics of different modalities. The text embedding sequence refers to the ordered set of vectors formed after vector mapping of the user question, which has the same dimensions as the multimodal feature semantic space.

[0081] As an optional implementation, semantic parsing is performed on the user question to break it down into the core words and phrases, remove redundant words without practical meaning, and extract the core semantic units that reflect the user's needs. Next, the dimensional standard of the multimodal feature semantic space is determined, and a vector mapping rule matching this dimension is selected. Following the logical order of the core semantic units in the question, each unit is transformed into a corresponding vector through vector mapping operations. These corresponding vectors are arranged according to the logical order of the semantic units to form an initial text embedding sequence. Finally, the initial sequence undergoes semantic consistency verification, comparing the semantic attributes of each vector in the sequence with those of the multimodal feature semantic space. The vector values ​​are fine-tuned to ensure that the overall sequence is consistent with the semantic logic of the semantic space, resulting in the final text embedding sequence. This method is suitable for scenarios where the semantic structure of the user question is complex and contains multiple key needs, and the generated sequence can meticulously match the needs of each part of the question.

[0082] Step D12: Using the text embedding sequence as the retrieval benchmark, calculate the semantic similarity between the text embedding sequence and each semantic unit in the multimodal features, and filter out semantic units with similarity higher than a preset threshold to form a candidate semantic segment set.

[0083] In this embodiment, the retrieval benchmark refers to a reference standard used to measure the relevance of semantic units in multimodal features. Each semantic unit in a multimodal feature refers to a basic component that carries specific semantic information. Semantic similarity is a quantitative indicator of the degree of semantic matching between a text embedding sequence and a semantic unit. The preset threshold is a pre-set similarity threshold used to determine whether semantic units are related. The candidate semantic segment set is a set containing all semantic units that meet the threshold requirements after similarity filtering.

[0084] As an optional implementation, the text embedding sequence is split into several sub-vectors, each corresponding to a core semantic point in the user's question. Each semantic unit in the multimodal features is traversed, and the semantic similarity between that unit and each sub-vector is calculated, recording all similarity values. For each semantic unit, the average similarity between it and all sub-vectors is taken as the unit's comprehensive similarity. The comprehensive similarity is compared with a preset threshold, and semantic units with a comprehensive similarity higher than the threshold are selected. These semantic units are arranged in chronological order in the multimodal features to form a candidate semantic fragment set. This method can accurately capture detailed units that match the core semantic point of the user's question, and the selected results have strong semantic specificity.

[0085] As an alternative implementation, multimodal features are divided into several consecutive semantic blocks according to semantic themes, with each semantic block containing multiple closely related semantic units. The overall semantic similarity between the text embedding sequence and each semantic block is calculated, and semantic blocks with an overall similarity higher than a preset threshold are selected. For each selected semantic block, the individual similarity between each semantic unit within the block and the text embedding sequence is further calculated, retaining units with unit-level similarity higher than the threshold. All eligible units are concatenated according to their original position order in the multimodal features to form a candidate semantic fragment set. This method, by selecting from thematic blocks and then refining to units, reduces unnecessary computation, has high processing efficiency, and preserves the overall association of semantic blocks.

[0086] Step D13: Extract historical context information related to the user's question from the historical context semantic database, conduct a secondary evaluation of the logical coherence and demand matching degree of the candidate semantic fragment set, and retain the qualified fragments as target-related semantic fragments.

[0087] In this embodiment, the historical context semantic database refers to a database that stores dialogue semantics, demand preferences, and related information generated during past user interactions with the system. The candidate semantic fragment set refers to the set of semantic units related to the text embedding sequence in multimodal features after semantic similarity filtering. Demand matching degree refers to the degree to which the candidate fragment matches the core demand of the current user problem. Secondary evaluation refers to a further filtering and judgment of the candidate fragment set based on historical context information. Target-related semantic fragments refer to the semantic fragments that are highly relevant to the user problem and are ultimately retained after passing the secondary evaluation.

[0088] As an optional implementation, based on the core semantic theme of the current user question, historical context information containing the same or similar themes is retrieved from the historical context semantic database, extracting key content such as needs preferences and dialogue logic. Two core dimensions for secondary evaluation are determined: logical coherence. This is assessed by evaluating whether the candidate segments are sequentially coherent, semantically connected, and matching in terms of needs, thus evaluating the degree to which the candidate segments fit the user's core needs and historical preferences. Each segment in the candidate semantic segment set is analyzed one by one, and its compliance with the evaluation dimensions is determined by combining historical context information: if the segment's temporal sequence is consistent with historical dialogue logic, its semantics connect previous and subsequent segments, and its core content matches the user's current and historical needs, it is marked as qualified. After all segments have been evaluated, all qualified segments are collected and arranged according to their original temporal order in multimodal features to form target-related semantic segments. This method directly combines historical context information to evaluate each segment, accurately matching the user's current and historical needs and avoiding the selection of segments that are disconnected from past interaction logic.

[0089] Step D14: Using the target-related semantic fragments as core semantic support and combining them with historical context information, an autoregressive generation mode is adopted to gradually generate preliminary response text.

[0090] In this embodiment, core semantic support refers to the basic information that serves as the primary semantic source and determines the core content of the answer when generating the response text. Historical context information refers to past interactive dialogues and demand preference information related to the current user's question, stored in the semantic database. The autoregressive generation mode refers to the generation method that constructs the answer sentence by sentence / word by word according to the text generation order, with the preceding generated content providing semantic logical support for the subsequent content. The preliminary response text refers to the initial response content generated autoregressively based on core semantics and historical information, but without final optimization.

[0091] As an optional implementation method, core semantic points are extracted from target-related semantic fragments and arranged according to the time sequence and causal relationship in a preset semantic logic to generate a core semantic framework for the answer. Historical context information is retrieved, and past demand expressions and dialogue logic related to the core semantic framework are selected and integrated into the core framework as semantic supplements. An autoregressive generation mode is initiated, generating text sentence by sentence, starting from the first semantic point of the core framework. For each generated sentence, its semantic coherence with historical context information is compared, and word choice and sentence structure are adjusted to ensure natural flow. After the text transformation of all core semantic points is completed, transitional statements are added to ensure overall logical flow, generating a preliminary answer text. This method, by adjusting sentence by sentence in conjunction with historical context, can precisely guarantee the semantic coherence of the answer with past interactions, and the logical arrangement of core semantic points is rigorous, resulting in high answer accuracy.

[0092] Step D15: Semantic coherence detection and factual accuracy verification of the preliminary answer text, correcting content with logical contradictions or information deviations, and generating the answer content corresponding to the user question.

[0093] In this embodiment, semantic coherence detection refers to the process of checking whether the logical connections between sentences in the preliminary response text are smooth and whether the semantics are coherent and uninterrupted. Factual accuracy verification refers to verifying whether the information in the preliminary response text is consistent with objective facts and the core semantics of multimodal features, and eliminating erroneous information. Logical contradiction refers to logical problems such as semantic conflicts, reversed causes and effects, and disordered timing between sentences in the preliminary response text. Information deviation refers to erroneous content in the preliminary response text that does not conform to objective facts and the core information of target-related semantic fragments. The answer content corresponding to the user's question refers to the final answer that is logically coherent and accurately informative after coherence detection and accuracy verification.

[0094] As an optional implementation, the initial response text is divided into multiple paragraphs based on semantic themes. Each paragraph's sentences are checked to ensure they revolve around the same theme and that the logic between sentences is consistent. Transitional sentences between paragraphs are also checked for natural theme shifts, and inconsistencies in logic within paragraphs and lack of connection between paragraphs are identified. The core factual points of each paragraph are then extracted and double-checked using historical context and target-related semantic fragments: the core points are compared to the historical context for consistency, and to ensure they conform to the core semantics of the target-related semantic fragments. Discrepancies between both are identified. For paragraphs with logical inconsistencies, the sentence order is reorganized and additional explanations are added. For paragraphs lacking connection, transitional paragraphs are added. Discrepancies are corrected based on target-related semantic fragments to ensure consistency with the historical context. Finally, the overall text logic and information accuracy are verified to generate the response to the user's question. This method generates response text with clear overall logic, high consistency with historical interactions, and is suitable for multi-turn dialogue scenarios, ensuring the coherence and overall consistency of the response.

[0095] For example, in a user-question dialogue scenario, the Large Language Model (LLM) first retrieves the most relevant semantic fragments from multimodal features based on the question content, such as specific actions, scenes, or audio content in a video. Subsequently, the model progressively generates answers in natural language form during the decoding process. Each generation step relies on the semantic association between the currently generated content and the historical context, as well as the semantic support provided by the multimodal features. The entire generation process is autoregressive, meaning that each new word generated is based on a joint understanding of previously generated words and multimodal information, thus ensuring the coherence and accuracy of the output content. Furthermore, the Large Language Model also possesses certain logical reasoning, common sense understanding, and contextual modeling capabilities, enabling it to perform advanced cognitive tasks such as causal reasoning, temporal judgment, and object recognition by combining multimodal information. For instance, when a user asks "What happened in the video?" or "When did someone start speaking?", the model can comprehensively infer a reasonable answer by combining visual action changes, time information, and audio content. Ultimately, the large language model outputs a well-structured, semantically coherent, and user-intent-aligned natural language response, which can be used in various application scenarios such as video content understanding, video question answering, intelligent dialogue, and summary generation. The entire process achieves a complete closed loop from raw video input to multimodal understanding and then to natural language output, fully demonstrating the system's capabilities in cross-modal semantic modeling and intelligent generation.

[0096] Thanks to the powerful contextual understanding and generation capabilities of the large language model, as well as the information integration mechanism of the multimodal cue aggregator, the system can still stably output high-quality, semantically consistent dialogue content when faced with noisy and structurally complex real-world video scenarios.

[0097] Based on any of the above embodiments, in Embodiment Six of this application, before step S10, steps E11 to E13 are further included: Step E11: Extract the video data frame by frame at fixed time intervals, split the continuous video stream into multiple independent image frames, and obtain the image frame set corresponding to the video data.

[0098] In this embodiment, fixed time interval frame extraction refers to the operation of acquiring a single frame from a continuous video stream at a pre-set uniform time interval. A continuous video stream refers to a dynamic sequence of video data presented uninterruptedly in the time dimension. An independent image frame refers to a single frame extracted from the video stream that can exist as a static image on its own. The set of image frames corresponding to the video data refers to the whole formed by arranging all the extracted independent image frames in their original temporal order.

[0099] As an optional implementation, the time interval for each extracted image frame is determined according to a preset fixed time interval standard. The total duration and original frame rate of the video data are read, and the total number of image frames to be extracted and the timestamp corresponding to each frame are calculated based on the fixed time interval. Starting from the beginning of the video stream, the corresponding frames are located and extracted sequentially according to the calculated timestamps, ensuring that each extracted frame strictly corresponds to the set time interval. Each extracted independent image frame undergoes integrity verification, checking for issues such as missing pixels or color anomalies, and valid image frames are retained. All valid image frames are arranged according to their original temporal order in the video stream, forming a set of image frames corresponding to the video data. This method generates an image frame set with strong temporal consistency, suitable for scenarios requiring accurate reconstruction of the time sequence information of video at each interval.

[0100] As an alternative implementation, the video data is preprocessed to remove unstable segments caused by image jitter and sudden changes in lighting, retaining only clear and color-stable valid video segments. After determining a fixed time interval standard, the number of image frames to be extracted and the timestamp range within each valid video segment are calculated based on its duration. Near each set timestamp, a small time interval is selected before and after it, and the clarity of all images within each interval is compared. The image with the highest clarity is selected as the extraction frame corresponding to that time interval. Color consistency adjustment is performed on each selected individual image frame to ensure that the color parameters of different frames are within a similar range. Finally, all adjusted image frames are arranged in the original temporal order of the valid video segments to form the image frame set corresponding to the video data. The image frame set generated by this method has excellent image quality and is suitable for scenarios requiring detailed analysis based on high-quality frames.

[0101] Step E12: Divide the image frames evenly according to a preset fixed size to obtain an initial image block group corresponding to each image frame.

[0102] In this embodiment, the preset fixed size refers to a pre-defined uniform size standard used for segmenting image frames. Uniform division refers to the operation of dividing an image frame into multiple parts without overlap or omission according to the fixed size rules. An image frame refers to an independent static image extracted from a video stream. The initial image block group refers to the set of all independent image blocks with fixed sizes obtained after uniformly dividing the image frame.

[0103] As an optional implementation, based on preset fixed size parameters, the width and height of a single image block are determined. The overall resolution of the image frame is read, and the number of blocks that can be divided along the width direction of the image frame is calculated. If the image frame width is divisible by the fixed width, it is directly divided according to the fixed width; otherwise, the image frame width is padded to an integer multiple of the fixed width, and the padded area is filled with a value similar to the edge pixel color before dividing according to the fixed width. Similarly, the number of blocks in the height direction is calculated and processed. Starting from the upper left corner of the image frame, the image frame is divided into multiple independent image blocks by cutting along the width direction one block at a time, moving down a fixed height after completing a row, and cutting the next row. The size of each block strictly matches the preset fixed size. All image blocks are arranged according to the cutting order to form the initial image block group corresponding to the image frame. The block group generated by this method has strong uniformity in size and is suitable for batch standardization processing scenarios.

[0104] Step E13: Based on the temporal order of the image frames in the video data and the spatial order of each image frame, summarize the initial image block group corresponding to the image frames to generate the image block sequence corresponding to the video data.

[0105] In this embodiment, the temporal order of image frames in video data refers to the order in which image frames are arranged after being extracted from the video stream, according to the original playback order of the video. The spatial order corresponding to each image frame refers to the arrangement order of each block in the initial image block group of a single image frame from left to right and from top to bottom within the image frame after the frame is uniformly divided.

[0106] As an optional implementation, the temporal identifiers of all image frames are obtained, and the extraction time point of each frame in the video is recorded. The frames are then sorted by temporal identifier in ascending order to determine the overall temporal order of the image frames. For each sorted image frame, its corresponding initial image block group is retrieved, and image blocks are extracted one by one according to their spatial position within the frame. Starting from the first temporal frame, all its image blocks are arranged into subsequences according to their spatial position. The subsequence of the next temporal frame is then directly concatenated to the end of the subsequence of the previous frame, ensuring that the last block of the previous frame's subsequence is temporally continuous with the first block of the next frame's subsequence. This concatenation operation is repeated until the subsequences of all frames are integrated. Finally, the temporal assignment and spatial position marker of each block are verified to ensure no blocks are missing or out of order, generating the image block sequence corresponding to the video data. The sequence generated by this method is suitable for scenarios requiring detailed analysis of frame-by-frame and block-by-block details, providing comprehensive spatiotemporal information support for subsequent feature extraction.

[0107] For example, in a multimodal feature generation scenario, the video data is extracted frame by frame at fixed time intervals. First, a preset fixed time interval is determined, the total duration and original frame rate of the video data are read, the total number of independent image frames to be extracted and the timestamp corresponding to each frame are calculated, starting from the start time of the continuous video stream, the images are located and extracted sequentially according to the timestamps, and the pixel integrity and color consistency of each extracted independent image frame are checked. Qualified frames are retained and arranged in the original playback sequence of the video to obtain the set of image frames corresponding to the video data. Then, the image frames are evenly divided according to a preset fixed size, the width and height parameters of a single image block are determined, and the number of blocks that can be divided in the width and height directions of each image frame is calculated. If the resolution of the image frame cannot be divided by the fixed size, edge pixel color filling is used to fill it to an integer multiple of the fixed size. Starting from the upper left corner of the image frame, it is divided according to the spatial position order of "first cutting along the width direction block by block, then cutting along the height direction row by row". The divided independent image blocks are arranged according to the cutting order to obtain the initial image block group corresponding to each image frame. Finally, based on the temporal order of the image frames in the video data, the image frame set is sorted by timestamp from smallest to largest. For each sorted image frame, its initial image block group is retrieved and formed into a block sub-sequence according to spatial position. The block sub-sequence of the next frame is directly connected to the end of the block sub-sequence of the previous frame to ensure the temporal continuity between frames and the spatial order within blocks. The temporal assignment and spatial position mark of all blocks are verified to generate the image block sequence corresponding to the video data.

[0108] By performing frame extraction at fixed time intervals, the problem of temporal disorder in traditional frame extraction is solved, effectively supporting the accuracy and efficiency of subsequent multimodal processing.

[0109] Based on any of the above embodiments, in Embodiment Seven of this application, referring to Figure 4 , Figure 4 This is a flowchart illustrating the seventh embodiment of the dialogue generation method based on multimodal cue aggregation in this application. Before step S30, steps F11-F14 are also included: Step F11: Initialize the basic framework of the multimodal cue aggregator, build the basic network pathway for feature transfer, and obtain the initial framework of the aggregator.

[0110] In this embodiment, the basic framework of the multimodal cue aggregator refers to the core structural system supporting multimodal cue processing and integration, including the basic layout and collaborative logic of each functional module. The basic network path for feature flow refers to the channel used to realize the transmission, interaction, and transformation of different modal features within the aggregator, ensuring that features can flow according to preset logic. The initial framework of the aggregator refers to the initial structure with basic multimodal feature processing capabilities formed after the basic framework initialization and feature flow path construction are completed.

[0111] As an optional implementation, the modality types that the multimodal cue aggregator needs to process are determined, and a basic framework module is defined according to a three-layer structure of input, processing, and output: the input layer sets up feature receiving modules corresponding to each modality, the processing layer is divided into feature preprocessing and feature interaction sub-modules, and the output layer sets up an aggregated feature output module. Next, the basic network pathways for feature flow are constructed: pathways are led out from each modality receiving module in the input layer, connecting to the corresponding preprocessing sub-module in the processing layer; the output pathways of the preprocessing sub-modules connect to the feature interaction sub-modules; and the interaction sub-modules then connect to the output layer modules through pathways. Finally, the integrity of the connection between each module and pathway is checked. After confirming that there are no broken pathways or isolated modules, the module layout and pathway logic are solidified, resulting in the initial framework of the aggregator. The initial framework generated by this method is suitable for scenarios with fixed modality types and standardized feature processing procedures.

[0112] Step F12: Based on the initial aggregator framework, embed cross-modal alignment components, dynamic weighted fusion components, and temporal modeling components to generate the aggregator framework.

[0113] In this embodiment, the cross-modal alignment component refers to a functional component used to map features from different modalities to a unified semantic space, eliminating modal differences and achieving semantic matching. The dynamic weighted fusion component refers to a functional component that dynamically allocates weights and integrates features based on the semantic contribution of different modal features. The temporal modeling component refers to a functional component that captures the correlation between multimodal features in the time dimension and strengthens temporal semantics. The aggregator framework refers to a complete multimodal processing structure that, after embedding the three types of components, possesses the capabilities of cross-modal alignment, dynamic fusion, and temporal modeling.

[0114] As an optional implementation, the initial three-layer structure of the aggregator framework—input, processing, and output—is analyzed first. After embedding the cross-modal alignment component into the feature interaction submodule of the processing layer, it receives preprocessed multimodal features, transforms different modal features into a unified dimension through semantic mapping rules within the component, and outputs aligned features. A dynamic weighted fusion component is connected to the output of the cross-modal alignment component. This component dynamically calculates and assigns fusion weights based on the semantic clarity of each aligned feature, integrating the aligned features into preliminary fused features according to the weights. Finally, a temporal modeling component is embedded between the processing and output layers. This component receives the preliminary fused features, captures the temporal dependencies of the features through temporal correlation analysis logic, strengthens the temporal semantic expression, and transmits the processed features to the output layer, completing the embedding of all components and generating the aggregator framework. The aggregator framework generated by this method is suitable for scenarios with fixed modality types and standardized feature processing procedures.

[0115] Step F13: Initialize the similarity calculation threshold of the cross-modal alignment component, the attention mechanism scaling factor of the dynamic weighted fusion component, and the network layer parameters of the temporal modeling component, and output the aggregator model to be trained.

[0116] In this embodiment, the similarity calculation threshold of the cross-modal alignment component refers to the critical value used in the cross-modal alignment component to determine whether the semantic matching degree of different modal features such as visual and textual features meets the standard. The attention mechanism scaling factor of the dynamic weighted fusion component refers to the key parameter in the dynamic weighted fusion component that adjusts the range of attention weight allocation and avoids excessive concentration or dispersion of weights. The network layer parameters of the temporal modeling component refer to the numerical parameters in the temporal modeling component that control the network operation logic and feature processing method. The aggregator training model refers to the aggregator model that has basic multimodal processing functions after the key parameters of each component have been initialized, but whose parameters need to be optimized through training.

[0117] As an optional implementation, an empirical range for the similarity calculation threshold of the cross-modal alignment component is determined. Considering the characteristics of visual and text modalities, the median value of this range is selected as the initial threshold to ensure that feature pairs with moderate matching degrees can be initially screened. Next, an attention mechanism scaling factor is set based on the number of modalities accessed by the dynamic weighted fusion component; the more modalities, the larger the scaling factor value, to balance the weight distribution of each modality. Then, the network layer parameters of the temporal modeling component are assigned initial values ​​to each computational unit using a basic initialization method to ensure that the parameters are within a reasonable computational range. Finally, the initialization parameters of the three components are checked to ensure they are compatible with the aggregator framework interface. After confirming that no parameters exceed the component's processing range, all components and parameters are integrated, and the aggregator model to be trained is output. This method does not rely on training data, and the initialization process is fast and direct, making it suitable for scenarios lacking initial data or requiring rapid model prototype building. Step F14: Train the aggregator training model based on the labeled multimodal training data, optimize the parameter configuration of the aggregator training model through the loss function, and generate the target multimodal cue aggregator.

[0118] In this embodiment, labeled multimodal training data refers to a set of training samples containing multiple modal features such as visual and textual features, and accompanied by corresponding semantic labels. Parameter configuration refers to all numerical parameters of the cross-modal alignment, dynamic weighted fusion, and temporal modeling components in the aggregator model. As an optional implementation, the labeled multimodal training data is split into multiple subsets based on modality type: visual and text, and visual and audio. Each subset corresponds to a modality combination scenario. Using the aggregator's training model as a foundation, training is first performed using the visual and text subsets: after inputting data, the model's prediction and annotation losses are calculated, and the alignment and fusion parameters for the corresponding modalities are optimized. After completing the visual and text modality training, the optimized parameters are fixed, and then training is performed using the visual and audio subsets, adjusting only the parameters related to the audio modality. Following an incremental training logic, all subsets are trained sequentially. After training each subset, the performance is verified using validation data for that modality combination. After all subsets are trained, the overall model parameters are fine-tuned to adapt to multimodal collaborative processing. Once the performance is confirmed to meet the standards for each modality combination scenario, the parameters are saved to generate the target multimodal cue aggregator. The target aggregator generated by this method is suitable for handling general scenarios with fixed modality combinations and large data scales.

[0119] For example, in the scenario of training a multimodal cue aggregator, the basic framework of the multimodal cue aggregator is initialized, and modules are defined in a three-layer structure of input, processing, and output: the input layer sets up feature receiving modules for visual, text, and other modalities; the processing layer is divided into feature preprocessing and feature interaction sub-modules; and the output layer sets up an aggregated feature output module, establishing a basic network path for feature flow. Pathways are led out from each modal receiving module to connect to the corresponding preprocessing sub-module, the preprocessing sub-module path is connected to the feature interaction sub-module, and the interaction sub-module path is connected to the output layer module. After checking the integrity of the module and path connections, the initial framework of the aggregator is obtained. Based on the initial framework of the aggregator, a cross-modal alignment component (used to map features of different modalities to a unified semantic space) is embedded after the feature interaction sub-module of the processing layer. A dynamic weighted fusion component (used to allocate weights according to semantic contribution and integrate features) is connected to the output of the cross-modal alignment component. A temporal modeling component (used to capture the temporal correlation of features) is embedded between the processing layer and the output layer. After confirming the compatibility between the components and the framework path, the aggregator framework is generated. Initialize the similarity calculation threshold of the cross-modal alignment component, selecting the median value from the empirical range of common modal matching scenarios; the scaling factor of the attention mechanism of the dynamic weighted fusion component, set according to the number of access modalities (the more modalities, the larger the value); and the network layer parameters of the temporal modeling component, allocating values ​​using the basic initialization method. After verifying that the parameters do not exceed the processing range of the components, output the aggregator model to be trained; train the aggregator model to be trained based on labeled multimodal training data (including visual and text pairing features and semantic labels), dividing the data into training and validation sets according to a preset ratio, batch inputting the training set data into the model, the model performs cross-modal alignment, dynamic weighted fusion, and temporal modeling, and outputs prediction results; calculate the difference between the prediction results and the labeled labels using the loss function, optimize the model parameter configuration according to the gradient descent rule, monitor the performance on the validation set, stop training if the loss value does not decrease for several consecutive rounds, save the optimal parameter configuration, and generate the target multimodal cue aggregator.

[0120] Because multimodal cue aggregators can effectively integrate information from multiple modalities such as visual, audio, and text, and improve overall semantic understanding through cross-modal alignment and collaborative modeling strategies, dialogue generation becomes more natural and coherent.

[0121] Based on any of the above embodiments, in Embodiment 8 of this application, after step S40, steps G11~G13 are further included: Step G11 involves parsing the user's input and follow-up questions to extract key issues and core needs, thus obtaining user feedback information.

[0122] In this embodiment, user input refers to text expressing the user's views, evaluations, or suggestions regarding specific content. Follow-up questions refer to text expressing further questions raised by the user after the initial interaction. Key issues refer to the most crucial and unanswered points in the user's follow-up questions. Core needs refer to the fundamental demands that the user implicitly hopes to fulfill through their opinions and questions. User feedback information refers to a structured set of information formed by integrating key issues and core needs after text parsing.

[0123] As an optional implementation, the user-input text is divided into an opinion section and a follow-up question section based on content type, with the boundaries clearly defined by identifying interjections in the text. For the opinion section, the core object described and the evaluation tendency are extracted, summarizing the user's core needs regarding that object. For the follow-up question section, modifying statements are removed, retaining the core statements containing interrogative words, and breaking down the key issues corresponding to each question. Finally, the summarized core needs and the broken-down key issues are organized according to the correspondence between needs and questions, ensuring that each issue is associated with a corresponding need, forming user feedback information. This method is suitable for scenarios where the user text structure is clear and the distinction between opinions and questions is well-defined, generating highly structured feedback information.

[0124] Step G12: Based on the user feedback information, associate the adjustable parameters of the model, analyze the adjustment direction and magnitude of the adjustable parameters, and generate a parameter optimization scheme.

[0125] In this embodiment, adjustable parameters of the model refer to numerical parameters in the model whose performance can be changed through adjustment. The adjustment direction of the adjustable parameters refers to the direction in which parameters need to be added, removed, or optimized to address user feedback issues. The adjustment magnitude of the adjustable parameters refers to the range or proportion of numerical changes in the adjustable parameters according to the adjustment direction. The parameter optimization scheme refers to the specific execution plan that integrates adjustable parameters, adjustment direction, and magnitude to optimize model performance to meet user needs.

[0126] As an optional implementation method, a correspondence table between user feedback information and adjustable model parameters is established, clarifying the associated parameters for each key issue and core requirement. For each piece of feedback, the severity of the problem is analyzed, and adjustment levels are set according to the severity. The direction of adjustment is determined based on the reasonable range of parameter values. All parameters, directions, and magnitudes corresponding to each feedback are reviewed one by one, checking for conflicts between parameters. If conflicts exist, adjustments to parameters that address the core requirement are prioritized, and finally, an optimized parameter solution is obtained through integration. This method directly corresponds to the feedback and parameters, has clear adjustment goals, and generates solutions efficiently.

[0127] Step G13: Adjust the model parameters according to the parameter optimization scheme, verify the effect with corresponding dialogue scenario data, and output the optimized model parameters.

[0128] In this embodiment, the corresponding dialogue scenario data refers to scenario-related data that matches the dialogue scenario in which the current model is applied and is used to verify the effect of parameter adjustments. Verifying the effect means testing the performance of the adjusted model using scenario data to determine whether it meets the core needs in user feedback.

[0129] As an optional implementation, the adjustable parameters in the parameter optimization scheme are sorted by adjustment priority, and the current value, adjustment direction, and magnitude of each parameter are read one by one. Parameters are adjusted sequentially according to the order. If the direction is to increase, the value corresponding to the magnitude is added to the current value; if it is to decrease, the value corresponding to the magnitude is subtracted from the current value. The values ​​before and after each parameter adjustment are recorded. After parameter adjustment, corresponding dialogue scenario data is prepared, and the data is input into the model according to the scenario logic. The model is run, and key performance indicators are recorded. The indicators are compared with the required standards in user feedback. If the indicators meet the standards, the parameter adjustment is confirmed to be effective, and the adjusted model parameters are directly output. If the indicators do not meet the standards, the process is reversed, and the parameters that did not meet the standards are adjusted again according to the fine-tuning ratio of the original scheme. This process is repeated until the indicators meet the standards, and finally, the optimized model parameters are output. This method is suitable for simple dialogue scenarios, can quickly output optimized parameters, meets basic user needs, and shortens the parameter optimization cycle.

[0130] For example, in a user question-and-answer scenario, the system parses the user's input, such as "The model has low accuracy in recognizing the derivation of physics formulas in educational video Q&A, and we hope to improve the accuracy of the answers. We also report that the response time is too long," along with the follow-up question. The system first divides the input into the opinion part ("low accuracy, hope to improve accuracy," "response time is too long") and the implicit question part ("how to adjust to improve these two problems") based on tone words. Modifiers such as "I think" and "can it be done?" are removed, and the key issues are extracted as "improving the accuracy of recognizing the derivation of physics formulas in educational video Q&A" and "shortening the response time." The core requirement is "optimizing model parameters to achieve high-precision and high-efficiency educational video Q&A," thus obtaining the user feedback information. Based on the user feedback, the adjustable parameters of the associated model were analyzed: the similarity calculation threshold of the cross-modal alignment component, the attention mechanism scaling factor of the dynamic weighted fusion component, and the number of network layer iterations of the temporal modeling component. The direction and magnitude of the adjustments were analyzed: for "low recognition accuracy", the similarity calculation threshold was adjusted to "decrease" to expand the effective matching range; for "excessive response time", the attention mechanism scaling factor was adjusted to "increase" to simplify the weight calculation process, and the number of network layer iterations was adjusted to "decrease" to shorten the computation cycle. A parameter optimization scheme was generated. The model parameters were adjusted according to the parameter optimization scheme, and the effect was verified with data from the corresponding dialogue scenario of "physics education video Q&A". After inputting the data, the performance indicators were tested: the recognition accuracy increased from 75% to 88%, and the response time decreased from 0.9 seconds / time to 0.6 seconds / time, confirming that the core requirements were met. The optimized model parameters were then output.

[0131] By using a question-answering model that aggregates multimodal clues, the model solves the problems of traditional model parameter adjustments lacking user feedback and relying solely on data iteration, leading to a disconnect between performance optimization and actual user needs. This achieves a precise match between model performance and user needs, enhancing the model's practicality in educational question-answering scenarios.

[0132] This application provides a dialogue generation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to execute the dialogue generation method based on multimodal cue aggregation in the first embodiment described above.

[0133] The following is for reference. Figure 5The diagram illustrates a structural schematic suitable for implementing the dialogue generation device in the embodiments of this application. The dialogue generation device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, in-vehicle intelligent assistants, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), intelligent assistants, etc., as well as fixed terminals such as wearable devices, desktop computers, etc. Figure 5 The dialogue generation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0134] like Figure 5 As shown, the dialogue generation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the dialogue generation device. The processing unit 1001, the ROM 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the dialogue generation device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show dialogue generation devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0135] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0136] The dialogue generation device provided in this application employs the dialogue generation method based on multimodal cue aggregation in the above embodiments, which can solve the technical problem of being unable to stably generate high-quality natural language dialogue content. Compared with the prior art, the beneficial effects of the dialogue generation device provided in this application are the same as those of the dialogue generation method based on multimodal cue aggregation provided in the above embodiments, and other technical features in this dialogue generation device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0137] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0138] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0139] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the dialogue generation method based on multimodal cue aggregation in the above embodiments.

[0140] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0141] The aforementioned computer-readable storage medium may be included in the dialogue generation device; or it may exist independently and not assembled into the dialogue generation device.

[0142] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a dialogue generation device, cause the dialogue generation device to: encode image blocks corresponding to video data, then concatenate the encoded embedding sequences according to the temporal order of the video data to obtain visual features; extract and transcribe audio from the video data to obtain text features corresponding to the video data; align the visual features and the text features in the semantic space based on a target multimodal cue aggregator to generate multimodal features; retrieve relevant semantic fragments of the user question from the multimodal features, and generate the answer content corresponding to the user question based on the semantic association between the retrieval results and the historical context.

[0143] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0144] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described dialogue generation method based on multimodal cue aggregation, thereby solving the technical problem of the inability to stably generate high-quality natural language dialogue content. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the dialogue generation method based on multimodal cue aggregation provided in the above embodiments, and will not be repeated here.

[0145] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A dialogue generation method based on multimodal cue aggregation, characterized in that, The method includes: After encoding the image blocks corresponding to the video data, the encoded embedding sequence is concatenated according to the temporal order of the video data to obtain visual features; The audio is extracted and transcribed from the video data to obtain the text features corresponding to the video data; Based on the target multimodal cue aggregator, the visual features and the text features are aligned in the semantic space to generate multimodal features; Retrieve relevant semantic fragments of the user's question from the multimodal features, and generate the answer content corresponding to the user's question based on the semantic association between the retrieval results and the historical context.

2. The dialogue generation method based on multimodal cue aggregation as described in claim 1, characterized in that, After encoding the image blocks corresponding to the video data, the steps of concatenating the encoded embedding sequence according to the temporal order of the video data to obtain visual features include: Each image patch is linearly embedded into a fixed-dimensional vector, and the embedding vector corresponding to each image patch is output based on global context modeling. Arrange the embedding vectors corresponding to the image blocks in the same frame according to their spatial position to obtain the frame embedding sequence corresponding to the same frame. By concatenating the frame embedding sequence of all frames according to the temporal order of the video data, the visual features containing video spatiotemporal information are obtained.

3. The dialogue generation method based on multimodal cue aggregation as described in claim 1, characterized in that, The step of extracting and transcribing audio from the video data to obtain text features corresponding to the video data includes: Extract the audio track from the video data and divide the original audio signal corresponding to the video data into multiple audio segments of fixed length; The speech recognition process extracts the spectrogram of each audio segment and generates a text sequence and its embedded representation corresponding to each audio segment. The text sequence is concatenated according to the temporal order of the audio segments, and the embedded representation is concatenated according to the temporal order to generate the text features corresponding to the video data.

4. The dialogue generation method based on multimodal cue aggregation as described in claim 1, characterized in that, The step of aligning the visual features and textual features in the semantic space to generate multimodal features based on the target multimodal cue aggregator includes: Based on the target multimodal cue aggregator, the visual features and the text features are mapped to a common semantic space, and the feature distribution is adjusted by calculating the semantic correlation between modalities to obtain aligned visual features and text features. Calculate the association weights corresponding to the aligned visual features and text features, and integrate key semantic information in a weighted manner to obtain preliminary fused features; The preliminary fusion features are modeled based on temporal dependencies, and the feature representation is optimized through nonlinear transformation and normalization to generate multimodal features.

5. The dialogue generation method based on multimodal cue aggregation as described in claim 1, characterized in that, The step of retrieving relevant semantic fragments of the user question from the multimodal features and generating the answer content corresponding to the user question based on the semantic association between the retrieval results and the historical context includes: The user question is converted into a text embedding sequence consistent with the multimodal feature semantic space through vector mapping operations; Using the text embedding sequence as the retrieval benchmark, the semantic similarity between the text embedding sequence and each semantic unit in the multimodal features is calculated, and semantic units with similarity higher than a preset threshold are selected to form a candidate semantic fragment set. Extract historical context information related to the user's question from the historical context semantic database, conduct a secondary evaluation of the logical coherence and demand matching degree of the candidate semantic fragment set, and retain the qualified fragments as target-related semantic fragments; Using the target-related semantic fragments as core semantic support, combined with historical context information, an autoregressive generation mode is adopted to gradually generate preliminary answer text; The initial response text is checked for semantic coherence and factual accuracy. Content with logical contradictions or information discrepancies is corrected, and the response content corresponding to the user's question is generated.

6. The dialogue generation method based on multimodal cue aggregation as described in claim 1, characterized in that, Before the step of concatenating the encoded embedding sequence according to the temporal order of the video data to obtain visual features after encoding the image blocks corresponding to the video data, the dialogue generation method based on multimodal cue aggregation further includes: The video data is extracted at fixed time intervals, and the continuous video stream is split into multiple independent image frames to obtain the set of image frames corresponding to the video data. The image frames are evenly divided according to a preset fixed size to obtain an initial image block group corresponding to each image frame; Based on the temporal order of the image frames in the video data and the spatial order of each image frame, the initial image block group corresponding to the image frames is summarized to generate the image block sequence corresponding to the video data.

7. The dialogue generation method based on multimodal cue aggregation as described in claim 1, characterized in that, Before the step of aligning the visual features and text features in the semantic space to generate multimodal features based on the target multimodal cue aggregator, the dialogue generation method based on multimodal cue aggregation further includes: The basic framework of the multimodal cue aggregator is initialized, and the basic network pathway for feature transfer is built to obtain the initial framework of the aggregator. Based on the initial aggregator framework, cross-modal alignment components, dynamic weighted fusion components, and temporal modeling components are embedded to generate the aggregator framework. Initialize the similarity calculation threshold of the cross-modal alignment component, the attention mechanism scaling factor of the dynamic weighted fusion component, and the network layer parameters of the temporal modeling component, and output the aggregator model to be trained; The aggregator training model is trained based on labeled multimodal training data, and the parameter configuration of the aggregator training model is optimized through a loss function to generate the target multimodal cue aggregator.

8. The dialogue generation method based on multimodal cue aggregation as described in claim 1, characterized in that, After the steps of retrieving relevant semantic fragments of the user question from the multimodal features and generating the answer content corresponding to the user question based on the semantic association between the retrieval results and the historical context, the dialogue generation method based on multimodal cue aggregation further includes: By parsing user input and follow-up questions, key issues and core needs are extracted to obtain user feedback information; Based on the user feedback information, the adjustable parameters of the associated model are analyzed, and the adjustment direction and magnitude of the adjustable parameters are analyzed to generate a parameter optimization scheme. Adjust the model parameters according to the parameter optimization scheme, verify the effect with corresponding dialogue scenario data, and output the optimized model parameters.

9. A dialogue generation device, characterized in that, The dialogue generation device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the dialogue generation method based on multimodal cue aggregation as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the dialogue generation method based on multimodal cue aggregation as described in any one of claims 1 to 8.

Images