Multi-modal entity recognition method fusing video and voice

Abstract

The invention discloses a multi-mode entity recognition method fusing videos and voices. The method comprises the steps that firstly, a visual candidate entity set is extracted from a video stream, and an auditory candidate entity set is extracted from an audio stream; establishing a fine-grained time sequence corresponding relation between visual and auditory features through a learnable time sequence alignment network, and generating joint feature representation of cross-modal alignment; constructing a cross-modal heterogeneous graph containing various semantic relation edges by taking the candidate entities as nodes, and performing message propagation and node feature updating by utilizing a graph neural network; and finally, calculating cross-modal alignment confidence based on the updated node features, fusing multi-modal candidate entities referring to the same entity, and outputting the category, the position, the time interval and the confidence of each entity. According to the method, fine-grained time sequence alignment and deep fusion of the video and the voice are realized, and the accuracy and robustness of cross-modal entity recognition are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of multimodal information processing and artificial intelligence technology, specifically relating to a multimodal entity recognition method that integrates video and voice. Background Technology

[0002] With the rapid development of artificial intelligence and multimedia processing technologies, video and voice, as two core modalities of information transmission, are increasingly widely used in fields such as security monitoring, intelligent interaction, content analysis, and smart healthcare. How to effectively identify entities (such as people, objects, locations, and events) from video and voice, and achieve the fusion and alignment of cross-modal information, has become a key challenge in the field of multimodal intelligent understanding.

[0003] Currently, entity recognition technology has made significant progress in single-modal applications. In vision, deep learning-based object detection (such as YOLO and Faster R-CNN), face recognition, and optical character recognition (OCR) technologies can accurately extract entities and their spatiotemporal information from video frames. In hearing, automatic speech recognition (ASR) and named entity recognition (NER) technologies can identify text entities and their timestamps from audio streams. However, most existing methods are designed for single-modal applications and lack system modeling of the inherent relationships between multiple modalities, leading to the following problems in practical applications: First, temporal alignment is difficult. Video and audio signals often exhibit asynchrony during acquisition, transmission, and processing. Traditional multimodal fusion methods (such as feature splicing, early fusion, or late fusion) often ignore the correspondence between the two on a fine-grained timeline, resulting in low cross-modal entity matching accuracy. Second, semantic association is lacking. Visual entities in videos and mentioned entities in audio not only have temporal co-occurrence relationships but also rich spatial, semantic, and referential associations. Existing methods mostly rely on manual rules or simple similarity calculations for association, making it difficult to model complex cross-modal semantic dependencies, especially limiting their performance in complex scenarios involving multiple people, multiple objects, and multiple referents. Third, information fusion is insufficient. Existing systems often employ a two-stage independent recognition and fusion strategy, i.e., first recognizing entities from video and audio separately, and then performing post-processing alignment. This approach cannot utilize the contextual information of the other modality during recognition, easily leading to information loss and error propagation. In addition, existing cross-modal learning methods often rely on large-scale labeled data and have fixed model structures, making it difficult to adapt to the actual application needs of different scenarios, devices, and languages, and their generalization and adaptive capabilities are limited.

[0004] Therefore, there is an urgent need for a multimodal entity recognition method that can achieve fine-grained alignment of video and speech, deep fusion of cross-modal semantic relationships, and good scalability and adaptability, so as to improve the accuracy, consistency and practicality of entity recognition in complex scenarios. Summary of the Invention

[0005] In view of this, this invention proposes a multimodal entity recognition method that integrates video and speech. Through a learnable temporal alignment network, it innovatively performs fine-grained dynamic alignment of video and speech at the temporal level, solving the problem of difficult entity matching caused by temporal misalignment between modalities in traditional methods. By constructing a cross-modal heterogeneous graph and introducing a graph neural network, it achieves deep fusion of visual and auditory features in terms of semantics and spatial relationships, significantly improving the accuracy and robustness of cross-modal entity recognition.

[0006] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a multimodal entity recognition method that integrates video and audio, comprising: Perform visual analysis on the input video stream to generate a set of visual candidate entities containing spatiotemporal information; The input audio stream is subjected to speech recognition and text analysis to generate a set of auditory candidate entities containing time information; Feature sequences of visual and auditory candidate entities are extracted, and a fine-grained temporal correspondence between visual and acoustic features is established through a learnable temporal alignment network to generate a cross-modal aligned joint feature representation. Using visual and auditory candidate entities as nodes, a cross-modal heterogeneous graph containing multiple semantic relationship edges is constructed based on joint feature representation; The cross-modal heterogeneous graph is input into the cross-modal graph neural network for message propagation and node feature update. Based on the updated node features, the cross-modal alignment confidence of any two candidate entities referring to the same real entity is calculated. Based on the cross-modal alignment confidence, multimodal candidate entities referring to the same real entity are fused, and the category, spatial location, time interval, and cross-modal confidence of each identified entity are output.

[0007] Preferably, visual analysis is performed on the input video stream to generate a set of visual candidate entities containing spatiotemporal information, including: The YOLO or Faster R-CNN deep learning-based detection model is used to detect objects in video frames. The detected target boxes and their tracking trajectories obtained by the SORT or DeepSort algorithm are used as visual candidate entities. And / or, use CRNN or Transformer-based models to perform optical character recognition on video frames, and use the recognized text regions and their occurrence periods as visual candidate entities; And / or, use MTCNN or RetinaFace models for face detection, and use face features for clustering and association, and use the detected face regions and their occurrence time periods as visual candidate entities.

[0008] Preferably, the input audio stream is subjected to speech recognition and text analysis to generate a set of auditory candidate entities containing time information, including: The audio stream is recognized using the end-to-end speech recognition model Conformer or Wav2Vec 2.0, and the text sequence with character-level or word-level timestamps is output. Entity recognition is performed on text sequences with timestamps using BERT or RoBERTa-based named entity recognition models. Each identified named entity is associated with the time interval in which it appears in the audio stream to form auditory candidate entities.

[0009] The preferred, learnable temporal alignment network specifically performs the following operations: Visual candidate entity sequences and auditory candidate entity sequences are encoded into feature vector sequences using a visual encoder and an acoustic encoder, respectively. and ; A differentiable dynamic time warping algorithm is used to calculate the visual feature sequence. Acoustic feature sequence Optimal soft alignment path and alignment weight matrix between ; Based on the alignment weight matrix Temporal attention fusion is performed on the visual feature sequence and the acoustic feature sequence to generate a joint feature vector that is cross-modal aligned at each time step.

[0010] Preferably, the differentiable dynamic time warping algorithm is implemented using Soft-DTW, which calculates the cumulative distance matrix recursively. ,in: Define the local distance matrix Local distance matrix elements t and s represent the index positions in the two sequences, namely the t-th time step in the visual feature sequence V and the s-th time step in the acoustic feature sequence A; initialization ,for To T, To S, and For all locations, recursively calculate the cumulative distance using a differentiable softmin operation. Let represent the minimum cumulative distance sum among all possible paths from the starting point (1,1) to the current point (t,s): in, For smoothing parameters, For activation function, Indicates input parameters; Based on the calculated cumulative distance Construct a cumulative distance matrix R, and determine the optimal soft alignment path for aligning the entire visual sequence with the entire acoustic sequence based on the cumulative distance matrix R. ; The local distance matrix of the optimal soft alignment path is calculated using automatic differentiation or backpropagation algorithms. Each element Find the gradient and construct the gradient matrix. , the gradient matrix After normalization, the final alignment weight matrix is ​​obtained. .

[0011] Preferably, when constructing a cross-modal heterogeneous graph, the multiple semantic relation edges include at least the following types: Temporal co-occurrence edge: connects any two candidate entity nodes whose overlap in time interval exceeds the first threshold; Spatial proximity edge: connects visual candidate entity nodes whose spatial intersection and union ratio exceeds a second threshold; Semantic similarity edge: any two nodes whose cosine similarity to their node feature vectors exceeds a third threshold; Referential link edges: Based on the pronoun resolution analysis of audio text, connect pronoun nodes with candidate entity nodes that they may refer to.

[0012] Preferably, the weight of the referential link edge is calculated in the following way: Dependency parsing is performed on the speech recognition text to identify pronouns. and its context; Calculation pronouns With each candidate entity node Overall matching score : in, These are learnable or preset weight coefficients. For time overlap, For semantic similarity, The visual saliency score at the corresponding time point; Will After being normalized by the sigmoid function, it is used as the initial weight representing the linked edge.

[0013] Preferably, the cross-modal graph neural network adopts a hierarchical message passing mechanism, and the node feature update process is as follows: For each node in the heterogeneous graph In the The layer update process is as follows: Intramodal aggregation: Aggregation originates from neighbors in the same modality. News : Cross-modal aggregation: Aggregation from different modal neighbors Message: Node update: Update the node representation by combining its own characteristics and aggregated messages: in, This indicates that the j-th neighbor node is in the... The feature vector output by the layer, Represents the intramodal aggregation function. The AGGREGATE function represents a cross-modal aggregation function. The AGGREGATE function represents mean pooling, attention pooling, or graph attention networks. The UPDATE function represents a gated recurrent unit or a fully connected layer.

[0014] Preferably, the calculation of cross-modal alignment confidence is as follows: For any two nodes and final layer features and Its cross-modal alignment confidence Calculated using a bilinear function: in, For a learnable parameter matrix, For bias terms, It is the sigmoid activation function.

[0015] Preferably, the entity fusion process is as follows: Cross-modal alignment confidence Candidate entity node pairs that exceed the preset fusion threshold are determined to refer to the same real entity; The set of all nodes that are determined to be the same entity The entity features after fusion and spatial location Calculated by weighted average: in, For nodes Modal confidence or alignment confidence, Let be the feature vector of the k-th node in . Its spatial location.

[0016] The present invention has achieved at least the following beneficial effects: 1. This invention innovatively performs fine-grained dynamic alignment of video and audio at the temporal level through a learnable temporal alignment network, solving the problem of entity matching difficulties caused by temporal misalignment between modalities in traditional methods. By constructing a cross-modal heterogeneous graph and introducing a graph neural network, it achieves deep fusion of visual and auditory features in semantics and spatial relationships, significantly improving the accuracy and robustness of cross-modal entity recognition.

[0017] 2. This invention not only supports the recognition of entities in a single modality, but also unifies the candidate representations of the same entity in different modalities through cross-modal alignment confidence calculation and entity fusion mechanisms, effectively solving the ambiguity and omission problems existing in single-modal recognition. Especially in complex scenarios such as multi-person dialogue, dynamic backgrounds, and noise interference, the system can still achieve high consistency and high recall in entity recognition.

[0018] 3. The heterogeneous graph structure and graph neural network model proposed in this invention support flexible expansion of various semantic relationship edges, adapting to the needs of different application scenarios. Furthermore, the system can incorporate an online learning mechanism to continuously optimize based on user feedback and real-world scenario data, exhibiting strong transfer and adaptive capabilities, making it suitable for multiple fields such as security monitoring, intelligent conferencing, and human-computer interaction.

[0019] Other advantages, objectives, and features of the invention will be set forth in the following description and will be apparent to those skilled in the art in some respects, or may be learned by practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0020] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the following figures are provided for illustration: Figure 1 This is a flowchart illustrating the steps of a multimodal entity recognition method that integrates video and audio in an embodiment of the present invention. Detailed Implementation

[0021] The specific embodiments of the present invention will be described in detail below with reference to the technical solutions provided by the present invention. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the scope of protection of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0022] First preferred embodiment This invention provides a multimodal entity recognition method that integrates video and audio, referring to... Figure 1 This includes the following steps: S1: Perform visual analysis on the input video stream to generate a set of visual candidate entities containing spatiotemporal information; S2: Perform speech recognition and text analysis on the input audio stream to generate a set of auditory candidate entities containing time information; S3: Extract feature sequences of visual and auditory candidate entities, establish fine-grained temporal correspondence between visual and acoustic features through a learnable temporal alignment network, and generate cross-modal aligned joint feature representations; S4: Using visual and auditory candidate entities as nodes, construct a cross-modal heterogeneous graph containing multiple semantic relationship edges based on joint feature representation; S5: Input the cross-modal heterogeneous graph into the cross-modal graph neural network for message passing and node feature updating, and calculate the cross-modal alignment confidence of any two candidate entities referring to the same real entity based on the updated node features; S6: Based on the cross-modal alignment confidence, multimodal candidate entities referring to the same real entity are fused, and the category, spatial location, time interval and cross-modal confidence of each identified entity are output.

[0023] In this embodiment, "video stream" refers to a continuous sequence of images provided by a camera, screen recording software, or video file, such as surveillance video, conference recordings, or multimedia files. "Audio stream" refers to audio signals captured synchronously with or independently of the video, such as microphone recordings, narration tracks, or dialogue audio.

[0024] In this embodiment, the "visual analysis" in step S1 includes multiple sub-tasks such as object detection, tracking, optical character recognition, and face detection. The system employs a multi-model parallel processing architecture to analyze video frames. Object detection models (such as YOLOv8 or Faster R-CNN) are responsible for detecting objects (such as people, vehicles, and items) in the frame and outputting their bounding boxes. Tracking algorithms (such as DeepSort) associate the same object in different frames, forming a "visual trajectory" with a time interval. Simultaneously, OCR models (such as CRNN or TRBA) identify text regions (such as subtitles and signs) in the video, and face detection models (such as RetinaFace) detect and cluster regions where the same face appears at different times. All these detection results (objects, text, and faces) and their spatiotemporal information together constitute a "visual candidate entity set".

[0025] In this embodiment, the "speech recognition and text analysis" step S2 uses an end-to-end speech recognition model (such as Conformer or Wav2Vec 2.0) to convert the audio stream into a text sequence with word-level timestamps. Subsequently, a named entity recognition model (based on BERT or RoBERTa) identifies entities (such as person names, place names, and organization names) from the text. The system associates each identified entity with the time interval (start time, end time) of its appearance in the audio, forming an "auditory candidate entity set". For example, if "Zhang San" is identified to appear between the 10th and 15th seconds in a meeting recording, then "Zhang San" is considered an auditory candidate entity with timestamps [10s, 15s].

[0026] In this embodiment, the "learnable temporal alignment network" in step S3 is the core module for achieving cross-modal alignment in this method. Its input is a sequence of visual features. and auditory feature sequences The cumulative distance matrix is ​​extracted by a visual encoder (such as ResNet-50) and an acoustic encoder (such as VGGish), respectively. The network employs a differentiable dynamic time warping algorithm (Soft-DTW) to recursively compute the cumulative distance matrix. Find the optimal soft alignment path between the two sequences and generate the alignment weight matrix. .based on The network uses a temporal attention mechanism to... and Weighted fusion is performed to generate a "cross-modal aligned joint feature vector" at each time step. This process enables soft alignment of objects appearing in the vision with entities mentioned in the speech on the time axis, laying the foundation for subsequent cross-modal association.

[0027] In this embodiment, the "cross-modal heterogeneous graph" in step S4 is a graph structure with all visual and auditory candidate entities as nodes. The graph contains four types of edges: "temporal co-occurrence edges" connect nodes with high temporal overlap, "spatial proximity edges" connect visual nodes that are spatially close, "semantic similarity edges" connect nodes with similar feature vectors, and "referential link edges" connect pronouns with the entity nodes they may refer to. These edges collectively model the rich relationships between entities, providing structured contextual information for the graph neural network.

[0028] In this embodiment, the "cross-modal graph neural network" in step S5 employs a hierarchical message passing mechanism. For each node, the network aggregates messages from neighbors of the same modality (intra-modal aggregation) and neighbors of different modalities (cross-modal aggregation), and updates them in conjunction with the node's own features. After multiple layers of propagation, the node features incorporate information from multi-hop neighbors, enabling a better representation of the entity and its context. Based on the final layer node features, the system calculates the "cross-modal alignment confidence" between any two nodes using a bilinear matching function. This value reflects the probability that they point to the same real entity.

[0029] In this embodiment, the "entity fusion" in step S6 is a confidence-based clustering process. The system sets a fusion threshold (e.g., 0.7) and classifies node pairs with confidence scores higher than this threshold as the same entity. For each entity category, the system performs a weighted average of the features and locations of all its nodes (which may come from visual and auditory sources) to obtain the fused entity representation. Finally, it outputs the "category" (e.g., person, location), "spatial location" (bounding box coordinates), "time interval" (the time period appearing in the video), and "cross-modal confidence score" (the fused confidence score) for each entity. For example, in a conference video, the system might output "Entity: Zhang San, Category: Person, Location: [x1, y1, x2, y2], Time: [10s, 25s], Confidence: 0.92".

[0030] Second preferred embodiment Based on the first preferred embodiment, in step S3, the learnable temporal alignment network is specifically implemented as follows: defining a local distance matrix. , of which elements , indicating visual features Acoustic characteristics The square of the Euclidean distance between them; Initialize the cumulative distance matrix ,make For all other positions Recursive calculation: in For a smooth minimum value function, For smoothing parameters; The optimal alignment path is calculated using automatic differentiation. The gradient is obtained by finding the gradient matrix. After normalization, the alignment weight matrix is ​​obtained. ; based on Calculate the joint feature vector at each time step: in This indicates a feature splicing operation.

[0031] In this embodiment, "Soft-DTW" is a differentiable timing alignment algorithm. Compared with traditional DTW, it introduces a smoothing parameter. This makes the entire alignment process differentiable, allowing it to be embedded in neural networks for end-to-end training. The algorithm recursively calculates the cumulative distance matrix. It simulates the cumulative distance of all possible paths from the start point to the end point of the sequence and implements soft selection of paths through the softmin operation.

[0032] In this embodiment, the "alignment weight matrix" The physical meaning of "" is the correspondence strength between visual frames and speech segments. The larger the value, the more significant the th... Frame visual content and the first The higher the relevance of the audio content, the better. For example, the higher the relevance between video frames showing the movement of someone's lips as they speak and the corresponding audio segments. The value will be significantly higher than that of other frames.

[0033] In this embodiment, "joint feature vector" "It integrates visual and auditory information and is built on an aligned timeline. This means that even with small time shifts between visual and auditory signals (such as audio-visual asynchrony), the network can still establish correct correspondences, thereby generating robust multimodal representations."

[0034] Third preferred embodiment Based on the first or second preferred embodiment, a semantic association enhancement step based on referential resolution is also included: When constructing a cross-modal heterogeneous graph, dependency parsing is performed on the speech recognition text to identify all pronoun nodes (such as "he", "she", "it", "this" etc.). For each pronoun node Calculate its relationship with each candidate entity node Overall matching score: in, For time overlap, For semantic similarity, The score is based on visual saliency. The scores are normalized to weights using the sigmoid function and used as pronoun nodes. With entity nodes The initial weights of the "referring to the connecting edges"; In the message passing process of graph neural networks, referential link edges allow information to flow from entity nodes to pronoun nodes, thereby enhancing the feature representation of pronoun nodes and ultimately resolving pronouns into specific entities through the node alignment process.

[0035] In this embodiment, "referential resolution" is a classic problem in natural language understanding, and this method models it as a link prediction problem in a graph structure. By constructing "referential link edges" between pronouns and entities and assigning them learnable weights, the system can dynamically resolve the referents of pronouns during graph reasoning.

[0036] In this embodiment, the "comprehensive matching score" is designed considering three dimensions: temporal, semantic, and visual. Temporal overlap ensures that pronouns and entities co-occur in time; semantic similarity is calculated using word vectors to measure the distance between pronoun and entity names in the semantic space; and visual saliency score reflects the prominence of the entity in the image (such as the size of the target box and the degree of centering). The weighted combination of these three dimensions can comprehensively evaluate the referential probability between pronouns and entities.

[0037] In this embodiment, during the message passing process of the graph neural network, referential link edges enable pronoun nodes to aggregate information from potentially referential entity nodes. After multiple iterations, the features of the pronoun nodes gradually approach the features of the entities they refer to, thus enabling the pronoun nodes to match the correct entity nodes with high confidence during the node alignment stage, achieving accurate referential resolution. For example, if someone in a video says "put it on the table," the system can use graph reasoning to resolve "it" as the "cup" being held in the frame.

[0038] Fourth preferred embodiment Based on any of the first to third preferred embodiments, a model adaptive update step based on online learning is also included: After system deployment, continuously collect multimodal data (video, audio) from real-world scenarios and user feedback on recognition results (such as corrections and confirmations). Once enough new data has been collected, an incremental training dataset is constructed, and the temporal alignment network and graph neural network are fine-tuned using continuous learning or online learning strategies. Introduce catastrophic forgetting mitigation mechanisms, such as elastic weight consolidation or replay buffers, to ensure that the model does not forget knowledge of old scenarios while adapting to new scenarios; The system periodically evaluates the model's recognition performance in new scenarios, and automatically triggers the model update process when the performance drops below a threshold.

[0039] In this embodiment, real-world application scenarios are complex and diverse, potentially including areas, accents, lighting conditions, or entity categories not covered by the training data. Through an "online learning" mechanism, the system can continuously optimize itself during operation, gradually improving its recognition accuracy in specific scenarios.

[0040] In this embodiment, "user feedback" serves as a valuable supervisory signal. Users can correct (e.g., change entity categories, merge incorrectly segmented entities) or confirm the entities identified by the system through an interactive interface (such as video analytics software). This feedback is converted into labeled training samples for incremental learning of the model.

[0041] In this embodiment, "catastrophic forgetting" is a key challenge in continuous learning. To avoid the model overwriting old knowledge when learning new knowledge, the system adopts a "flexible weight consolidation" strategy, imposing constraints on important parameters in the network (parameters that have a significant impact on the performance of old tasks) to limit their update magnitude. At the same time, the system maintains a "replay buffer" to store a small number of typical samples of old scenes, which are mixed with new data in the new round of training to maintain the model's ability to recognize old scenes.

[0042] In this embodiment, the entire online learning and update process can be automated. The system monitors performance indicators such as recognition accuracy and F1 score. Once a significant performance decline is detected (e.g., a decline exceeding 5%), the system automatically initiates the data collection, incremental training, and model update process without manual intervention, thereby achieving a long-term stable and adaptive multimodal entity recognition service.

[0043] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.

Claims

1. A multimodal entity recognition method integrating video and audio, characterized in that, include: Perform visual analysis on the input video stream to generate a set of visual candidate entities containing spatiotemporal information; The input audio stream is subjected to speech recognition and text analysis to generate a set of auditory candidate entities containing time information; Feature sequences of visual and auditory candidate entities are extracted, and a fine-grained temporal correspondence between visual and acoustic features is established through a learnable temporal alignment network to generate a cross-modal aligned joint feature representation. Using visual and auditory candidate entities as nodes, a cross-modal heterogeneous graph containing multiple semantic relationship edges is constructed based on joint feature representation; The cross-modal heterogeneous graph is input into the cross-modal graph neural network for message propagation and node feature update. Based on the updated node features, the cross-modal alignment confidence of any two candidate entities referring to the same real entity is calculated. Based on the cross-modal alignment confidence, multimodal candidate entities referring to the same real entity are fused, and the category, spatial location, time interval, and cross-modal confidence of each identified entity are output.

2. The multimodal entity recognition method integrating video and audio as described in claim 1, characterized in that, Visual analysis is performed on the input video stream to generate a set of visual candidate entities containing spatiotemporal information, including: The YOLO or Faster R-CNN deep learning-based detection model is used to detect objects in video frames. The detected target boxes and their tracking trajectories obtained by the SORT or DeepSort algorithm are used as visual candidate entities. And / or, use CRNN or Transformer-based models to perform optical character recognition on video frames, and use the recognized text regions and their occurrence periods as visual candidate entities; And / or, use MTCNN or RetinaFace models for face detection, and use face features for clustering and association, and use the detected face regions and their occurrence time periods as visual candidate entities.

3. The multimodal entity recognition method integrating video and audio according to claim 1, characterized in that, The input audio stream is subjected to speech recognition and text analysis to generate a set of auditory candidate entities containing time information, including: The audio stream is recognized using the end-to-end speech recognition model Conformer or Wav2Vec 2.0, and the text sequence with character-level or word-level timestamps is output. Entity recognition is performed on text sequences using named entity recognition models based on BERT or RoBERTa. Each identified named entity is associated with the time interval in which it appears in the audio stream to form auditory candidate entities.

4. The multimodal entity recognition method integrating video and audio according to claim 1, characterized in that, Learnable temporal alignment networks specifically perform the following operations: Visual candidate entity sequences and auditory candidate entity sequences are encoded into feature vector sequences using a visual encoder and an acoustic encoder, respectively. and ; A differentiable dynamic time warping algorithm is used to calculate the visual feature sequence. Acoustic feature sequence Optimal soft alignment path and alignment weight matrix between ; Based on the alignment weight matrix Temporal attention fusion is performed on the visual feature sequence and the acoustic feature sequence to generate a joint feature vector that is cross-modal aligned at each time step.

5. The multimodal entity recognition method integrating video and audio according to claim 4, characterized in that, The differentiable dynamic time warping algorithm is implemented using Soft-DTW, which calculates the cumulative distance matrix recursively. ,in: Define the local distance matrix Local distance matrix elements t and s represent the index positions in the two sequences, namely the t-th time step in the visual feature sequence V and the s-th time step in the acoustic feature sequence A; initialization ,for To T, To S, and For all locations, recursively calculate the cumulative distance using a differentiable softmin operation. Let represent the minimum cumulative distance sum among all possible paths from the starting point (1,1) to the current point (t,s): ， ， in, For smoothing parameters, For activation function, Indicates input parameters; Based on the calculated cumulative distance Construct a cumulative distance matrix R, and determine the optimal soft alignment path for aligning the entire visual sequence with the entire acoustic sequence based on the cumulative distance matrix R. ; The local distance matrix of the optimal soft alignment path is calculated using automatic differentiation or backpropagation algorithms. Each element Find the gradient and construct the gradient matrix. , the gradient matrix After normalization, the final alignment weight matrix is ​​obtained. .

6. The multimodal entity recognition method integrating video and audio according to claim 1, characterized in that, When constructing a cross-modal heterogeneous graph, the edges of various semantic relations should include at least the following types: Temporal co-occurrence edge: connects any two candidate entity nodes whose overlap in time interval exceeds the first threshold; Spatial proximity edge: connects visual candidate entity nodes whose spatial intersection and union ratio exceeds a second threshold; Semantic similarity edge: any two nodes whose cosine similarity to their node feature vectors exceeds a third threshold; Referential link edges: Based on the pronoun resolution analysis of audio text, connect pronoun nodes with candidate entity nodes that they may refer to.

7. The multimodal entity recognition method integrating video and audio according to claim 6, characterized in that, The weight of the referential link edge is calculated in the following way: Dependency parsing is performed on the speech recognition text to identify pronouns. and its context; Calculation pronouns With each candidate entity node Overall matching score : ， in, These are learnable or preset weight coefficients. For time overlap, For semantic similarity, The visual saliency score at the corresponding time point; Will After being normalized by the sigmoid function, it is used as the initial weight representing the linked edge.

8. The multimodal entity recognition method integrating video and audio according to claim 1, characterized in that, Cross-modal graph neural networks employ a hierarchical message passing mechanism, and the node feature update process is as follows: For each node in the heterogeneous graph In the The layer update process is as follows: Intramodal aggregation: Aggregation originates from neighbors in the same modality. News : ， Cross-modal aggregation: Aggregation from different modal neighbors Message: ， Node update: Update the node representation by combining its own characteristics and aggregated messages: ， in, This indicates that the j-th neighbor node is in the... The feature vector output by the layer, Represents intramodal aggregation functions. The AGGREGATE function represents a cross-modal aggregation function. The AGGREGATE function represents mean pooling, attention pooling, or graph attention networks. The UPDATE function represents a gated recurrent unit or a fully connected layer.

9. A multimodal entity recognition method integrating video and audio according to claim 8, characterized in that, The specific steps for calculating cross-modal alignment confidence are as follows: For any two nodes and final layer features and Its cross-modal alignment confidence Calculated using a bilinear function: ， in, For a learnable parameter matrix, For bias terms, It is the sigmoid activation function.

10. The method according to claim 9, characterized in that, The entity fusion process is as follows: Cross-modal alignment confidence Candidate entity node pairs that exceed the preset fusion threshold are determined to refer to the same real entity; The set of all nodes that are determined to be the same entity The entity features after fusion and spatial location Calculated by weighted average: , in, For nodes Modal confidence or alignment confidence, Let be the feature vector of the k-th node in . Its spatial location.

Images