Scene understanding-oriented time sequence audio-visual knowledge graph automatic generation method

Abstract

The invention provides a scene understanding-oriented time sequence audio-visual knowledge graph automatic generation method, and belongs to the field of computers. According to the invention, video and audio data are used as input for scene understanding and are segmented into continuous audiovisual segments based on time synchronization. Next, a background time period without an audio event is identified by segmenting the audio signal, and segments in which an event occurs are sampled. Subsequently, an audiovisual knowledge graph is generated at each sampling time using an entity and relationship detection technique. And finally, performing relation completion on the generated time sequence audio-visual knowledge graph to improve the integrity and accuracy of the generated time sequence audio-visual knowledge graph. According to the method, the integrity and practicability of the knowledge graph are improved, and the system can automatically deduce and complement related audio-visual relationships, so that scene understanding and intelligent development of application are better supported.

Description

Technical Field

[0001] The present invention relates to a method for automatically generating a knowledge graph and belongs to the field of computers. Background Art

[0002] At present, with the continuous development of artificial intelligence technology, scene understanding has received widespread attention as an important research field. Traditional scene understanding mainly focuses on the visual perception of static images or videos. Although this method performs well in some scenarios, it has certain limitations when processing complex scenes and multimodal information. In actual application scenarios, the information in the environment is not limited to vision, but also includes multiple perception modes such as sound and touch. Therefore, relying solely on visual information for scene understanding can no longer meet the demand for diversified information.

[0003] In this context, audio-visual information has become an important part of scene understanding. Audio-visual information has rich environmental information and cognitive value, and can provide a more comprehensive and in-depth scene understanding. Therefore, combining audio-visual information for scene understanding has become one of the research hotspots. However, the current research on audio-visual scene understanding is still relatively rudimentary, especially in the processing and modeling of temporal information. There are still certain challenges.

[0004] In response to the above problems, the present invention takes audio-visual information as the main research object, and aims to construct a knowledge graph of dynamic audio-visual scene entities and their relationships, so as to achieve a deeper and more comprehensive scene understanding. Specifically, the present invention proposes a method for automatically generating a temporal audio-visual knowledge graph, which analyzes and processes audio-visual data to achieve a comprehensive perception and understanding of the temporal evolution of the scene. This method will bring new ideas and methods to the research and application of scene understanding, and is expected to play an important role in the fields of intelligent interaction, intelligent monitoring, etc. Summary of the invention

[0005] The main purpose of the present invention is to propose a method for automatically generating a temporal audio-visual knowledge graph for scene understanding, so as to solve the problem that there is a lack of a method for automatically generating a temporal knowledge graph for scene understanding in the prior art.

[0006] In order to achieve the above object, the present invention provides a method for automatically generating a temporal audio-visual knowledge graph for scene understanding, comprising the following steps:

[0007] S1: Acquire audio-visual perception data of the same time interval through audio-visual sensors;

[0008] S2: performing time domain segmentation of audiovisual events according to the spectrum amplitude of audio data in the audiovisual perception data to obtain synchronized audiovisual segments;

[0009] S3: Evenly divide the synchronized audiovisual segments in time to obtain T continuous synchronized audiovisual segments of equal length and without overlap;

[0010] S4: extracting synchronized visual-auditory features from the video segments in the continuous synchronized audio-visual segments and the audio segments in the corresponding time intervals, and enhancing the video frame features and the audio features;

[0011] S5: Calculate the similarity of the continuous synchronized audiovisual segments after feature enhancement through the segment cross-attention mechanism, use the similarity between the continuous synchronized audiovisual segments to judge the semantic relevance between the synchronized audiovisual segments, delete the continuous synchronized audiovisual segments with negative and weak relevance, and obtain the continuous synchronized audiovisual segments with strong relevance;

[0012] S6: Perform scene graph recognition of video frames and category recognition of audio segments for the highly correlated continuous synchronized audio-visual segments to construct an audio-visual knowledge graph based on the timestamps of the video frames;

[0013] S7: Temporally align the video frame scene graph information and audio segment category information of the highly correlated continuous synchronized audiovisual segments, and align them to the same timeline according to the timestamps of the videos; then associate the entities with high semantic similarity in the two modalities through semantic alignment; and construct a temporal multimodal knowledge graph through entity alignment;

[0014] S8: Take the audio-visual knowledge graph at the timestamps corresponding to the continuous video frames in the temporal multimodal knowledge graph as the continuous audio-visual knowledge graph within the time window, and complete the link between any two given entities based on the continuous knowledge graph; if the identified link already exists, if not, the completion of the temporal audio-visual knowledge graph is completed.

[0015] Furthermore, the time domain segmentation of the audiovisual perception data in the above step S2 includes the following contents:

[0016] S2.1: Iteratively calculate the audio adaptive energy threshold using the short-time energy of the audio frame. The formula is as follows

[0017] E th =a·E mean +(1-α)·E prev_th ,

[0018] Where E th Indicates the audio adaptability threshold of the current iteration calculation. α is an attenuation factor between 0 and 1, which is used to balance the current average energy E mean and the threshold E obtained in the previous iteration prev_th ;

[0019] S2.2: Use the audio adaptive energy threshold to determine the start and end of the audio-visual event of the audio-visual perception data, determine the segmentation points of the audio frames in the audio-visual perception data, and perform synchronous cutting on the audio frames and the corresponding video frames, so as to extract a series of non-overlapping and audio-visual information matching fragment pairs.

[0020] Furthermore, the synchronized visual-auditory features extracted in the above step S4 include the following:

[0021] S4.1: Use the VGG19 network model pre-trained on ImageNet to extract the feature map of the sampled frames per second, perform average pooling on the feature map, and obtain the fragment-level visual encoding features

[0022] Where T represents the duration of the video clip in seconds, k represents the spatial dimension of the feature map extracted from each frame, and d v Represents the dimension of visual encoding features;

[0023] S4.2: For each one-second audio clip, use the VGG-like network model pre-trained on AudioSet to obtain the segment-level auditory encoding features where d a Dimensions representing auditory encoding features;

[0024] S4.3: Visual features encoded using the cross-attention mechanism and audio features

[0025] in and The calculation formula is as follows:

[0026]

[0027] where α t and a′ t represents the attention weight vector, which is calculated using a multilayer perceptron with a Softmax activation function;

[0028] S4.4: Visual features Audio Features Input into two separate LSTMs, the enhanced visual features and audio features are represented as and where d l is the dimension of the enhanced visual feature or audio feature.

[0029] Furthermore, the above step S5 specifically includes the following steps:

[0030] S5.1: Determining the visual segment H in consecutive synchronized audiovisual segments by calculating similarity v and audio clip H a The strength of the connection between them is calculated as follows:

[0031]

[0032] in and is the training parameter, ω va and ω av ∈R T×T is the similarity matrix;

[0033] S5.2: Use the ReLU activation function to eliminate negative connections within consecutive synchronized audiovisual segments, using the set threshold τ 1 To eliminate the weak connections within the fragments below the threshold, the final similarity matrix η is obtained va and η av ∈R T×T ;Threshold τ 1 is a constant;

[0034] S5.3: By formula Get the time index of strongly connected fragments in Represents the similarity matrix η va diagonal elements of .

[0035] Furthermore, the entity alignment in the above step S7 includes the following contents:

[0036] S7.1 Temporal alignment: Align the frame-level scene graph and the clip-level audio categories to the same timeline according to the video timestamps, thereby integrating the temporal visual-audio knowledge graph information under the same video frame Where M represents the total number of video frames, Respectively represent the audio knowledge graph information and visual knowledge graph information in the i-th frame;

[0037] S7.2 Semantic alignment: The embedding vector of the entity is obtained through the pre-trained GloVe model, and the cosine similarity is used to measure the similarity between visual-audio entities in the same video frame. The calculation formula is as follows:

[0038]

[0039] in Indicates that in video frame I i GloVe embedding vectors of the visual and audio entities at ;

[0040] When the similarity of some entity pairs exceeds the specified threshold τ 2, a semantic alignment is established, expressed as a “related to” relationship: Generate a new triple Threshold τ 2 is a constant;

[0041] S7.3: After entity alignment, construct the final temporal multimodal knowledge graph

[0042] Furthermore, the above step S8 includes the following contents:

[0043] S8.1: Apply the traditional translation-based method TransE to learn knowledge graph representations;

[0044] S8.2: TransE defines the energy of a triple by calculating the distance between embedding vectors: E(s,r,o)=d(s+r,o)=|s+ro|; where s, r, and o represent the temporal multimodal knowledge graph The feature vectors of the subject, relation, and object in ;

[0045] S8.3: Use element-by-element multiplication of entity embedding vectors to obtain interaction information between entities, and complete the links through a logistic regression classifier; if the identified link already exists, if not, complete the completion of the temporal audio-visual knowledge graph.

[0046] Furthermore, the above method also includes: in order to improve the completeness and accuracy of the generated temporal audio-visual knowledge graph, an optimization function is proposed: The optimization function It consists of three parts: classification loss Visual-Audio Clip Similarity Loss and embedding vector loss

[0047] Among them, the classification loss Visual-audio segment similarity loss is used to improve the prediction ability of synchronized audio-visual segment events. Used to improve the ability to judge the similarity between synchronized audio-visual clips, embedding vector loss Used to improve the ability to learn entity semantic information.

[0048] The beneficial technical effects of the present invention are as follows:

[0049] Integrating visual and auditory information sources to construct a knowledge graph of dynamic audio-visual scene entities and their relationships solves the limitation of visual-dominant scene understanding, thereby more comprehensively perceiving and understanding the environment. Through a simple and efficient audio data time domain segmentation method, the audio-visual event fragments are preliminarily distinguished and background noise is filtered, reducing the complexity of subsequent processing. Further strengthening the visual and auditory features, and combining the cross-attention mechanism to calculate the similarity of audio-visual fragments, intelligently identifying and retaining strongly related information, and eliminating redundant content, greatly improving the accuracy of constructing the temporal multimodal knowledge graph. This method not only ensures the accuracy and consistency of data at the temporal level, but also enriches the understanding of complex events and scenes at the semantic level. Finally, the highly correlated video frame scene graph information and audio fragment category information are integrated, and a unified temporal multimodal knowledge graph is constructed through temporal and semantic alignment to achieve the connection and association between multimodal data, and more deeply express and understand complex events and scenes. Based on this graph, the continuous audio-visual knowledge graph completes the links between entities, further improving the integrity and practicality of the knowledge graph, enabling the system to automatically infer and complete relevant audio-visual relationships, thereby better supporting the intelligent development of scene understanding and applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Flowchart of a method for automatically generating temporal audio-visual knowledge graphs for scene understanding.

[0051] Figure 2 The figure is a schematic diagram of the structure of a method for automatically generating temporal audio-visual knowledge graphs for scene understanding. DETAILED DESCRIPTION

[0052] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] like Figure 1 As shown, the present invention is a method for automatically generating a temporal audio-visual knowledge graph for scene understanding, comprising the following steps:

[0054] S1: Acquire audio-visual perception data of the same time interval through audio-visual sensors;

[0055] S2: performing time domain segmentation of audiovisual events according to the spectrum amplitude of audio data in the audiovisual perception data to obtain synchronized audiovisual segments;

[0056] S3: Evenly divide the synchronized audiovisual segments in time to obtain T continuous synchronized audiovisual segments of equal length and without overlap;

[0057] As a specific embodiment of the present invention, continuous visual segments and audio segments are represented as: Each segment lasts for one second, and 16 frames are uniformly sampled in each one-second visual segment. Here, and Represent the continuous visual segments and audio segments of the sequence data S respectively.

[0058] S4: extracting synchronized visual-auditory features from the video segments in the continuous synchronized audio-visual segments and the audio segments in the corresponding time intervals, and enhancing the video frame features and the audio features;

[0059] S5: Calculate the similarity of the continuous synchronized audiovisual segments after feature enhancement through the segment cross-attention mechanism, use the similarity between the continuous synchronized audiovisual segments to judge the semantic relevance between the synchronized audiovisual segments, delete the continuous synchronized audiovisual segments with negative and weak relevance, and obtain the continuous synchronized audiovisual segments with strong relevance;

[0060] S6: Perform scene graph recognition of video frames and category recognition of audio segments for the highly correlated continuous synchronized audio-visual segments to construct an audio-visual knowledge graph based on the timestamps of the video frames;

[0061] S7: Temporally align the video frame scene graph information and audio segment category information of the highly correlated continuous synchronized audiovisual segments, and align them to the same timeline according to the timestamps of the videos; then associate the entities with high semantic similarity in the two modalities through semantic alignment; and construct a temporal multimodal knowledge graph through entity alignment;

[0062] S8: Take the audio-visual knowledge graph at the timestamps corresponding to the continuous video frames in the temporal multimodal knowledge graph as the continuous audio-visual knowledge graph within the time window, and complete the link between any two given entities based on the continuous knowledge graph; if the identified link already exists, if not, the completion of the temporal audio-visual knowledge graph is completed.

[0063] As a preferred embodiment of the present invention, the time domain segmentation of the audiovisual perception data in step S2 includes the following contents:

[0064] S2.1: Iteratively calculate the audio adaptive energy threshold using the short-time energy of the audio frame. The formula is as follows

[0065] E th =α·E mean +(1-α)·E prev_th ,

[0066] Where E th Indicates the audio adaptability threshold of the current iteration calculation. α is an attenuation factor between 0 and 1, which is used to balance the current average energy E mean and the threshold E obtained in the previous iteration prev_th ;

[0067] S2.2: Use the audio adaptive energy threshold to determine the start and end of the audio-visual event of the audio-visual perception data, determine the segmentation points of the audio frames in the audio-visual perception data, and perform synchronous cutting on the audio frames and the corresponding video frames, so as to extract a series of non-overlapping and audio-visual information matching fragment pairs.

[0068] As a preferred embodiment of the present invention, extracting synchronized visual-auditory features in step S4 includes the following contents:

[0069] S4.1: Use the VGG19 network model pre-trained on ImageNet to extract the feature map of the sampled frames per second, perform average pooling on the feature map, and obtain the fragment-level visual encoding features

[0070] Where T represents the duration of the video clip in seconds, k represents the spatial dimension of the feature map extracted from each frame, and d v Represents the dimension of visual encoding features;

[0071] S4.2: For each one-second audio clip, use the VGG-like network model pre-trained on AudioSet to obtain the segment-level auditory encoding features where d a Dimensions representing auditory encoding features;

[0072] S4.3: Visual features encoded using the cross-attention mechanism and audio features

[0073] in and The calculation formula is as follows:

[0074]

[0075] where α t and α′ t represents the attention weight vector, which is calculated using a multilayer perceptron with a Softmax activation function;

[0076] S4.4: Visual features Audio Features Input into two separate LSTMs, the enhanced visual features and audio features are represented as and where d l is the dimension of the enhanced visual feature or audio feature.

[0077] As a preferred embodiment of the present invention, step S5 specifically includes the following steps:

[0078] S5.1: Determining the visual segment H in consecutive synchronized audiovisual segments by calculating similarity v and audio clip H a The strength of the connection between them is calculated as follows:

[0079]

[0080] in and is the training parameter, ω va and ω av ∈R T×T is the similarity matrix;

[0081] in and is the training parameter, ω va and ω av ∈R T×T is the similarity matrix;

[0082] S5.2: Use the ReLU activation function to eliminate negative connections within consecutive synchronized audiovisual segments, using the set threshold τ 1 To eliminate the weak connections within the fragments below the threshold, the final similarity matrix η is obtained va and η av ∈R T×T ;Threshold τ 1 is a constant;

[0083] To achieve this, the ReLU activation function is used to eliminate negative connections within a segment and a row-wise l 1 Normalized similarity matrix This process is formalized as: Where LayerNorm(·) is represented by l 1 Normalized, ReLU(·) represents the ReLU activation function. Then a threshold is used to filter out those weak connections with relatively small similarity. After applying the threshold, the l is performed row by row again. 1 Normalization, the process is formalized as:

[0084]

[0085] where τ 1 is a hyperparameter threshold, indicating that connections with similarity values ​​below this threshold will be filtered out. is an indicator function. When the condition is met, the function outputs 1, otherwise it outputs 0, generating the final similarity matrix η va and η va ∈R T×T Finally, time indexes with strong connections can be identified by the following calculation:

[0086] in, represents the set of all time indexes, and the η corresponding to these indexes va The diagonal elements of are greater than 0.

[0087] S5.3: By formula Get the time index of strongly connected fragments in Represents the similarity matrix η va diagonal elements of .

[0088] As a preferred embodiment of the present invention, the scene graph recognition of the video frame in step S6 specifically includes:

[0089] S6.1.1: Use Faster-RCNN to propose object regions in video frames and preprocess the relationship feature vectors.

[0090] S6.1.2: Taking the relation feature vector as input, use the spatial encoder to extract the spatial context information of each frame.

[0091] S6.1.3: Taking the output of the spatial encoder as input, a temporal decoder is used to capture temporal dependencies, and a linear classifier is used for relation prediction.

[0092] Specifically, Faster-RCNN is used as the backbone network to train a neural network with M frames of visual content S v =[I 1 ,I 2 ,…,I M ], the frame at time t is denoted as I t , the detector extracts visual features Bounding Box Object type distribution Where N(t) represents the number of objects in the frame. There is a set of relationship vectors between the N(t) objects in each frame. Between the i-th and j-th objects Representation vector corresponding to the relationship It includes visual information, spatial information and semantic embedding, and the formula is as follows:

[0093]

[0094] Among them, <,> represents the connection operation, Represents a flat operation, represents element-by-element operation, W s ,W o ∈R 2048×512 , W u ∈R 12544×512 A linear matrix representing the dimension compression. Represents the feature map of the union box calculated by RoIAlign, f box is a transformation of the subject and object bounding boxes into Function of the complete feature with the same shape. Semantic embedding vector Determined by the object categories of the subject and object. In the spatial-temporal Transformer, the relation representation exchanges spatial and temporal information.

[0095] The spatial-temporal Transformer maintains the original encoder-decoder architecture. The difference is that the encoder and decoder are given more specific tasks.

[0096] The spatial encoder focuses on the spatial context of each frame, and its input is a single The query Q, key K and value V share the same input, and the output of the n-th layer encoder is represented as:

[0097] The encoder consists of N identical self-attention layers Att enc They are stacked in sequence, and the input of the (n)th layer is equal to the output of the (n-1)th layer.

[0098] The frame encoding is further customized to inject temporal position into the relational representation. Since the number of embedding vectors in the temporal decoder is fixed and relatively short, the frame encoding E f is constructed using the learned embedding parameters. E f =[e 1 ,…,e β ], where e 1 ,…,e β ∈R 1936 Yes and have the same absolute vector.

[0099] The temporal decoder captures the temporal dependencies between frames. dec and Att enc The multi-head self-attention layer is removed. The sliding window β represents the spatial context [X 1 ,…,X M ], the i-th generated input batch is represented as: Z i =[X i ,…,X i+β-1 ], i∈{1,…,M-β+1}, window size β≤M. The decoder is the same as the encoder and consists of N identical self-attention layers Att dec Stacked in sequence, the first layer is: Q = K = Z i +E f , V=Zi ,

[0100]

[0101] In the formula Q = K = Z i +E f In , the frame encoding is added to the relation representation. The output of the last layer decoder is used for the final prediction. Due to the existence of the sliding window, the relation in a frame has different representations in different batches, and the representation that appears earliest on the window is selected.

[0102] According to the above method, from the visual segment S v =[I 1 ,I 2 ,…,I M ] extracts a temporal visual knowledge graph here, Represents frame I i The scene graph, where V i is a collection of entities, E i is a set of relations. i The entity and E i The relations in the two together constitute the set of visual triples.

[0103] As a preferred embodiment of the present invention, the category identification of the audio clip in step S6 specifically includes:

[0104] S6.2.1: Convert the audio segment S a The spectrogram is converted into a 1024×128 size and split into 16×16 patches with no overlap.

[0105] S6.2.2: Use the Audio Spectrogram Transformer (AST) model to extract the embedding information of the patch.

[0106] S6.2.3: Randomly select a patch, generate a mask based on the clustering factor, and apply the mask to the patch embedding information. Specifically, randomly extract an index from the range {1,512} and generate a set of neighboring indexes based on the clustering factor. Repeat this process until the set of indexes contains the specified number of patch locations.

[0107] S6.2.4: Input the masked patch embedding information and position information into the Transformer model to generate the final audio classification result.

[0108] As a preferred embodiment of the present invention, the entity alignment in step S7 includes the following contents:

[0109] S7.1 Temporal alignment: Align the frame-level scene graph and the clip-level audio categories to the same timeline according to the video timestamps to integrate the temporal visual-audio knowledge graph information under the same video frame. Where M represents the total number of video frames, Respectively represent the audio knowledge graph information and visual knowledge graph information in the i-th frame;

[0110] S7.2 Semantic alignment: The embedding vector of the entity is obtained by using the pre-trained GloVe model; the cosine similarity is used to measure the similarity between visual-audio entities in the same video frame. The calculation formula is as follows:

[0111]

[0112] in Indicates that in video frame I i GloVe embedding vectors of the visual and audio entities at ;

[0113] When the similarity of some entity pairs exceeds the specified threshold τ 2 , they will establish semantic alignment, represented by a "related to" relationship: Specifically, if Greater than τ 2 , then a new triple is generated Threshold τ 2 is a constant.

[0114] S7.3: After entity alignment, construct the final temporal multimodal knowledge graph

[0115] As a preferred embodiment of the present invention, step S8 includes the following contents:

[0116] S8.1: Apply the traditional translation-based method TransE to learn knowledge graph representation. In this method, a knowledge graph Contains triples (s, r, o), where s, r, and o represent subject, relation, and object, respectively. Feature vectors s, r, and o are used to represent these entities and relations. The basic idea of ​​TransE is to learn embedding vectors such that for valid triples, s+r is roughly equal to o. As an energy-based model, TransE defines the energy of a triple by calculating the distance between embedding vectors: E(s, r, o) = d(s+r, o) = |s+ro|. This framework enables the effective capture of semantic relations embedded in knowledge graphs.

[0117] S8.2: TransE defines the energy of a triple by calculating the distance between embedding vectors: E(s,r,o)=d(s+r,o)=|s+ro|; where s, r, and o represent the temporal multimodal knowledge graph The feature vectors of the subject, relation, and object in ;

[0118] S8.3: Use element-by-element multiplication of entity embedding vectors to obtain interaction information between entities, and complete the links through a logistic regression classifier; if the identified link already exists, if not, complete the completion of the temporal audio-visual knowledge graph.

[0119] As a preferred embodiment of the present invention, the present invention further includes: in order to improve the completeness and accuracy of the generated temporal audio-visual knowledge graph, an optimization function is proposed: The optimization function It consists of three parts: classification loss Visual-Audio Clip Similarity Loss and embedding vector loss

[0120] Among them, the classification loss Visual-audio segment similarity loss is used to improve the prediction ability of synchronized audio-visual segment events. Used to improve the ability to judge the similarity between synchronized audio-visual clips, embedding vector loss Used to improve the ability to learn entity semantic information.

[0121] As a specific embodiment of the present invention, in order to improve the prediction ability of video clip event categories, a cross entropy classification loss function is used. It is used to measure the difference between the predicted event label and the true label. The formula is as follows: where Y tc Describes the temporal category of the visual-audio segment at time t.

[0122] Visual-Audio Clip Similarity Loss The formula is as follows: Where S cos is the normalized cosine similarity between visual and audio features, G = {g t ∈{0,1}|t=1,2,…,T}∈R 1×T Describes whether a visual-audio clip pair contains an event, Represents the mean squared error between two vectors.

[0123] When learning the embedding information, the entity embedding is further optimized by minimizing the energy difference between positive and negative samples, the embedding vector loss The formula is as follows:

[0124]

[0125] Where γ is a specific boundary value to ensure that the energy of the positive sample is at least higher than this boundary value. Specifically, for each pair of positive and negative samples, the energy difference between them is calculated. If the difference is less than the boundary value, the loss is 0; otherwise, it is the difference minus the boundary value.

[0126] like Figure 2 The structure system of the method for automatically generating a temporal audiovisual knowledge graph for scene understanding proposed by the present invention is shown, which specifically includes several key modules: first, the audiovisual data acquisition module for a specific scene, then the time domain segmentation module based on the audio spectrum, followed by the audiovisual segment feature extraction and encoding module and the highly correlated audiovisual segment extraction module. The outputs of these modules are highly correlated audiovisual segments, which are input into the temporal visual knowledge graph construction module and the temporal audio knowledge graph construction module respectively. Furthermore, these knowledge graphs are input into the temporal audiovisual knowledge graph alignment module and the temporal multimodal knowledge graph completion module to achieve a comprehensive understanding and analysis of the scene.

Claims

1. A method for automatically generating a temporal audio-visual knowledge graph for scene understanding, characterized in that: The following steps are involved: S1: Acquire audio-visual perception data of the same time interval through audio-visual sensors; S2: performing time domain segmentation of audiovisual events according to the spectrum amplitude of audio data in the audiovisual perception data to obtain synchronized audiovisual segments; S3: Evenly divide the synchronized audiovisual segments in time to obtain T continuous synchronized audiovisual segments of equal length and without overlap; S4: extracting synchronized visual-auditory features from the video segments in the continuous synchronized audio-visual segments and the audio segments in the corresponding time intervals, and enhancing the video frame features and the audio features; S5: Calculate the similarity of the continuous synchronized audiovisual segments after feature enhancement through the segment cross-attention mechanism, use the similarity between the continuous synchronized audiovisual segments to judge the semantic relevance between the synchronized audiovisual segments, delete the continuous synchronized audiovisual segments with negative and weak relevance, and obtain the continuous synchronized audiovisual segments with strong relevance; S6: Perform scene graph recognition of video frames and category recognition of audio segments for the highly correlated continuous synchronized audio-visual segments to construct an audio-visual knowledge graph based on the timestamps of the video frames; S7: Temporally align the video frame scene graph information and audio segment category information of the highly correlated continuous synchronized audiovisual segments, and align them to the same timeline according to the timestamps of the videos; then associate entities with high semantic similarity in the two modalities through semantic alignment; Construct a temporal multimodal knowledge graph through entity alignment; S8: Take the audio-visual knowledge graph at the timestamps corresponding to the continuous video frames in the temporal multimodal knowledge graph as the continuous audio-visual knowledge graph within the time window, and complete the link between any two given entities based on the continuous knowledge graph; if the identified link already exists, if not, the completion of the temporal audio-visual knowledge graph is completed.

2. The method for automatically generating a temporal audio-visual knowledge graph for scene understanding according to claim 1, characterized in that: The time domain segmentation of the audiovisual perception data in S2 includes the following contents: S2.1: Iteratively calculate the audio adaptive energy threshold using the short-time energy of the audio frame. The formula is as follows E th =α·E mean +(1-a)·E prev_th , Where E th Indicates the audio adaptability threshold of the current iteration calculation. α is an attenuation factor between 0 and 1, which is used to balance the current average energy E mean and the threshold E obtained in the previous iteration prev_th ; S2.2: Use the audio adaptive energy threshold to determine the start and end of the audiovisual event of the audiovisual perception data, determine the segmentation points of the audio frames in the audiovisual perception data, and perform synchronous cutting on the audio frames and the corresponding video frames, so as to extract a series of non-overlapping and audiovisual information matching fragment pairs.

3. The method for automatically generating a temporal audio-visual knowledge graph for scene understanding according to claim 1, characterized in that: The synchronized visual-auditory features extracted in S4 include the following: S4.1: Use the VGG19 network model pre-trained on ImageNet to extract the feature map of the sampled frames per second, perform average pooling on the feature map, and obtain the fragment-level visual encoding features Where T represents the duration of the video clip in seconds, k represents the spatial dimension of the feature map extracted from each frame, and d v Represents the dimension of visual encoding features; S4.2: For each one-second audio clip, use the VGG-like network model pre-trained on AudioSet to obtain the segment-level auditory encoding features where d a Represents the dimension of auditory encoding features; S4.3: Visual features encoded using the cross-attention mechanism and audio features in and The calculation formula is as follows: where α t and α t ′ represents the attention weight vector, which is calculated using a multilayer perceptron with a Softmax activation function; S4.4: Visual features Audio Features Input into two separate LSTMs, the enhanced visual features and audio features are represented as and where d l is the dimension of the enhanced visual feature or audio feature.

4. The method for automatically generating a temporal audio-visual knowledge graph for scene understanding according to claim 1, characterized in that: The S5 specifically includes the following steps: S5.1: Determining the visual segment H in consecutive synchronized audiovisual segments by calculating similarity v and audio clip H a The strength of the connection between them is calculated as follows: in and is the training parameter, ω va and ω av ∈R T×T is the similarity matrix; S5.2: Use the ReLU activation function to eliminate negative connections within continuous synchronized audiovisual segments, and use the set threshold τ1 to eliminate weak connections within segments below the threshold, and obtain the final similarity matrix η va and η av ∈R T×T ;Threshold τ1 is a constant; S5.3: By formula Get the time index of strongly connected fragments in Represents the similarity matrix η va diagonal elements of .

5. The method for automatically generating a temporal audio-visual knowledge graph for scene understanding according to claim 1, characterized in that: The entity alignment in step S7 includes the following contents: S7.1 Temporal alignment: Align the frame-level scene graph and the clip-level audio categories to the same timeline according to the video timestamps, thereby integrating the temporal visual-audio knowledge graph information under the same video frame Where M represents the total number of video frames, Respectively represent the audio knowledge graph information and visual knowledge graph information in the i-th frame; S7.2 Semantic alignment: The embedding vector of the entity is obtained through the pre-trained GloVe model, and the cosine similarity is used to measure the similarity between visual-audio entities in the same video frame. The calculation formula is as follows: in Indicates that in video frame I i GloVe embedding vectors of the visual and audio entities at ; When the similarity of some entity pairs exceeds the specified threshold τ2, semantic alignment is established, expressed as a "related to" relationship: Generate a new triple The threshold τ2 is a constant; S7.3: After entity alignment, construct the final temporal multimodal knowledge graph 6. The method for automatically generating a temporal audio-visual knowledge graph for scene understanding according to claim 1, characterized in that: The step S8 includes the following contents: S8.1: Apply the traditional translation-based method TransE to learn knowledge graph representations; S8.2: TransE defines the energy of a triple by calculating the distance between embedding vectors: E(s,r,o)=d(s+r,o)=|s+ro|; where s, r, and o represent the temporal multimodal knowledge graph The feature vectors of the subject, relation, and object in ; S8.3: Use element-by-element multiplication of entity embedding vectors to obtain interaction information between entities, and complete the links through a logistic regression classifier; if the identified link already exists, if not, complete the completion of the temporal audio-visual knowledge graph.

7. The method for automatically generating a temporal audio-visual knowledge graph for scene understanding according to claim 1, characterized in that: This method also includes proposing an optimization model to improve the completeness and accuracy of the generated temporal audio-visual knowledge graph: The optimization function It consists of three parts: classification loss Visual-Audio Clip Similarity Loss and embedding vector loss Among them, the classification loss Visual-audio segment similarity loss is used to improve the prediction ability of synchronized audio-visual segment events. Used to improve the ability to judge the similarity between synchronized audio-visual clips, embedding vector loss Used to improve the ability to learn entity semantic information.

Images