A method for automatically generating a time-series audio-visual knowledge graph for scene understanding

Abstract

The application provides a time sequence audio-visual knowledge graph automatic generation method for scene understanding, belonging to the computer field.The application takes video and audio data as the input of scene understanding, and divides them into continuous audio-visual segments based on time synchronization.Then, the background time period of the audio event is identified by segmenting the audio signal, and the segments with event occurrence are sampled.Subsequently, the entity and relationship detection technology is used to generate the audio-visual knowledge graph at each sampling time.Finally, the relationship completion is performed on the generated time sequence audio-visual knowledge graph to improve its integrity and accuracy.The application improves the integrity and practicability of the knowledge graph, enables the system to automatically infer and complete the related audio-visual relationship, and better supports the intelligent development of scene understanding and application.

Description

Technical Field

[0001] This invention relates to a method for automatically generating knowledge graphs, belonging to the field of computer science. Background Technology

[0002] Currently, with the continuous development of artificial intelligence technology, scene understanding has received widespread attention as an important research area. Traditional scene understanding mainly focuses on the visual perception of static images or videos. While this method performs well in certain scenarios, it has limitations when handling complex scenes and multimodal information. In practical applications, the information in the environment is not limited to vision but also includes multiple sensory modes such as sound and touch. Therefore, relying solely on visual information for scene understanding can no longer meet the needs for diverse information.

[0003] Against this backdrop, audiovisual information has become a crucial component of scene understanding. Audiovisual information possesses rich environmental information and cognitive value, enabling a more comprehensive and in-depth understanding of the scene. Therefore, combining audiovisual information with scene understanding has become a research hotspot. However, current research on audiovisual scene understanding is still relatively rudimentary, particularly in the processing and modeling of temporal information, where certain challenges remain.

[0004] To address the above problems, this invention focuses on audiovisual information, aiming to construct a dynamic knowledge graph of audiovisual scene entities and their relationships to achieve a deeper and more comprehensive understanding of scenes. Specifically, this invention proposes a method for automatically generating a temporal audiovisual knowledge graph. By analyzing and processing audiovisual data, it achieves a comprehensive perception and understanding of the temporal evolution of scenes. This method will bring new ideas and approaches to research and applications in the field of scene understanding, and is expected to play an important role in areas such as intelligent interaction and intelligent monitoring. Summary of the Invention

[0005] The main objective of this invention is to propose an automatic generation method for time-series audiovisual knowledge graphs based on scene understanding, in order to solve the problem that there is a lack of such an automatic generation method for time-series knowledge graphs based on scene understanding in the existing technology.

[0006] To achieve the above objectives, this invention provides a method for automatically generating a temporal audiovisual knowledge graph for scene understanding, comprising the following steps: S1: Acquire audiovisual perception data within the same time interval through audiovisual sensors; S2: Perform time-domain segmentation of audio-visual events based on the spectral amplitude of audio data in the audio-visual perception data to obtain synchronized audio-visual segments; S3: Perform time-uniform segmentation on the synchronized audiovisual segments to obtain... A series of equal-length, non-overlapping, continuous synchronized audiovisual segments; S4: Extract synchronized visual-audio features from video segments and corresponding audio segments in continuous synchronized audiovisual segments, and enhance video frame features and audio features. S5: Calculate the similarity of consecutive synchronized audiovisual segments after feature enhancement through the segment cross-attention mechanism, use the similarity between consecutive synchronized audiovisual segments to judge the semantic correlation between synchronized audiovisual segments, delete consecutive synchronized audiovisual segments with negative or weak correlation, and obtain consecutive synchronized audiovisual segments with strong correlation. S6: Perform scene graph recognition on video frames and category recognition on audio segments for highly correlated continuous synchronous audiovisual segments in order to construct an audiovisual knowledge graph on the timestamps of video frames; S7: Temporally align the video frame scene graph information and audio segment category information of closely related continuous synchronized audiovisual segments, aligning them to the same timeline according to the video timestamps; then, through semantic alignment, associate entities with high semantic similarity in the two modalities; and construct a temporal multimodal knowledge graph through entity alignment. S8: Use the audiovisual knowledge graphs on the timestamps corresponding to consecutive video frames in the temporal multimodal knowledge graph as the continuous audiovisual knowledge graph within the time window, and complete the links between any two given entities based on the continuous knowledge graph; if the identified links already exist, the completion of the temporal audiovisual knowledge graph is completed.

[0007] Furthermore, the temporal segmentation of the audiovisual perception data in step S2 above includes the following: S2.1: The audio adaptive energy threshold is calculated using the short-time energy iteration of audio frames, as shown in the following formula. , in This represents the audio adaptive capability threshold calculated in the current iteration. It is a decay factor between 0 and 1, used to balance the current average energy. The threshold obtained from the previous iteration ; S2.2: The start and end of the audiovisual events in the audiovisual perception data are determined by using the audio adaptive energy threshold, the segmentation point of the audio frame in the audiovisual perception data is determined, and the audio frame and the corresponding video frame are synchronously cut to extract a series of non-overlapping segments with matching audiovisual information.

[0008] Furthermore, the synchronized visual-auditory features extracted in step S4 above include the following: S4.1: Extract feature maps from the sampled frames per second using a VGG19 network model pre-trained on ImageNet, and perform average pooling on the feature maps to obtain fragment-level visual encoding features. , in Indicates the duration of a video clip, in seconds. This represents the spatial dimension of the feature map extracted from each frame. Dimensions representing visual encoded features; S4.2: For each one-second audio segment, use a VGG-like network model pre-trained on AudioSet to obtain segment-level auditory coding features. ,in Dimensions representing auditory coding features; S4.3: Visual features encoded using cross-attention mechanism and audio features ;in and The calculation formula is as follows: , ,in This represents the attention weight vector, which is computed using a multilayer perceptron with a softmax activation function. S4.4: Visual features Audio features The inputs are fed into two separate LSTMs, and the enhanced visual and audio features are represented as follows: and ,in It is the dimension of the enhanced visual or audio features.

[0009] Furthermore, step S5 specifically includes the following steps: S5.1: Determine visual segments in consecutive synchronized audiovisual segments by calculating similarity. and audio clips The strength of the connection between them is calculated using the following formula: , ,in and For training parameters, and It is a similarity matrix; S5.2: Utilize the ReLU activation function to eliminate negative connections within continuously synchronized audiovisual segments, using a set threshold. This eliminates weak connections within segments below a certain threshold, yielding the final similarity matrix. and Threshold It is a constant; S5.3: Through formula Obtain the time index of the strong connection segment ,in Representing similarity matrices The diagonal elements.

[0010] Furthermore, the entity alignment in step S7 above includes the following: S7.1 Temporal Alignment: Aligns frame-level scene graphs and segment-level audio categories to the same timeline according to video timestamps, thereby integrating temporal visual-audio knowledge graph information from the same video frame. ;in This indicates the total number of video frames. They represent the first time. Audio knowledge graph information and visual knowledge graph information for each frame; S7.2 Semantic Alignment: The embedding vectors of entities are obtained through a pre-trained GloVe model. Cosine similarity is used to measure the similarity between visual and audio entities in the same video frame. The calculation formula is as follows: ,in Indicates in video frame GloVe embedding vectors of visual and audio entities at a given location; when the similarity of certain entity pairs exceeds a specified threshold. In this case, semantic alignment is established, represented as a "related to" relationship: Generate a new triplet threshold ; S7.3: After entity alignment, construct the final temporal multimodal knowledge graph. .

[0011] Furthermore, step S8 above includes the following: S8.1: Applying the translation-based method TransE for knowledge graph representation learning; S8.2: TransE defines the energy of a triple by calculating the distance between the embedding vectors: ;in , and To represent temporal multimodal knowledge graphs Feature vectors of subject, relation, and object in the text; S8.3: Element-wise multiplication of entity embedding vectors is used to obtain interaction information between entities, and the links are completed by a logistic regression classifier; if the identified links already exist, the temporal audiovisual knowledge graph is completed if they do not exist.

[0012] Furthermore, the above method also includes: proposing an optimization function: The optimization function It consists of three parts, namely classification loss. Visual-audio segment similarity loss and embedding vector loss Among them, classification loss Visual-audio segment similarity loss is used to improve the predictive ability of synchronized audiovisual segment events. Embedded vector loss is used to improve the ability to judge the similarity between synchronized audiovisual segments. Used to improve the ability to learn semantic information about entities.

[0013] The beneficial technical effects of the present invention are as follows: This method integrates visual and auditory information sources to construct a knowledge graph of dynamic audiovisual scene entities and their relationships, overcoming the limitations of visual-centric scene understanding and enabling a more comprehensive perception and cognition of the environment. A simple and efficient audio data temporal segmentation method is used to initially distinguish audiovisual event segments and filter background noise, reducing the complexity of subsequent processing. Visual and auditory features are further enhanced, and a cross-attention mechanism is combined to calculate the similarity of audiovisual segments, intelligently identifying and retaining strongly correlated information while eliminating redundant content, significantly improving the construction accuracy of the temporal multimodal knowledge graph. This method not only ensures data accuracy and consistency at the temporal level but also enriches the understanding of complex events and scenes at the semantic level. Finally, by integrating strongly correlated video frame scene graph information and audio segment category information, a unified temporal multimodal knowledge graph is constructed through temporal and semantic alignment, realizing the connection and association between multimodal data and enabling a deeper expression and understanding of complex events and scenes. Based on this graph, the continuous audiovisual knowledge graph completes the links between entities, further improving the completeness and practicality of the knowledge graph. This enables the system to automatically infer and complete relevant audiovisual relationships, thereby better supporting the intelligent development of scene understanding and applications. Attached Figure Description

[0014] Figure 1 A flowchart of a method for automatically generating temporal audiovisual knowledge graphs based on scene understanding.

[0015] Figure 2 This is a schematic diagram of the structure of a method for automatically generating temporal audiovisual knowledge graphs based on scene understanding. Detailed Implementation

[0016] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0017] like Figure 1 As shown, this invention is a method for automatically generating temporal audiovisual knowledge graphs based on scene understanding, comprising the following steps: S1: Acquire audiovisual perception data within the same time interval through audiovisual sensors; S2: Perform time-domain segmentation of audio-visual events based on the spectral amplitude of audio data in the audio-visual perception data to obtain synchronized audio-visual segments; S3: Perform time-uniform segmentation on the synchronized audiovisual segments to obtain... A series of equal-length, non-overlapping, continuous synchronized audiovisual segments; As a specific embodiment of the present invention, continuous visual segments and audio segments are represented as follows: Each segment lasts one second, and 16 frames are uniformly sampled within each one-second visual segment. Here, and Representing sequence data respectively A series of visual and audio segments.

[0018] S4: Extract synchronized visual-audio features from video segments and corresponding audio segments in continuous synchronized audiovisual segments, and enhance video frame features and audio features. S5: Calculate the similarity of consecutive synchronized audiovisual segments after feature enhancement through the segment cross-attention mechanism, use the similarity between consecutive synchronized audiovisual segments to judge the semantic correlation between synchronized audiovisual segments, delete consecutive synchronized audiovisual segments with negative or weak correlation, and obtain consecutive synchronized audiovisual segments with strong correlation. S6: Perform scene graph recognition on video frames and category recognition on audio segments for highly correlated continuous synchronous audiovisual segments in order to construct an audiovisual knowledge graph on the timestamps of video frames; S7: Temporally align the video frame scene graph information and audio segment category information of closely related continuous synchronized audiovisual segments, aligning them to the same timeline according to the video timestamps; then, through semantic alignment, associate entities with high semantic similarity in the two modalities; and construct a temporal multimodal knowledge graph through entity alignment. S8: Use the audiovisual knowledge graphs on the timestamps corresponding to consecutive video frames in the temporal multimodal knowledge graph as the continuous audiovisual knowledge graph within the time window, and complete the links between any two given entities based on the continuous knowledge graph; if the identified links already exist, the completion of the temporal audiovisual knowledge graph is completed.

[0019] In a preferred embodiment of the present invention, the temporal segmentation of the audiovisual perception data in step S2 includes the following: S2.1: The audio adaptive energy threshold is calculated using the short-time energy iteration of audio frames, as shown in the following formula. , in This represents the audio adaptive capability threshold calculated in the current iteration. It is a decay factor between 0 and 1, used to balance the current average energy. The threshold obtained from the previous iteration ; S2.2: The start and end of the audiovisual events in the audiovisual perception data are determined by using the audio adaptive energy threshold, the segmentation point of the audio frame in the audiovisual perception data is determined, and the audio frame and the corresponding video frame are synchronously cut to extract a series of non-overlapping segments with matching audiovisual information.

[0020] In a preferred embodiment of the present invention, the extraction of synchronized visual-auditory features in step S4 includes the following: S4.1: Extract feature maps from the sampled frames per second using a VGG19 network model pre-trained on ImageNet, and perform average pooling on the feature maps to obtain fragment-level visual encoding features. , in Indicates the duration of a video clip, in seconds. This represents the spatial dimension of the feature map extracted from each frame. Dimensions representing visual encoded features; S4.2: For each one-second audio segment, use a VGG-like network model pre-trained on AudioSet to obtain segment-level auditory coding features. ,in Dimensions representing auditory coding features; S4.3: Visual features encoded using cross-attention mechanism and audio features ;in and The calculation formula is as follows: , , in This represents the attention weight vector, which is computed using a multilayer perceptron with a softmax activation function. S4.4: Visual features Audio features The inputs are fed into two separate LSTMs, and the enhanced visual and audio features are represented as follows: and ,in It is the dimension of the enhanced visual or audio features.

[0021] In a preferred embodiment of the present invention, step S5 specifically includes the following steps: S5.1: Determine visual segments in consecutive synchronized audiovisual segments by calculating similarity. and audio clips The strength of the connection between them is calculated using the following formula: , ,in and For training parameters, and Let be a similarity matrix; where and For training parameters, and It is a similarity matrix; S5.2: Utilize the ReLU activation function to eliminate negative connections within continuously synchronized audiovisual segments, using a set threshold. This eliminates weak connections within segments below a certain threshold, yielding the final similarity matrix. and Threshold It is a constant; To achieve this goal, the ReLU activation function is used to eliminate negative connections within segments, and line-by-line activation is applied. Normalization yields the similarity matrix This process can be formalized as follows: ,in Represented as Normalization This represents the ReLU activation function. Then, a threshold is used to filter out weak connections with relatively low similarity. After applying the threshold, the process is repeated row by row. Normalization, the process of which is formalized as follows: , ,in It is a hyperparameter threshold that indicates that connections with similarity values ​​below this threshold will be filtered out. It is an index function that outputs 1 when a certain condition is met, and 0 otherwise, generating the final similarity matrix. and Finally, time indices with strong connections can be identified through the following calculation: ,in, Represents the set of all time indices, corresponding to these indices. The diagonal elements are greater than 0; S5.3: Through formula Obtain the time index of the strong connection segment ,in Representing similarity matrices The diagonal elements.

[0022] In a preferred embodiment of the present invention, the scene graph recognition of the video frame in step S6 specifically includes: S6.1.1: Use Faster-RCNN to extract object regions in video frames and preprocess the relation feature vectors; S6.1.2: Using the relational feature vector as input, extract the spatial context information of each frame using a spatial encoder; S6.1.3: Using the output of the spatial encoder as input, a temporal decoder is used to capture temporal dependencies, and a linear classifier is used to predict relationships.

[0023] Specifically, using Faster-RCNN as the backbone network, for networks with... Visual content of a frame ,time The frame is represented as The detector extracts visual features bounding box Distribution of object types ,in This indicates the number of objects in that frame. (In each frame) There is a set of relationship vectors between the objects. , No. and Between objects The representation vector corresponding to the relation It includes visual information, spatial information, and semantic embedding, and the formula is as follows: ,in, Indicates a connection operation. This indicates a flattening operation. This indicates element-wise operation. , Represents a linear matrix with dimensionality compression. This represents the feature map of the union box calculated by RoIAlign. It is a function that transforms the bounding boxes of the subject and object into a syntax similar to... Functions of complete features with the same shape. Semantic embedding vectors. The object category is determined by the subject and object. In the spatial-temporal Transformer, the relation representation exchanges spatial and temporal information.

[0024] The Spatial-Temporal Transformer retains the original encoder-decoder architecture. The difference is that the encoder and decoder are given more specific tasks.

[0025] The spatial encoder focuses on the spatial context of each frame, and its input is a single... Query ,key Sum Sharing the same input, the first The output of the layer encoder is represented as: The encoder consists of N identical self-attention layers. The first is formed by stacking them sequentially. The input of the layer is equal to the first layer. The layer's output. Further customization of frame encoding to inject temporal location into the relational representation. This is due to the windowing in the temporal decoder. The number of determined embedding vectors is fixed and relatively short, therefore frame coding It is built using learned embedding parameters. ,in Is and They have the same absolute vector.

[0026] The time decoder captures the temporal dependencies between frames. and The multi-head self-attention layer was also removed. (Sliding window) In spatial context representation Executed on the sequence, the first Each generated input batch is represented as: Window size The decoder, like the encoder, consists of N identical self-attention layers. It is composed of layers stacked sequentially, with the first layer being: , , .

[0027] In the formula In this process, frame encoding is incorporated into the relation representation. The output of the last decoder layer is used for final prediction. Due to the existence of a sliding window, relations in a frame have different representations in different batches; the representation that appears earliest in the window is selected.

[0028] Based on the above method, from visual fragments A temporal visual knowledge graph was extracted. .here, Represents a frame Scene diagrams, in which It is a collection of entities. It is a set of relations. Entities and The relationships in these elements together constitute the set of visual triples. .

[0029] In a preferred embodiment of the present invention, the category identification of the audio segment in step S6 specifically includes: S6.2.1: Transfer audio clips Convert the spectrum to a size of 1024×128 and divide it into patches of 16×16 blocks without overlap.

[0030] S6.2.2: Use the Audio Spectrogram Transformer (AST) model to extract the embedding information of the patch.

[0031] S6.2.3: Randomly select a patch, generate a mask based on the clustering factor, and apply the mask to the patch embedding information. Specifically, randomly extract indices from the range {1, 512}, and generate a set of neighboring indices based on the clustering factor. Repeat this process until the index set contains a specified number of patch locations; S6.2.4: Input the mask patch embedding information and position information into the Transformer model to generate the final audio classification result.

[0032] In a preferred embodiment of the present invention, step S7 regarding entity alignment includes the following: S7.1 Temporal Alignment: Aligns frame-level scene graphs and segment-level audio categories onto the same timeline according to video timestamps, thereby integrating temporal visual-audio knowledge graph information from the same video frame. ,in This indicates the total number of video frames. They represent the first time. Audio knowledge graph information and visual knowledge graph information for each frame; S7.2 Semantic Alignment: Entity embedding vectors are obtained using a pre-trained GloVe model; cosine similarity is used to measure the similarity between visual and audio entities on the same video frame, calculated as follows: ,in Indicates in video frame GloVe embedding vectors of visual and audio entities at a given location; when the similarity of certain entity pairs exceeds a specified threshold. At that time, they will establish semantic alignment, represented as a "related to" relationship: Specifically, if Greater than Then a new triple is generated. threshold It is a constant.

[0033] S7.3: After entity alignment, construct the final temporal multimodal knowledge graph. .

[0034] In a preferred embodiment of the present invention, step S8 includes the following: S8.1: Applying the translation-based method TransE for knowledge graph representation learning. In this method, a knowledge graph... Includes triples ,in , and These represent the subject, relation, and object, respectively. Feature vectors are used. , and To represent these entities and relationships. The basic idea of ​​TransE is to learn embedding vectors such that for valid triples, Roughly equal to As an energy-based model, TransE defines the energy of a triple by calculating the distance between embedding vectors: This framework enables the efficient capture of semantic relationships embedded in knowledge graphs; S8.2: TransE defines the energy of a triple by calculating the distance between the embedding vectors: ;in , and To represent temporal multimodal knowledge graphs Feature vectors of subject, relation, and object in the text; S8.3: Element-wise multiplication of entity embedding vectors is used to obtain interaction information between entities, and the links are completed by a logistic regression classifier; if the identified links already exist, the temporal audiovisual knowledge graph is completed if they do not exist.

[0035] As a preferred embodiment of the present invention, the present invention further includes: proposing an optimization function to improve the completeness and accuracy of the generated temporal audiovisual knowledge graph: The optimization function It consists of three parts, namely classification loss. Visual-audio segment similarity loss and embedding vector loss Among them, classification loss Visual-audio segment similarity loss is used to improve the predictive ability of synchronized audiovisual segment events. Embedded vector loss is used to improve the ability to judge the similarity between synchronized audiovisual segments. Used to improve the ability to learn semantic information about entities.

[0036] As a specific embodiment of the present invention, in order to improve the prediction ability of video segment event categories, a cross-entropy classification loss function is adopted. This is used to measure the difference between the predicted event label and the actual label, and the formula is as follows: ,in Description of time Temporal categories of visual-audio segments. Visual-audio segment similarity loss. The formula is as follows: .in It is the cosine similarity between normalized visual and audio features. Describe whether a pair of visual-audio clips contains an event. This represents the mean square error between two vectors.

[0037] When learning the embedding information, entity embedding is further optimized by minimizing the energy difference between positive and negative samples, resulting in a loss of embedding vector. The formula is as follows: ,in This is a specific boundary value to ensure that the energy of positive samples is at least higher than this boundary value. Specifically, for each pair of positive and negative samples, the energy difference between them is calculated. If the difference is less than the boundary value, the loss is 0; otherwise, the difference is subtracted from the boundary value.

[0038] like Figure 2 The diagram illustrates the structural system of the proposed automatic generation method for temporal audiovisual knowledge graphs for scene understanding. It comprises several key modules: first, a module for acquiring audiovisual data for a specific scene; second, a temporal segmentation module based on audio spectrum; third, a module for extracting and encoding audiovisual segment features; and fourth, a module for extracting highly correlated audiovisual segments. The outputs of these modules are highly correlated audiovisual segments, which are input into the temporal visual knowledge graph construction module and the temporal audio knowledge graph construction module, respectively. Furthermore, these knowledge graphs are input into the temporal audiovisual knowledge graph alignment module and the temporal multimodal knowledge graph completion module to achieve a comprehensive understanding and analysis of the scene.

Claims

1. A method for automatically generating temporal audiovisual knowledge graphs based on scene understanding, characterized in that, Includes the following steps: S1: Acquire audiovisual perception data within the same time interval through audiovisual sensors; S2: Perform time-domain segmentation of audio-visual events based on the spectral amplitude of audio data in the audio-visual perception data to obtain synchronized audio-visual segments; S3: Perform time-uniform segmentation on the synchronized audiovisual segments to obtain... A series of equal-length, non-overlapping, continuous synchronized audiovisual segments; S4: Extract synchronized visual-audio features from video segments and corresponding audio segments in continuous synchronized audiovisual segments, and enhance video frame features and audio features. S5: Calculate the similarity of consecutive synchronized audiovisual segments after feature enhancement through the segment cross-attention mechanism, use the similarity between consecutive synchronized audiovisual segments to judge the semantic correlation between synchronized audiovisual segments, delete consecutive synchronized audiovisual segments with negative or weak correlation, and obtain consecutive synchronized audiovisual segments with strong correlation. S6: Perform scene graph recognition on video frames and category recognition on audio segments for highly correlated continuous synchronous audiovisual segments in order to construct an audiovisual knowledge graph on the timestamps of video frames; S7: Perform temporal alignment of video frame scene graph information and audio segment category information of closely related continuous synchronized audiovisual segments, aligning them to the same timeline according to the video timestamps; then, through semantic alignment, associate entities with high semantic similarity in the two modalities. Construct a temporal multimodal knowledge graph through entity alignment; S8: Use the audiovisual knowledge graphs on the timestamps corresponding to consecutive video frames in the temporal multimodal knowledge graph as the continuous audiovisual knowledge graph within the time window, and complete the links between any two given entities based on the continuous knowledge graph; if the identified links already exist, the completion of the temporal audiovisual knowledge graph is completed if they do not exist. S5 specifically includes the following steps: S5.1: Determine visual segments in consecutive synchronized audiovisual segments by calculating similarity. and audio clips The strength of the connection between them is calculated using the following formula: , ,in and For training parameters, and It is a similarity matrix; S5.2: The ReLU activation function is used to eliminate negative connections within continuously synchronized audiovisual segments, using a set threshold. This eliminates weak connections within segments below a certain threshold, yielding the final similarity matrix. and Threshold It is a constant; S5.3: Through formula Obtain the time index of the strong connection segment ,in Representing similarity matrices The diagonal elements.

2. The method for automatically generating a temporal audiovisual knowledge graph based on scene understanding as described in claim 1, characterized in that, The temporal segmentation of the audiovisual perception data in S2 includes the following: S2.1: The audio adaptive energy threshold is calculated using the short-time energy iteration of audio frames, as shown in the following formula: ,in This represents the audio adaptive capability threshold calculated in the current iteration. It is a decay factor between 0 and 1, used to balance the current average energy. The threshold obtained from the previous iteration ; S2.2: Use the audio adaptive energy threshold to determine the start and end of audiovisual events in the audiovisual perception data, determine the segmentation point of the audio frame in the audiovisual perception data, and perform synchronous cutting on the audio frame and the corresponding video frame to extract a series of non-overlapping segments with matching audiovisual information.

3. The method for automatically generating a temporal audiovisual knowledge graph based on scene understanding as described in claim 1, characterized in that, The synchronized visual-auditory features extracted in S4 include the following: S4.1: Extract feature maps from the sampled frames per second using a VGG19 network model pre-trained on ImageNet, and perform average pooling on the feature maps to obtain fragment-level visual encoding features. ,in Indicates the duration of a video clip, in seconds. This represents the spatial dimension of the feature map extracted from each frame. Dimensions representing visual encoded features; S4.2: For each one-second audio segment, use a VGG-like network model pre-trained on AudioSet to obtain segment-level auditory coding features. ,in Dimensions representing auditory coding features; S4.3: Visual features encoded using cross-attention mechanism and audio features ; in and The calculation formula is as follows: , ,in This represents the attention weight vector, which is computed using a multilayer perceptron with a softmax activation function. S4.4: Visual features Audio features The inputs are fed into two separate LSTMs, and the enhanced visual and audio features are represented as follows: and ,in It is the dimension of the enhanced visual or audio features.

4. The method for automatically generating a temporal audiovisual knowledge graph based on scene understanding as described in claim 1, characterized in that, The entity alignment in step S7 includes the following: S7.1 Temporal Alignment: Aligns frame-level scene graphs and segment-level audio categories to the same timeline according to video timestamps, thereby integrating temporal visual-audio knowledge graph information from the same video frame. ;in This indicates the total number of video frames. They represent the first time. Audio knowledge graph information and visual knowledge graph information for each frame; S7.2 Semantic Alignment: The embedding vectors of entities are obtained through a pre-trained GloVe model. Cosine similarity is used to measure the similarity between visual and audio entities in the same video frame. The calculation formula is as follows: ,in Indicates in video frame GloVe embedding vectors of visual and audio entities at a given location; when the similarity of certain entity pairs exceeds a specified threshold. In this case, semantic alignment is established, represented as a "related to" relationship: Generate a new triplet threshold ; S7.3: After entity alignment, construct the final temporal multimodal knowledge graph. .

5. The method for automatically generating a temporal audiovisual knowledge graph based on scene understanding as described in claim 1, characterized in that, Step S8 includes the following: S8.1: Applying the translation-based method TransE for knowledge graph representation learning; S8.2: TransE defines the energy of a triple by calculating the distance between the embedding vectors: ;in , and To represent temporal multimodal knowledge graphs Feature vectors of subject, relation, and object in the text; S8.3: Element-wise multiplication of entity embedding vectors is used to obtain interaction information between entities, and the links are completed by a logistic regression classifier; if the identified links already exist, the temporal audiovisual knowledge graph is completed if they do not exist.

6. The method for automatically generating a temporal audiovisual knowledge graph based on scene understanding as described in claim 1, characterized in that, An optimization model is proposed: Optimization function It consists of three parts, namely classification loss. Visual-audio segment similarity loss and embedding vector loss Among them, classification loss Visual-audio segment similarity loss is used to improve the predictive ability of synchronized audiovisual segment events. Embedded vector loss is used to improve the ability to determine the similarity between synchronized audiovisual segments. Used to improve the ability to learn semantic information about entities.

Images