Multi-modal fusion panoramic video intelligent content analysis method

Abstract

The invention relates to the technical field of panoramic video processing, in particular to a multi-modal fusion panoramic video intelligent content analysis method, which comprises the following steps of: acquiring a panoramic video frame and a synchronous audio thereof; performing spherical projection transformation on the panoramic image data, and converting the equidistant projection image into a spherical image; generating a plurality of plane images based on the spherical image; performing candidate target position detection based on the plurality of plane images; based on a candidate target position detection result, an image target detection model is utilized to realize instance segmentation of a target, and target features are extracted to generate a target embedded description vector sequence; based on the synchronous audio, an audio recognition processing module is used for carrying out time sequence information extraction, and an acoustic attention sequence is formed; a spectrum sensor is adopted to realize association fusion of the target embedded description vector sequence and the acoustic attention sequence, and an audio and video collaborative attention sequence is generated; and based on the audio and video collaborative attention sequence, establishing an audio and visual event identification model, and obtaining event content described by the panoramic content.

Description

Technical Field

[0001] The present invention relates to the technical field of panoramic video processing, and more specifically, to a method for intelligent content analysis of panoramic videos with multi-modal fusion. Background Art

[0002] With the rapid development of technologies such as virtual reality, augmented reality, and intelligent surveillance, panoramic videos are playing an increasingly important role in various application scenarios. With its 360-degree visual experience, panoramic videos provide users with an immersive experience. However, the massive amount of information contained in panoramic videos also poses a huge challenge to content analysis. Traditional video analysis methods are often unable to cope with panoramic videos. It is difficult to effectively deal with the geometric distortion caused by spherical projection, and it is also impossible to fully utilize the rich contextual information in panoramic videos.

[0003] In recent years, with the advancement of deep learning technology, video analysis methods based on convolutional neural networks have made significant progress. These methods have shown excellent performance in tasks such as object detection and behavior recognition. However, when applied to panoramic videos, these methods still have many limitations. First, conventional convolutional neural networks are designed to process flat images and are difficult to be directly applied to panoramic videos with spherical projection. Secondly, most existing methods only focus on visual information and ignore important clues contained in other modal data such as audio. Furthermore, mainstream video analysis methods often separate different analysis tasks, making it difficult to fully utilize the correlation between tasks.

[0004] In addition, existing video analysis methods also face challenges when processing long-time videos. Although traditional recurrent neural networks can model temporal relationships, they are prone to gradient vanishing or gradient exploding problems when processing long videos. Although long short-term memory networks (LSTM) have alleviated this problem to a certain extent, they still have difficulty capturing long-term dependencies in videos.

[0005] Another issue worth noting is that existing video analysis methods often lack deep semantic understanding of video content. Most methods only stay at the level of identifying "what is seen" and cannot understand high-level semantic information such as "why it happened". This shallow understanding limits the application of video analysis systems in complex scenarios.

[0006] In view of the above problems, a new method is urgently needed to comprehensively, deeply and efficiently analyze the content of panoramic videos. The ideal method should be able to effectively process spherical projection data, fuse multimodal information, capture long-term temporal dependencies, understand deep semantics, and handle multiple related tasks simultaneously. The present invention is proposed to solve these technical problems. Summary of the invention

[0007] The present invention provides a multi-modal fusion panoramic video intelligent content analysis method, which aims to solve many problems existing in the existing technology when processing panoramic videos. The method cleverly combines advanced technologies such as spherical convolutional networks, multi-modal fusion, long-series spatiotemporal graph convolutional networks, knowledge graph reasoning and multi-task learning to achieve a comprehensive, in-depth and efficient analysis of panoramic video content.

[0008] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0009] The multi-modal fusion panoramic video intelligent content analysis method includes:

[0010] The acquisition steps include:

[0011] Get panoramic video frames and their synchronized audio;

[0012] Performing spherical projection transformation on the panoramic image data to convert the equidistant projection image into a spherical image;

[0013] Processing steps include:

[0014] Based on the spherical image, generating a plurality of planar images;

[0015] Based on the multiple planar images, performing candidate target position detection;

[0016] Based on the candidate target position detection result, an image target detection model is used to implement instance segmentation of the target, and target features are extracted to generate a target embedding description vector sequence;

[0017] Based on the synchronized audio, extracting timing information using an audio recognition processing module to form an acoustic attention sequence;

[0018] A spectral sensor is used to realize the association fusion of the target embedding description vector sequence and the acoustic attention sequence to generate an audio-video collaborative attention sequence;

[0019] Based on the audio-visual collaborative attention sequence, establishing an audio-visual event recognition model;

[0020] Output steps include:

[0021] Using the audio-visual event recognition model, generating a spatiotemporal trajectory of the audio-visual event;

[0022] Establishing a long-term spatiotemporal graph convolutional network to detect the causal relationship in the spatiotemporal trajectory of the audio-visual event;

[0023] The spatiotemporal event trajectory is semantically parsed to obtain event content described by the panoramic content.

[0024] Preferably, the candidate target position detection specifically includes:

[0025] Performing 2D convolution feature extraction using the planar image;

[0026] Performing 2D object detection processing on the plane image using a region-based object detection model to obtain object bounding box coordinates and corresponding scores;

[0027] Converting the planar projection image into a spherical image, and mapping the coordinates of each pixel point to a spherical coordinate system according to an isometric projection;

[0028] According to the spherical coordinates, the spherical image is converted into a plurality of plane images by sampling, thereby realizing the detection of the spherical target.

[0029] Preferably, the target instance segmentation specifically includes:

[0030] Through the convolutional network, an image feature map is formed;

[0031] Using multiple feature fusion methods to perform detection on the feature map to extract features of the regional target;

[0032] The non-maximum suppression method is used to remove redundant frames to obtain the final detection result;

[0033] For the detected target, using an instance segmentation model, generating a pixel-level segmentation result of the instance based on semantic segmentation and the image feature map;

[0034] The segmentation results within the target bounding box are used to generate the target's bounding box based on the coordinates of the bounding box, completing the fusion of image and spatial semantic information.

[0035] Preferably, the generation of the target embedding description vector specifically includes:

[0036] Generate feature concatenation combination using target semantic information and feature information;

[0037] Mapping the feature concatenation combination into a semantic embedding vector through a multi-layer perceptron;

[0038] Construct a target description vector with target semantic information.

[0039] Preferably, the generation of the audio-video collaborative attention sequence specifically includes:

[0040] Performing spectrogram conversion on the synchronized audio to convert the audio into a time-series spectrogram;

[0041] After multi-layer spectrogram convolution decomposition, the time series spectrum graph is transformed from low-dimensional to high-dimensional spectrum sequence to form an audio feature map;

[0042] Processing the target embedding vector and the spectrogram-converted audio feature map through a spectral sensor;

[0043] The generated audio attention sequence is combined with the target embedding vector to generate an audio-video collaborative attention sequence.

[0044] Preferably, the event detection of the audio-visual video clip specifically includes:

[0045] Input the audio-video collaborative attention sequence into an event classifier based on audio-video attention and audio-visual joint learning;

[0046] The audio and video attention are formed by convolutional neural networks respectively;

[0047] The audio and video feature maps are obtained through convolution, ReLU activation function, pooling, and fully connected network respectively;

[0048] splicing the audio and video feature maps to generate an audio-visual joint feature map;

[0049] The result sequence of audio-visual event detection is generated through the softmax layer.

[0050] Preferably, a spherical convolution network step is also included:

[0051] Design a two-stream parallel spherical convolution structure, one branch captures spatial features and the other captures semantic features;

[0052] A spherical reshaping and rearrangement (SRL) module is introduced in the spatial feature branch to enhance spatial features.

[0053] In the semantic feature branch, 1×1 convolution and global pooling operations are used to highlight key semantic information;

[0054] The features of the two branches are combined to obtain the final spherical convolution feature map.

[0055] Preferably, the establishment of the long-term spatiotemporal graph convolutional network specifically includes:

[0056] Treat each video frame as a node in the graph and use graph convolutional networks to model the relationship between frames;

[0057] Adding temporal information to the graph convolutional layer is done by pooling only the convolutional layer before the current time step and then multiplying it with the features of the previous time step;

[0058] Use multi-layer graph convolution and ReLU activation function to enhance the nonlinear expression ability of the model;

[0059] The features of each node are converted into a classification probability vector through the softmax layer to obtain the final classification result.

[0060] As a preference, the following scene understanding steps based on knowledge graph are also included:

[0061] Build knowledge graphs for objects and events in videos, combining visual content with semantic knowledge;

[0062] Use graph neural networks to reason about knowledge graphs and achieve deep semantic understanding of video content;

[0063] Combine event prediction with entity recognition to improve the accuracy of scene understanding;

[0064] Through knowledge graph reasoning and event prediction, we can obtain the labels of entities and descriptions of events in the video.

[0065] Preferably, a multi-task learning step is also included:

[0066] Design a lightweight multi-task shared convolutional layer to extract common features of multiple tasks;

[0067] Design a multi-task learning mechanism based on the dependencies between tasks and optimize the model structure;

[0068] Integrate multiple tasks such as object detection, semantic segmentation, object tracking and event classification into one framework;

[0069] Through an end-to-end joint optimization method, multiple modules are jointly trained to enable the modules to adapt to and optimize each other.

[0070] Specifically, the beneficial effects of the present invention are mainly reflected in the following aspects:

[0071] First, by introducing spherical convolutional networks and spherical projection transformation, this method effectively solves the geometric distortion problem in panoramic video processing. This design enables the model to extract features directly in spherical space, avoiding the information loss and computational redundancy caused by splitting spherical images into multiple plane images in traditional methods. This not only improves the accuracy of feature extraction, but also significantly improves computational efficiency.

[0072] Secondly, the present invention innovatively integrates visual and auditory information. By designing a clever spectral sensor, a deep integration of audio information and visual features is achieved. This multimodal fusion method can make full use of the complementarity between different modal data, greatly improving the accuracy of event detection and scene understanding. For example, in a noisy street scene, it may be difficult to accurately identify the person who is speaking based on visual information alone, but combining audio information can easily solve this problem.

[0073] Furthermore, the long-sequence spatiotemporal graph convolutional network proposed in this paper effectively solves the problem of long video analysis. The network can simultaneously capture the spatial and temporal dependencies in the video, especially when processing videos with long time spans. This enables the method to identify and understand complex long-term event sequences, such as abnormal behavior patterns in surveillance videos.

[0074] In addition, the present invention introduces a scene understanding module based on knowledge graph, which greatly improves the semantic understanding ability of the system. By combining visual content with prior knowledge, this method can not only identify objects and events in the video, but also understand the semantic relationship between them. This deep understanding enables the system to answer high-level questions such as "why did it happen", providing strong support for intelligent decision-making.

[0075] Finally, the multi-task learning framework adopted by the present invention realizes the collaborative optimization of multiple related tasks. By sharing the underlying features and learning multiple tasks in parallel, this method not only improves the computational efficiency, but also improves the overall performance through mutual promotion between tasks. For example, the collaborative learning of object detection and semantic segmentation tasks can provide useful spatial information to each other, thereby improving the accuracy of the two tasks.

[0076] In general, the method proposed in the present invention realizes a comprehensive, in-depth and efficient analysis of panoramic video content through a number of innovative designs. This method has broad application prospects in many fields such as intelligent monitoring, autonomous driving, virtual reality, etc. For example, in smart city management, this method can provide valuable decision support for city managers by analyzing panoramic monitoring videos, detecting traffic congestion, illegal parking and other problems in real time, and understanding the causes of these problems. In the field of autonomous driving, this method can help vehicles perceive the surrounding environment more comprehensively, predict the behavioral intentions of other road users, and make safer and smarter driving decisions.

[0077] By solving many challenges faced by the prior art, the present invention opens up a new research direction for intelligent content analysis of panoramic videos and makes important contributions to the technological progress in related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 The figure is an overall flow chart of the method of the present invention.

[0079] Figure 2 The logical block diagram of the process of generating the target embedding description vector of the present invention.

[0080] Figure 3 The logical block diagram of the generation process of the audio-video collaborative attention sequence of the present invention.

[0081] Figure 4The present invention is a logic block diagram of the event detection process of the audio-visual video clip. DETAILED DESCRIPTION

[0082] like Figure 1-4 As shown, the present invention provides a multi-modal fusion panoramic video intelligent content analysis method. The method cleverly combines visual and auditory information to achieve in-depth understanding and analysis of panoramic video content. The specific implementation steps of the method will be described in detail below.

[0083] First, the method includes an acquisition step. In this step, a panoramic video frame and its synchronous audio are acquired. Preferably, the resolution of the panoramic video frame can be 4K (3840×2160 pixels) or 8K (7680×4320 pixels) to ensure high-quality visual information. The synchronous audio usually adopts a sampling rate of 44.1kHz or 48kHz to capture rich acoustic details.

[0084] Next, the method performs a spherical projection transformation on the acquired panoramic image data to convert the equidistant projection image into a spherical image. This step is crucial because it can preserve the spatial relationship of the panoramic image. In one embodiment, an equi-angle spherical projection method can be used, which can be expressed by the following formula:

[0085] x=Rcosφcosθ

[0086] y=Rcosφsinθ

[0087] z=Rsinφ

[0088] Among them, R is the radius of the sphere, θ is the azimuth angle, and φ is the elevation angle. Through this transformation, each point on the plane image can be mapped to the corresponding position on the sphere.

[0089] In the processing step, the method first generates multiple plane images based on the spherical image. The purpose of this step is to convert the spherical information into a more processable plane form. Preferably, a cubic projection method can be used to divide the sphere into 6 faces, each face corresponding to a plane image. This method can effectively reduce image distortion and improve the accuracy of subsequent processing.

[0090] Subsequently, the method performs candidate object location detection based on the generated multiple planar images. At this stage, a region-based convolutional neural network (R-CNN) can be used to achieve efficient object detection. For example, the FasterR-CNN model can be used, which performs well in object detection tasks. Specifically, Faster R-CNN contains a region proposal network (RPN) and a Fast R-CNN detector. RPN can generate candidate regions by the following formula:

[0091] p i =softmax(Wx i +b)

[0092] Among them, x i is the input feature, W and b are weight and bias parameters respectively, p i is the probability of the generated candidate region.

[0093] Based on the detection results of candidate target locations, this method uses the image target detection model to achieve instance segmentation of the target and extracts target features to generate a sequence of target embedding description vectors. At this stage, the Mask R-CNN model can be used, which adds a branch for instance segmentation based on FasterR-CNN. For each detected target, a feature vector f can be obtained. i , this vector can be obtained as follows:

[0094] f i =CNN(ROI i )

[0095] Among them, ROI i represents region of interest and CNN represents convolutional neural network.

[0096] At the same time, this method is based on synchronized audio and uses the audio recognition processing module to extract time sequence information to form an acoustic attention sequence. In this process, the long short-term memory network (LSTM) can be used to capture the time sequence characteristics of the audio. The core formula of LSTM is as follows:

[0097] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0098] i t =σ(W i ·[h t-1 ,x t ]+b i )

[0099]

[0100] o t =σ(W o ·[h t-1 ,x t ]+b o )

[0101] h t =o t *tanh(C t )

[0102] Among them, f t 、i t and t They are forget gate, input gate and output gate, C t is the cell state and h_t is the hidden state.

[0103] Next, this method uses a spectral perceptron to realize the association fusion of the target embedding description vector sequence and the acoustic attention sequence to generate an audio-video collaborative attention sequence. The spectral perceptron can be expressed by the following formula:

[0104]

[0105] where Q, K, and V are query, key, and value matrices, respectively, and d k is the dimension of the key. This attention mechanism can effectively capture the association between visual and auditory information.

[0106] Based on the generated audio-video co-attention sequence, this method builds an audio-visual event recognition model. At this stage, a 3D convolutional neural network (3D CNN) can be used to process spatiotemporal data. The convolution operation of 3D CNN can be expressed as:

[0107]

[0108] Among them, x is the input, w is the convolution kernel, and y is the output feature map.

[0109] In the output step, the method first uses the audio-visual event recognition model to generate the spatiotemporal trajectories of audio-visual events. These trajectories can be represented by a series of timestamps and spatial coordinates, such as (t i ,x i ,y i ,z i ), where t i is the time, (x i ,y i ,z i ) are spatial coordinates.

[0110] Subsequently, this method builds a long-term spatiotemporal graph convolutional network to detect causal relationships in the spatiotemporal trajectories of audio-visual events. The core operation of the graph convolutional network can be expressed as:

[0111]

[0112] Among them, A is the adjacency matrix with self-loops added, D is the corresponding degree matrix, and H (l) is the node feature of the lth layer, W (l) is a learnable weight matrix.

[0113] Finally, this method performs semantic analysis on the spatiotemporal event trajectory to obtain the event content described by the panoramic content. At this stage, a sequence-to-sequence model based on Transformer can be used to generate event descriptions. The core of Transformer is the multi-head attention mechanism, which can be expressed as:

[0114] MultiHead(Q,K,V)=Concat(head1,...,headh)W O

[0115] Among them, each head i They are all outputs of an attention mechanism.

[0116] Through the above steps, the method of the present invention can comprehensively analyze the content of the panoramic video, not only capturing visual information, but also combining audio information for a deeper understanding. This multimodal fusion method has broad application prospects in practical applications, such as video surveillance, autonomous driving, virtual reality and other fields. Continuing to explore the multimodal fusion panoramic video intelligent content analysis method of the present invention in depth, the generation process of the target embedding description vector will be described in detail. In a preferred embodiment of the present invention, the generation of the target embedding description vector includes multiple carefully designed steps to fully capture the semantic information and visual features of the target.

[0117] First, the method uses the target semantic information and feature information to generate a feature splicing combination. The core of this step is to effectively fuse information from different sources. Specifically, the target semantic information may come from a pre-trained semantic segmentation model, while the feature information may come from the intermediate layer output of a deep convolutional neural network. Preferably, the present invention uses ResNet-101 as the backbone network for feature extraction, which is pre-trained on the ImageNet dataset and can provide rich visual features.

[0118] In the feature concatenation process, this method adopts a dynamic weighting mechanism. Assuming that the semantic information is represented by vector s and the feature information is represented by vector f, the concatenated feature x can be expressed as:

[0119] x=αs+(1-α)f

[0120] Among them, α is a learnable parameter ranging from 0 to 1. This approach allows the model to dynamically adjust the importance of semantic information and visual features according to different inputs.

[0121] Next, the method of the present invention maps the feature concatenation combination into a semantic embedding vector through a multi-layer perceptron. This process can be expressed by the following mathematical formula:

[0122] e=MLP(x)=W n(σ(W n-1 (...σ(W1x+b1)...)+b n-1 ))+b n

[0123] Among them, W i and b i are the weight matrix and bias vector of the i-th layer respectively, and σ is the activation function. In a preferred embodiment of the present invention, ReLU is used as the activation function, that is, σ = max(0, x). The number of layers of the multilayer perceptron can be adjusted according to the complexity of the specific task. Generally, 3 to 5 layers can achieve good results.

[0124] Finally, the method constructs a target description vector with target semantic information. This target description vector not only contains the original visual features, but also incorporates high-level semantic information. Preferably, the present invention adopts the idea of ​​residual connection, adding the original features to the features processed by the multi-layer perceptron to obtain the final target description vector d:

[0125] d=e+x

[0126] This design can enhance the semantic expression ability while retaining the original information.

[0127] In practical applications, the method of the present invention can effectively handle various complex scenarios. For example, in an autonomous driving scenario, the method can accurately identify and describe various objects on the road, such as vehicles, pedestrians, traffic signs, etc. By generating rich target description vectors, the autonomous driving system can better understand the surrounding environment and make safer and smarter decisions.

[0128] Next, the generation process of the audio-video collaborative attention sequence in the present invention will be described in detail. This process is the core of multimodal fusion, which aims to effectively combine visual and auditory information.

[0129] In one embodiment of the present invention, the synchronous audio is first converted to a spectrogram, and the audio is converted into a time-series spectrogram. The purpose of this step is to convert the time domain signal into a frequency domain representation in order to better capture the characteristics of the audio. Specifically, the method uses a short-time Fourier transform (STFT) to generate a spectrogram. The STFT can be expressed as:

[0130]

[0131] Wherein, x[n] is the input signal, w[n] is the window function, m is the time step, and w is the angular frequency. In a preferred embodiment of the present invention, a Hamming window is used as the window function, the window size is 512 sampling points, and the overlap rate is 50%.

[0132] Next, this method uses multi-layer spectrogram convolution to decompose the time series spectrogram from low-dimensional to high-dimensional spectrum sequence to form an audio feature map. This process can be expressed by the following formula:

[0133]

[0134] Among them, A is the adjacency matrix, D is the degree matrix, and H (l) is the node feature of the lth layer, W (l) is a learnable weight matrix, and σ is an activation function. In the present invention, GELU is preferably used as the activation function because it can provide a smoother gradient while maintaining nonlinearity.

[0135] Subsequently, the method of the present invention processes the target embedding vector and the audio feature map converted by spectrogram through a spectral sensor. The spectral sensor is a specially designed attention mechanism that can effectively capture the association between different modalities. Its core formula is as follows:

[0136]

[0137] where Q, K, and V are query, key, and value matrices, respectively, and d k is the dimension of the key. In the present invention, Q comes from the target embedding vector, while K and V come from the audio feature map.

[0138] Finally, this method combines the generated audio attention sequence with the target embedding vector to generate an audio-video collaborative attention sequence. This process can be expressed as:

[0139] C=Concat(E,A)W

[0140] Where E is the target embedding vector, A is the audio attention sequence, and W is the learnable weight matrix.

[0141] In this way, the method of the present invention can effectively fuse visual and auditory information to generate expressive audio-video collaborative attention sequences. This multimodal fusion method performs well in practical applications, for example, in tasks such as video content analysis and abnormal event detection, it can achieve better performance than single-modal methods.

[0142] In another embodiment of the present invention, the event detection process of the audio-visual video clip is described in detail. This process is a key link in the panoramic video content analysis and directly affects the accuracy and reliability of the final analysis result.

[0143] First, this method inputs the audio-video collaborative attention sequence into an event classifier based on audio-video attention and audio-visual joint learning. The design of this classifier incorporates the latest deep learning technology and can effectively process multimodal data. Specifically, the classifier adopts a hierarchical structure to process audio and video information separately, and then fuse them at the high level.

[0144] When processing audio information, the method of the present invention uses an improved temporal convolutional network (TCN). Compared with traditional recurrent neural networks, TCN can better capture long-term dependencies and is more efficient to train. The core operation of TCN can be expressed as:

[0145]

[0146] Among them, f1,...,f k are k parallel one-dimensional convolution kernels, d is the dilation factor, and * represents the convolution operation. In a preferred embodiment of the present invention, a multi-layer TCN is used, and the dilation factor of each layer increases exponentially to capture temporal patterns of different scales.

[0147] For the processing of video information, this method forms video attention by convolutional neural network. Here, a 3D convolutional network is used, which can capture information in both spatial and temporal dimensions. The operation of 3D convolution can be expressed as:

[0148]

[0149] Among them, u is the input tensor, w is the convolution kernel, and v is the output feature map.

[0150] Next, the method of the present invention obtains audio and video feature mapping through convolution, ReLU activation function, pooling, and fully connected network. This process can be expressed as:

[0151] h = FC(Pool(ReLU(Conv(x))))

[0152] Where x is the input, Conv, ReLU, Pool, and FC represent convolution, ReLU activation, pooling, and fully connected operations, respectively.

[0153] Subsequently, this method concatenates the audio and video feature maps to generate a joint audio-visual feature map. This process can be expressed as:

[0154] f = Concat(h audio ,h video )

[0155] Among them, h audio and h video are the feature maps for audio and video respectively.

[0156] Finally, the method of the present invention generates a result sequence of audio-visual event detection through a softmax layer. The softmax function is defined as follows:

[0157]

[0158] Among them, z i is the raw score of the ith category.

[0159] In this way, the method of the present invention can effectively detect various events in panoramic videos, such as character interactions, object movements, environmental changes, etc. This event detection method based on multimodal fusion performs well in practical applications, for example, in the fields of intelligent security, autonomous driving, video content analysis, etc., and can provide more accurate and comprehensive event recognition results. In another key embodiment of the present invention, the establishment process of the long-series spatiotemporal graph convolutional network is carefully designed to effectively capture the complex spatiotemporal dependencies in panoramic videos. The innovation of this network structure is that it can simultaneously process long time series and rich spatial information, providing strong support for a comprehensive understanding of video content.

[0160] First, this method treats each video frame as a node in the graph and uses a graph convolutional network to model the relationship between frames. The advantage of this method is that it can flexibly process video sequences of different lengths while maintaining the spatial association between frames. Specifically, for a given video sequence, a graph G = (V, E) can be constructed, where V is a set of nodes representing video frames, and E is a set of edges representing the connection relationship between frames. Preferably, the present invention adopts a dynamic graph construction method, that is, the weight of the edge is dynamically determined according to the visual similarity between frames. This can be achieved by the following formula:

[0161]

[0162] Among them, A ij is an element in the adjacency matrix, f(x i ,x j ) is a function that measures the similarity between frame i and frame j. is the neighborhood set of frame i.

[0163] In the graph convolution layer, the method of the present invention innovatively adds temporal information. This is achieved by only pooling the convolution layer before the current time step and then multiplying it with the features of the previous time step. This design can effectively capture the causal relationship in the video and avoid information leakage. The specific update rule can be expressed as:

[0164]

[0165] in, is the feature of node i at layer l, W (l) and b (l) are the weight matrix and bias vector of the lth layer respectively, and σ is the activation function. In a preferred embodiment of the present invention, LeakyReLU is used as the activation function to introduce nonlinearity while avoiding the gradient vanishing problem.

[0166] In order to enhance the nonlinear expression ability of the model, this method uses multi-layer graph convolution and ReLU activation function. This deep structure can abstract higher-level feature representation layer by layer. In practice, the present invention finds that using 3 to 5 layers of graph convolution can usually achieve good results, and the specific number of layers can be adjusted according to the complexity of the task.

[0167] Finally, the method of the present invention converts the features of each node into a classification probability vector through a softmax layer to obtain the final classification result. The softmax function is defined as follows:

[0168]

[0169] Among them, W k and b k are the weight vector and bias of the kth class respectively.

[0170] This method based on long-term spatiotemporal graph convolutional networks has performed well in practical applications. For example, in long video understanding tasks, it can effectively capture long-term dependencies in videos and identify complex event sequences. In anomaly detection scenarios, this method can accurately locate spatiotemporal anomalies and provide reliable early warnings for security systems.

[0171] Furthermore, the present invention also includes a knowledge graph-based scene understanding step, which is an innovative design that aims to combine visual content with semantic knowledge to achieve a deeper level of video understanding.

[0172] First, the method constructs a knowledge graph for objects and events in the video. This process involves multiple subtasks such as entity recognition, relationship extraction, and knowledge representation. In a preferred embodiment of the present invention, a dynamic knowledge graph construction method is adopted. Specifically, for each frame in the video, the target detection algorithm is first used to identify the key objects, and then these objects are mapped to the entities in the knowledge graph based on the predefined ontology model. At the same time, the relationship between entities is inferred by analyzing the spatial relationship and temporal changes between objects. This process can be expressed by the following formula:

[0173] G=(E,R,T)

[0174] Among them, G is the constructed knowledge graph, E is the entity set, R is the relationship set, T is the triple set, and each triple (h, r, t) indicates that the head entity h is connected to the tail entity t through the relationship r.

[0175] Next, the method of the present invention uses a graph neural network to reason about the knowledge graph and achieve deep semantic understanding of the video content. Here, a graph neural network based on an attention mechanism is used, which can effectively capture the complex interactions between entities. Its core operation can be expressed as:

[0176]

[0177] in, is the representation of node i at layer l, α ij is the attention weight, W (l) is a learnable parameter matrix. Attention weight α ij The calculation method is as follows:

[0178]

[0179] Among them, a is a learnable attention vector and || represents the concatenation operation.

[0180] The method of the present invention innovatively combines event prediction with entity recognition, improving the accuracy of scene understanding. Specifically, when predicting events, not only visual features are considered, but also semantic information in the knowledge graph is incorporated. This multimodal fusion method can be expressed by the following formula:

[0181] P(e|V,G)=softmax(W e [f V (V)||f G (G)]+b e )

[0182] Among them, e is the predicted event, V is the video feature, G is the knowledge graph feature, and f V and f G are the feature extraction functions of video and graph, W e and b e are learnable parameters.

[0183] Finally, through knowledge graph reasoning and event prediction, this method obtains the labels of entities in the video and the description of events. This scene understanding method based on knowledge graph performs well in practical applications, especially in scenarios that require deep semantic understanding. For example, in intelligent monitoring systems, it can not only identify abnormal behaviors, but also provide semantic explanations of the behaviors, greatly improving the interpretability and usability of the system.

[0184] Finally, the present invention also includes an innovative multi-task learning step, which is designed to analyze panoramic video content more comprehensively and efficiently. The core idea of ​​multi-task learning is to improve the generalization ability and learning efficiency of the model by sharing representations and learning multiple related tasks in parallel.

[0185] First, this method designs a lightweight multi-task shared convolutional layer to extract common features of multiple tasks. The design of this shared layer is based on an important observation: different video analysis tasks often share underlying visual features. By sharing these features, model parameters can be significantly reduced and computational efficiency can be improved. In a preferred embodiment of the present invention, a progressive feature sharing strategy is adopted. Specifically, the shallow network is fully shared, while the deep network is partially shared based on the similarity of the tasks. This can be expressed by the following formula:

[0186]

[0187] in, is the feature of the i-th task at layer l, is the shared weight, are task-specific weights and x are input features.

[0188] Next, the method of the present invention designs a multi-task learning mechanism based on the dependency relationship between tasks and optimizes the model structure. Here, a graph-based task relationship modeling method is adopted. Specifically, each task is regarded as a node in the graph, and the dependency relationship between tasks is represented as an edge. Then, a graph attention network is used to learn the interaction between tasks. Its core operation can be expressed as:

[0189]

[0190] Among them, z i is the final representation of task i, h j is the feature of the adjacent task, α ij is the attention weight and W is the learnable parameter matrix.

[0191] This method innovatively integrates multiple tasks such as object detection, semantic segmentation, object tracking, and event classification into a single framework. This multi-task framework not only improves the efficiency of the model, but also improves the overall performance through mutual promotion between tasks. For example, object detection and semantic segmentation can provide useful spatial information to each other, while object tracking can provide temporal context for event classification.

[0192] Finally, the method of the present invention jointly trains multiple modules through an end-to-end joint optimization method, so that each module can adapt and optimize each other. A multi-task learning algorithm based on gradient balancing is adopted here. Specifically, for each task i, its loss function can be expressed as:

[0193]

[0194] middle, is the task-specific loss, g i and g j are the gradients of tasks i and j respectively, cos represents the cosine similarity, and λ is the balancing parameter.

[0195] This multi-task learning method performs well in practical applications. For example, in intelligent transportation systems, it can simultaneously perform vehicle detection, lane segmentation, vehicle tracking, and traffic event classification, providing comprehensive road condition analysis. In video surveillance systems, it can achieve multiple functions such as personnel detection, behavior recognition, and abnormal event warning, greatly improving the practicality and efficiency of the system.

[0196] In general, the multimodal fusion panoramic video intelligent content analysis method proposed in this paper realizes comprehensive, in-depth and efficient analysis of panoramic video content by innovatively combining long-term spatiotemporal graph convolutional networks, scene understanding based on knowledge graphs and multi-task learning. This method has demonstrated excellent performance in multiple practical application scenarios, providing new research directions and solutions for the field of video content understanding and analysis.

[0197] The above description is only a preferred specific implementation manner of the present invention; however, the protection scope of the present invention is not limited thereto; any person familiar with the art who, within the scope disclosed by the present invention, makes equivalent replacements or changes based on the scheme and improved concepts of the present invention shall be covered within the protection scope of the present invention.

Claims

1. A multi-modal fusion panoramic video intelligent content analysis method, characterized in that: include: The acquisition steps include: Get panoramic video frames and their synchronized audio; Performing spherical projection transformation on the panoramic image data to convert the equidistant projection image into a spherical image; Processing steps include: Based on the spherical image, generating a plurality of planar images; Based on the multiple planar images, performing candidate target position detection; Based on the candidate target position detection result, an image target detection model is used to implement instance segmentation of the target, and target features are extracted to generate a target embedding description vector sequence; Based on the synchronized audio, extracting timing information using an audio recognition processing module to form an acoustic attention sequence; A spectral sensor is used to realize the association fusion of the target embedding description vector sequence and the acoustic attention sequence to generate an audio-video collaborative attention sequence; Based on the audio-visual collaborative attention sequence, establishing an audio-visual event recognition model; Output steps include: Using the audio-visual event recognition model, generating a spatiotemporal trajectory of the audio-visual event; Establishing a long-term spatiotemporal graph convolutional network to detect the causal relationship in the spatiotemporal trajectory of the audio-visual event; The spatiotemporal event trajectory is semantically parsed to obtain event content described by the panoramic content.

2. The multi-modal fusion panoramic video intelligent content analysis method according to claim 1 is characterized in that: The candidate target position detection specifically includes: Performing 2D convolution feature extraction using the planar image; Performing 2D object detection processing on the plane image using a region-based object detection model to obtain object bounding box coordinates and corresponding scores; Converting the planar projection image into a spherical image, and mapping the coordinates of each pixel point to a spherical coordinate system according to an isometric projection; According to the spherical coordinates, the spherical image is converted into a plurality of plane images by sampling, thereby realizing the detection of the spherical target.

3. The multi-modal fusion panoramic video intelligent content analysis method according to claim 1 is characterized in that: The target instance segmentation specifically includes: Through the convolutional network, an image feature map is formed; Using multiple feature fusion methods to perform detection on the feature map to extract features of the regional target; The non-maximum suppression method is used to remove redundant frames to obtain the final detection result; For the detected target, using an instance segmentation model, generating a pixel-level segmentation result of the instance based on semantic segmentation and the image feature map; The segmentation results within the target bounding box are used to generate the target's bounding box based on the coordinates of the bounding box, completing the fusion of image and spatial semantic information.

4. The multi-modal fusion panoramic video intelligent content analysis method according to claim 1 is characterized in that: The generation of the target embedding description vector specifically includes: Generate feature concatenation combination using target semantic information and feature information; Mapping the feature concatenation combination into a semantic embedding vector through a multi-layer perceptron; Construct a target description vector with target semantic information.

5. The multi-modal fusion panoramic video intelligent content analysis method according to claim 1 is characterized in that: The generation of the audio-video collaborative attention sequence specifically includes: Performing spectrogram conversion on the synchronized audio to convert the audio into a time-series spectrogram; After multi-layer spectrogram convolution decomposition, the time series spectrum graph is transformed from low-dimensional to high-dimensional spectrum sequence to form an audio feature map; Processing the target embedding vector and the spectrogram-converted audio feature map through a spectral sensor; The generated audio attention sequence is combined with the target embedding vector to generate an audio-video collaborative attention sequence.

6. The multi-modal fusion panoramic video intelligent content analysis method according to claim 1 is characterized in that: The event detection of the audio-visual video clip specifically includes: Input the audio-video collaborative attention sequence into an event classifier based on audio-video attention and audio-visual joint learning; The audio and video attention are formed by convolutional neural networks respectively; The audio and video feature maps are obtained through convolution, ReLU activation function, pooling, and fully connected network respectively; splicing the audio and video feature maps to generate an audio-visual joint feature map; The result sequence of audio-visual event detection is generated through the softmax layer.

7. The multi-modal fusion panoramic video intelligent content analysis method according to claim 1 is characterized in that: Also includes the spherical convolutional network step: Design a two-stream parallel spherical convolution structure, one branch captures spatial features and the other captures semantic features; A spherical reshaping and rearrangement (SRL) module is introduced in the spatial feature branch to enhance spatial features. In the semantic feature branch, 1×1 convolution and global pooling operations are used to highlight key semantic information; The features of the two branches are combined to obtain the final spherical convolution feature map.

8. The multi-modal fusion panoramic video intelligent content analysis method according to claim 1, characterized in that: The establishment of the long-term spatiotemporal graph convolutional network specifically includes: Treat each video frame as a node in the graph and use graph convolutional networks to model the relationship between frames; Adding temporal information to the graph convolutional layer is done by pooling only the convolutional layer before the current time step and then multiplying it with the features of the previous time step; Use multi-layer graph convolution and ReLU activation function to enhance the nonlinear expression ability of the model; The features of each node are converted into a classification probability vector through the softmax layer to obtain the final classification result.

9. The multi-modal fusion panoramic video intelligent content analysis method according to claim 1, characterized in that: It also includes scene understanding steps based on knowledge graphs: Build knowledge graphs for objects and events in videos, combining visual content with semantic knowledge; Use graph neural networks to reason about knowledge graphs and achieve deep semantic understanding of video content; Combine event prediction with entity recognition to improve the accuracy of scene understanding; Through knowledge graph reasoning and event prediction, we can obtain the labels of entities and descriptions of events in the video.

10. The multi-modal fusion panoramic video intelligent content analysis method according to claim 1, characterized in that: It also includes multi-task learning steps: Design a lightweight multi-task shared convolutional layer to extract common features of multiple tasks; Design a multi-task learning mechanism based on the dependencies between tasks and optimize the model structure; Integrate multiple tasks such as object detection, semantic segmentation, object tracking and event classification into one framework; Through an end-to-end joint optimization method, multiple modules are jointly trained to enable the modules to adapt to and optimize each other.

Images