Video analysis method and related equipment based on deep learning platform

Abstract

The present invention provides a video analysis method and related equipment based on a deep learning platform, wherein the method comprises: preprocessing and enhancing the input raw video data to obtain a standardized video clip sequence and an enhanced audio spectrum diagram. Then, a fused feature vector containing a spatiotemporal action sequence is obtained by using multimodal deep feature extraction technology. Next, hierarchical semantic understanding and content analysis are performed to generate a temporal semantic graph. Based on the graph, marketing strategy analysis and effect prediction are performed to obtain a marketing strategy evaluation report. Finally, personalized content matching and recommendation are performed in combination with dynamic user portraits. This method achieves a deep understanding of video content through refined action segmentation and multimodal analysis, provides strong support for the formulation of accurate video marketing strategies, and effectively improves the accuracy and effect of video marketing.

Description

Technical Field

[0001] The present invention relates to the field of video analysis, and in particular to a video analysis method based on a deep learning platform and related equipment. Background Art

[0002] With the booming development of short video platforms, video marketing has become an important means of corporate promotion. Existing video marketing analysis methods mainly rely on the overall understanding of video content, such as video theme recognition, sentiment analysis, etc. However, these methods often ignore the specific action sequences in the video and cannot accurately capture the key links in the product usage process or service process.

[0003] For example, in a product demonstration video, the user's action sequence is crucial to understanding the product's functions and usage experience. Traditional methods may only be able to identify that the general theme of the video is "product demonstration", but it is difficult to distinguish and analyze the details of different operation steps. This coarse-grained analysis limits the accuracy and personalization of marketing strategies, resulting in the inefficient use of marketing resources and affecting the overall effectiveness of video marketing. Summary of the invention

[0004] The main purpose of the present invention is to solve the technical problem that video analysis in the existing marketing field cannot accurately capture the key links in the product use process or service process, resulting in the inefficient use of marketing resources and affecting the overall effect of video marketing.

[0005] A first aspect of the present invention provides a video analysis method based on a deep learning platform, and the video analysis method based on the deep learning platform includes:

[0006] Preprocess and enhance the raw video data input into the deep learning platform to obtain standardized video clip sequences and enhanced audio spectrograms;

[0007] According to the video clip sequence and the audio spectrogram, multimodal deep feature extraction processing is performed on the video content of the original video data to obtain a multimodal fusion feature vector containing a spatiotemporal action sequence;

[0008] Based on the multimodal fusion feature vector, hierarchical semantic understanding and content analysis processing are performed on the video content to obtain a temporal semantic graph representing the video structure and semantic information;

[0009] According to the temporal semantic graph, the marketing strategy analysis and effect prediction processing are performed on the video content to obtain a marketing strategy evaluation report;

[0010] Based on the strategy evaluation report and the preset dynamic user portrait, the video content is personalized matched and recommended to obtain a personalized content recommendation list for a specific user group.

[0011] Optionally, in a first implementation of the first aspect of the present invention, the preprocessing and enhancement processing of the original video data input into the deep learning platform to obtain a standardized video clip sequence and an enhanced audio spectrogram includes:

[0012] Performing multi-resolution decomposition processing on the original video data input into the deep learning platform to obtain pyramid-structured video data containing features of different scales, and performing adaptive frame rate adjustment processing on the original video data based on the pyramid-structured video data to obtain a multi-level video sequence with a dynamic frame rate;

[0013] Performing spatiotemporal consistency optimization processing on the multi-level video sequence of the dynamic frame rate to obtain a spatiotemporally coherent standardized video clip sequence;

[0014] Performing blind source separation processing on the audio part of the original video data to obtain multiple independent audio channels, and performing adaptive equalization and reverberation elimination processing on the multiple independent audio channels to obtain multi-channel audio with enhanced clarity;

[0015] The multi-channel audio with enhanced clarity is processed by multi-scale wavelet transform to obtain a multi-resolution audio spectrogram, and the frequency spectrum features in the multi-resolution audio spectrogram are processed by nonlinear enhancement and noise suppression to obtain an enhanced audio spectrogram.

[0016] Optionally, in a second implementation of the first aspect of the present invention, performing multimodal deep feature extraction processing on the video content of the original video data according to the video clip sequence and the audio spectrogram to obtain a multimodal fusion feature vector containing a spatiotemporal action sequence includes:

[0017] Processing the video clip sequence with a three-dimensional convolutional neural network to obtain a spatiotemporal feature representation of the video modality, and processing the audio spectrogram with a deep audio feature extraction network to obtain a spectral feature representation of the audio modality;

[0018] According to the video clip sequence, a human body posture estimation network is used to extract a human body key point sequence, and the human body key point sequence is processed by a spatiotemporal graph convolution network to obtain a motion feature representation of a skeletal modality;

[0019] Performing cross-modal attention fusion processing on the spatiotemporal feature representation of the video modality, the spectral feature representation of the audio modality, and the motion feature representation of the skeletal modality to obtain preliminary fused multi-modal features;

[0020] According to the preliminarily fused multimodal features, a multi-scale temporal convolutional network is constructed to perform multi-granularity temporal modeling on the feature sequence to obtain a temporal feature representation with long-term and short-term dependencies;

[0021] Processing the temporal feature representation with long-term and short-term dependencies with a spatiotemporal attention mechanism to obtain attention enhancement features that highlight key spatiotemporal regions;

[0022] The attention enhancement feature is subjected to residual connection processing with the initially fused multimodal feature to obtain a multimodal fusion feature vector containing a spatiotemporal action sequence.

[0023] Optionally, in a third implementation of the first aspect of the present invention, extracting a human key point sequence using a human posture estimation network according to the video clip sequence, and performing spatiotemporal graph convolutional network processing on the human key point sequence to obtain a skeletal modality action feature representation includes:

[0024] Performing multi-scale human body detection processing on the video clip sequence to obtain candidate regions containing human body bounding box information;

[0025] The video frame is cropped according to the candidate area to obtain a human-centered image sequence;

[0026] Performing multi-stage posture estimation network processing on the human-centered image sequence to obtain an initial human key point coordinate sequence;

[0027] Constructing a spatiotemporal skeleton graph structure according to the initial human body key point coordinate sequence, and performing graph convolution network processing on the spatiotemporal skeleton graph structure to obtain local joint features;

[0028] Performing hierarchical spatiotemporal aggregation processing on the local joint features to obtain a global human posture representation;

[0029] Adaptively fusing the global human posture representation with the local joint features to obtain multi-scale skeleton features;

[0030] The multi-scale skeleton features are subjected to temporal modeling processing to obtain motion feature representation of the skeleton modality.

[0031] Optionally, in a fourth implementation of the first aspect of the present invention, the performing of a spatiotemporal attention mechanism on the temporal feature representation having long-term and short-term dependencies to obtain an attention enhancement feature that highlights a key spatiotemporal region includes:

[0032] Performing time series decomposition processing on the time series feature representation to obtain feature subsequences of multiple time scales, and performing self-attention calculation processing on each of the feature subsequences to obtain an attention weight matrix of the time dimension;

[0033] According to the attention weight matrix of the time dimension, weighted aggregation processing is performed on the feature subsequence to obtain a feature representation enhanced by time attention;

[0034] Performing spatial dimension decoupling processing on the temporal attention enhanced feature representation to obtain feature maps of multiple spatial channels;

[0035] Performing non-local correlation calculation processing on the feature map of each of the spatial channels to obtain an attention response map of the spatial dimension;

[0036] According to the attention response map of the spatial dimension, the feature map of the spatial channel is selectively enhanced to obtain a feature representation of spatial attention enhancement;

[0037] The feature representation of spatial attention enhancement and the feature representation of temporal attention enhancement are adaptively fused to obtain attention enhancement features that highlight key spatiotemporal areas.

[0038] Optionally, in a fifth implementation of the first aspect of the present invention, performing hierarchical semantic understanding and content analysis processing on the video content based on the multimodal fusion feature vector to obtain a temporal semantic graph representing video structure and semantic information includes:

[0039] Performing time-series segmentation processing on the multimodal fusion feature vector to obtain multiple sub-segment feature sequences, and performing hierarchical semantic coding processing on each of the sub-segment feature sequences to obtain feature representations at different semantic levels;

[0040] According to the feature representations of the different semantic levels, a multi-level semantic relationship graph is constructed, and the multi-level semantic relationship graph is processed by a graph neural network to obtain node features with enhanced semantic associations;

[0041] Performing temporal dynamic modeling processing on the node features with enhanced semantic association to obtain a semantic representation sequence with temporal context;

[0042] According to the semantic representation sequence, a hierarchical temporal semantic tree structure is constructed, and the hierarchical temporal semantic tree structure is processed by a tree convolutional network to obtain multi-scale temporal semantic features;

[0043] Performing cross-scale feature fusion processing on the multi-scale temporal semantic features and the initially fused multi-modal features to obtain a comprehensive feature representation that fuses global and local information;

[0044] The comprehensive feature representation is subjected to semantic segmentation and boundary detection processing to obtain a temporal semantic graph representing the video structure and semantic information.

[0045] Optionally, in a sixth implementation of the first aspect of the present invention, the performing of marketing strategy analysis and effect prediction processing on the video content according to the temporal semantic graph to obtain a marketing strategy evaluation report includes:

[0046] Performing user attention analysis on the temporal semantic graph to obtain an audience attraction score, and identifying the best delivery point for the video content based on the audience attraction score to obtain optimized time point recommendations for advertisement insertion or product display;

[0047] Performing sentiment trend analysis on the temporal semantic graph to obtain a sentiment change curve related to the product or service experience;

[0048] According to the emotion change curve, brand impression association analysis is performed on the video content to obtain brand perception enhancement suggestions for the original video data;

[0049] The optimization time point suggestions and brand perception enhancement suggestions are integrated to obtain a marketing strategy evaluation report.

[0050] A second aspect of the present invention provides a video analysis device based on a deep learning platform, the video analysis device based on the deep learning platform comprising:

[0051] The data processing module is used to pre-process and enhance the raw video data input into the deep learning platform to obtain a standardized video clip sequence and an enhanced audio spectrogram;

[0052] A feature extraction module, configured to perform multimodal deep feature extraction processing on the video content of the original video data according to the video clip sequence and the audio spectrogram, so as to obtain a multimodal fusion feature vector containing a spatiotemporal action sequence;

[0053] A feature analysis module, used to perform hierarchical semantic understanding and content analysis processing on the video content based on the multimodal fusion feature vector, and obtain a temporal semantic graph representing the video structure and semantic information;

[0054] A marketing evaluation module, used to perform marketing strategy analysis and effect prediction processing on the video content according to the temporal semantic graph, and obtain a marketing strategy evaluation report;

[0055] The analysis and recommendation module is used to perform personalized matching and recommendation processing on the video content based on the strategy evaluation report and the preset dynamic user portrait, and obtain a personalized content recommendation list for a specific user group.

[0056] The third aspect of the present invention provides a video analysis device based on a deep learning platform, comprising: a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected through lines; the at least one processor calls the instructions in the memory so that the video analysis device based on the deep learning platform performs the steps of the above-mentioned video analysis method based on the deep learning platform.

[0057] A fourth aspect of the present invention provides a computer-readable storage medium, which stores instructions that, when executed on a computer, enable the computer to execute the steps of the above-mentioned video analysis method based on a deep learning platform.

[0058] The above-mentioned video analysis method and related equipment based on the deep learning platform obtain a standardized video clip sequence and an enhanced audio spectrum diagram by preprocessing and enhancing the input raw video data. Then, the multimodal deep feature extraction technology is used to obtain a fused feature vector containing a spatiotemporal action sequence. Next, hierarchical semantic understanding and content analysis are performed to generate a temporal semantic graph. Based on the graph, marketing strategy analysis and effect prediction are performed to obtain a marketing strategy evaluation report. Finally, personalized content matching and recommendation are performed in combination with dynamic user portraits. This method achieves a deep understanding of video content through refined action segmentation and multimodal analysis, provides strong support for the formulation of accurate video marketing strategies, and effectively improves the accuracy and effectiveness of video marketing.

[0059] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0060] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a schematic diagram of a first embodiment of a video analysis method based on a deep learning platform in an embodiment of the present invention;

[0062] Figure 2 A schematic diagram of an embodiment of a video analysis device based on a deep learning platform in an embodiment of the present invention;

[0063] Figure 3 Schematic diagram of an embodiment of a video analysis device based on a deep learning platform in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] The terms "including" and "having" and any variations thereof mentioned in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device end including a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or device ends.

[0066] To facilitate understanding of this embodiment, a video analysis method based on a deep learning platform disclosed in an embodiment of the present invention is first described in detail. Figure 1 As shown, the method comprises the following steps:

[0067] 101. Preprocess and enhance the original video data input into the deep learning platform to obtain a standardized video clip sequence and an enhanced audio spectrogram;

[0068] In one embodiment of the present invention, the preprocessing and enhancement processing of the original video data input to the deep learning platform to obtain a standardized video clip sequence and an enhanced audio spectrum diagram includes: performing multi-resolution decomposition processing on the original video data input to the deep learning platform to obtain pyramid structure video data containing features of different scales, and according to the pyramid structure video data, performing adaptive frame rate adjustment processing on the original video data to obtain a multi-level video sequence with a dynamic frame rate; performing spatiotemporal consistency optimization processing on the multi-level video sequence with a dynamic frame rate to obtain a spatiotemporally coherent standardized video clip sequence; performing blind source separation processing on the audio part of the original video data to obtain multiple independent audio channels, and performing adaptive equalization and reverberation elimination processing on the multiple independent audio channels to obtain multi-channel audio with enhanced clarity; performing multi-scale wavelet transform processing on the multi-channel audio with enhanced clarity to obtain a multi-resolution audio spectrum diagram, and performing nonlinear enhancement and noise suppression processing on the spectral features in the multi-resolution audio spectrum diagram to obtain an enhanced audio spectrum diagram.

[0069] Specifically, first, the original video data is subjected to multi-resolution decomposition processing. This step uses the Gaussian pyramid algorithm to downsample the original video layer by layer. The specific operation is to apply a Gaussian filter to the original video frame, and then perform 2x2 area average downsampling, repeat this process multiple times, generate video sequences of different resolutions, and form pyramid structure video data. Then, adaptive frame rate adjustment is performed based on the pyramid structure video data. This step calculates the structural similarity index (SSIM) between adjacent frames. When the SSIM value is lower than a preset threshold, the frame is retained, otherwise it is discarded, thereby obtaining a multi-level video sequence with a dynamic frame rate. Subsequently, the multi-level video sequence with a dynamic frame rate is optimized for spatiotemporal consistency. This step uses an optical flow estimation algorithm, such as the Lucas-Kanade method, to calculate the motion vector between adjacent frames. Then, the video frames are interpolated and corrected using the calculated optical flow field to ensure that the video content at different levels and different frame rates remains coherent in time and space. In terms of audio processing, the audio part of the original video data is first subjected to blind source separation. This step uses the independent component analysis (ICA) algorithm to separate the mixed audio signal into multiple independent audio channels. In the specific implementation, the input signal is first pre-whitened, and then the separation matrix is ​​estimated through an iterative optimization algorithm. Next, these independent audio channels are processed by adaptive equalization and reverberation elimination. Adaptive equalization analyzes the spectrum of each channel through fast Fourier transform (FFT), and then uses a feedback control algorithm to dynamically adjust the gain coefficients of each frequency band. Reverberation elimination uses spectral subtraction to estimate the room impulse response and subtract the reverberation component in the frequency domain. Finally, the processed multi-channel audio is subjected to multi-scale wavelet transform. This step uses discrete wavelet transform (DWT) to select appropriate wavelet basis functions (such as Daubechies wavelet) to decompose the audio signal into subbands of different frequencies and time scales. Then, the obtained wavelet coefficients are thresholded to remove coefficients less than a certain threshold to achieve noise suppression. Finally, the signal is reconstructed by inverse discrete wavelet transform to obtain an enhanced audio spectrum.

[0070] 102. Perform multimodal deep feature extraction processing on the video content of the original video data according to the video clip sequence and the audio spectrogram to obtain a multimodal fusion feature vector containing a spatiotemporal action sequence;

[0071] In one embodiment of the present invention, the multimodal deep feature extraction processing is performed on the video content of the original video data according to the video clip sequence and the audio spectrum graph to obtain a multimodal fusion feature vector containing a spatiotemporal action sequence, including: performing a three-dimensional convolutional neural network processing on the video clip sequence to obtain a spatiotemporal feature representation of the video modality, and performing a deep audio feature extraction network processing on the audio spectrum graph to obtain a spectral feature representation of the audio modality; according to the video clip sequence, a human body posture estimation network is used to extract a human body key point sequence, and a spatiotemporal graph convolution network is performed on the human body key point sequence to obtain an action feature representation of the skeletal modality; The spatiotemporal feature representation of the video modality, the spectral feature representation of the audio modality and the action feature representation of the skeletal modality are subjected to cross-modal attention fusion processing to obtain preliminary fused multimodal features; based on the preliminary fused multimodal features, a multi-scale temporal convolutional network is constructed, and multi-granularity temporal modeling is performed on the feature sequence to obtain a temporal feature representation with long-term and short-term dependencies; the temporal feature representation with long-term and short-term dependencies is subjected to spatiotemporal attention mechanism processing to obtain attention enhancement features that highlight key spatiotemporal areas; the attention enhancement features are subjected to residual connection processing with the preliminary fused multimodal features to obtain a multimodal fusion feature vector containing a spatiotemporal action sequence.

[0072] Specifically, first, the video clip sequence is processed by a three-dimensional convolutional neural network. In the specific implementation, a pre-trained 3D-CNN model such as I3D or C3D is used to segment the video clip sequence into fixed time windows (such as 16 frames) and input the network for forward propagation. The network contains multiple layers of 3D convolution, pooling and fully connected layers, and extracts features in both time and space dimensions through convolution operations. The output of the last fully connected layer is used as the spatiotemporal feature representation of the video modality. At the same time, the audio spectrogram is processed by a deep audio feature extraction network. Here, a deep convolutional neural network structure similar to VGGish is used, and the audio spectrogram is input as a two-dimensional image. The network contains multiple convolutional layers, pooling layers and fully connected layers, and the spectral features of the audio are extracted layer by layer. The output of the last fully connected layer is used as the spectral feature representation of the audio modality. Next, based on the video clip sequence, a human posture estimation network is used to extract a sequence of human key points. This step uses pre-trained models such as OpenPose or HRNet to detect human key points for each frame of the image. The network outputs the coordinates of each detected human key point (such as 18 key points) to form a time series. Then, the sequence of human key points is processed by the spatiotemporal graph convolutional network. In the specific implementation, a spatiotemporal graph structure is constructed, in which nodes represent key points and edges represent the spatial and temporal connections between key points. The ST-GCN (Spatial Temporal Graph Convolutional Networks) model is used to perform convolution operations on this graph structure to extract the action feature representation of the skeletal modality. After obtaining the feature representation of the three modalities, cross-modal attention fusion processing is performed. This step implements a multi-head attention mechanism, and each modality is input into the attention calculation unit as a query, key, and value. Specifically, the features of the three modalities are first mapped to the same dimensional space, and then the attention weight matrix between different modalities is calculated. Finally, the features are weighted summed based on these weights to obtain the preliminary fused multimodal features. Based on the preliminary fused multimodal features, a multi-scale temporal convolutional network is constructed. This network consists of multiple parallel one-dimensional convolutional layers, each of which uses different convolution kernel sizes and dilation rates to capture features at different time scales. In specific implementation, three parallel one-dimensional convolutional layers can be used, with convolution kernel sizes of 3, 5, and 7 and dilation rates of 1, 2, and 4, respectively. The outputs of these parallel layers are combined through the concat operation and then fused through a 1x1 convolutional layer to obtain a temporal feature representation with long-term and short-term dependencies. The temporal feature representation with long-term and short-term dependencies is processed by the spatiotemporal attention mechanism.This step implements a dual attention mechanism: first, a self-attention mechanism is applied in the time dimension to calculate the correlation between different time steps; then a channel attention mechanism is applied in the feature dimension, such as the Squeeze-and-Excitation module in SENet, to weight the importance of different feature channels. The outputs of these two attention mechanisms are combined by element-wise multiplication to obtain attention-enhanced features that highlight key spatiotemporal regions. Finally, the attention-enhanced features are residually connected with the preliminarily fused multimodal features. In specific implementation, the attention-enhanced features are dimensionally adjusted through a fully connected layer, and then element-wise added with the preliminarily fused multimodal features. This residual connection structure facilitates lossless transmission of information. Finally, after a normalization layer and a nonlinear activation function (such as ReLU), the final multimodal fusion feature vector containing the spatiotemporal action sequence is obtained.

[0073] Furthermore, according to the video clip sequence, a human body posture estimation network is used to extract a human body key point sequence, and the human body key point sequence is processed by a spatiotemporal graph convolutional network to obtain a skeletal modal motion feature representation, which includes: performing multi-scale human body detection processing on the video clip sequence to obtain a candidate area containing human body bounding box information; cropping the video frames according to the candidate area to obtain a human body-centered image sequence; performing multi-stage posture estimation network processing on the human body-centered image sequence to obtain an initial human body key point coordinate sequence; constructing a spatiotemporal skeleton graph structure according to the initial human body key point coordinate sequence, and performing graph convolutional network processing on the spatiotemporal skeleton graph structure to obtain local joint features; performing hierarchical spatiotemporal aggregation processing on the local joint features to obtain a global human body posture representation; adaptively fusing the global human body posture representation with the local joint features to obtain multi-scale skeleton features; performing temporal modeling processing on the multi-scale skeleton features to obtain a skeletal modal motion feature representation.

[0074] Specifically, first, multi-scale human detection is performed on the video clip sequence. This step uses target detection algorithms such as FasterR-CNN or YOLOv3 to process each frame of the image. In specific implementation, the input image is scaled to multiple different scales (such as 0.5 times, 1 times, and 1.5 times the original size), and then passed through the detection network respectively. The detection network contains a feature extraction backbone (such as ResNet50), a region proposal network (RPN), and a detection head. The network outputs the bounding box coordinates and confidence scores of each detected human body, and the final human candidate region is obtained by non-maximum suppression (NMS). Next, the video frame is cropped according to the candidate region. This step uses an affine transformation to crop and scale the detected human bounding box region to a fixed size (such as 256x256 pixels). In order to make the human body located in the center of the image, the center point of the bounding box is calculated and aligned to the center of the output image. In this way, a series of human-centered image sequences are obtained. Then, the human-centered image sequence is processed by a multi-stage posture estimation network. Here, a multi-stage pose estimation model such as Cascaded Pyramid Network (CPN) or HRNet is used. In the specific implementation, the image features are first extracted through a feature extraction network (such as ResNet), and then passed through multiple cascaded pose estimation modules. Each module contains multiple convolutional layers and upsampling layers to gradually refine the prediction of key point positions. The last module outputs a heat map, which represents the possible position of each key point. By finding the maximum value point of the heat map, the initial human key point coordinate sequence is obtained. The spatiotemporal skeleton graph structure is constructed based on the initial human key point coordinate sequence. This step first defines the connection relationship of the human skeleton (such as the 17 key points and 16 bone connections defined by the COCO dataset), and then uses the key point coordinates of each frame as the nodes of the graph, the bone connections as the spatial edges, and the connections between the same key points in adjacent frames as the temporal edges. In this way, a three-dimensional spatiotemporal graph structure is constructed to perform graph convolutional network processing on the spatiotemporal skeleton graph structure. Here, the ST-GCN (Spatial Temporal Graph Convolutional Networks) model is used. In the specific implementation, the graph convolution operation is defined, including spatial convolution and temporal convolution. Spatial convolution aggregates features of connected key points in each frame, and temporal convolution aggregates features of the same key point in the temporal dimension. Local joint features are obtained by stacking multiple layers of ST-GCN blocks. Next, the local joint features are hierarchically aggregated in time and space. This step implements a bottom-up feature aggregation process. First, the hierarchical structure of the human body (such as limbs, torso, head) is defined, and then pooling operations or attention mechanisms are used to aggregate joint features belonging to the same part. This process can be iterated to eventually obtain global features that represent the posture of the entire human body.Adaptively fuse the global human posture representation with the local joint features. This step implements a feature fusion module, which can use a gating mechanism or an attention mechanism. Specifically, use global features as queries, local features as keys and values, calculate attention weights, and then perform weighted summation on the local features. This process can be seen as weighting the importance of local features according to global information to obtain multi-scale skeletal features. Finally, perform temporal modeling on the multi-scale skeletal features. This step can use a long short-term memory network (LSTM) or a temporal convolutional network (TCN). Taking TCN as an example, multiple layers of one-dimensional convolution are used, and each layer uses a different dilation rate to capture dependencies at different time scales. In specific implementation, a 3-layer TCN can be used, with dilation rates of 1, 2, and 4, respectively, and each layer contains convolution, batch normalization, and ReLU activation functions. The output of the last layer is the action feature representation of the skeletal modality.

[0075] Furthermore, the temporal feature representation with long-term and short-term dependencies is processed by a spatiotemporal attention mechanism to obtain attention enhancement features that highlight key spatiotemporal areas, including: performing temporal decomposition processing on the temporal feature representation to obtain feature subsequences of multiple time scales, and performing self-attention calculation processing on each of the feature subsequences to obtain an attention weight matrix of the time dimension; performing weighted aggregation processing on the feature subsequences according to the attention weight matrix of the time dimension to obtain a feature representation enhanced by time attention; performing spatial dimension decoupling processing on the feature representation enhanced by time attention to obtain feature maps of multiple spatial channels; performing non-local correlation calculation processing on the feature map of each of the spatial channels to obtain an attention response map of the spatial dimension; performing selective enhancement processing on the feature map of the spatial channel according to the attention response map of the spatial dimension to obtain a feature representation enhanced by spatial attention; adaptively fusing the feature representation enhanced by spatial attention with the feature representation enhanced by temporal attention to obtain attention enhancement features that highlight key spatiotemporal areas.

[0076] Specifically, first, the time series feature representation is decomposed. This step uses wavelet transform or multi-scale decomposition technology to decompose the input time series feature representation into feature subsequences of multiple time scales. In specific implementation, discrete wavelet transform (DWT) can be used to select appropriate wavelet basis functions (such as Haar wavelet or Daubechies wavelet) to perform multi-layer decomposition on the input sequence. Each layer of decomposition obtains an approximate coefficient sequence and a detail coefficient sequence, which together constitute feature subsequences of multiple time scales. Next, self-attention calculation is performed on each feature subsequence. This step implements the self-attention mechanism in Transformer. Specifically, for each feature subsequence, query, key, and value matrices are first generated through linear transformation. Then, the dot product of Query and Key is calculated and divided by a scaling factor (usually the square root of the Key dimension), and then the attention weight is obtained through the softmax function. Finally, these weights are multiplied by Value to obtain the weighted feature representation. This process can be expressed as: Attention(Q, K, V) = softmax(QK^T / sqrt(d_k))V, where d_k is the dimension of Key. In this way, a corresponding time dimension attention weight matrix is ​​obtained for each feature subsequence of each time scale. Then, the feature subsequences are weighted and aggregated according to the attention weight matrix of the time dimension. This step applies the attention weights to the original features to achieve the weighted importance of the features. In specific implementation, the attention weight matrix of each time scale is matrix multiplied with the corresponding feature subsequence, and then all weighted feature subsequences are added to obtain the feature representation enhanced by time attention. Next, the feature representation enhanced by time attention is decoupled in spatial dimension. This step aims to separate the spatial information in the feature representation. In specific implementation, the feature representation can be reshaped into a three-dimensional tensor, where the first dimension represents the time step, and the second and third dimensions represent the spatial dimensions (such as height and width). Then, the features of each time step can be transformed using 1x1 convolution or fully connected layers to obtain feature maps of multiple spatial channels. The feature map of each spatial channel is processed by non-local correlation calculation. This step implements the core operation in non-local neural networks. Specifically, for each spatial position, calculate its correlation with all other positions. This can be achieved by the following steps: first, use 1x1 convolution to generate query, key and value feature maps; then, perform matrix multiplication of the query feature map with the key feature map to obtain the correlation matrix; then, perform softmax normalization on the correlation matrix; finally, multiply the normalized correlation matrix with the value feature map to obtain the attention response map of the spatial dimension.According to the attention response map of the spatial dimension, the feature map of the spatial channel is selectively enhanced. This step applies spatial attention to the original features. In specific implementation, the attention response map can be element-wise multiplied with the original feature map, and then added to the original feature map (similar to residual connection) to obtain the feature representation enhanced by spatial attention. Finally, the feature representation enhanced by spatial attention is adaptively fused with the feature representation enhanced by temporal attention. This step aims to integrate the attention information of both time and space dimensions. In specific implementation, a gating mechanism or an attention mechanism can be used. For example, a small neural network (such as a multi-layer perceptron) can be used to generate fusion weights, and then the temporal attention features and spatial attention features are weighted summed according to these weights. Alternatively, the two feature representations can be concatenated together and then reduced in dimension through a fully connected layer to obtain the final fused features.

[0077] 103. Based on the multimodal fusion feature vector, the video content is hierarchically understood and analyzed to obtain a temporal semantic graph representing the video structure and semantic information;

[0078] In one embodiment of the present invention, the hierarchical semantic understanding and content analysis processing of the video content based on the multimodal fusion feature vector to obtain a temporal semantic graph representing the video structure and semantic information includes: performing temporal segmentation processing on the multimodal fusion feature vector to obtain multiple sub-segment feature sequences, and performing hierarchical semantic coding processing on each of the sub-segment feature sequences to obtain feature representations of different semantic levels; constructing a multi-level semantic relationship graph based on the feature representations of different semantic levels, and performing graph neural network processing on the multi-level semantic relationship graph to obtain a semantic association enhanced temporal semantic graph. Node features; performing temporal dynamic modeling processing on the node features with enhanced semantic associations to obtain a semantic representation sequence with a temporal context; constructing a hierarchical temporal semantic tree structure based on the semantic representation sequence, and performing tree convolutional network processing on the hierarchical temporal semantic tree structure to obtain multi-scale temporal semantic features; performing cross-scale feature fusion processing on the multi-scale temporal semantic features and the initially fused multimodal features to obtain a comprehensive feature representation that integrates global and local information; performing semantic segmentation and boundary detection processing on the comprehensive feature representation to obtain a temporal semantic map that represents the video structure and semantic information.

[0079] Specifically, first, the multimodal fusion feature vector is segmented in time series. This step uses the sliding window technology to segment the long sequence feature vector into multiple overlapping sub-segments. In the specific implementation, a fixed window size (such as 64 frames) and step size (such as 16 frames) are set, and the window slides along the time dimension to extract a sub-segment feature sequence each time. In this way, a series of local feature representations can be obtained while retaining certain context information.

[0080] Next, hierarchical semantic encoding is performed on the feature sequence of each sub-segment. This step uses a hierarchical encoder structure, such as a hierarchical LSTM or Transformer encoder. Taking hierarchical LSTM as an example, first use an LSTM layer to process the original feature sequence to obtain a low-level semantic representation; then, the low-level semantic representation is used as input and sent to another LSTM layer to obtain a mid-level semantic representation; finally, it passes through another LSTM layer to obtain a high-level semantic representation. In this way, each sub-segment obtains feature representations at different semantic levels. Then, a multi-level semantic relationship graph is constructed based on the feature representations at different semantic levels. This step regards the feature representation of each semantic level as a node in the graph and establishes edge connections based on feature similarity. In specific implementation, cosine similarity or dot product similarity can be used to calculate the similarity between nodes, and a threshold is set. When the similarity exceeds the threshold, an edge connection is established between nodes. In this way, a semantic relationship graph containing multiple levels is obtained. The constructed multi-level semantic relationship graph is processed by a graph neural network. This step uses a graph attention network (GAT) or a graph convolutional network (GCN) to process graph structure data. Taking GAT as an example, first, the attention weights of each node with its neighboring nodes are calculated, and then the features of the neighboring nodes are weighted and aggregated according to these weights. This process can be iterated multiple times, and each iteration will update the feature representation of the node. In this way, each node can obtain the information of other related nodes, thereby obtaining node features with enhanced semantic associations. Next, the node features with enhanced semantic associations are processed by temporal dynamic modeling. This step can use a bidirectional LSTM or Transformer encoder to capture long-term dependencies. In specific implementation, the enhanced node features are arranged in chronological order and input into the bidirectional LSTM. The forward propagation of LSTM captures the information flow from the past to the present, and the back propagation captures the information flow from the future to the present. Finally, the hidden states in both directions are spliced ​​to obtain a sequence of semantic representations with temporal context. Then, a hierarchical temporal semantic tree structure is constructed based on the semantic representation sequence. This step uses a bottom-up clustering method, such as a hierarchical clustering algorithm. First, the semantic representation of each time step is used as a leaf node. Then, the most similar node pairs are gradually merged to form a new parent node. This process is repeated until the predetermined number of layers is reached or only one root node is left. In this way, a semantic tree representing the hierarchical structure of the video content is obtained. The constructed hierarchical temporal semantic tree structure is processed by a tree convolutional network. This step uses a tree convolutional neural network (Tree-CNN) to process tree structure data. The convolution operation of Tree-CNN is performed on the local structure of the tree, and can consider the characteristics of the nodes and the structural information of the tree at the same time. In the specific implementation, starting from the leaf node, the convolution operation is performed layer by layer, and the convolution kernel of each layer acts on the current node and its child nodes.Through multi-layer tree convolution, we finally get the multi-scale temporal semantic features representing the entire video. Then, we perform cross-scale feature fusion processing on the multi-scale temporal semantic features and the preliminarily fused multimodal features. This step can use the idea of ​​Feature Pyramid Network (FPN). In specific implementation, we first adjust the multi-scale temporal semantic features to the same time scale as the preliminarily fused multimodal features through upsampling or downsampling. Then, we use 1x1 convolution to adjust the number of feature channels so that the dimensions of the two features are consistent. Finally, we fuse the two features through element-level addition or concatenation to obtain a comprehensive feature representation that integrates global and local information. Finally, we perform semantic segmentation and boundary detection on the comprehensive feature representation. This step can use the fully convolutional network (FCN) or U-Net structure. In specific implementation, we first obtain the feature map with the same number of input video frames through multi-layer convolution and upsampling operations. Then, we use 1x1 convolution to map the feature map to the predefined number of semantic categories to obtain the semantic segmentation result of each frame. At the same time, we use another branch network to detect the action boundary, which can be regarded as a binary classification problem (boundary / non-boundary). Finally, the semantic segmentation results and boundary detection results are combined to obtain a temporal semantic graph that represents the video structure and semantic information.

[0081] 104. According to the time-series semantic graph, the marketing strategy analysis and effect prediction processing of the video content are carried out to obtain a marketing strategy evaluation report;

[0082] In one embodiment of the present invention, the marketing strategy analysis and effect prediction processing of the video content according to the temporal semantic graph to obtain a marketing strategy evaluation report includes: performing user attention analysis processing on the temporal semantic graph to obtain an audience attraction score, and based on the audience attraction score, performing optimal delivery point identification processing on the video content to obtain optimized time point suggestions for advertisement insertion or product display; performing sentiment trend analysis processing on the temporal semantic graph to obtain a sentiment change curve related to the product or service experience; performing brand impression association analysis processing on the video content according to the sentiment change curve to obtain brand perception enhancement suggestions for the original video data; integrating the optimized time point suggestions and brand perception enhancement suggestions to obtain a marketing strategy evaluation report.

[0083] Specifically, first, the user attention analysis is performed on the temporal semantic graph. This step adopts a method that combines visual saliency model and action recognition technology. In specific implementation, a pre-trained visual saliency network (such as SalNet) is used to process each frame of the video to obtain a saliency map for each frame. At the same time, the action information in the temporal semantic graph is used to identify the key action segments in the video. Then, the saliency information is combined with the action information to calculate the attention score of each time period by weighted average. This score reflects the audience's possible attention to the video content, that is, the audience attractiveness score. Next, according to the audience attractiveness score, the best delivery point for the video content is identified. This step uses peak detection algorithm and dynamic programming method. Specifically, the attractiveness score sequence is first smoothed using the sliding window method to reduce the impact of noise. Then, the peak detection algorithm (such as finding the local maximum) is applied to find the peak points of the attractiveness score, which are regarded as potential advertising insertion points. Next, considering the actual constraints of advertising delivery (such as the minimum interval time, the maximum number of advertisements, etc.), the dynamic programming algorithm is used to select the optimal subset among these potential points as the optimized time point recommendation for advertising insertion or product display. Then, the sentiment trend analysis is performed on the temporal semantic graph. This step uses natural language processing and multimodal sentiment analysis technology. In specific implementation, text information (such as dialogue content, subtitles) and visual features (such as character expressions, scene atmosphere) are first extracted from the temporal semantic graph. Then, the text information is processed using a pre-trained text sentiment analysis model (such as a BERT-based model) to obtain the sentiment polarity and intensity of the text. At the same time, the visual sentiment recognition model (such as a CNN-based expression recognition model) is used to process the visual features to obtain visual sentiment information. Finally, the text and visual sentiment information are fused, and the sentiment change curve that changes over time is obtained through temporal smoothing. This curve reflects the sentiment change trend related to the product or service experience. According to the sentiment change curve, the video content is analyzed for brand impression association. This step uses time series analysis and association rule mining technology. In specific implementation, the sentiment change curve is first aligned with the content information in the temporal semantic graph to identify the key content elements that cause sentiment changes (such as specific product features, service processes, etc.). Then, the sliding window method is used to analyze the correlation between sentiment changes and content elements at different time scales. Apply association rule mining algorithms (such as the Apriori algorithm) to discover strong association rules between positive changes in sentiment and specific content elements. Based on these rules, generate brand perception enhancement suggestions, such as emphasizing certain product features associated with positive sentiment, or improving certain service links that cause negative sentiment. Finally, integrate the optimization time point suggestions and brand perception enhancement suggestions. This step uses a multi-objective optimization algorithm and a decision tree model.In the specific implementation, the optimization time point suggestions and brand perception enhancement suggestions are first represented as a series of feature vectors. Then, multiple optimization objectives are defined, such as maximizing advertising effectiveness, minimizing user experience interference, maximizing brand positive impression, etc. A multi-objective optimization algorithm (such as NSGA-II) is used to generate a series of non-dominated solutions, each of which represents a possible marketing strategy combination. Next, a decision tree model is used to classify and interpret these solutions to generate easy-to-understand decision rules. Finally, these decision rules, corresponding data analysis results, and visualization charts are integrated into a structured marketing strategy evaluation report.

[0084] 105. Based on the strategy evaluation report and the preset dynamic user portrait, the video content is personalized matched and recommended to obtain a personalized content recommendation list for a specific user group.

[0085] In one embodiment of the present invention, first, a pre-trained feature extractor is used to process the strategy evaluation report, extract key information (such as the best delivery time point, brand perception enhancement suggestions, etc.), and convert it into a fixed-dimensional feature vector. At the same time, the feature vector of the target user is obtained from the preset dynamic user portrait database, including the user's basic information, interest preferences, historical behavior, etc. Then, a pre-defined feature fusion module, such as feature splicing or weighted summation, is used to fuse the video content features, strategy evaluation features and user features into a unified representation. Next, a pre-trained deep matching model, such as a deep factorization machine (DeepFM) or a neural collaborative filtering (NCF) model, is called. This model has been trained offline on large-scale historical data and can perform online reasoning quickly. The fused feature vector is input into the model to obtain the user's interest prediction score for each candidate video content. Then, based on the predicted interest score, the candidate video content is sorted. The top-N videos are selected as a preliminary recommendation list. In order to increase the diversity of recommendations, a pre-defined diversity algorithm, such as the maximum marginal relevance (MMR) algorithm, is used to re-sort the preliminary list. This process seeks a balance between relevance and diversity to avoid overly simplistic recommendations. Finally, a preset personalized rule engine is applied to fine-tune the recommendation list based on the user's specific attributes (such as age group, occupation, geographic location, etc.) and the current context (such as time, device type, etc.). For example, the difficulty level of video content can be adjusted for users of different age groups, or the clarity and length of the video can be adjusted based on the user's device type. This rule engine can quickly apply some hard rules based on business logic to further enhance the personalization of recommendations.

[0086] In this embodiment, by preprocessing and enhancing the input raw video data, a standardized video clip sequence and an enhanced audio spectrum are obtained. Then, a fused feature vector containing a spatiotemporal action sequence is obtained by using multimodal deep feature extraction technology. Next, hierarchical semantic understanding and content analysis are performed to generate a temporal semantic graph. Based on the graph, marketing strategy analysis and effect prediction are performed to obtain a marketing strategy evaluation report. Finally, personalized content matching and recommendation are performed in combination with dynamic user portraits. This method achieves a deep understanding of video content through refined action segmentation and multimodal analysis, provides strong support for the formulation of accurate video marketing strategies, and effectively improves the accuracy and effectiveness of video marketing.

[0087] The above describes the video analysis method based on the deep learning platform in the embodiment of the present invention. The following describes the video analysis device based on the deep learning platform in the embodiment of the present invention. Figure 2 , an embodiment of a video analysis device based on a deep learning platform in an embodiment of the present invention includes:

[0088] The data processing module 201 is used to pre-process and enhance the original video data input into the deep learning platform to obtain a standardized video clip sequence and an enhanced audio spectrogram;

[0089] A feature extraction module 202 is used to perform multimodal deep feature extraction processing on the video content of the original video data according to the video clip sequence and the audio spectrogram to obtain a multimodal fusion feature vector containing a spatiotemporal action sequence;

[0090] A feature analysis module 203 is used to perform hierarchical semantic understanding and content analysis processing on the video content based on the multimodal fusion feature vector to obtain a temporal semantic graph representing the video structure and semantic information;

[0091] The marketing evaluation module 204 is used to perform marketing strategy analysis and effect prediction processing on the video content according to the temporal semantic graph to obtain a marketing strategy evaluation report;

[0092] The analysis and recommendation module 205 is used to perform personalized matching and recommendation processing on the video content based on the strategy evaluation report and the preset dynamic user portrait, and obtain a personalized content recommendation list for a specific user group.

[0093] In an embodiment of the present invention, the video analysis device based on the deep learning platform runs the above-mentioned video analysis method based on the deep learning platform, and the video analysis device based on the deep learning platform obtains a standardized video clip sequence and an enhanced audio spectrum diagram by preprocessing and enhancing the input raw video data. Then, a fusion feature vector containing a spatiotemporal action sequence is obtained by using multimodal deep feature extraction technology. Next, hierarchical semantic understanding and content analysis are performed to generate a temporal semantic graph. Based on the graph, marketing strategy analysis and effect prediction are performed to obtain a marketing strategy evaluation report. Finally, personalized content matching and recommendation are performed in combination with dynamic user portraits. This method achieves a deep understanding of video content through refined action segmentation and multimodal analysis, provides strong support for the formulation of accurate video marketing strategies, and effectively improves the accuracy and effect of video marketing.

[0094] above Figure 2 The video analysis device based on the deep learning platform in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The video analysis device based on the deep learning platform in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0095] Figure 3 3 is a schematic diagram of the structure of a video analysis device based on a deep learning platform provided by an embodiment of the present invention. The video analysis device 300 based on the deep learning platform may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 310 (for example, one or more processors) and a memory 320, and one or more storage media 330 (for example, one or more mass storage device terminals) storing application programs 333 or data 332. Among them, the memory 320 and the storage medium 330 can be short-term storage or persistent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the video analysis device 300 based on the deep learning platform. Furthermore, the processor 310 may be configured to communicate with the storage medium 330, and execute a series of instruction operations in the storage medium 330 on the video analysis device 300 based on the deep learning platform to implement the steps of the above-mentioned video analysis method based on the deep learning platform.

[0096] The video analysis device 300 based on the deep learning platform may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input and output interfaces 360, and / or one or more operating systems 331, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It can be understood by those skilled in the art that Figure 3 The structure of the video analysis device based on the deep learning platform shown does not constitute a limitation on the video analysis device based on the deep learning platform provided by the present invention, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0097] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the video analysis method based on the deep learning platform.

[0098] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0099] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0100] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video analysis method based on a deep learning platform, It is characterized in that The video analysis method based on the deep learning platform includes: The original video data input to the deep learning platform is preprocessed and enhanced to obtain a standardized video clip sequence and an enhanced audio spectrum diagram; specifically, the method includes: performing multi-resolution decomposition processing on the original video data input to the deep learning platform to obtain pyramid structure video data containing features of different scales, and performing adaptive frame rate adjustment processing on the original video data according to the pyramid structure video data to obtain a multi-level video sequence with a dynamic frame rate; performing spatiotemporal consistency optimization processing on the multi-level video sequence with a dynamic frame rate to obtain a spatiotemporally coherent standardized video clip sequence; performing blind source separation processing on the audio part of the original video data to obtain multiple independent audio channels, and performing adaptive equalization and reverberation elimination processing on the multiple independent audio channels to obtain multi-channel audio with enhanced clarity; performing multi-scale wavelet transform processing on the multi-channel audio with enhanced clarity to obtain a multi-resolution audio spectrum diagram, and performing nonlinear enhancement and noise suppression processing on the spectral features in the multi-resolution audio spectrum diagram to obtain an enhanced audio spectrum diagram; According to the video clip sequence and the audio spectrum graph, multimodal deep feature extraction processing is performed on the video content of the original video data to obtain a multimodal fusion feature vector containing a spatiotemporal action sequence; specifically including: performing three-dimensional convolutional neural network processing on the video clip sequence to obtain a spatiotemporal feature representation of the video modality, and performing deep audio feature extraction network processing on the audio spectrum graph to obtain a spectral feature representation of the audio modality; performing multi-scale human body detection processing on the video clip sequence to obtain a candidate area containing human body bounding box information; cropping the video frames according to the candidate areas to obtain a human-centered image sequence; performing multi-stage posture estimation network processing on the human-centered image sequence to obtain an initial human body key point coordinate sequence; constructing a spatiotemporal skeleton graph structure according to the initial human body key point coordinate sequence, and performing graph convolution network processing on the spatiotemporal skeleton graph structure to obtain local joint features; Perform hierarchical spatiotemporal aggregation processing to obtain a global human posture representation; perform adaptive fusion processing on the global human posture representation and the local joint features to obtain multi-scale skeleton features; perform temporal modeling processing on the multi-scale skeleton features to obtain the motion feature representation of the skeleton modality; perform cross-modal attention fusion processing on the spatiotemporal feature representation of the video modality, the spectral feature representation of the audio modality and the motion feature representation of the skeleton modality to obtain a preliminarily fused multi-modal feature; construct a multi-scale temporal convolutional network based on the preliminarily fused multi-modal feature, perform multi-granularity temporal modeling on the feature sequence, and obtain a temporal feature representation with long-term and short-term dependencies; perform spatiotemporal attention mechanism processing on the temporal feature representation with long-term and short-term dependencies to obtain an attention enhancement feature that highlights the key spatiotemporal area; perform residual connection processing on the attention enhancement feature and the preliminarily fused multi-modal feature to obtain a multi-modal fusion feature vector containing a spatiotemporal action sequence; Based on the multimodal fusion feature vector, hierarchical semantic understanding and content analysis processing are performed on the video content to obtain a temporal semantic graph representing the video structure and semantic information; According to the temporal semantic graph, the marketing strategy analysis and effect prediction processing are performed on the video content to obtain a marketing strategy evaluation report; Based on the strategy evaluation report and the preset dynamic user portrait, the video content is personalized matched and recommended to obtain a personalized content recommendation list for a specific user group.

2. The video analysis method based on the deep learning platform according to claim 1, It is characterized in that The processing of the temporal feature representation with long-term and short-term dependencies by a spatiotemporal attention mechanism to obtain attention enhancement features that highlight key spatiotemporal regions includes: Performing time series decomposition processing on the time series feature representation to obtain feature subsequences of multiple time scales, and performing self-attention calculation processing on each of the feature subsequences to obtain an attention weight matrix of the time dimension; According to the attention weight matrix of the time dimension, weighted aggregation processing is performed on the feature subsequence to obtain a feature representation enhanced by time attention; Performing spatial dimension decoupling processing on the temporal attention enhanced feature representation to obtain feature maps of multiple spatial channels; Performing non-local correlation calculation processing on the feature map of each of the spatial channels to obtain an attention response map of the spatial dimension; According to the attention response map of the spatial dimension, the feature map of the spatial channel is selectively enhanced to obtain a feature representation of spatial attention enhancement; The feature representation of spatial attention enhancement and the feature representation of temporal attention enhancement are adaptively fused to obtain attention enhancement features that highlight key spatiotemporal areas.

3. The video analysis method based on the deep learning platform according to claim 1, It is characterized in that The step of performing hierarchical semantic understanding and content analysis on the video content based on the multimodal fusion feature vector to obtain a temporal semantic graph representing the video structure and semantic information includes: Performing time-series segmentation processing on the multimodal fusion feature vector to obtain multiple sub-segment feature sequences, and performing hierarchical semantic coding processing on each of the sub-segment feature sequences to obtain feature representations at different semantic levels; According to the feature representations of the different semantic levels, a multi-level semantic relationship graph is constructed, and the multi-level semantic relationship graph is processed by a graph neural network to obtain node features with enhanced semantic associations; Performing temporal dynamic modeling processing on the node features with enhanced semantic association to obtain a semantic representation sequence with temporal context; According to the semantic representation sequence, a hierarchical temporal semantic tree structure is constructed, and the hierarchical temporal semantic tree structure is processed by a tree convolutional network to obtain multi-scale temporal semantic features; Performing cross-scale feature fusion processing on the multi-scale temporal semantic features and the initially fused multi-modal features to obtain a comprehensive feature representation that fuses global and local information; The comprehensive feature representation is subjected to semantic segmentation and boundary detection processing to obtain a temporal semantic graph representing the video structure and semantic information.

4. The video analysis method based on the deep learning platform according to claim 3, It is characterized in that The marketing strategy analysis and effect prediction processing of the video content is performed according to the temporal semantic graph to obtain a marketing strategy evaluation report including: Performing user attention analysis on the temporal semantic graph to obtain an audience attraction score, and identifying the best delivery point for the video content based on the audience attraction score to obtain optimized time point recommendations for advertisement insertion or product display; Performing sentiment trend analysis on the temporal semantic graph to obtain a sentiment change curve related to the product or service experience; According to the emotion change curve, brand impression association analysis is performed on the video content to obtain brand perception enhancement suggestions for the original video data; The optimization time point suggestions and brand perception enhancement suggestions are integrated to obtain a marketing strategy evaluation report.

5. A video analysis device based on a deep learning platform, It is characterized in that The video analysis device based on the deep learning platform includes: The data processing module is used to pre-process and enhance the original video data input to the deep learning platform to obtain a standardized video clip sequence and an enhanced audio spectrum diagram; specifically comprising: performing multi-resolution decomposition processing on the original video data input to the deep learning platform to obtain pyramid structure video data containing features of different scales, and according to the pyramid structure video data, performing adaptive frame rate adjustment processing on the original video data to obtain a multi-level video sequence with a dynamic frame rate; performing spatiotemporal consistency optimization processing on the multi-level video sequence with a dynamic frame rate to obtain a spatiotemporally coherent standardized video clip sequence; performing blind source separation processing on the audio part of the original video data to obtain multiple independent audio channels, and performing adaptive equalization and reverberation elimination processing on the multiple independent audio channels to obtain multi-channel audio with enhanced clarity; performing multi-scale wavelet transform processing on the multi-channel audio with enhanced clarity to obtain a multi-resolution audio spectrum diagram, and performing nonlinear enhancement and noise suppression processing on the spectral features in the multi-resolution audio spectrum diagram to obtain an enhanced audio spectrum diagram; The feature extraction module is used to perform multimodal deep feature extraction processing on the video content of the original video data according to the video clip sequence and the audio spectrum graph to obtain a multimodal fusion feature vector containing a spatiotemporal action sequence; specifically comprising: performing three-dimensional convolutional neural network processing on the video clip sequence to obtain a spatiotemporal feature representation of the video modality, and performing deep audio feature extraction network processing on the audio spectrum graph to obtain a spectral feature representation of the audio modality; performing multi-scale human body detection processing on the video clip sequence to obtain a candidate area containing human body bounding box information; cropping the video frames according to the candidate areas to obtain a human-centered image sequence; performing multi-stage posture estimation network processing on the human-centered image sequence to obtain an initial human body key point coordinate sequence; constructing a spatiotemporal skeleton graph structure according to the initial human body key point coordinate sequence, and performing graph convolution network processing on the spatiotemporal skeleton graph structure to obtain local joint features; The hierarchical spatiotemporal aggregation processing of the local joint features is performed to obtain the global human posture representation; the global human posture representation is adaptively fused with the local joint features to obtain the multi-scale skeleton features; the multi-scale skeleton features are subjected to time series modeling processing to obtain the motion feature representation of the skeleton modality; the spatiotemporal feature representation of the video modality, the spectral feature representation of the audio modality and the motion feature representation of the skeleton modality are subjected to cross-modal attention fusion processing to obtain the preliminarily fused multi-modal features; based on the preliminarily fused multi-modal features, a multi-scale temporal convolutional network is constructed to perform multi-granularity time series modeling on the feature sequence to obtain the temporal feature representation with long-term and short-term dependencies; the temporal feature representation with long-term and short-term dependencies is subjected to the spatiotemporal attention mechanism processing to obtain the attention enhancement feature that highlights the key spatiotemporal areas; the attention enhancement feature is subjected to residual connection processing with the preliminarily fused multi-modal features to obtain the multi-modal fusion feature vector containing the spatiotemporal action sequence; A feature analysis module, used to perform hierarchical semantic understanding and content analysis processing on the video content based on the multimodal fusion feature vector, and obtain a temporal semantic graph representing the video structure and semantic information; A marketing evaluation module, used to perform marketing strategy analysis and effect prediction processing on the video content according to the temporal semantic graph, and obtain a marketing strategy evaluation report; The analysis and recommendation module is used to perform personalized matching and recommendation processing on the video content based on the strategy evaluation report and the preset dynamic user portrait, and obtain a personalized content recommendation list for a specific user group.

6. A video analysis device based on a deep learning platform, It is characterized in that The video analysis device based on the deep learning platform includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor calls the instructions in the memory so that the video analysis device based on the deep learning platform performs the steps of the video analysis method based on the deep learning platform as described in any one of claims 1-4.

7. A computer-readable storage medium having instructions stored thereon, It is characterized in that When the instructions are executed by the processor, the steps of the video analysis method based on the deep learning platform as described in any one of claims 1 to 4 are implemented.

Images