Video context understanding and recognition method based on multi-modal signal fusion

Abstract

The application discloses a video context understanding and recognition method based on multi-modal signal fusion and concretely relates to the technical field of artificial intelligence, and is used for solving the problems of poor content understanding and poor context matching in existing programmed advertisement video analysis. The method comprises the following steps: extracting a key frame sequence from a video segment, generating a dynamic convolution kernel group based on a current key frame, a reference key frame, a frame difference graph and a normalized cross-correlation graph, combining multi-scale local information entropy feature extraction, time sequence consistency constraint, scene semantic prior injection and text record fusion recognition, so as to improve the video content deep context understanding ability, reduce the content context mismatch in the programmed advertisement scene, improve the video content recognition accuracy and semantic consistency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a video context understanding and recognition method based on multimodal signal fusion. Background Technology

[0002] This invention relates to technologies related to video content parsing and contextual understanding. With the increasing application of video content in information distribution, content recommendation, and programmatic advertising, existing technologies typically require first performing object detection, scene classification, or semantic segmentation on video frames, and then extracting object categories, scene types, and some semantic information from the video to support subsequent content matching, tag generation, and ad placement decisions. Common approaches in existing implementations include extracting inter-frame features based on fixed convolutional structures, describing the motion relationships between adjacent frames based on optical flow information, enhancing spatial representation capabilities based on multi-scale convolutional or pooling structures, and supplementing video content with subtitles, transcribed text, or descriptive text. These technologies can perform basic object recognition and scene parsing in general video understanding tasks and constitute the main technological foundation of current video content processing.

[0003] However, in application scenarios such as programmatic advertising, which have high requirements for the context of video content, existing technologies still largely remain at the level of extracting single-frame targets, local scenes, or shallow labels, and are insufficient in representing the dynamic relationships, behavioral processes, semantic evolution of scenes, and contextual situations between targets in the video. On the one hand, some methods are insufficient in modeling the temporal correlation between consecutive keyframes, are easily affected by static background interference, and are difficult to stably reflect the process of video content changing over time; on the other hand, their ability to represent the spatial structure of complex texture regions, geometrically deformed regions, and boundary regions is limited, easily leading to inaccurate local recognition and inconsistent results between consecutive frames. At the same time, although existing technologies also introduce textual information to assist in understanding video content, most of them still use visual results and textual results side by side, lacking a unified analysis around the overall video context, resulting in output results that are more biased towards listing target labels, rather than reflecting the scene semantics and expressive context corresponding to the video content.

[0004] Therefore, existing technologies still suffer from a significant challenge: the difficulty in continuously, stably, and consistently analyzing target objects, behavioral processes, and scene semantics within video content, and in further developing a structured video content description that can be directly used for subsequent content matching. Under this problem, video parsing results are prone to temporal inconsistencies, discrepancies between local recognition and the overall scene, and weak integration of textual and visual information, thus affecting the accurate understanding of the video content context by subsequent systems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a video context understanding and recognition method based on multimodal signal fusion, in order to solve the problems mentioned in the background section.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a video context understanding and recognition method based on multimodal signal fusion, comprising: S1. Acquire video clips, extract keyframe sequences, and establish pixel-level semantic labels, scene category labels, and text records; S2. Based on the current keyframe, reference keyframe, frame difference map, and normalized cross-correlation map, generate a dynamic convolutional kernel group, perform temporal context encoding on the current keyframe, and obtain a temporal context feature map. S3. Based on the temporal context feature map, calculate the multi-scale local information entropy map, generate a deformable sampling offset field and modulation factor field, and extract the local entropy weighted feature map. S4. Based on the temporal context feature map, local entropy weighted feature map, historical segmentation confidence information, and scene category vector, execute temporal consistency constraints and semantic prior injection to generate a semantic segmentation probability map. S5. Based on the semantic segmentation probability map, local information entropy map, and optical flow alignment features, construct edge-aware weighted loss and spatiotemporal consistency loss, and train the keyframe semantic segmentation model. S6. Call the trained keyframe semantic segmentation model to generate keyframe segmentation results for the video to be processed, combine the text records to perform multimodal fusion recognition, and output a structured video content description.

[0007] Furthermore, S1 includes: Obtain complete video sequences and videos of specific scenes; Extract video clips and keyframe sequences; Pixel-level semantic labels, scene category labels, and text records are created for keyframe sequences; Establish optical flow fields for adjacent keyframes; The training dataset is divided based on the video sequences.

[0008] Furthermore, S2 includes: Construct a frame difference map based on the current keyframe and the reference keyframe; Calculate the normalized cross-correlation graph based on the current keyframe and the reference keyframe; Based on the channel stitching results of the current keyframes, frame difference maps, and normalized cross-correlation maps, dynamic convolutional kernel groups are generated using a kernel generator network.

[0009] Furthermore, S2 also includes: A spatial attention mask corresponding to the dynamic convolutional kernel group is generated based on the frame difference map and the normalized cross-correlation map. Perform convolution operations on the current keyframe using a dynamic convolution kernel group; The convolutional response is modulated element-wise and summed based on the spatial attention mask to obtain the temporal context feature map.

[0010] Furthermore, S3 includes: Calculate the local information entropy map at each scale based on the temporal context feature map; The offset field and modulation factor field at each scale are generated based on the temporal context feature map and the local information entropy map; Based on the offset field, modulation factor field, and local information entropy map, deformable local entropy weighted convolution is performed on the temporal context feature map to obtain local entropy weighted feature maps at each scale.

[0011] Furthermore, S4 includes: Based on the temporal context feature map, the local entropy weighted feature map, and the historical segmentation confidence information, cross-scale fusion and temporal aggregation are performed to obtain multi-scale spatiotemporal feature maps. Generate scene category vectors based on keyframe sequences; Semantic prior injection is performed on the multi-scale spatiotemporal feature map based on the scene category vector to obtain the semantic prior feature map.

[0012] Furthermore, S4 also includes: Edge enhancement terms are generated based on the local information entropy map; Semantic segmentation prediction is performed based on the semantic prior feature map and edge enhancement terms to obtain a semantic segmentation probability map; Among them, the edge enhancement term is obtained by calculating the gradient magnitude and exponential weighting of the local information entropy map.

[0013] Furthermore, S5 includes: A composite loss function is constructed based on semantic segmentation probability maps, local information entropy maps, and optical flow alignment features. Backpropagation and parameter updates are performed on the keyframe semantic segmentation model based on the composite loss function; The model parameters are determined based on the evaluation results of the validation set. The composite loss function is composed of edge-aware weighted cross-entropy loss and multi-scale spatiotemporal consistency loss.

[0014] Furthermore, S6 includes: Call the keyframe semantic segmentation model to generate keyframe semantic segmentation results; A list of target object categories is generated on the visual side based on the semantic segmentation results of keyframes; A set of semantic keywords on the text side is formed based on the text records; A fusion recognition process is performed based on the visual object category list and the text-based semantic keyword set. A structured video content metadata file is generated based on the fusion recognition results.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. By extracting keyframe sequences from video clips and generating dynamic convolutional kernel groups based on the current keyframe, reference keyframe, frame difference map, and normalized cross-correlation map, and then combining multi-scale local information entropy feature extraction, temporal consistency constraints, scene semantic prior injection, and text record fusion recognition, the system aims to improve the deep contextual understanding of video content, reduce content context mismatch in programmatic advertising scenarios, and improve the accuracy and semantic consistency of video content recognition.

[0016] 2. By adaptively generating dynamic convolutional kernel groups based on the content differences between the current keyframe and the reference keyframe, and using spatial attention masks to modulate the convolutional responses of each group, the system achieves the effects of enhancing the representation of temporal features of moving regions, suppressing static background interference, and improving the efficiency and robustness of temporal context coding of consecutive keyframes.

[0017] 3. By calculating a multi-scale local information entropy map based on the temporal context feature map and generating a deformable sampling offset field and modulation factor field, deformable local entropy weighted convolution is performed on the temporal context feature map, thereby enhancing the feature response of complex texture regions and geometrically deformed regions, improving the multi-scale spatial structure representation ability and semantic segmentation accuracy.

[0018] 4. By performing temporal consistency constraints based on historical segmentation confidence information, performing semantic prior injection based on scene category vectors, and training the keyframe semantic segmentation model by combining edge-aware weighted loss and spatiotemporal consistency loss, the keyframe segmentation results and text records are then fused and recognized in a multimodal manner. This achieves the effects of improving the temporal stability and global semantic consistency of the segmentation results, enhancing the quality of boundary region recognition, and realizing the output of structured video content description. Attached Figure Description

[0019] Fig. 1 This is a schematic diagram of the overall process of a video context understanding and recognition method based on multimodal signal fusion; Fig. 2 A schematic diagram of dynamic convolutional kernel generation and temporal context encoding; Fig. 3 This diagram illustrates the calculation of multi-scale local information entropy and the extraction of deformable local entropy weighted features. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example: Combined with Appendix Figs. 1-3 This embodiment provides a video context understanding and recognition method based on multimodal signal fusion, including: S1. Acquire video clips, extract keyframe sequences, and establish pixel-level semantic labels, scene category labels, and text records. The specific implementation is as follows: First, the raw video clip data for video context understanding is obtained. The raw video clip data consists of complete video sequences from publicly available large-scale video semantic segmentation datasets and self-collected videos of specific scenes. The self-collected part is recorded in real-world environments using high-definition cameras. The collected scenes cover streets, offices, and natural landscapes. The collected content includes the movement, interaction, and occlusion of various target objects to ensure that subsequent samples are representative under indoor scenes, outdoor scenes, complex backgrounds, and rich motion patterns.

[0022] Subsequently, short, consecutive frame segments are extracted from each complete video sequence as video segments. The length of each video segment is set to 64 to 128 frames to ensure that the scene evolution or target movement process is complete within a single video segment. Keyframe sequences are then extracted from each video segment using a fixed-interval sampling method. The sampling interval is determined based on the original video frame rate and the rate of content change. When the original video frame rate is higher than a preset frame rate threshold, keyframes are extracted at a preset downsampling interval. When the average pixel difference or structural similarity change between adjacent frames exceeds a change threshold, the sampling interval is reduced. When the change between adjacent frames is lower than the change threshold, the sampling interval is expanded so that the retained keyframes can represent the main content and significant changes of the video. The extracted keyframe sequences are then uniformly preprocessed, adjusting each frame image to a fixed spatial resolution, for example, a width of 640 pixels and a height of 360 pixels, to unify the subsequent data organization and balance computational cost with detail preservation.

[0023] After the keyframe sequence is formed, dense pixel-level semantic segmentation and annotation are performed on each keyframe image. That is, a semantic category label is assigned to each pixel in the frame. The annotation work is completed by professional annotators with the help of auxiliary tools, and multiple rounds of verification are performed to eliminate missing labels, incorrect labels and category drift. The semantic category labels can at least include pedestrians, vehicles, roads, buildings, sky, vegetation and traffic signs according to the scene.

[0024] After pixel-level semantic tags are established, an overall scene category tag is then established for each video segment. The scene category tag is formed by uniformly assigning values ​​to the entire video segment, such as city streets, parks, and indoor meetings, which are used to represent the global scene semantics of the video segment.

[0025] Meanwhile, to provide a foundation for subsequent multimodal semantic association of video segments, text records corresponding to each video segment are extracted and established. These text records include at least one of the following: original video subtitles, automatically transcribed text from speech recognition, and external descriptive text. They are stored in association according to video segment identifiers and keyframe sequence time ranges to ensure that the text content is consistent with the video segments in the time dimension. Subtitle text and automatically transcribed text from speech recognition are segmented by timestamps, and text segments falling within the start and end time ranges of the keyframe sequence are aggregated. Descriptive text without timestamps is associated as a whole segment according to the video segment identifier.

[0026] Furthermore, inter-frame motion reference information is pre-established for adjacent frames in the keyframe sequence. The motion reference information is provided directly by the dataset or calculated by existing optical flow estimation algorithms to obtain the optical flow field between adjacent frames, which is used as a supervision signal or reference information.

[0027] Finally, video clips, keyframe sequences, pixel-level semantic labels, scene category labels, text records, and optical flow fields of adjacent frames are organized according to a unified sample structure and divided into training, validation, and test sets. The training, validation, and test sets are independently divided according to the original video sequence to which the video clip belongs. Video clips in the same original video sequence are only included in one of the datasets to avoid data leakage caused by adjacent clips being included in different datasets at the same time. Each sample contains at least one keyframe sequence corresponding to the video clip, the pixel-level ground truth segmentation label corresponding to the keyframe sequence, the scene category label corresponding to the video clip, the text record corresponding to the video clip, and the optical flow information between adjacent frames of the keyframe, thus forming a training dataset that can be directly used by subsequent video context understanding models.

[0028] S2. Based on the current keyframe, reference keyframe, frame difference map, and normalized cross-correlation map, generate a dynamic convolutional kernel group, perform temporal context encoding on the current keyframe, and obtain a temporal context feature map. Specifically, this is implemented as follows: In this step, the current keyframe is first determined. and its reference keyframe ,in Indicates a time index. Indicates the time offset. It is a positive integer used to control the temporal span of the reference keyframe relative to the current keyframe. In one implementation... Current keyframe With reference keyframe All images are three-channel images, and their spatial dimensions are consistent with the dimensions of the preprocessed keyframes. Based on the current keyframe and the reference keyframe, a frame difference map is first constructed. ,in The frame difference map is used to characterize the motion intensity between the current keyframe and the reference keyframe pixel by pixel; then a normalized cross-correlation map is constructed. The normalized cross-correlation map is used to characterize the local structural similarity between two frames. Specifically, a local window is selected at each spatial location, and the normalized cross-correlation between the current keyframe and the reference keyframe is calculated within that local window. ,

[0029] in, Indicated by A local window centered on the center. This represents the mean value of the current keyframe within this local window. This represents the mean of the reference keyframe within this local window. This represents a minimal constant to prevent the denominator from being zero.

[0030] Then, a negative exponential mapping is performed on the obtained result to get

[0031] in The cross-correlation attenuation coefficient is taken as [value missing] in one embodiment. Local windows can be used Window; After obtaining the current keyframe, reference keyframe, frame difference map, and normalized cross-correlation map, a kernel generator network is constructed to generate dynamic convolutional kernel groups. The kernel generator network uses channel-stitched results. As input, the kernel parameters are first processed through a first convolutional layer and linear rectified activation to achieve dimensionality reduction and preliminary feature fusion. Then, the kernel parameters are output through a second convolutional layer. The mathematical expression is as follows:

[0032] in, The convolutional weights are used to perform dimensionality reduction and feature fusion on the concatenated input, and the number of intermediate feature channels can be chosen as follows. , The convolution weights are used to map intermediate features to unfolded kernel parameters. In one implementation, Conv1 and Conv2 employ two-dimensional convolutions with a stride of 1 and zero padding to maintain the same spatial dimensions. The kernel size is [missing information]. The independent convolutions performed on the frame difference map and the normalized cross-correlation map also employ two-dimensional convolutions with a stride of 1 and zero padding to maintain the same spatial dimensions; The dynamic convolutional kernel group is obtained after reshaping. Its dimensions can be reshaped into ,in This represents the number of dynamic convolutional kernel groups, which in one implementation is taken as... , This represents the number of input channels, which is taken from the initial keyframe image. , This represents the number of output channels for each dynamic convolution kernel, which in one implementation is taken as... , and These represent the kernel height and kernel width, respectively, and in one embodiment, both are taken as... After generating dynamic convolutional kernel groups, spatial attention masks corresponding to each group of dynamic convolutional kernels are generated based on the frame difference map and the normalized cross-correlation map. Specifically, independent convolutions are first applied to the frame difference map and the normalized cross-correlation map to extract the channel slices corresponding to the group indices. Then, spatial attention masks are generated through convolution mapping and sigmoid activation. , No. The spatial attention mask is only used to modulate the first... The convolutional response of a group of dynamic convolutional kernels, which is not weighted in the summation of the responses of other groups of dynamic convolutional kernels, can be expressed as:

[0033] in, This represents the Sigmoid activation function, used to constrain the output to... to interval, The first frame difference map obtained after independent convolution is represented as the frame difference map. Each channel feature slice The first part represents the result obtained after independent convolution of the normalized cross-correlation graph. Each channel feature slice and These represent the convolutional weights and biases used to generate the spatial attention mask. Based on this, each group of dynamic convolutional kernels is used to perform parallel convolution operations on the current keyframe. Then, the corresponding spatial attention mask is used to modulate the convolutional responses of each group element-wise. Finally, the modulation results of all groups are summed to obtain the temporal context feature map. Its mathematical expression is:

[0034] in This represents the convolution operation. This represents element-wise multiplication; the temporal context feature map is used to characterize the temporal motion information and scene evolution information contained in the current keyframe under the condition of comparison with the reference keyframe. Since the dynamic convolution kernel group is jointly driven by the current keyframe, frame difference map, and normalized cross-correlation map, the convolution sampling weight can adaptively change with the content difference between consecutive keyframes. Combined with the spatial attention mask, the motion region is enhanced and the static background region is suppressed. Thus, without relying on high-cost optical flow-dominated modeling, the temporal context information of the current keyframe is effectively encoded, providing a unified temporal context feature input for subsequent multi-scale spatial structure modeling and semantic segmentation.

[0035] S3. Based on the temporal context feature map, calculate the multi-scale local information entropy map, generate a deformable sampling offset field and modulation factor field, and extract the local entropy weighted feature map. The specific implementation is as follows: The temporal context feature map obtained in the previous step As input, local information entropy maps are calculated at multiple preset scales to generate deformable sampling offset fields and modulation factor fields, and local entropy weighted feature maps are extracted accordingly; specifically, scale indices are first set. In one implementation, the total number of scales (S) is taken as (3). Then, for each scale (s) and each spatial location ((x,y)) on the temporal context feature map, a local spatial window centered on that location is selected. When the local window exceeds the boundary of the feature map, the window content is supplemented by boundary copying and filling. All channel feature values ​​within the window are normalized to the interval ([0,1]) and the probability distribution of falling into each histogram interval is statistically analyzed. The normalized feature values ​​are divided into a predetermined number of histogram intervals of equal width from 0 to 1, and the proportion of feature values ​​in each interval to the total number of feature values ​​in the local window is calculated as follows. Where (b) represents the histogram interval index; then, the local information entropy map at the (s)th scale is calculated according to the definition of information entropy. Its mathematical expression is !

[0036] in To prevent the smallest positive constant of the logarithm of zero, in one implementation, one can take... ; It is used to quantify the local complexity of a location ((x,y)) at the corresponding scale. The higher the entropy value, the richer the texture and the more complex the structure of the region.

[0037] In one implementation, each scale corresponds to a different local spatial window size, with the first scale corresponding to... Local window, corresponding to the second scale Local window, corresponding to the third scale Local window; after obtaining the local information entropy maps at each scale, the temporal context feature map is... Local information entropy map at the corresponding scale By concatenating the layers along the channel dimension and inputting the parameters at the (s)th scale, a convolutional layer is generated, resulting in a deformable convolutional parameter tensor, whose expression is:

[0038] in This represents the convolutional layer weights used to generate deformable convolution parameters at the (s)th scale. This represents the corresponding bias; after slicing the tensor of this parameter, the offset field at the (s)th scale is obtained. and modulation factor field ,in The dimension is , The dimension is , This represents the total number of sampling points in the convolutional kernel at this scale. Each sampling point contains (1) values ​​for the offset in the (x) direction and the offset in the (y) direction, and (1) corresponding importance weight values ​​in the modulation factor field. Subsequently, based on the offset field and the modulation factor field, deformable local entropy weighted convolution is performed on the temporal context feature map. When the coordinates are non-integer, bilinear interpolation is used to obtain the feature values ​​at the corresponding positions. Local entropy-weighted feature map at the (s)th scale satisfy ! !

[0039] in This represents the coordinates of the (n)th sampling point in the standard convolutional regular grid. Let represent the standard weight matrix of the convolution kernel at scale (s) at sampling position (n). Indicates position The spatial offset of the (n)th sampling point, Indicates position The modulation factor at the (n)th sampling point This represents the weight adjustment function based on local information entropy, and its expression is: !

[0040] in To control the hyperparameter of the weight enhancement magnitude, in one implementation, it is set to (0.5). To control the hyperparameter of weight enhancement steepness, in one implementation, it is taken as (1.0). To control the hyperparameter of the weighted activation threshold, in one embodiment, it is set to (2.0). Through the above processing, the sampling points of the convolutional kernel are no longer fixed on the regular grid, but are adaptively shifted according to the temporal context features and local complexity. The contribution of each sampling point is controlled by the modulation factor, and the information-rich region is enhanced by the local information entropy weight adjustment function. Finally, local entropy weighted feature maps are output at each preset scale. It is used to characterize the multi-scale spatial structure features after geometric deformation adaptation and complex texture enhancement.

[0041] S4. Based on the temporal context feature map, local entropy weighted feature map, historical segmentation confidence information, and scene category vector, perform temporal consistency constraints and semantic prior injection to generate a semantic segmentation probability map. Specifically, the implementation is as follows: The temporal context feature map obtained in the previous step Local entropy weighted feature maps at various scales Historical segmentation confidence information and scene category vector As input, during the training phase, the historical segmentation confidence information uses the segmentation probability map obtained from the previous keyframe through model forward propagation and subjected to gradient truncation. During the inference phase, the historical segmentation confidence information uses the predicted segmentation probability map of the previous keyframe, and multi-scale spatiotemporal feature aggregation, scene semantic prior injection, and adaptive edge-aware segmentation map generation are performed sequentially. Specifically, the temporal context feature map is first... Weighted feature maps of local entropy at various scales Cross-scale fusion is performed, and the learned scale fusion convolutional layer is used to weight and filter features at each scale. At the same time, a temporal aggregation term is introduced to modulate the feature representation of the current frame using the segmentation confidence information of the previous frame, resulting in a multi-scale spatiotemporal feature map. Its expression is:

[0042] in The weight coefficient for the temporal aggregation term is 0.3 in one implementation, and the number of fused feature channels can be 128; the temporal aggregation function. This is used to inject historical segmentation information into the current feature, and can be written as:

[0043] in For temporal aggregation of convolution weights, To control the rate of feature similarity decay, a coefficient of 1.0 is used in one embodiment. For a learnable linear projection layer to align historical temporal context features, the following must be satisfied:

[0044] After obtaining the multi-scale spatiotemporal feature maps, global scene semantic information is extracted from the keyframe sequence to generate scene category vectors. The scene category vector is obtained from the visual features of the keyframe sequence through global average pooling and linear mapping, and is supervised and constrained during the training phase by the scene category labels of the corresponding video segments. Its expression is:

[0045] in This represents the total number of keyframes used to generate scene vectors. This represents a global average pooling operation, where the scene feature vector dimension can be 64. Subsequently, a gating mechanism is used to inject the scene category vector into a multi-scale spatiotemporal feature map, resulting in a semantic prior feature map. Its expression is:

[0046] in This represents element-wise multiplication. Represents the scene embedding matrix. Represents a matrix of all ones used for space broadcasting, and the gate function. A spatial attention map is generated based on the scene category vector and fused features, which can be written as:

[0047] in This represents the Sigmoid activation function. This involves broadcasting the scene category vector along the spatial dimension and then jointly modulating it with the fused features. First, the scene category vector is linearly expanded to match the number of channels in the multi-scale spatiotemporal feature map. Then, it is broadcast along the spatial dimension and concatenated with the multi-scale spatiotemporal feature map channel by channel before being input into a gated convolutional layer. After semantic prior injection, the final semantic segmentation prediction is performed based on the semantic prior feature map. An edge enhancement term generated from the multi-scale local entropy map is introduced to improve the segmentation accuracy of regions with complex textures and blurred boundaries, resulting in a semantic segmentation probability map. Its expression is:

[0048] in and These are the convolutional weights and biases of the segmentation layer, respectively. The weight coefficient for the edge perception enhancement term is set to 0.2 in one implementation. For the first The edge enhancement function generated by the local information entropy map at each scale is expressed as follows:

[0049] in This indicates that the Sobel operator is used to calculate the gradient magnitude. The Sobel operator consists of a horizontal convolution kernel. and vertical convolution kernel constitute, To control the influence of local information entropy on edge enhancement intensity, the coefficient is set to 0.5 in one embodiment. Through the above processing, the temporal consistency constraint is achieved by using historical segmentation confidence information on the basis of cross-scale fusion, the global semantic prior is injected by using scene category vector, and the edge perception enhancement is achieved by using local information entropy map. Finally, the semantic segmentation probability map in the form of normalized probability distribution of each pixel in each semantic category is output.

[0050] S5. Based on the semantic segmentation probability map, local information entropy map, and optical flow alignment features, construct edge-aware weighted loss and spatiotemporal consistency loss, and train the keyframe semantic segmentation model. The specific implementation is as follows: First, use the semantic segmentation probability graph obtained from the aforementioned forward propagation. Local information entropy maps at various scales Using historical features aligned with the optical flow field as input, a composite loss function is constructed to supervise the training of the keyframe semantic segmentation model. This composite loss function consists of a weighted loss function of edge-aware weighted cross-entropy loss and a multi-scale spatiotemporal consistency loss. Specifically, to address the requirements of single-frame semantic segmentation accuracy and optimization of complex region boundaries, an edge-aware weighted cross-entropy loss function is first constructed. Its expression is

[0051] Where (H) and (W) represent the height and width of the semantic segmentation probability map, respectively. This represents the true label of category (l) at position ((x,y)) at time (t), using one-hot encoding. Let represent the predicted probability value of category (l) at position ((x,y)) at time (t). This represents the local information entropy weighting coefficient, which in one implementation is set to (0.5) to increase the weight of complex texture regions in the loss. This represents the global maximum value in the local information entropy graph at the (s)th scale, used to complete intra-scale normalization; This represents the edge gradient weighting coefficient, which is taken as (0.3) in one implementation. The edge gradient weights at position ((x,y)) are expressed as follows: !

[0052] in The gradient magnitude scaling factor is taken as (10) in one embodiment. and These represent the gradient operators in the horizontal and vertical directions, respectively. Let L represent the Euclidean norm. Through the above construction, locations with high local information entropy and drastic changes in predicted edges receive higher penalty weights during training, thereby strengthening the segmentation supervision of object boundaries and complex texture regions. Subsequently, to address the temporal smoothness and cross-scale feature consistency requirements of segmentation results between adjacent keyframes in the video sequence, a multi-scale spatiotemporal consistency loss (L{consist}) is constructed, whose expression is: ,in, Let represent the local entropy-weighted feature map at the th scale at time (t). This represents the local entropy-weighted feature map at the (t-1)th time and the (s)th scale. This represents the number of feature channels at the (s)th scale. Let represent the optical flow estimation field from frame (t) to frame (t-1), with dimension . , The feature warping function is used to align the feature map of the previous frame to the coordinate system of the current frame through bilinear sampling based on the optical flow estimation field. The standard deviation of the eigenvalue at scale (s) is used as a normalization factor to balance the magnitude of loss at different scales. To prevent extremely small constants with a denominator of zero; This represents the weight coefficient of the total variational regularization term, which in one implementation is taken as (0.1). The total variational regularization term of the semantic segmentation probability graph is expressed as follows:

[0053] This is used to promote spatial smoothness of the segmentation results and suppress fragmentation noise. The optical flow estimation field can be predicted by a lightweight optical flow subnetwork, which consists of two convolutional layers. It takes the current keyframe and adjacent reference keyframes as input and outputs a two-dimensional optical flow field. The lightweight optical flow subnetwork uses a two-layer continuous convolutional structure. The first layer input is the channel concatenation result of the current keyframe and adjacent reference keyframes, and the second layer outputs a two-channel optical flow component map, corresponding to the horizontal and vertical displacement components, respectively. After the composite loss is constructed, the edge-aware weighted cross-entropy loss (L{seg}) and the multi-scale spatiotemporal consistency loss (L*{consist}) are weighted and summed according to a preset ratio to obtain the total loss (L) of the current training batch. The total loss satisfies... ,in and The preset loss weight coefficient.

[0054] Then, end-to-end training is performed on the training dataset under the deep learning framework. In each iteration, one batch of video clip data is randomly sampled. Each training batch uses video clips as the smallest sampling unit. Each sample in the same batch contains a complete keyframe sequence and its corresponding label information. The batch data contains at least the keyframe sequence, pixel-level real segmentation labels, scene category labels, and optical flow information between adjacent frames. First, forward computation is performed, including dynamic convolutional kernel encoding, multi-scale deformable local entropy feature extraction, multi-scale spatiotemporal feature aggregation, scene semantic prior injection, and adaptive edge-aware segmentation map generation, to obtain the semantic segmentation probability map prediction value of the current batch of video keyframes. Then, backpropagation is performed based on the total loss to calculate the gradient of the total loss with respect to all trainable parameters. Finally, the stochastic gradient descent optimization algorithm is used to update all trainable parameters at once to minimize the total loss.

[0055] During training, the learning rate is dynamically adjusted using a cosine annealing strategy with a warm-up phase to stabilize training and promote model convergence. At the same time, after each round of training, the average intersection-union ratio (IU) is evaluated based on the validation set. If the IU of the validation set does not improve for 10 consecutive rounds, early stopping is performed and the best model parameters are saved. The maximum number of training rounds is preset based on the convergence trend of the IU of the validation set and is fixed before training begins. If the preset maximum number of training rounds is reached, training is forcibly stopped, and the model parameters corresponding to the best performance on the validation set are also saved. In addition, the training loss and validation loss change curves are monitored simultaneously. When the training loss continues to decrease while the validation loss increases significantly, it is used as an auxiliary criterion for overfitting to ensure that a keyframe semantic segmentation model that can be used for subsequent keyframe semantic segmentation inference is finally obtained.

[0056] S6. Call the trained keyframe semantic segmentation model to generate keyframe segmentation results for the video to be processed. Combine this with text records to perform multimodal fusion recognition and output a structured video content description. The specific implementation is as follows: First, the keyframe semantic segmentation model that has been trained and whose optimal parameters have been saved is called. Then, the preprocessing operation is performed on the input video to be processed, which is the same as that in the training stage. Specifically, the complete video is cut into video segments in a preset way, or keyframes are extracted directly at fixed intervals to form a keyframe sequence arranged in chronological order. Finally, each keyframe image is uniformly scaled to the fixed spatial resolution required by the model.

[0057] The keyframe semantic segmentation model was then set to evaluation mode, in which computational steps enabled only during the training phase were disabled, and forward inference was performed frame by frame on the keyframe sequence. For any given keyframe... Select its preceding reference keyframe according to the time offset determined during the training phase. As input, the current keyframe and the reference keyframe are fed into the keyframe semantic segmentation model. The model sequentially performs dynamic convolutional kernel encoding of temporal context information, deformable local entropy feature extraction of multi-scale spatial structure, and collaborative optimization segmentation processing based on temporal consistency and semantic prior, outputting the semantic segmentation probability map corresponding to the current keyframe. .

[0058] For cases where the starting position of a video sequence cannot meet the selection criteria for a reference keyframe due to insufficient historical frames, a copy-paste method or a zero-value initialization method is used to supplement the reference frame information to ensure that the model input format remains consistent. After obtaining the semantic segmentation probability map, maximum probability selection is performed along the category dimension to obtain the discrete semantic segmentation result of the current keyframe, the mathematical expression of which is:

[0059] in (x, y) represents the semantic category label at position (x, y), and (c) represents the semantic category index. This generates a semantic segmentation mask map with the same spatial size as the input keyframe. The semantic segmentation mask map is output in the form of a color map or an index map. Each color value or index value in the map corresponds to a specific semantic category. After performing the above processing on all keyframes of the entire input video, a set of keyframe semantic segmentation results arranged in chronological order is obtained. The results are used to clearly label the object category or region category to which each pixel in the scene belongs at each moment of the video.

[0060] Based on this, temporal aggregation analysis is performed on the semantic segmentation sequence of keyframes to count the semantic categories that appear stably, frequently, or in prominent positions within the entire video segment or a specific time period. Combined with the positional continuity of the target object in adjacent keyframes, the appearance, persistence, and disappearance process of the target object are initially tracked to form a visual target object category list. The visual target object category list includes at least the target object category name, first appearance time, duration, frequency of appearance, and spatial salience position.

[0061] Subsequently, text records associated with video clips are introduced. These text records include at least one of the following: subtitle text, text transcribed by automatic speech recognition, and descriptive text. The text records are then input into a pre-trained natural language understanding model to extract key entities, action descriptors, and scene descriptors from the text, forming a set of text-side semantic keywords. This set of text-side semantic keywords includes at least a set of entity words, a set of action words, a set of scene words, and a set of emotion descriptors.

[0062] A multimodal fusion analysis module is then constructed to associate, align, and jointly infer the visual target object category list with the text semantic keyword set. Candidate matching pairs are established based on the semantic correspondence between the visual target object categories and the text entity words. Inconsistent matching items are filtered out based on the consistency between the appearance time of the visual target object and the appearance time of the text action words. Then, scene constraints are applied to the candidate matching pairs based on the scene word set to obtain the fusion recognition result. For example, when the visual side identifies ball-like objects and running people, and the text side extracts shooting action descriptions, the corresponding video content can be identified as a football match scene, and the target objects can be refined into footballs and athletes.

[0063] In one implementation, a list of visual targets and a text summary can be simultaneously input into an open-source multimodal large model. Based on semantic understanding and reasoning capabilities, the model generates a comprehensive semantic description of the corresponding video segments. The visual targets with the highest frequency and a duration exceeding a preset threshold are identified as the primary targets. The scene type is determined by matching the scene label corresponding to the scene category vector with the scene words in the text. The core action is determined by matching the temporal behavior of the visual targets with the action words in the text. The emotional description words in the text and the semantics of the visual scene are jointly determined as the emotional tone. Finally, according to the requirements of the programmatic advertising industry standard protocol, the structured information obtained from the fusion analysis is mapped and filled into the fields specified in the protocol. The structured information includes at least the primary targets, scene type, core action, and emotional tone, generating a machine-readable structured video content metadata file. The structured video content metadata file is stored in key-value pairs with corresponding field names and values ​​to achieve a continuous processing flow from the original video to be processed to keyframe segmentation, target identification, scene understanding, and standardized content output.

[0064] In the operational scenario shown in this embodiment, taking the video context understanding and recognition scenario for programmatic advertising as an example, a batch of urban life videos from short video content platforms are analyzed and structured for output. First, the original video segment data for model training is obtained. The original data consists of complete video sequences from publicly available video semantic segmentation datasets and self-collected videos. The self-collected videos are recorded using high-definition cameras in real environments such as city streets, commercial pedestrian areas, office building squares, and park trails. The video content includes dynamic situations such as pedestrian traffic, vehicle movement, road changes, traffic sign appearances, human interaction, ball games, and outdoor billboard appearances.

[0065] Each complete video sequence is cut into video segments of 96 frames in length, and key frame sequences are extracted based on the original video frame rate and the amplitude of image changes. When the original video frame rate is 30 frames per second and the amplitude of changes between adjacent frames is within the normal range, one key frame is extracted every 3 frames. When the average pixel difference between adjacent frames exceeds the preset change threshold, the sampling interval is reduced to 2 frames. When the continuous image changes smoothly, the sampling interval is increased to 4 frames.

[0066] The extracted keyframes are uniformly scaled to The resolution was determined, and pixel-level semantic annotations were performed on each keyframe by annotators to obtain category labels such as pedestrians, vehicles, roads, buildings, sky, vegetation, and traffic signs. At the same time, the entire video segment was assigned the scene category label "city street," and subtitle text, automatic speech recognition transcription text, or external description text corresponding to the video segment were extracted and text records were created according to the video segment identifier and start and end times. Optical flow fields were generated for adjacent keyframes using an optical flow estimation algorithm, and these, together with the keyframe sequence, pixel-level semantic labels, scene category labels, and text records, were organized into training samples, which were independently divided into training set, validation set, and test set according to the original video sequence.

[0067] Subsequently, during the model training phase, for any current keyframe Select time offset Corresponding reference keyframe Construct frame difference map and in Calculate normalized cross-correlation values ​​within a local window Press again Generate a normalized cross-correlation plot, where The current keyframe, frame difference map, and normalized cross-correlation map are concatenated by channel and then input into the kernel generation network, which is then processed through two layers. Convolution generates dynamic convolutional kernel groups, with 4 groups and 8 output channels per group. Spatial attention masks, corresponding one-to-one with each dynamic convolutional kernel group, are generated jointly by a frame difference map branch and a normalized cross-correlation map branch. Finally, based on...

[0068] Obtain the temporal context feature map. Then, using the temporal context feature map as input, construct [the following] at three scales respectively. , , Local windows are used to statistically analyze the probability distribution of normalized eigenvalues ​​across histogram intervals, and then...

[0069] Calculate the local information entropy graph, where After concatenating the temporal context feature map with the local information entropy map, a deformable sampling offset field and modulation factor field at various scales are generated through a convolutional layer, and then based on... Extract the local entropy-weighted feature map, where , , , When the offset coordinate is a non-integer position, bilinear interpolation is used to obtain the corresponding feature value.

[0070] Subsequently, the temporal context feature map, local entropy weighted feature maps at each scale, segmentation confidence information from the previous keyframe, and scene category vector are jointly input into the collaborative optimization segmentation network. Through multi-scale spatiotemporal feature aggregation, the desired segmentation result is obtained. The weight coefficient of the time-series aggregation term Then, a scene category vector with a dimension of 64 is generated from the keyframe sequence through global average pooling and linear mapping. During the training phase, the scene label of "city street" is used to supervise and constrain the training; then, the scene category vector is injected into the multi-scale spatiotemporal feature map through a gating mechanism to obtain the semantic prior feature map. and based on

[0071] Generate a semantic segmentation probability graph, where , , In the loss construction stage, a combination of edge-aware weighted cross-entropy loss and multi-scale spatiotemporal consistency loss is used to train the model. The edge-aware weighted cross-entropy loss utilizes the gradient of the local information entropy map and the probability map to increase the penalty weights in the boundary region, where the local information entropy weighting coefficients... Marginal gradient weighting coefficients Gradient scaling factor Multi-scale spatiotemporal consistency loss utilizes the optical flow field to align the local entropy-weighted feature map of the previous frame to the coordinate system of the current frame, constrains the feature differences at each scale, and introduces a total variational regularization term, with weights for the regularization term. .

[0072] During training, video clips are used as the minimum batch sampling unit, and the total loss is set to... All parameters are updated using a stochastic gradient descent optimization algorithm, and a cosine annealing strategy with a warm-up phase is adopted for the learning rate. When the average intersection-union ratio of the validation set has not improved for 10 consecutive rounds, early stopping is performed and the optimal model parameters are saved.

[0073] After the model training is completed, a video of a city street to be processed is selected as the application object. The video is 18 seconds long and shows roads, vehicles, pedestrians crossing the street, roadside beverage billboards, and people holding drinks and talking.

[0074] The video is processed using the same rules as during training. Keyframe sequences are extracted and input into the trained keyframe semantic segmentation model. Semantic segmentation probability maps are output frame by frame, and then based on... Generate a keyframe semantic segmentation mask.

[0075] After temporal aggregation of all keyframe semantic segmentation results, the first occurrence time, duration, frequency of occurrence, and salient position of "pedestrians," "vehicles," "roads," "billboards," and "handheld objects" in the segment are statistically obtained, and a visual target object category list is formed. At the same time, the subtitle text and speech transcription text of the video are extracted to obtain entity words, action words, scene words, and emotion description words such as "iced drink," "refreshing," "street," and "gathering," forming a text-side semantic keyword set.

[0076] In the multimodal fusion analysis module, candidate matching pairs are established between "holding an object" and "people gathering and talking" and "iced drinks" and "refreshing". Valid matches are retained based on the consistency between the time of appearance of the visual target and the time of appearance of the text description. Then, scene constraints are applied by combining the scene vector of "city street" and the scene words of "street". Finally, the main target objects of the video clip are identified as beverage products and people, the scene type is urban street social scene, the core actions are walking, talking and holding a drink, and the emotional tone is relaxed and refreshing.

[0077] Finally, the above results are mapped to programmatic advertising protocol fields, generating a structured video content metadata file in key-value pair format, which includes " Beverage products, pedestrians, : "", : "", : The content of fields such as "" is provided so that the subsequent advertising bidding system can directly call it in the placement of similar beverage advertisements, urban lifestyle advertisements and social scene advertisements.

[0078] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video context understanding and recognition method based on multimodal signal fusion, characterized in that, include: S1. Acquire video clips, extract keyframe sequences, and establish pixel-level semantic labels, scene category labels, and text records; S2. Based on the current keyframe, reference keyframe, frame difference map, and normalized cross-correlation map, generate a dynamic convolutional kernel group, perform temporal context encoding on the current keyframe, and obtain a temporal context feature map. S3. Based on the temporal context feature map, calculate the multi-scale local information entropy map, generate a deformable sampling offset field and modulation factor field, and extract the local entropy weighted feature map. S4. Based on the temporal context feature map, local entropy weighted feature map, historical segmentation confidence information, and scene category vector, execute temporal consistency constraints and semantic prior injection to generate a semantic segmentation probability map. S5. Based on the semantic segmentation probability map, local information entropy map, and optical flow alignment features, construct edge-aware weighted loss and spatiotemporal consistency loss, and train the keyframe semantic segmentation model. S6. Call the trained keyframe semantic segmentation model to generate keyframe segmentation results for the video to be processed, combine the text records to perform multimodal fusion recognition, and output a structured video content description.

2. The video context understanding and recognition method based on multimodal signal fusion according to claim 1, characterized in that, S1 includes: Obtain complete video sequences and videos of specific scenes; Extract video clips and keyframe sequences; Pixel-level semantic labels, scene category labels, and text records are created for keyframe sequences; Establish optical flow fields for adjacent keyframes; The training dataset is divided based on the video sequences.

3. The video context understanding and recognition method based on multimodal signal fusion according to claim 1, characterized in that, S2 include: Construct a frame difference map based on the current keyframe and the reference keyframe; Calculate the normalized cross-correlation graph based on the current keyframe and the reference keyframe; Based on the channel stitching results of the current keyframes, frame difference maps, and normalized cross-correlation maps, dynamic convolutional kernel groups are generated using a kernel generator network.

4. The video context understanding and recognition method based on multimodal signal fusion according to claim 3, characterized in that, S2 also includes: A spatial attention mask corresponding to the dynamic convolutional kernel group is generated based on the frame difference map and the normalized cross-correlation map. Perform convolution operations on the current keyframe using a dynamic convolution kernel group; The convolutional response is modulated element-wise and summed based on the spatial attention mask to obtain the temporal context feature map.

5. The video context understanding and recognition method based on multimodal signal fusion according to claim 1, characterized in that, S3 includes: Calculate the local information entropy map at each scale based on the temporal context feature map; The offset field and modulation factor field at each scale are generated based on the temporal context feature map and the local information entropy map; Based on the offset field, modulation factor field, and local information entropy map, deformable local entropy weighted convolution is performed on the temporal context feature map to obtain local entropy weighted feature maps at each scale.

6. The video context understanding and recognition method based on multimodal signal fusion according to claim 1, characterized in that, S4 includes: Based on the temporal context feature map, the local entropy weighted feature map, and the historical segmentation confidence information, cross-scale fusion and temporal aggregation are performed to obtain multi-scale spatiotemporal feature maps. Generate scene category vectors based on keyframe sequences; Semantic prior injection is performed on the multi-scale spatiotemporal feature map based on the scene category vector to obtain the semantic prior feature map.

7. The video context understanding and recognition method based on multimodal signal fusion according to claim 6, characterized in that, S4 also includes: Edge enhancement terms are generated based on the local information entropy map; Semantic segmentation prediction is performed based on the semantic prior feature map and edge enhancement terms to obtain a semantic segmentation probability map; Among them, the edge enhancement term is obtained by calculating the gradient magnitude and exponential weighting of the local information entropy map.

8. The video context understanding and recognition method based on multimodal signal fusion according to claim 1, characterized in that, S5 include: A composite loss function is constructed based on semantic segmentation probability maps, local information entropy maps, and optical flow alignment features. Backpropagation and parameter updates are performed on the keyframe semantic segmentation model based on the composite loss function; The model parameters are determined based on the evaluation results of the validation set. The composite loss function is composed of edge-aware weighted cross-entropy loss and multi-scale spatiotemporal consistency loss.

9. The video context understanding and recognition method based on multimodal signal fusion according to claim 1, characterized in that, S6 include: Call the keyframe semantic segmentation model to generate keyframe semantic segmentation results; A list of target object categories is generated on the visual side based on the semantic segmentation results of keyframes; A set of semantic keywords on the text side is formed based on the text records; A fusion recognition process is performed based on the visual object category list and the text-based semantic keyword set. A structured video content metadata file is generated based on the fusion recognition results.

Images