Video content understanding method for multimodal advertising inventory intelligent matching system

Abstract

The present application relates to the technical field of video content understanding, and specifically to a video content understanding method for a multimodal advertising inventory intelligent matching system, the method comprising: taking the video frame of the original advertising video as a benchmark, analyzing the similarity of adjacent video frames and the discrete degree of the feature matrix elements corresponding to the voice segment in the shot where the timestamp of each video frame is located, and determining the first search window; at the same time, analyzing the frequency of occurrence and duration of text segments in a unit time, and confirming the second search window; based on the search window, analyzing the feature similarity of the voice segment and text segment with the video frame in the corresponding window, and obtaining the optimal alignment path; aligning the timestamps of the voice segment and text segment with the timestamp of the video frame, and realizing multi-level semantic parsing through multimodal fusion technology and deep learning models. The present application aims to synchronize the temporal relationship between audio, text and video frames, and improve the accuracy of video content understanding.

Description

Technical Field

[0001] The present application relates to the technical field of video content understanding, and in particular to a video content understanding method for a multimodal advertising inventory intelligent matching system. Background Art

[0002] Multimodal advertising integrates visual, auditory, and text-based media formats, conveying brand value and product characteristics through the integration of multi-dimensional information. The Multimodal Advertising Inventory Intelligent Matching System is an automated platform built on artificial intelligence technology. Its core function is to accurately match massive ad inventory with media-side video traffic. By analyzing the multimodal characteristics of video content, it achieves efficient integration between ads, content, and users, improving the relevance and conversion rate of advertising.

[0003] As the core carrier of multimodal advertising, the unstructured information contained in video needs to be converted into computable structured features to semantically match it with the label system of advertising inventory. This avoids the disconnect between ads and video scenes, which can lead to a poor user experience and wasted advertising budget. Traditional video content understanding methods typically first extract visual, audio, and text features separately, then use models to fuse these features, and finally rely on keyword matching or rule engines for semantic analysis and matching.

[0004] However, traditional methods do not take into account that multimodal signals such as visual images, audio speech, and text subtitles in videos often have timing asynchrony problems due to post-editing, equipment acquisition differences, etc. It is usually assumed that multimodal signals are synchronous. If they are directly processed without alignment, it will lead to cross-modal feature misalignment, resulting in deviations in semantic parsing, and thus causing the matching logic of advertising context targeting to fail. Summary of the Invention

[0005] In view of the above, it is necessary to provide a video content understanding method for a multimodal advertising inventory intelligent matching system to solve the above problems.

[0006] One embodiment of the present application provides a video content understanding method for a multimodal advertising inventory intelligent matching system, the method comprising:

[0007] Extract video frames and segment shots from the original advertising video; extract audio signals from the original advertising video to obtain voice segments; read text segments from the original advertising video;

[0008] Divide the video frames, voice segments, and text segments of corresponding timestamps based on the start and end time of each shot segmentation;

[0009] Taking the video frame as a benchmark, in the shot corresponding to the timestamp of each video frame, the similarity between adjacent video frames and the discreteness of the spectral information corresponding to the speech segment are analyzed, a dynamic balance factor corresponding to the timestamp of each video frame is determined, and the preset benchmark window size corresponding to the timestamp of each video frame is adjusted to obtain a first search window;

[0010] In the shot corresponding to the timestamp of each video frame, the occurrence frequency and duration of the text segment corresponding to the timestamp in unit time are analyzed to determine the text dynamic factor corresponding to the timestamp of each video frame, and the preset reference window size corresponding to the timestamp of each video frame is adjusted to obtain a second search window;

[0011] Based on the first search window and the second search window, analyzing the feature similarity between the speech segment and the text segment and the video frame in the corresponding window, obtaining a first sparse matrix and a second sparse matrix, and obtaining a first optimal alignment path and a second optimal alignment path;

[0012] Based on the obtained optimal alignment path, the timestamps of the speech segments and text segments are aligned with the timestamps of the video frames. The timestamp-aligned video frames, speech segments, and text segment information are integrated through multimodal fusion technology, and multi-level semantic analysis is achieved through a deep learning model.

[0013] Preferably, the dynamic balance factor corresponding to the timestamp of each video frame is specifically:

[0014] Obtain the average value of the cosine similarities between all adjacent video frames in the shot where the timestamp of each video frame corresponds to as the visual motion intensity of the timestamp of each video frame;

[0015] Extract the Mel spectrum matrix of each speech segment, and use the sequence of Mel spectrum matrices of all speech segments as the audio feature sequence;

[0016] Obtain the standard deviation of each element in the audio feature sequence corresponding to the shot at the timestamp of each video frame and calculate the mean to obtain the audio energy change rate of each video frame corresponding to the timestamp;

[0017] The absolute value and sum of the difference between the visual motion intensity and the audio energy change rate corresponding to each video frame are calculated respectively, and the negative correlation mapping result of the sum value is positively fused with the absolute value of the difference to obtain the dynamic balance factor corresponding to the timestamp of each video frame.

[0018] Preferably, the preset reference window size corresponding to the timestamp of each video frame is adjusted to obtain the first search window, and the specific formula is: ,in, represents the width of the first search window corresponding to the timestamp of the t-th video frame; t represents the timestamp corresponding to the t-th video frame, Indicates the preset base window size; Represents the Sigmoid function; Indicates a rounded value. represents the visual motion intensity of the corresponding timestamp of the t-th video frame; represents the audio energy change rate of the corresponding timestamp of the t-th video frame; Indicates the dynamic balance factor corresponding to the timestamp of the t-th video frame.

[0019] Preferably, the text dynamic factor for confirming the timestamp corresponding to each video frame is specifically:

[0020] The sequence composed of all text segments is regarded as a text segment sequence;

[0021] Obtaining the frequency of occurrence of the text segment corresponding to the timestamp in a unit time within the shot corresponding to the timestamp of each video frame, and the duration of the text segment corresponding to the timestamp, and multiplying the two to obtain a first product;

[0022] Obtain the maximum occurrence frequency and maximum duration of the text segment corresponding to the timestamp in the text segment sequence, and multiply them to obtain a second product;

[0023] A result of forward fusion of the negative correlation mapping result of the second product and the first product is used as a text dynamic factor corresponding to the timestamp of each video frame.

[0024] Preferably, the preset reference window size corresponding to the timestamp of each video frame is adjusted to obtain the second search window, specifically:

[0025] In the shot corresponding to the timestamp of each video frame, the similarity of the feature information between the video frame and the text segment is analyzed to determine the semantic similarity of the timestamp corresponding to each video frame;

[0026] The width of the second search window corresponding to the timestamp of the t-th video frame is recorded as , its formula form is: ,in: express; Indicates the semantic similarity of the timestamp corresponding to the t-th video frame; Indicates the preset activation factor; represents the activation function; Indicates a rounded value. The text dynamic factor representing the timestamp of the t-th video frame; Indicates the preset base window size.

[0027] Preferably, the semantic similarity is obtained by:

[0028] For each video frame, the visual embedding vector is extracted through the pre-trained visual model, and the sequence of all visual embedding vectors is used as the visual feature vector;

[0029] Generate a contextual semantic vector for each text segment, and use the sequence of all contextual semantic vectors as the text segment feature sequence;

[0030] In the shot corresponding to the timestamp of each video frame, the mean cosine similarity of all elements at the same position in the text segment feature sequence and the visual feature sequence is calculated as the semantic similarity of the timestamp corresponding to each video frame.

[0031] Preferably, the obtaining of the first sparse matrix and the second sparse matrix is ​​specifically as follows:

[0032] For each video frame, traverse the speech segments within the first search window, and calculate the cosine similarity between the elements of the visual feature sequence and the elements of the audio feature sequence within the window as the elements of the first sparse matrix;

[0033] For each video frame, the text segments within its second search window are traversed, and the normalized Euclidean distances between the elements of the visual feature sequence within the window and the elements of the text segment feature sequence are used as elements of the second sparse matrix.

[0034] Preferably, the first optimal alignment path is obtained by applying an optimal path algorithm to the first sparse matrix; and the second optimal alignment path is obtained by applying an optimal path algorithm to the second sparse matrix.

[0035] Preferably, the first optimal path is specifically an optimal alignment path consisting of timestamp index pairs of video frames and voice segments; the second optimal path is specifically an optimal alignment path consisting of timestamp index pairs of video frames and text segments.

[0036] Preferably, the timestamps of the speech segments and text segments are aligned with the timestamps of the video frames based on the obtained optimal alignment path, specifically:

[0037] Based on the optimal alignment path, the timestamps of the speech segments and the text segments are mapped to the time axis of all video frames respectively;

[0038] If the timestamps of the speech segment and text fragment in the optimal alignment path completely match those of the video frame, the corresponding speech segment and text fragment are directly taken; if the timestamps do not match, new speech segments and text fragments are generated through linear interpolation to obtain speech segments and text fragments that are aligned with the timestamps of the video frame.

[0039] This application has at least the following beneficial effects:

[0040] This application uses the timestamp of the video frame as a benchmark. In the shot where the timestamp of each video frame is located, the similarity of adjacent video frames and the discreteness of the feature matrix elements corresponding to the voice segment are analyzed to determine the first search window. This helps to improve the time synchronization accuracy of the video frame and the voice segment, optimize the matching effect of multimodal data, and thus enhance the accuracy and efficiency of video content understanding. At the same time, the frequency and duration of the text segment in the shot where the timestamp of each video frame is located are analyzed to determine the second search window, more accurately capture the temporal alignment relationship between the text and the video frame, improve the accuracy of multimodal fusion, and make the matching between text information and visual content more natural and smooth. Based on the search window, the feature similarity of the voice segment and the text segment with the video frame in the corresponding window is analyzed to obtain a sparse matrix and obtain the optimal alignment path, which helps to accurately synchronize the temporal relationship between audio, text and video frames, reduce data redundancy, optimize computational efficiency, and thus provide a more accurate basis for multimodal information fusion and subsequent semantic analysis. Based on the obtained optimal alignment path, the timestamps of the voice segment and text segment are aligned with the video frame timestamp, and the aligned video frame, voice segment and text segment information are integrated through multimodal fusion technology. Multi-level semantic analysis can be achieved through deep learning models, which helps to improve the fusion accuracy of multimodal data, ensure semantic consistency between different information sources, enhance content understanding capabilities, and further improve data-driven decision support and application effects.

[0041] This application transforms the traditional dynamic time warping algorithm's globally intensive computation into a locally sparse computation through shot segmentation and dynamic search windows, significantly reducing time complexity. Furthermore, the shot boundary constraints and adaptive window design ensure precise alignment of cross-modal signals within a semantically coherent local range, maintaining the temporal correlation of multimodal signals while significantly reducing memory usage and computing resource consumption. This provides efficient and precise temporal alignment support for applications such as real-time advertising and preprocessing of massive video inventories. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of the video content understanding method of the multimodal advertising inventory intelligent matching system provided in this application;

[0043] Figure 2 A flow chart for obtaining the dynamic balance factor provided in this application;

[0044] Figure 3 This is a flowchart for obtaining text dynamic factors provided in this application. DETAILED DESCRIPTION

[0045] In the description of the embodiments of this application, words such as "exemplary," "or," and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "or," and "for example" is intended to present the relevant concepts in a concrete manner.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art in the art of this application. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0047] It should also be noted that the terms "first" and "second" in this application and its accompanying drawings are used to distinguish similar objects, rather than to describe a specific order or precedence. The methods disclosed in the embodiments of this application or the methods shown in the flowcharts include one or more steps for implementing the methods. Without departing from the scope of protection of this application, the order of execution of multiple steps can be interchanged with each other, and some steps can also be deleted.

[0048] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0049] This application proposes a video content understanding method for a multimodal advertising inventory intelligent matching system, which is applied in the field of video content understanding technology. Figure 1 , the method comprises the following steps:

[0050] S1: Extract video frames and segment shots of the original advertising video; extract audio signals from the original advertising video to obtain voice segments; and read text segments from the original advertising video.

[0051] In video content understanding tasks, raw video, as unstructured data, contains a mixture of visual images, audio signals, and text information. Lacking a unified time base and modal division, it cannot be directly used for cross-modal time series analysis. Therefore, multimodal signal analysis is first required to convert unstructured video into structured time series data that can be processed by computers.

[0052] S101: Visual modality processing, the specific steps are as follows:

[0053] (1) Video frame sequence parsing and timestamp extraction: The original video is traversed at a preset fixed frame rate, extracting each frame from the beginning to the end of the video. In this embodiment, it is set to 30fps, that is, one frame is extracted every 1 / 30 second. An absolute timestamp is generated for each extracted frame in milliseconds. For variable frame rate videos, the timestamps are synchronized through linear interpolation or the nearest neighbor method to ensure temporal continuity. The final result is a video frame sequence, in which each element is a video frame and its corresponding timestamp.

[0054] (2) Shot segmentation and scene boundary marking: Shot segmentation is performed based on histogram differences or deep learning models, and the start and end timestamps of each shot are recorded to generate shot-level time intervals (for example, shot 1 corresponds to timestamps [0ms, 2000ms], and shot 2 corresponds to [2001ms, 5000ms]). The shot segmentation sequence is obtained, in which each element is the start and end timestamps of a shot.

[0055] (3) Visual feature extraction: For the extracted video frames, feature vectors are extracted using a pre-trained visual model. In this embodiment, ResNet-18 pre-trained on ImageNet is used to scale the video frames to 224×224 pixels, and the visual embedding vector corresponding to the video frame is output. A visual feature sequence is obtained, in which each element is the visual embedding vector of the corresponding video frame.

[0056] S102: Perform audio mode processing. The specific steps are as follows:

[0057] (1) Voice activity detection and voice segmentation: extract a mono audio signal from the video in WAV format with a sampling rate of 16kHz, apply a bandpass filter to remove low-frequency noise and high-frequency interference. The frequency range of the bandpass filter used in this embodiment is 300Hz to 8kHz; use energy detection or machine learning models to detect the time period in which voice exists. In this embodiment, voice activity detection is performed by combining PyAudio and WebRTC VAD modules; merge continuous voice frames to generate voice segments with start and end timestamps, and filter out segments with too short duration. In this embodiment, segments less than 50ms are filtered; finally, a voice segment sequence is obtained, in which each element is a voice segment and its corresponding start and end timestamps.

[0058] (2) Speech segment feature extraction: Perform short-time Fourier transform on each speech segment to generate a spectrogram; convert the linear frequency into Mel frequency through a Mel filter bank (usually 40 to 80 filters), use the Mel spectrum as the underlying acoustic representation of the audio, and obtain an audio feature sequence, where each element is a Mel spectrum matrix of a speech segment, and the dimension is the number of frames. The number of Mel channels refers to the number of speech frames in a speech segment, and the number of Mel channels refers to the number of Mel filter banks.

[0059] S103: Perform text modal processing. The specific steps are as follows:

[0060] (1) Obtaining subtitle text segments: Read the subtitle file, split each subtitle segment by entry, obtain the start and end timestamps of the subtitle segment, and remove special symbols and repeated spaces.

[0061] (2) OCR text segment acquisition: Perform OCR on each video frame to identify the text content in the picture, sort the OCR text according to the timestamp of the video frame, and remove duplicate text.

[0062] (3) Fusion and feature generation of text fragment sequences: The subtitles and OCR text are merged and sorted by timestamp to generate a unified text fragment sequence, where each element is a text fragment and its corresponding start and end timestamps; each text fragment is embedded and encoded using a pre-trained language model to generate a contextual semantic vector, resulting in a text fragment feature sequence, where each element is a semantic embedding vector of a text fragment and its corresponding start and end timestamps. In this embodiment, the pre-trained language model used is BERT (Bidirectional Encoder Representations from Transformers), which is a well-known technology and is not described in detail in this application. In other embodiments, the RoBERTa (A Robustly Optimized BERT Pretraining Approach) model can also be used.

[0063] S2: Divide the video frames, voice segments, and text segments of corresponding timestamps based on the start and end time of each shot segmentation.

[0064] Traditional temporal alignment methods only model the timing of a single modality and fail to explicitly address cross-modal asynchrony. Alternatively, they sample features from each modality at fixed intervals for alignment, ignoring actual temporal misalignment and leading to semantic association errors. This solution, however, utilizes an improved DTW algorithm to align multimodal videos. Through dynamic elastic matching, it allows for local bending of the time axis, adaptively handles various asynchronous scenarios, and minimizes alignment accuracy errors, providing a more reliable temporal calibration foundation for intelligent matching of multimodal advertising inventory.

[0065] Shots are the basic semantic units of video content. Visual, audio, and text signals within a shot have strong spatiotemporal correlations, while shot switching is often accompanied by discontinuities in scene, action, or semantics. Limiting Dynamic Time Warping (DTW) alignment to a single shot avoids invalid cross-shot matches and effectively reduces the computational complexity of traditional DTW. This results in greater semantic consistency among multimodal signals within a shot, making it easier for dynamic time warping paths to converge to true temporal correspondences and avoiding distortion errors caused by semantic discontinuities across shots. Combining forced alignment of shot boundaries with dynamic search window restrictions further compresses invalid computational regions, optimizing computational efficiency while ensuring accuracy.

[0066] According to the start and end timestamps of each shot in the shot segmentation sequence, the elements in the video frame sequence, voice segment sequence, and text segment sequence are matched to the corresponding shot segments according to the timestamps, ensuring that each segment only contains multimodal signals within the same shot, such as the audio segment timestamp completely falls within a certain shot time interval, or the audio segment across shots is divided into independent sub-segments according to the shot boundaries, and finally the multimodal segments under each shot are obtained.

[0067] S3: Taking the video frame as the benchmark, in the shot corresponding to the timestamp of each video frame, analyze the similarity between adjacent video frames and the discreteness of the spectral information corresponding to the speech segment, determine the dynamic balance factor corresponding to the timestamp of each video frame, and adjust the preset benchmark window size corresponding to the timestamp of each video frame to obtain the first search window.

[0068] When calculating time series similarity, the dynamic search window defines an allowable horizontal offset range for each alignment point, ensuring that the alignment path extends only within a strip centered on the diagonal line. This application reduces the time complexity of traditional DTW by pruning invalid alignment paths within the dynamic search window, significantly reducing the computational effort while ensuring alignment rationality.

[0069] Because time series often exhibit local temporal dependencies, meaning that elements at the current moment only reasonably match elements at nearby moments, dynamic windows effectively eliminate meaningless matches over long distances by limiting the lateral offset range of the alignment path, reducing redundant computations in global searches. The window width is dynamically adjusted based on the characteristics of the sequence, ensuring that alignment paths that conform to temporal logic are retained within the permitted offset range, avoiding alignment distortion caused by excessive pruning.

[0070] This application uses video frames as a benchmark to align the timing of audio and text. Visual information plays a central role in ad contextual targeting because it provides semantic representation of the scene, and video frames, with their fixed frame rate and continuous timestamps, provide a stable temporal reference. Therefore, ad contextual targeting primarily relies on visual semantic representations, with audio and text serving as supplementary information.

[0071] When the video frames are aligned with the speech segments, the average value of the cosine similarities between all adjacent video frames in the shot where the timestamp of each video frame is located is obtained as the visual motion intensity of the timestamp corresponding to each video frame, which is used to reflect the dynamic degree of the picture. The range is 0~2. The larger the value, the more drastic the picture change. The standard deviation of each element in the audio feature sequence corresponding to the shot where the timestamp of each video frame is located is obtained and the mean is calculated to obtain the audio energy change rate of the timestamp corresponding to each video frame, which is used to reflect the dynamic range of the audio signal. The larger the value, the more drastic the audio energy fluctuation. The absolute value and sum of the difference between the visual motion intensity and the audio energy change rate corresponding to each video frame are calculated respectively, and the negative correlation mapping result of the sum value is positively fused with the absolute value of the difference to obtain the dynamic balance factor of the timestamp corresponding to each video frame.

[0072] In this embodiment, the absolute value of the difference is recorded as A, and the sum is recorded as B. Then the formula of the dynamic balance factor is: Where, It represents the preset parameter, which is set to 0.01 to prevent the denominator from being zero. It should be understood that the window expansion amplitude is adjusted by the difference between the visual and audio dynamic characteristics. When the difference between the visual motion intensity and the audio energy change rate is large, the dynamic balance factor approaches 1, and the window width increases significantly to adapt to the large span distortion of the single mode. When the two are close, the dynamic balance factor approaches 0, and the window width is smoothly adjusted based on the average dynamic characteristics. Among them, the flow chart for obtaining the dynamic balance factor is as follows Figure 2 shown.

[0073] The width of the first search window is calculated based on the visual motion intensity and the audio energy change rate. The formula is: ,in, represents the width of the first search window corresponding to the timestamp of the t-th video frame; t represents the timestamp corresponding to the t-th video frame, Indicates the reference window size, which is used to define the minimum allowable search range for temporal alignment to ensure basic alignment accuracy in static scenarios. Its value range is 30 to 100 frames. In this embodiment, the value is 50 frames. Represents the Sigmoid function, The value range of is mapped to 0~1 to avoid sudden changes in window width and ensure smooth transition; Indicates a rounded value. represents the visual motion intensity of the corresponding timestamp of the t-th video frame; represents the audio energy change rate of the corresponding timestamp of the t-th video frame; Indicates the dynamic balance factor corresponding to the timestamp of the t-th video frame.

[0074] It should be understood that by fusing the visual motion intensity and the audio energy change, the width of the search window is dynamically adjusted. When the picture switches quickly or the audio changes drastically, that is, the larger the value of the visual motion intensity or the audio energy change rate is, the width of the first search window is larger. The expansion allows for a wider range of time warping to capture true temporal correspondences in complex clips (such as slow motion and voiceovers). When the scene is static, the window width is close to the base width, reducing invalid search areas and lowering computational effort. A dynamic balance factor balances the dynamic differences between visual and audio, avoiding excessive window expansion caused by drastic changes in a single modality and finding the optimal balance between efficiency and accuracy. The sigmoid function ensures that the window width adjusts gradually with scene changes, avoiding frequent recalculations caused by minor fluctuations within the shot and improving algorithm stability.

[0075] S4: In the shot where the timestamp of each video frame is located, analyze the frequency of occurrence and duration of the text segment corresponding to the timestamp per unit time, confirm the text dynamic factor of the timestamp of each video frame, adjust the preset benchmark window size of the timestamp of each video frame, and obtain the second search window.

[0076] When aligning video frames with text segments, obtain the frequency of occurrence of the text segment corresponding to the timestamp in the shot corresponding to each video frame per unit time, as well as the duration of the text segment corresponding to the timestamp, and multiply the two together to obtain a first product; obtain the maximum frequency of occurrence and the maximum duration of the text segment corresponding to the timestamp in the text segment sequence, and multiply them to obtain a second product; the negative correlation mapping result of the second product is forwardly fused with the result of the first product as the text dynamic factor corresponding to the timestamp of each video frame. In this embodiment, the first product is denoted as C, the second product is denoted as D, and the formula form of the text dynamic factor is: Where, It represents the preset parameter, with a value of 0.01 to prevent the denominator from being zero. The flowchart for obtaining the text balance factor is as follows: Figure 3 shown.

[0077] Next, the width of the second search window is calculated based on the semantic relevance of the text, and the formula is: ,in: represents the width of the second search window corresponding to the timestamp of the t-th video frame; Represents the semantic similarity of the timestamp corresponding to the t-th video frame, that is, the mean cosine similarity of all elements in the same position of the text segment feature sequence and the visual feature sequence in the shot where the timestamp is located. The larger the value, the stronger the semantic association between the text and the picture content; Indicates the preset activation factor, ranging from 0 to 1. In this embodiment, the value is 0.5. Expand the window when reading to avoid invalid calculations caused by meaningless text; The text dynamic factor representing the timestamp of the t-th video frame; Indicates the preset base window size; represents the activation function, when When the result is less than 0, output 0; when the result is greater than 0, output ; The calculation of semantic similarity first requires L2 normalization of the semantic embedding vector of the text segment and the visual embedding vector of the video frame to eliminate the difference in modulus length so that cosine similarity only reflects the consistency of the vector direction. Then, a projection layer (such as a fully connected layer) is used to map the text and visual features to the same dimension, ensuring the validity of the cosine similarity calculation for corresponding vectors at the same position.

[0078] It should be understood that when the text and visual content are semantically strongly related, the second search window width Follow The super-threshold part is linearly expanded, allowing a wider range of time series distortion to adapt to the non-strict synchronization of text description and action, maintain the basic window in weakly related scenes, and limit the calculation range; high-frequency long text is Enlarge the window expansion range to cope with continuous picture changes corresponding to dense semantics, and limit window expansion for low-frequency short texts to avoid excessive searching.

[0079] It should be noted that since the dynamic search is performed within the shot clip, when the width of the dynamic search window exceeds the shot clip range, only the content within the shot clip is retained.

[0080] S5: Based on the first search window and the second search window, analyze the feature similarity between the speech segment and the text segment and the video frame in the corresponding window, obtain the first sparse matrix and the second sparse matrix, and obtain the first optimal alignment path and the second optimal alignment path.

[0081] For each video frame of each shot fragment in the video frame sequence, the corresponding first search window and second search window are calculated according to step S2 and step S3 respectively, the speech segment or text segment is mapped to the video frame time axis, and the corresponding speech segment index range and text segment index range in the window are determined.

[0082] To save storage space, this application uses a sparse matrix format to store only non-infinite similarity values ​​and avoid storing all zero elements. First, initialize the visual-audio matrix and the visual-text matrix to a sparse matrix structure, where the number of rows in the matrix is ​​the number of video frames, the number of columns is the number of speech segments or text fragments, and all elements are initially set to infinity, indicating that cross-window alignment is not allowed by default.

[0083] Then, for each video frame, the speech segments within the first search window are traversed, and the cosine similarity between the elements of the visual feature sequence and the elements of the audio feature sequence within the window is calculated, and the result is stored in the corresponding position of the first sparse matrix. For each video frame, the text segments within the second search window are traversed, and the normalized Euclidean distance between the elements of the visual feature sequence and the elements of the text segment feature sequence within the window is stored in the corresponding position of the second sparse matrix.

[0084] It is a well-known technology to use dynamic programming to search for the optimal path of the first sparse matrix and the second sparse matrix. The specific steps are not repeated here. Finally, the optimal alignment path consisting of the timestamp index pairs of the video frame and the voice segment and the video frame and the text segment is output, which satisfies the temporal continuity and maximizes the cumulative similarity.

[0085] S6: Based on the obtained optimal alignment path, the timestamps of the speech segments and text segments are aligned with the timestamps of the video frames. The timestamp-aligned video frames, speech segments, and text segments are integrated through multimodal fusion technology, and multi-level semantic analysis is achieved through a deep learning model.

[0086] Based on the optimal alignment path, the timestamps of the speech segment sequence and the text fragment sequence are mapped to the time axis of the video frame sequence. That is, a mapping is established for each point in the path. If the timestamps of the speech segment and text fragment completely match the video frame, the corresponding element in the sequence is directly taken; if the timestamps do not match, new elements are generated through linear interpolation. For speech segments and text fragments outside the optimal alignment path, the nearest boundary value is used to fill or discard them, and finally a synchronized multimodal sequence aligned with the video frame sequence timestamp is output.

[0087] Based on synchronized multimodal sequences, cross-modal semantic information is integrated through multimodal fusion techniques (such as Transformer and gating mechanisms). Visual features capture the subject, scene, and action, audio features analyze speech content, emotional rhythm, and ambient sound, and text features complement the semantic logic of subtitles and OCR text.

[0088] Deep learning models are then used to implement multi-level semantic analysis, including video theme classification, dynamic scene recognition, emotional tone determination, and target user profile mapping. Ultimately, these fine-grained semantic representations provide precise scene matching for contextual programmatic advertising targeting, ensuring ad relevance while improving user experience and delivery efficiency. Multimodal fusion technology and semantic analysis are well-known techniques, so the specific details and detailed steps will not be elaborated here.

[0089] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action, or may be implemented by a combination of dedicated hardware and computer instructions.

[0090] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A video content understanding method for a multimodal advertising inventory intelligent matching system, characterized in that: The method comprises the following steps: Extract video frames and segment shots from the original advertising video; extract audio signals from the original advertising video to obtain voice segments; read text segments from the original advertising video; Divide the video frames, voice segments, and text segments of corresponding timestamps based on the start and end time of each shot segmentation; Taking the video frame as a benchmark, in the shot corresponding to the timestamp of each video frame, the similarity between adjacent video frames and the discreteness of the spectral information corresponding to the speech segment are analyzed, a dynamic balance factor corresponding to the timestamp of each video frame is determined, and the preset benchmark window size corresponding to the timestamp of each video frame is adjusted to obtain a first search window; In the shot corresponding to the timestamp of each video frame, the occurrence frequency and duration of the text segment corresponding to the timestamp in unit time are analyzed to determine the text dynamic factor corresponding to the timestamp of each video frame, and the preset reference window size corresponding to the timestamp of each video frame is adjusted to obtain a second search window; Based on the first search window and the second search window, analyzing the feature similarity between the speech segment and the text segment and the video frame in the corresponding window, obtaining a first sparse matrix and a second sparse matrix, and obtaining a first optimal alignment path and a second optimal alignment path; Based on the obtained optimal alignment path, the timestamps of the speech segments and text segments are aligned with the timestamps of the video frames. The timestamp-aligned video frames, speech segments, and text segment information are integrated through multimodal fusion technology, and multi-level semantic analysis is achieved through a deep learning model.

2. The video content understanding method of the multimodal advertising inventory intelligent matching system according to claim 1, characterized in that: The dynamic balance factor of each video frame corresponding to the timestamp is specifically: Obtain the average value of the cosine similarities between all adjacent video frames in the shot where the timestamp of each video frame corresponds to as the visual motion intensity of the timestamp of each video frame; Extract the Mel spectrum matrix of each speech segment, and use the sequence composed of the Mel spectrum matrices of all speech segments as the audio feature sequence; Obtain the standard deviation of each element in the audio feature sequence corresponding to the shot at the timestamp of each video frame and calculate the mean to obtain the audio energy change rate of each video frame corresponding to the timestamp; The absolute value and sum of the difference between the visual motion intensity and the audio energy change rate corresponding to each video frame are calculated respectively, and the negative correlation mapping result of the sum value is positively fused with the absolute value of the difference to obtain the dynamic balance factor corresponding to the timestamp of each video frame.

3. The video content understanding method of the multimodal advertising inventory intelligent matching system according to claim 2, characterized in that: The preset reference window size corresponding to the timestamp of each video frame is adjusted to obtain the first search window. The specific formula is: ,in, represents the width of the first search window corresponding to the timestamp of the t-th video frame; t represents the timestamp corresponding to the t-th video frame, Indicates the preset base window size; Represents the Sigmoid function; Indicates a rounded value. represents the visual motion intensity of the corresponding timestamp of the t-th video frame; represents the audio energy change rate of the corresponding timestamp of the t-th video frame; Indicates the dynamic balance factor corresponding to the timestamp of the t-th video frame.

4. The video content understanding method of the multimodal advertising inventory intelligent matching system according to claim 1, characterized in that: The text dynamic factor for confirming the timestamp of each video frame is specifically: The sequence composed of all text segments is regarded as a text segment sequence; Obtaining the frequency of occurrence of the text segment corresponding to the timestamp in a unit time within the shot corresponding to the timestamp of each video frame, and the duration of the text segment corresponding to the timestamp, and multiplying the two to obtain a first product; Obtain the maximum occurrence frequency and maximum duration of the text segment corresponding to the timestamp in the text segment sequence, and multiply them to obtain a second product; A result of forward fusion of the negative correlation mapping result of the second product and the first product is used as a text dynamic factor corresponding to the timestamp of each video frame.

5. The video content understanding method of the multimodal advertising inventory intelligent matching system according to claim 2, characterized in that: The preset reference window size corresponding to the timestamp of each video frame is adjusted to obtain the second search window, specifically: In the shot corresponding to the timestamp of each video frame, the similarity of the feature information between the video frame and the text segment is analyzed to determine the semantic similarity of the timestamp corresponding to each video frame; The width of the second search window corresponding to the timestamp of the t-th video frame is recorded as , its formula form is: ,in: express; Indicates the semantic similarity of the timestamp corresponding to the t-th video frame; Indicates the preset activation factor; represents the activation function; Indicates a rounded value. The text dynamic factor representing the timestamp of the t-th video frame; Indicates the preset base window size.

6. The video content understanding method of the multimodal advertising inventory intelligent matching system according to claim 5, characterized in that: The specific acquisition process of the semantic similarity is as follows: For each video frame, the visual embedding vector is extracted through the pre-trained visual model, and the sequence of all visual embedding vectors is used as the visual feature vector; Generate a contextual semantic vector for each text segment, and use the sequence of all contextual semantic vectors as the text segment feature sequence; In the shot corresponding to the timestamp of each video frame, the mean cosine similarity of all elements at the same position in the text segment feature sequence and the visual feature sequence is calculated as the semantic similarity of the timestamp corresponding to each video frame.

7. The video content understanding method of the multimodal advertising inventory intelligent matching system according to claim 6, characterized in that: The first sparse matrix and the second sparse matrix are obtained as follows: For each video frame, traverse the speech segments within the first search window, and calculate the cosine similarity between the elements of the visual feature sequence and the elements of the audio feature sequence within the window as the elements of the first sparse matrix; For each video frame, the text segments within its second search window are traversed, and the normalized Euclidean distances between the elements of the visual feature sequence within the window and the elements of the text segment feature sequence are used as elements of the second sparse matrix.

8. The video content understanding method of the multimodal advertising inventory intelligent matching system according to claim 1, characterized in that: The first optimal alignment path is obtained by applying an optimal path algorithm to the first sparse matrix; the second optimal alignment path is obtained by applying an optimal path algorithm to the second sparse matrix.

9. The video content understanding method of the multimodal advertising inventory intelligent matching system according to claim 1, characterized in that: The first optimal path is specifically an optimal alignment path consisting of timestamp index pairs of video frames and voice segments; the second optimal path is specifically an optimal alignment path consisting of timestamp index pairs of video frames and text segments.

10. The video content understanding method of the multimodal advertising inventory intelligent matching system according to claim 1, characterized in that: The method of aligning the timestamps of the speech segments and text segments with the timestamps of the video frames based on the obtained optimal alignment path is as follows: Based on the optimal alignment path, the timestamps of the speech segments and the text segments are mapped to the time axis of all video frames respectively; If the timestamps of the speech segment and text fragment in the optimal alignment path completely match those of the video frame, the corresponding speech segment and text fragment are directly taken; if the timestamps do not match, new speech segments and text fragments are generated through linear interpolation to obtain speech segments and text fragments that are aligned with the timestamps of the video frame.

Images