Television broadcast quality monitoring system based on multi-modal model

Abstract

The invention discloses a television broadcast quality monitoring system based on a multi-modal model, and mainly relates to the technical field of multi-modal information fusion. Comprising a data acquisition module used for acquiring multi-modal data in television broadcast signals in real time; the data preprocessing module is used for preprocessing the collected multi-modal data; the multi-modal fusion module is used for performing data alignment, feature extraction and feature fusion on the preprocessed multi-modal data; the quality evaluation module is used for evaluating the television broadcasting quality based on the fused multi-mode characteristics and judging whether broadcasting faults exist or not; and the feedback control module is used for generating a corresponding control instruction according to the quality evaluation result and alarming television broadcasting. The method has the beneficial effects that the problems of low monitoring accuracy and high false alarm rate of single-modal data in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal information fusion, and in particular to a television broadcast quality monitoring system based on a multimodal model. Background Art

[0002] With the continuous development of television media, the broadcast quality of television programs has attracted more and more attention. Existing television broadcast quality monitoring systems mostly rely on single-modal data (such as audio and images) for quality assessment. However, television signals contain data in multiple modes (such as video, audio, text subtitles, etc.), and there are complex correlations between these data. The main technical problems of single-modal data analysis are as follows: (1) Inaccurate fault detection: Existing unimodal systems may ignore problems that are obvious in other modalities. For example, the video may look normal, but the audio may be noisy or the subtitles may be out of sync. (2) High false alarm rate: Systems that rely only on single-modality data may raise an alarm when there is no actual fault. For example, a sudden change in audio may be mistakenly marked as a fault when there is a normal sound effect in the scene. (3) Lack of contextual understanding: Single-modal data cannot provide the contextual information of multimodal data. For example, subtitle text can help understand whether the voice quality in the audio matches the lip movements in the video. (4) Insufficient ability to handle complex scenarios: Abnormal TV broadcast quality in the real world may involve complex problems in multiple aspects, such as picture freeze, freeze or frame skipping, audio loss, audio interruption, audio and video asynchrony, and incorrect subtitles. Single-mode systems have difficulty handling such complex failures.

[0003] In view of the above problems, how to design a television broadcast quality monitoring system based on a multimodal model is an issue that needs to be urgently addressed in this field. Summary of the invention

[0004] The purpose of the present invention is to provide a television broadcast quality monitoring system based on a multimodal model, which solves the problems of low accuracy and high false alarm rate of single modal data monitoring in the prior art.

[0005] In order to achieve the above object, the present invention is implemented through the following technical solutions: A television broadcast quality monitoring system based on a multimodal model is provided, comprising: A data acquisition module is used to collect multimodal data from television broadcast signals in real time; A data preprocessing module, used to preprocess the collected multimodal data; The multimodal fusion module is used to perform data alignment, feature extraction and feature fusion on the preprocessed multimodal data; The quality assessment module is used to assess the quality of TV broadcasting based on the fused multimodal features and determine whether there are any broadcasting failures; The feedback control module is used to generate corresponding control instructions according to the quality evaluation results and issue an alarm for TV broadcasting.

[0006] Preferably, the data acquisition module collects multimodal data in the process of television broadcasting in real time through a television signal receiver and a web crawler, including: video data, audio data, text subtitle data, closed subtitle data and EPG information.

[0007] Preferably, the data preprocessing module performs the following steps: Extract key frames and scene transition frames from the video; De-duplication, denoising and enhancement pre-processing operations are performed on the extracted frame images to remove noise and interference factors; Enhance and reduce noise on audio signals to improve audio quality; Clean and segment text subtitles to extract useful information.

[0008] Preferably, the multimodal fusion module includes: Modal alignment unit, feature extraction unit and feature fusion unit; The modality alignment unit is used to align the timestamps of data of different modalities, and specifically performs the following process: Alignment of video data and audio data, alignment of text data, unified time reference, weight assignment of text data, and generation of unified time series.

[0009] Preferably, the timestamp of the video frame is set to , the timestamp of the audio is ,in, Represents the index of the video frame, Indicates the index of the audio frame; the alignment of the video data and the audio data, the specific process is: Calculate the time offset: ; Temporally adjust all video frames: ; Find audio frames: .

[0010] Preferably, the alignment of the text data includes: text with a timestamp and text without a timestamp; The specific execution process of the text with a timestamp is as follows: The text segment The time interval and Video frame alignment, if , then the video frame Corresponding text segment ; The specific execution process of the text without timestamp is as follows: Use Dynamic Time Warping to align text: ; Among them, when hour, ,otherwise , 0 means the text matches, 1 means the text does not match; represents the index of manually entered text, Indicates the index of the recognized text.

[0011] Preferably, the unified time reference is specifically implemented as follows: Select the time step of video frames As a baseline, audio frames are interpolated or downsampled to match the video frame rate: ; in, is the audio sampling rate, is the frame rate of the video, is the index of the video frame, The index of the audio frame.

[0012] Preferably, the weight distribution of the text data is specifically performed as follows: For text segments covering multiple video frames , assign weights : ; Or classify the video frame features and normalize the weights: ; in, , The indices of the start and end of the video frames covered by the text segment, respectively.

[0013] Preferably, the feature extraction unit is used to extract feature representations of video data, audio data and text data respectively; The feature extraction of the video data specifically performs the following process: Set the video frame sequence to , where each It is Frame images, using convolutional neural networks to extract visual features of each frame: ; The visual features extracted from each frame are input into the recurrent neural network to capture the timing information between video frames: ; The feature extraction of the audio data specifically performs the following process: Set the audio signal to , where each It is a sub-frame of audio, generating a Mel spectrum graph , use one-dimensional CNN to extract local features of the spectrogram: ; The local features of the extracted spectrogram are input into the RNN to extract the temporal features of the audio: ; The feature extraction of text data specifically performs the following process: Set the text data to , where each is a text segment, and the pre-trained language model BERT is used to extract the semantic features of each text segment: ; Select BERT's [CLS] token as the representation of the entire text segment and average all word embeddings: .

[0014] Preferably, the feature fusion unit is used to fuse the multimodal features, specifically performing the following process: Use the ReLU model to map the dimensions of the features of each modality so that they have the same dimension and map the video features. Specifically: ; Map the audio features, specifically: ; Map the text features, specifically: ; in, , , are learnable weights, , , is the bias term; The features of all modalities are concatenated, and the weight of each modality is calculated through a fully connected layer; the calculation method for concatenating the features of all modalities is: ; The weight of each modality is calculated as follows: ; in, is the weight matrix, is the bias term, is the modal weight vector; The modal features are weighted and summed according to the weights to obtain the fusion features: ; The fusion features of each time step Input into RNN or Transformer for sequence-level processing to capture timing information.

[0015] Preferably, the quality assessment module includes: A fault detection unit is used to detect faults in television broadcasting based on multimodal features, specifically: using an anomaly detection algorithm to analyze the fused multimodal features to determine whether there is an anomaly; A fault classification unit is used to classify the detected faults and determine the fault type and severity, specifically: using a classifier to classify the fault type; The quality scoring unit is used to comprehensively evaluate the quality of television broadcasting based on the results of fault detection and classification using a scoring model and to generate a detailed quality evaluation report.

[0016] Compared with the prior art, the beneficial effects of the present invention are: (1) Multimodal data fusion: By integrating data from multiple modalities such as video, audio, and text, the complementarity between the modalities can be fully utilized to improve the accuracy and robustness of television broadcast quality monitoring; (2) Fault detection and classification: Fault detection and classification based on multimodal features can more accurately identify various faults in television broadcasting and classify them; (3) Real-time monitoring and feedback: Real-time monitoring of TV broadcast quality, and generation of corresponding control instructions based on the monitoring results, to achieve real-time adjustment and alarm of TV broadcast, and ensure broadcast quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a structural block diagram of a television broadcast quality monitoring system based on a multimodal model provided by an embodiment of the present invention; Figure 2 is a flow chart of a television broadcast quality monitoring system based on a multimodal model provided by an embodiment of the present invention; Figure 3 It is a video preprocessing flow chart of a television broadcast quality monitoring system based on a multimodal model provided by an embodiment of the present invention; Figure 4 The present invention provides a flowchart of text preprocessing of a television broadcast quality monitoring system based on a multimodal model. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the application equally.

[0019] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention and should not be understood as limitations on the present invention.

[0020] Example: like Figure 1 As shown, this embodiment provides a television broadcast quality monitoring system based on a multimodal model, including: Data acquisition module, used for real-time acquisition of multi-modal data from television broadcast signals; A data preprocessing module, used for preprocessing the collected multimodal data; Multimodal fusion module, used to perform data alignment, feature extraction and feature fusion on the preprocessed multimodal data; The quality assessment module is used to assess the quality of TV broadcasting based on the fused multimodal features and determine whether there are any broadcasting failures; The feedback control module is used to generate corresponding control instructions according to the quality evaluation results and issue an alarm for TV broadcasting.

[0021] Based on the above modules, the monitoring process performed by the monitoring system of this embodiment is as follows: Figure 2 shown.

[0022] 1. The data acquisition module includes: collecting multimodal data in the TV broadcast process in real time through means such as TV signal receivers and web crawlers, including video, audio, text subtitles, closed subtitles, EPG information, etc. In this embodiment, the specific acquisition process is as follows: Video data acquisition: Use a TV signal receiver to collect TV program video data from satellite signals in real time, with a resolution of 1920x1080 and a frame rate of 30fps; Audio data acquisition: synchronously acquire audio data of TV programs, with a sampling rate of 44.1kHz and stereo; Text data collection: The subtitle data of TV programs is obtained through web crawlers in the form of SRT files, which contain timestamps and corresponding text content; Sample data: Video data: The captured video clip is 10 seconds long and contains 300 frames.

[0023] Audio data: The collected audio clip is 10 seconds long and contains 441,000 sampling points.

[0024] Text data: The collected subtitle data contains 2 subtitle segments, corresponding to the time intervals [0.0, 5.0] and [5.0, 10.0] seconds respectively.

[0025] 2. Data preprocessing module, used to preprocess the collected multimodal source data, including: (1) Extract key frames and scene transition frames from the video; (2) Perform preprocessing operations such as deduplication, denoising, and enhancement on the extracted frame images to remove noise and interference factors; (3) Enhance and reduce the noise of audio signals to improve audio quality; (4) Clean and segment text subtitles to extract useful information; In this implementation, the preprocessing process is: like Figure 3 As shown in the figure, the video preprocessing process is: Key frame extraction: Use OpenCV library or FFmpeg to extract one key frame per second; Image enhancement: Apply image enhancement algorithms (such as histogram equalization) to remove noise and improve image clarity; The audio preprocessing process is: Noise reduction: Use Audacity software to perform audio noise reduction and set the noise reduction intensity to 20dB; like Figure 4 As shown in the figure, the text preprocessing process is: Text cleaning: remove special characters and spaces in subtitles and unify text formats; Word segmentation: Use Jieba word segmentation tool to perform word segmentation on Chinese subtitles.

[0026] 3. The multimodal fusion module includes: a modal alignment unit, a feature extraction unit and a feature fusion unit. The modal alignment unit is used to align the timestamps of different modal data to ensure the consistency of multimodal data in the time dimension. The details are as follows: 1) Video and audio alignment: The timestamp of the video frame is , the timestamp of the audio is ,in Represents the index of the video frame, Represents the index of the audio frame, which is aligned by the following steps: (1) Calculate the time offset: ; (2) Temporally adjust all video frames: ; (3) Find the closest audio frame: ; Using the data collected by the data acquisition module in this embodiment, after calculation, the video frame timestamp is = [0.0, 0.033, 0.066, ..., 10.0] seconds; The audio frame timestamp is = [0.0, 0.022, 0.044, ..., 10.0] seconds; Calculating time offset , and interpolate the video frames to match the audio frame rate; 2) Text data alignment Case 1: Text with timestamps (such as subtitles) The text segment The time interval and Video frame alignment, if , then the video frame Corresponding text segment ; Case 2: Text without timestamp (such as manually entered narration) Text alignment using Dynamic Time Warping (DTW): ; Among them, when hour, ,otherwise , 0 means the text matches, 1 means the text does not match; represents the index of manually entered text, Indicates the index of the recognized text; 3) Unified time base Select the time step of video frames As a unified baseline, interpolate or downsample the audio frames to match the video frame rate: ; in, The sampling rate of the audio (unit: Hz, such as 44.1kHz); The frame rate of the video (unit: fps, such as 30fps); The index of the video frame (such as frame 0, frame 1, etc.); is the index of the audio frame (the index of the aligned audio data); 4) Text weight distribution For text segments covering multiple video frames , assign weights : ; Or classify according to video frame features, ensuring weight normalization: ; in, , The starting and ending indexes of the video frames covered by the text segment; 5) Final alignment result Generate a unified time series with each time step containing the corresponding video frame, audio frame, and text segment: { "timestep":0, "video_frame":frame_0, "audio_frame":audio_0, "text_segment":text_0, }, { "timestep":1, "video_frame":frame_1, "audio_frame":audio_1, "text_segment":text_1, }, The feature extraction unit is used to extract feature representations of video, audio and text data respectively: 1) Video feature extraction: Convolutional neural network (CNN) is used to extract visual features of video frames, and recurrent neural network is used to extract temporal features between video frames: Assume that the video frame sequence is , where each It is Frame image: ①Use convolutional neural network (CNN) to extract visual features of each frame: ; ② Input these features into a recurrent neural network (RNN) to capture the timing information between video frames: ; 2) Audio feature extraction: The acoustic features of the audio are extracted by combining the spectrogram with the recurrent neural network (RNN). Assuming that the audio signal is , where each It is a sub-frame of audio: ① Generate Mel spectrum , and then use one-dimensional CNN to extract local features of the spectrogram: ; ② Input these features into RNN to extract the timing features of the audio: ; 3) Text feature extraction: Use a pre-trained language model (such as BERT) to extract the semantic features of the text. Assume that the text data is , where each is a text segment: ① Use a pre-trained language model such as BERT to extract semantic features of each text segment: ; ② Select BERT's [CLS] token as the representation of the entire text segment and average all word embeddings: ; The feature fusion unit is used to fuse multimodal features: 1) Modal feature dimension mapping is to use the ReLU model to dimensionally map the features of each modality so that they have the same dimension: ① Video feature mapping: ; ②Audio feature mapping: ; ③Text feature mapping: ; in, , , are learnable weights, , , is the bias term; 2) Calculate the attention weights, specifically: concatenate the features of all modalities and calculate the weight of each modality through a fully connected layer, including: ①Splicing features ; ②Calculate weight ; in, is the weight matrix, is the bias term, is the modal weight vector; 3) Fusion features: Specifically, weighted summation of modal features is performed according to weights to obtain fusion features: ; 4) Time series level processing, specifically: input the fused features of each time step into RNN or Transformer, perform sequence-level processing, and capture timing information.

[0027] 4. Quality assessment modules include: (1) Fault detection unit: It is used to detect faults in TV broadcasting based on multimodal features, such as video freeze, audio distortion, subtitle confusion, etc. Specifically, it uses anomaly detection algorithms (such as isolation forest, autoencoder, etc.) to analyze the fused multimodal features to determine whether there are anomalies; (2) Fault classification unit: Used to classify detected faults and determine the fault type and severity; specifically: using a classifier (such as a support vector machine, deep neural network, etc.) to classify the fault type; (3) Quality scoring unit: The quality scoring unit uses a scoring model to comprehensively evaluate the quality of TV broadcasts based on the results of fault detection and classification, and generates a detailed quality assessment report that includes the scoring results and fault details.

[0028] 5. Feedback control module: Based on the quality assessment results, control instructions are generated, alarm information is sent to the monitoring center, and fault details are recorded.

[0029] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A television broadcast quality monitoring system based on a multimodal model, characterized in that: include: Data acquisition module, used for real-time acquisition of multi-modal data from television broadcast signals; A data preprocessing module, used for preprocessing the collected multimodal data; Multimodal fusion module, used to perform data alignment, feature extraction and feature fusion on the preprocessed multimodal data; The quality assessment module is used to assess the quality of TV broadcasting based on the fused multimodal features and determine whether there are any broadcasting failures; The feedback control module is used to generate corresponding control instructions according to the quality evaluation results and issue an alarm for TV broadcasting.

2. A television broadcast quality monitoring system based on a multimodal model according to claim 1, characterized in that: The data acquisition module collects multimodal data in the process of television broadcasting in real time through a television signal receiver and a web crawler, including: video data, audio data, text subtitle data, closed subtitle data and EPG information.

3. The TV broadcast quality monitoring system based on a multimodal model according to claim 1, characterized in that: The data preprocessing module performs the following steps: Extract key frames and scene transition frames from the video; De-duplication, denoising and enhancement pre-processing operations are performed on the extracted frame images to remove noise and interference factors; Enhance and reduce noise on audio signals to improve audio quality; Clean and segment text subtitles to extract useful information.

4. A television broadcast quality monitoring system based on a multimodal model according to claim 1, characterized in that: The multimodal fusion module comprises: Modal alignment unit, feature extraction unit and feature fusion unit; The modality alignment unit is used to align the timestamps of data of different modalities, and specifically performs the following process: Alignment of video data and audio data, alignment of text data, unified time reference, weight assignment of text data, and generation of unified time series.

5. A television broadcast quality monitoring system based on a multimodal model according to claim 4, characterized in that: Set the timestamp of the video frame to , the timestamp of the audio is ,in, Represents the index of the video frame, Indicates the index of the audio frame; the alignment of the video data and the audio data, the specific process is: Calculate the time offset: ； Temporally adjust all video frames: ； Find audio frames: 。 6. A television broadcast quality monitoring system based on a multimodal model according to claim 4, characterized in that: The alignment of the text data includes: text with a timestamp and text without a timestamp; The specific execution process of the text with a timestamp is as follows: The text segment The time interval and Video frame alignment, if , then the video frame Corresponding text segment ; The specific execution process of the text without timestamp is as follows: Use Dynamic Time Warping to align text: ； Among them, when hour, ,otherwise , 0 means the text matches, 1 means the text does not match; represents the index of manually entered text, Indicates the index of the recognized text.

7. A television broadcast quality monitoring system based on a multimodal model according to claim 4, characterized in that: The specific implementation process of the unified time base is as follows: Select the time step of video frames As a baseline, audio frames are interpolated or downsampled to match the video frame rate: ； in, is the audio sampling rate, is the frame rate of the video, is the index of the video frame, The index of the audio frame.

8. The TV broadcast quality monitoring system based on multimodal model according to claim 4, characterized in that: The specific implementation process of the weight distribution of the text data is as follows: For text segments covering multiple video frames , assign weights : ； Or classify the video frame features and normalize the weights: ； in, , The indices of the video frames covered by the text segment starting and ending, respectively.

9. A television broadcast quality monitoring system based on a multimodal model according to claim 4, characterized in that: The feature extraction unit is used to extract feature representations of video data, audio data and text data respectively; The feature extraction of the video data specifically performs the following process: Set the video frame sequence to , where each It is Frame images, using convolutional neural networks to extract visual features of each frame: ； The visual features of each frame are extracted and input into the recurrent neural network to capture the timing information between video frames: ； Extract the features of audio data and perform the following process: Set the audio signal to , where each It is a sub-frame of audio, generating a Mel spectrum graph , use one-dimensional CNN to extract local features of the spectrogram: ； The local features of the extracted spectrogram are input into the RNN to extract the temporal features of the audio: ； Extract the features of text data and perform the following process: Set the text data to , where each is a text segment, and the pre-trained language model BERT is used to extract the semantic features of each text segment: ； Select BERT's [CLS] token as the representation of the entire text segment and average all word embeddings: 。 10. A television broadcast quality monitoring system based on a multimodal model according to claim 4, characterized in that: The feature fusion unit is used to fuse multimodal features, and specifically performs the following process: Use the ReLU model to map the dimensions of the features of each modality so that they have the same dimension and map the video features. Specifically: ； Map the audio features, specifically: ； Map the text features, specifically: ； in, , , are learnable weights, , , is the bias term; The features of all modalities are concatenated, and the weight of each modality is calculated through a fully connected layer; the calculation method for concatenating the features of all modalities is: ； The weight of each modality is calculated as follows: ； in, is the weight matrix, is the bias term, is the modal weight vector; The modal features are weighted and summed according to the weights to obtain the fusion features: ； The fusion features of each time step Input into RNN or Transformer for sequence-level processing to capture timing information.

11. A television broadcast quality monitoring system based on a multimodal model according to claim 1, characterized in that: The quality assessment module includes: A fault detection unit is used to detect faults in television broadcasting based on multimodal features, specifically: using an anomaly detection algorithm to analyze the fused multimodal features to determine whether there is an anomaly; A fault classification unit is used to classify the detected faults and determine the fault type and severity, specifically: using a classifier to classify the fault type; The quality scoring unit is used to comprehensively evaluate the quality of television broadcasting based on the results of fault detection and classification using a scoring model and to generate a detailed quality evaluation report.

Images