Method, system, device and medium for video quality assessment of multi-modal parallel analysis

Abstract

The present disclosure belongs to the technical field of video quality evaluation, and particularly relates to a video quality evaluation method, system, device and medium based on multi-modal parallel analysis, which comprises the following steps: receiving a course video to be audited, loading user preset audit parameter configuration; unpacking the course video to be audited to obtain an initial video coding stream and an initial audio coding stream, simultaneously performing audio track integrity checking, terminating the audio analysis process based on the checking result, only performing video analysis, or simultaneously performing video and audio analysis audit processes; pre-processing the initial video coding stream and the initial audio coding stream; based on an asynchronous parallel architecture, performing multiple audit tasks on the standard image frame sequence and the standard audio data in parallel to obtain multiple output results; converging the multiple output results, performing cross-modal space-time alignment and fusion deduplication processing to obtain a processing result; and generating a structured audit report according to the processing result. The present application improves the efficiency and accuracy of video quality evaluation.

Description

Technical Field

[0001] This disclosure belongs to the field of video quality assessment technology, and in particular relates to video quality assessment methods, systems, devices and media using multimodal parallel analysis. Background Technology

[0002] With the deepening of digital transformation in education, online course resources have evolved from simple auxiliary teaching tools into digital hubs carrying core teaching content. Faced with the massive increase in course video data, ensuring the political security, compliance, and high-quality audiovisual experience of teaching content has become a major challenge for online education platforms.

[0003] Patent application number 202211665088.X discloses a video image content review method and terminal, relating to the field of content review. It divides video image content review into multiple sub-tasks for identifying specific violation types. Based on the characteristics of each sub-task, the recognition algorithm used is set to either object detection or multi-label classification. The model is trained using multi-task learning based on the set sub-tasks. The hybrid object detection and multi-label classification neural network model includes an object detection module, a multi-label classification module, and a shared backbone network. The video image is input into the backbone network for extracting image features to extract various dimensions of image features. These image features are then input into the object detection module and the multi-label classification module to determine whether the video image violates regulations and, if so, what type of violation it is. This method is fast, requires few computational resources, has high computational accuracy, and exhibits better generalization ability of the network model.

[0004] Current technologies generally lack automated detection modules for encoding corruption and audio signal anomalies. This deficiency means that videos that are compliant in content but of poor technical quality may be mistakenly released, severely impacting the learning experience for end users. Summary of the Invention

[0005] To address the aforementioned issues, this disclosure provides a video quality assessment method, system, device, and medium based on multimodal parallel analysis. It employs an asynchronous parallel architecture to process video and audio modal data simultaneously, enabling multi-dimensional quality testing of course videos and concurrent review of compliance-sensitive content. This effectively improves review efficiency and accuracy, ensuring the content compliance and technical quality of online course videos.

[0006] Firstly, this disclosure provides a video quality assessment method based on multimodal parallel analysis. The method includes, Receive course videos awaiting review and load user-preset review parameter configurations; The course video to be reviewed is decapsulated to obtain the initial video encoding stream and the initial audio encoding stream. At the same time, the audio track integrity verification is performed. Based on the verification result, the audio analysis process is terminated and only video analysis is performed, or the video and audio analysis review process is performed simultaneously. The initial video encoded stream and the initial audio encoded stream are preprocessed to generate a standard image frame sequence and standard audio data; Based on an asynchronous parallel architecture, the standard image frame sequence and the standard audio data are concurrently executed for multi-channel review tasks to obtain multiple output results; The multi-output results are aggregated, and cross-modal spatiotemporal alignment and fusion deduplication are performed to obtain the processing result; A structured audit report is generated based on the processing results.

[0007] Furthermore, The course videos to be reviewed are decapsulated to obtain initial video encoded streams and initial audio encoded streams, specifically including: The video of the course to be reviewed is opened using a multimedia processing tool, its container format is identified, and the format information is read. The media stream contained in the course video to be reviewed is detected and parsed to obtain the video stream index, audio stream index and their respective encoding parameter information; Based on the obtained video stream index and audio stream index, compressed data packets are read from the container in a loop, and video data packet sequences and audio data packet sequences are extracted according to the stream type to which the data packets belong. The video data packet sequence is used as the initial video encoded stream, and the audio data packet sequence is used as the initial audio encoded stream.

[0008] Furthermore, Perform audio track integrity verification. Based on the verification results, terminate the audio analysis process and perform only video analysis, or perform both video and audio analysis review processes simultaneously, specifically including: Feature extraction is performed on the initial audio encoded stream to obtain audio features; The audio features are compared with preset audio features, and the completeness of the audio track is determined based on the comparison results. When the audio track is complete, the video and audio analysis review process is executed simultaneously; When the audio track is incomplete, the audio analysis process is terminated, and only video analysis is performed. The audio stream missing anomaly is marked in the structured audit report.

[0009] Furthermore, Preprocessing the initial video encoded stream and the initial audio encoded stream specifically includes: The initial audio encoded stream is transcoded into an uncompressed audio format with a uniform sampling rate to obtain standard audio data; The initial video encoded stream is divided into several processing units according to a preset time length to obtain several segments; For each slice, perform temporal downsampling and frame extraction to obtain several extracted frame images. Convert the extracted frame images into the standard YUV color space to construct a standard image frame sequence.

[0010] Furthermore, Based on an asynchronous parallel architecture, multiple auditing tasks are performed concurrently on the standard image frame sequence and the standard audio data to obtain multiple output results, specifically including: A no-reference video quality assessment is performed on the standard image frame sequence, the actual multidimensional quality index data is calculated, the actual multidimensional quality index data is compared with a preset threshold, anomalies are judged based on the comparison results and the time period of quality anomalies is recorded, and the anomaly is used as the first output result. Text analysis is performed on the standard image frame sequence to extract the actual text content. The actual text content is then matched with a preset sensitive word library using multiple modes. Based on the matching results, the first sensitive content and its corresponding sensitive time coordinates are recorded and used as the second output result. The standard audio data is transcribed into a timestamped text stream. Sensitive words are retrieved from the text stream based on a preset sensitive word database. The second sensitive content and sensitive time period are recorded based on the retrieval results and used as the third output result. The standard audio data is subjected to signal feature analysis. Based on the analysis results, silence, sound interruption and low volume abnormalities are identified. The audio abnormality type and audio abnormality time period are recorded based on the identification results and used as the fourth output result.

[0011] Furthermore, The multiple output results are aggregated, and cross-modal spatiotemporal alignment and fusion deduplication processing are performed to obtain the processing result, which specifically includes: The first output result, the second output result, the third output result, and the fourth output result are combined, and the quality abnormal time period, the sensitive time coordinate, the sensitive time period, and the audio abnormal time period are spatiotemporally aligned. For the second output result and the third output result, perform cross-modal deduplication processing.

[0012] Furthermore, Perform no-reference video quality assessment on the standard image frame sequence and calculate actual multidimensional quality index data, specifically including: Extract the statistical features of each frame in the standard image frame sequence; The statistical features are input into a pre-trained no-reference video quality assessment model to obtain actual multidimensional quality index data.

[0013] Secondly, based on the same inventive concept, this disclosure provides a video quality assessment system for multimodal parallel analysis, the system comprising: The receiving module is used to receive course videos to be reviewed and load user-preset review parameter configurations. The unpacking and verification module is used to unpack the course video to be reviewed, obtain the initial video encoding stream and the initial audio encoding stream, and at the same time, perform audio track integrity verification. Based on the verification result, the audio analysis process is terminated and only video analysis is performed, or the video and audio analysis review process is performed simultaneously. The preprocessing module is used to preprocess the initial video encoded stream and the initial audio encoded stream to generate standard image frame sequences and standard audio data; The task execution module is used to concurrently execute multiple audit tasks on the standard image frame sequence and the standard audio data based on an asynchronous parallel architecture, and obtain multiple output results; The processing module is used to aggregate the multi-channel output results, perform cross-modal spatiotemporal alignment and fusion deduplication processing, and obtain the processing result; The report generation module is used to generate a structured audit report based on the processing results.

[0014] Thirdly, this disclosure also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of any of the multimodal parallel analysis video quality assessment methods described above.

[0015] Fourthly, this disclosure also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the multimodal parallel analysis video quality assessment methods described above.

[0016] Compared with the prior art, this disclosure has the following advantages: 1. This application utilizes audio track integrity verification to automatically identify missing or damaged audio and skip audio analysis when audio is unavailable, avoiding unnecessary calculations and enhancing processing efficiency. Employing an asynchronous parallel architecture to concurrently execute multiple review tasks, it can simultaneously process multiple dimensions such as video quality, video text, audio content, and audio signal anomalies, significantly reducing review time. Cross-modal spatiotemporal alignment and fusion deduplication integrate results from different analysis pathways, eliminating time deviations and duplicate alarms, and improving the accuracy and consistency of review results.

[0017] 2. This application achieves broad compatibility with course videos of different encapsulation formats by identifying container formats, parsing media streams, and extracting video and audio data packet sequences respectively, ensuring accurate acquisition of the initial encoded stream and laying a reliable foundation for subsequent processing.

[0018] 3. This application transforms the initial audio into an uncompressed format with a uniform sampling rate, while simultaneously downsampling and converting the video into segments and color spaces, thus forming standardized input data that can adapt to the input format requirements of different analysis tasks and ensure the stable execution of subsequent multi-channel review tasks.

[0019] 4. This application achieves multi-dimensional and comprehensive review of course videos by clearly defining four parallel tasks: video quality assessment, video text sensitive content detection, audio transcription sensitive content detection, and audio signal anomaly detection. Each task independently generates output results with time coordinates, enabling precise location of the time point where the problem occurred, providing refined input for cross-modal fusion. The parallel pipeline of each task significantly improves the overall throughput.

[0020] Other features and advantages of this disclosure will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the disclosure. The objects and other advantages of this disclosure may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a video quality assessment method based on multimodal parallel analysis according to an embodiment of the present disclosure is shown. Figure 2A flowchart illustrating a video quality assessment system based on multimodal parallel analysis according to an embodiment of the present disclosure is shown. Figure 3 A schematic diagram of an electronic device structure according to an embodiment of the present disclosure is shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0024] Figure 1 A flowchart illustrating a video quality assessment method based on multimodal parallel analysis according to an embodiment of the present disclosure is shown, such as... Figure 1 As shown, the video quality assessment method for multimodal parallel analysis according to embodiments of this disclosure includes, S1 receives course videos to be reviewed and loads user-preset review parameter configurations; In this embodiment, the system receives a request for a course video to be reviewed and loads user-defined review parameter configurations (including video screen distortion / black screen threshold, audio low volume threshold, and sensitive word library). The system first verifies whether the video resource is stored locally. If it exists, the system directly loads the file handle; if it does not exist, the system downloads the file using a preset URL (Uniform Resource Locator) address.

[0025] S2, decapsulate the course video to be reviewed to obtain the initial video encoding stream and the initial audio encoding stream. At the same time, perform audio track integrity verification. Based on the verification result, terminate the audio analysis process and perform only video analysis, or perform video and audio analysis review processes simultaneously. In this embodiment of the disclosure, step S2 specifically includes: S21, Open the course video to be reviewed using a multimedia processing tool, identify its container format and read the format information; S22, detect and parse the media stream contained in the course video to be reviewed, and obtain the video stream index and audio stream index as well as their respective encoding parameter information; S23, based on the obtained video stream index and audio stream index, read compressed data packets from the container in a loop, and extract the video data packet sequence and audio data packet sequence according to the stream type to which the data packets belong; S24, the video data packet sequence is used as the initial video encoded stream, and the audio data packet sequence is used as the initial audio encoded stream.

[0026] In this embodiment of the disclosure, multimedia processing tools (such as FFmpeg) are used to demuxing / unpackaging the composite bitstream of the video. This process aims to parse the video metadata and strip the composite signal in the audio and video container into independent single-mode data streams, that is, to separate the independent video encoded stream and audio encoded stream.

[0027] In this embodiment of the disclosure, step S2 further includes: S25, perform feature extraction on the initial audio encoded stream to obtain audio features; S26, compare the audio features with preset audio features, and determine whether the audio track is complete based on the comparison result; S27, When the audio track is complete, the video and audio analysis review process is executed simultaneously; S28. When the audio track is incomplete, terminate the audio analysis process, perform only video analysis, and mark the audio stream missing anomaly in the structured audit report.

[0028] In this embodiment of the disclosure, audio features may include key features such as audio duration, volume, and frequency.

[0029] In this embodiment of the disclosure, comparing the audio features with preset audio features may include determining that the audio track is incomplete if the audio duration is obviously abnormal, such as the audio duration is much shorter than the preset audio duration of a normal course video, or the volume is consistently at an extremely low level and below a set volume threshold, or the audio frequency has a large number of fluctuations that are not within the normal range.

[0030] In this embodiment of the disclosure, audio track integrity verification is performed synchronously at this stage: if an audio stream is detected to be missing, the system will throw an "audio stream missing" exception and terminate all subsequent audio analysis processes, and the system will automatically switch to a single-track mode that only retains video footage for review.

[0031] This application uses audio features to compare with preset features to determine the completeness of audio tracks, enabling accurate and rapid determination of audio validity. When audio is incomplete, the audio analysis process is terminated promptly, saving computational resources, avoiding meaningless analysis of invalid audio, and marking the anomaly in the report to ensure the controllability of the review process and the traceability of issues.

[0032] S3, preprocess the initial video encoded stream and the initial audio encoded stream to generate a standard image frame sequence and standard audio data; In this embodiment of the disclosure, step S3 specifically includes: S31, the initial audio encoded stream is transcoded into an uncompressed audio format with a uniform sampling rate to obtain standard audio data; S32, the initial video encoded stream is divided into several processing units according to a preset time length to obtain several segments; S33. Perform temporal downsampling and frame extraction on each slice to obtain several extracted frame images. Convert the extracted frame images into the standard YUV color space to construct a standard image frame sequence.

[0033] In this embodiment of the disclosure, format standardization and preprocessing are performed in parallel on the separated audio and video streams: On the audio side: the original audio stream is transcoded into an uncompressed format (such as WAV) with a uniform sampling rate suitable for speech analysis, in order to eliminate the impact of coding differences on signal analysis.

[0034] On the video side: A temporal segmentation strategy is executed (e.g., video is divided into processing units of 10 minutes), and temporal downsampling (frame extraction, such as setting the sampling rate to 1fps) is performed on each segment. The extracted frame images are then converted to the standard YUV (color encoding method) color space to construct a standard image frame sequence. YUV refers to a type of true-color color space; proper nouns such as Y'UV, YUV, YCbCr, and YPbPr can all be called YUV, and there is some overlap between them. "Y" represents luminance (Luminance, Luma), while "U" and "V" represent chrominance (Chrominance, Chroma).

[0035] This application transcodes audio into an uncompressed format with a uniform sampling rate, eliminating the differences caused by different audio encoding parameters and ensuring the stability of subsequent analysis. The video is downsampled in the temporal domain and frames are extracted and converted to the standard YUV color space, which not only compresses the amount of data processed and improves processing speed, but also eliminates color format differences through color space standardization, improving the accuracy and consistency of video quality assessment and text analysis.

[0036] S4, Based on an asynchronous parallel architecture, perform concurrent multi-channel review tasks on the standard image frame sequence and the standard audio data to obtain multiple output results; In this embodiment of the disclosure, step S4 specifically includes: S41, perform no-reference video quality assessment on the standard image frame sequence, calculate actual multidimensional quality index data, compare the actual multidimensional quality index data with a preset threshold, judge the anomaly based on the comparison result and record the time period of quality anomaly, and use it as the first output result. S42, perform text analysis on the standard image frame sequence, extract the actual text content, perform multi-mode matching between the actual text content and the preset sensitive word library, record the first sensitive content and the corresponding sensitive time coordinates based on the matching results, and use them as the second output result; S43, the standard audio data is transcribed into a text stream with timestamps, and sensitive words are retrieved from the text stream based on a preset sensitive word database. Based on the retrieval results, the second sensitive content and sensitive time period are recorded and used as the third output result. S44, perform signal feature analysis on the standard audio data, identify silence, sound interruption and low volume abnormalities based on the analysis results, record the audio abnormality type and audio abnormality time period based on the identification results, and use them as the fourth output result.

[0037] In this embodiment, video quality review involves performing a no-reference video quality assessment on the generated standard image frame sequence. An image quality assessment library (such as AGH-VQIS) is invoked to calculate multi-dimensional quality indicators (including 15 indicators such as blockiness, blurriness, and noise). For course scenarios, three types of anomalies are specifically monitored: black screen, blurriness, and screen tearing. The system compares the real-time calculated indicator values ​​with preset thresholds (which can be dynamically configured by the user based on the video type). If an indicator exceeds the threshold for a certain period of time (e.g., 4 consecutive seconds, or 4 consecutive frames), the review is deemed unsuccessful, and the corresponding anomaly type and time period are recorded.

[0038] In this embodiment of the disclosure, step S41 specifically includes: S411, Extract the statistical features of each frame in the standard image frame sequence; S412, The statistical features are input into the pre-trained no-reference video quality assessment model to obtain actual multidimensional quality index data.

[0039] In this embodiment of the disclosure, the statistical characteristics may include: Luminance distribution characteristics: Calculate the histogram, mean, variance, skewness, and kurtosis of the luminance components; Edge gradient features: Image edges are extracted using the Sobel or Canny operator, and the mean, standard deviation, and edge density of the gradient magnitude are calculated; Block boundary difference feature: Divide the image into 8×8 blocks and calculate the ratio of the pixel difference at the block boundary to the pixel variance inside the block to detect block artifacts; Frequency domain statistical features: Perform Discrete Cosine Transform (DCT) or wavelet transform on the image to extract the energy ratio of high-frequency coefficients, zero crossover rate, etc. Temporal motion information: Calculate the sum of absolute differences (SAD) or optical flow field between adjacent frames to obtain motion amplitude and motion consistency index.

[0040] In this embodiment of the disclosure, the statistical features are input into a pre-trained no-reference video quality assessment model to obtain actual multidimensional quality index data, specifically including: Build or invoke a pre-trained no-reference video quality assessment model, which can employ a lightweight neural network (such as a CNN (Convolutional Neural Network) + fully connected layers) or a traditional regression model (such as support vector regression). Supervised training is performed using a large number of distorted video frames labeled with subjective quality scores, with extracted statistical features as input and corresponding multidimensional quality indicators (blocking intensity, blur, noise level, black screen confidence, screen tearing confidence, etc.) as output. The extracted statistical feature vector of each frame is input into the trained model. After forward computation, the model outputs a multi-dimensional vector, where each dimension corresponds to a quality index (e.g., [blocking intensity = 0.12, blurriness = 0.78, noise level = 0.05, black screen confidence = 0.01, screen distortion confidence = 0.02]). For multiple consecutive frames, the output metrics of each frame can be smoothed in the time domain (such as by moving average) to obtain the final actual multidimensional quality metric data.

[0041] In this embodiment, video quality review involves performing a no-reference video quality assessment on the generated standard image frame sequence. An image quality assessment library (such as AGH-VQIS) is invoked to calculate multi-dimensional quality indicators (including 15 indicators such as blockiness, blurriness, and noise). For course scenarios, three types of anomalies are specifically monitored: black screen, blurriness, and screen tearing. The system compares the real-time calculated indicator values ​​with preset thresholds (which can be dynamically configured by the user based on the video type). If an indicator exceeds the threshold for a certain period of time (e.g., 4 consecutive seconds, or 4 consecutive frames), the review is deemed unsuccessful, and the corresponding anomaly type and time period are recorded.

[0042] In this embodiment, visual content compliance review involves performing deep text analysis on a video frame sequence. First, a text detection model is used to locate text regions (ROIs) in the image, and orientation detection and rotation correction are performed. Then, an optical character recognition (OCR) engine is invoked to extract all text content (including whiteboard writing, PPT slides, and subtitles) from the image, and the text is associated with the timestamp of the current frame. Finally, the Aho-Corasick automaton algorithm is used to perform multi-pattern matching between the extracted text and a sensitive word database. If a sensitive word is matched, the violation content and its time coordinates are recorded.

[0043] In this embodiment, the voice content compliance review includes: asynchronously invoking an Automatic Speech Recognition (ASR) service to transcribe a standardized audio stream into a timestamped text stream. Subsequently, an AC automaton algorithm is used to perform a single-scan retrieval of the transcribed text. This algorithm constructs a state transition graph based on a Trie tree structure and configures failure pointers for nodes. During the scan, the algorithm utilizes failure pointers to achieve rapid state transitions when a match fails, completing multi-modal matching without backtracking the text. This mechanism ensures that the system can process long texts with linear time complexity, significantly improving the real-time performance and throughput of multimodal review. If sensitive words are detected, the violating content and time period are recorded immediately.

[0044] In this embodiment of the disclosure, the audio technology quality audit includes: performing signal feature analysis on WAV (Waveform Audio File Format) audio waveform data, calculating indicators such as energy spectrum and zero-crossing rate, and focusing on detecting sound interruptions, silence, and low volume abnormalities.

[0045] (1) Interruption and silence detection: Based on the noise threshold, if there is no effective sound within 1 second and it occurs at least 3 times within a 2-second window, it is judged as sound interruption; if there are more than 10 interruptions lasting 5 seconds and the cumulative duration exceeds 5 minutes, it is judged as silence abnormality.

[0046] (2) Low volume detection: Calculate the average loudness of the audio and compare it with a user-defined threshold (recommended value such as -35dB). If it is lower than the threshold, it is judged as low volume abnormality. The system will record all detected audio quality problems and their time periods.

[0047] S5, aggregate the multi-channel output results, perform cross-modal spatiotemporal alignment and fusion deduplication processing to obtain the processing result; In this embodiment of the disclosure, step S5 specifically includes: S51, combine the first output result, the second output result, the third output result and the fourth output result, and perform spatiotemporal alignment processing on the quality abnormal time period, the sensitive time coordinate, the sensitive time period and the audio abnormal time period; S52, perform cross-modal deduplication processing on the second output result and the third output result.

[0048] In this embodiment of the disclosure, if the same sensitive word appears in both the second and third output results within the same time period, they are merged into one record according to a preset deduplication strategy to avoid duplicate alarms.

[0049] S6. Generate a structured audit report based on the processing results.

[0050] In this embodiment of the disclosure, a structured audit report containing detailed audit information is constructed based on the processing results after spatiotemporal alignment and deduplication. The report content covers the types of anomalies and corresponding time periods found in video technology quality audits, such as the start and end times of anomalies like black screen, blurry screen, and distorted screen; sensitive words and their time coordinates detected in visual content compliance audits; sensitive words and sensitive time periods detected in audio content compliance audits; and audio anomaly types identified in audio technology quality audits, such as silence, intermittent sound, and low volume anomalies, along with their corresponding time periods.

[0051] In this embodiment of the disclosure, the report can be in tabular form, categorized and displayed according to dimensions such as anomaly type, anomaly time, and anomaly description. It can also provide statistical data, such as the number of times each type of anomaly occurs and the total duration, in order to evaluate the overall quality of the course videos.

[0052] In this embodiment of the disclosure, by generating such a structured review report, the online education platform can comprehensively understand the quality of course videos, promptly identify and address any issues, and ensure the political security, compliance, and high-quality audiovisual experience of the teaching content. Simultaneously, the review report provides strong data support for the platform's subsequent course management and optimization, contributing to improved service quality and user satisfaction.

[0053] Figure 2 A flowchart illustrating a video quality assessment system for multimodal parallel analysis according to an embodiment of the present disclosure is shown. Based on the above method, embodiments of the present disclosure also provide a video quality assessment system for multimodal parallel analysis corresponding to the above method. See [link to relevant documentation]. Figure 2 As shown, the system includes: The receiving module 10 is used to receive course videos to be reviewed and load user-preset review parameter configurations. The unpacking and verification module 20 is used to unpack the course video to be reviewed, obtain the initial video encoding stream and the initial audio encoding stream, and at the same time, perform audio track integrity verification. Based on the verification result, the audio analysis process is terminated and only video analysis is performed, or the video and audio analysis review process is performed simultaneously. Preprocessing module 30 is used to preprocess the initial video encoded stream and the initial audio encoded stream to generate standard image frame sequences and standard audio data; The task execution module 40 is used to concurrently execute multiple audit tasks on the standard image frame sequence and the standard audio data based on an asynchronous parallel architecture, and obtain multiple output results; Processing module 50 is used to aggregate the multi-channel output results, perform cross-modal spatiotemporal alignment and fusion deduplication processing, and obtain the processing result; The report generation module 60 is used to generate a structured audit report based on the processing results.

[0054] Based on the same inventive concept, this disclosure also provides an electronic device. For example... Figure 3 As shown, the electronic device of this disclosure includes at least one processor and at least one memory electrically connected to the processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the video quality assessment method of multimodal parallel analysis as described above.

[0055] It should be noted that the electrical connections between the various units mentioned above do not necessarily represent the connections between lines. Any indirect connection method can be applied to the embodiments of this disclosure as long as it achieves the purpose of this disclosure.

[0056] Based on the same inventive concept, this disclosure also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the steps of the above method.

[0057] Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A video quality assessment method based on multimodal parallel analysis, characterized in that, The method includes, Receive course videos awaiting review and load user-preset review parameter configurations; The course video to be reviewed is decapsulated to obtain the initial video encoding stream and the initial audio encoding stream. At the same time, the audio track integrity verification is performed. Based on the verification result, the audio analysis process is terminated and only video analysis is performed, or the video and audio analysis review process is performed simultaneously. The initial video encoded stream and the initial audio encoded stream are preprocessed to generate a standard image frame sequence and standard audio data; Based on an asynchronous parallel architecture, the standard image frame sequence and the standard audio data are concurrently executed for multi-channel review tasks to obtain multiple output results; The multi-output results are aggregated, and cross-modal spatiotemporal alignment and fusion deduplication are performed to obtain the processing result; A structured audit report is generated based on the processing results.

2. The method according to claim 1, characterized in that, The course videos to be reviewed are decapsulated to obtain initial video encoded streams and initial audio encoded streams, specifically including: The video of the course to be reviewed is opened using a multimedia processing tool, its container format is identified, and the format information is read. The media stream contained in the course video to be reviewed is detected and parsed to obtain the video stream index, audio stream index and their respective encoding parameter information; Based on the obtained video stream index and audio stream index, compressed data packets are read from the container in a loop, and video data packet sequences and audio data packet sequences are extracted according to the stream type to which the data packets belong. The video data packet sequence is used as the initial video encoded stream, and the audio data packet sequence is used as the initial audio encoded stream.

3. The method according to claim 1, characterized in that, Perform audio track integrity verification. Based on the verification results, terminate the audio analysis process and perform only video analysis, or perform both video and audio analysis review processes simultaneously, specifically including: Feature extraction is performed on the initial audio encoded stream to obtain audio features; The audio features are compared with preset audio features, and the completeness of the audio track is determined based on the comparison results. When the audio track is complete, the video and audio analysis review process is executed simultaneously; When the audio track is incomplete, the audio analysis process is terminated, and only video analysis is performed. The audio stream missing anomaly is marked in the structured audit report.

4. The method according to claim 1, characterized in that, Preprocessing the initial video encoded stream and the initial audio encoded stream specifically includes: The initial audio encoded stream is transcoded into an uncompressed audio format with a uniform sampling rate to obtain standard audio data; The initial video encoded stream is divided into several processing units according to a preset time length to obtain several segments; For each slice, perform temporal downsampling and frame extraction to obtain several extracted frame images. Convert the extracted frame images into the standard YUV color space to construct a standard image frame sequence.

5. The method according to claim 1, characterized in that, Based on an asynchronous parallel architecture, multiple auditing tasks are performed concurrently on the standard image frame sequence and the standard audio data to obtain multiple output results, specifically including: A no-reference video quality assessment is performed on the standard image frame sequence, the actual multidimensional quality index data is calculated, the actual multidimensional quality index data is compared with a preset threshold, anomalies are judged based on the comparison results and the time period of quality anomalies is recorded, and the anomaly is used as the first output result. Text analysis is performed on the standard image frame sequence to extract the actual text content. The actual text content is then matched with a preset sensitive word library using multiple modes. Based on the matching results, the first sensitive content and its corresponding sensitive time coordinates are recorded and used as the second output result. The standard audio data is transcribed into a timestamped text stream. Sensitive words are retrieved from the text stream based on a preset sensitive word database. The second sensitive content and sensitive time period are recorded based on the retrieval results and used as the third output result. The standard audio data is subjected to signal feature analysis. Based on the analysis results, silence, sound interruption and low volume abnormalities are identified. The audio abnormality type and audio abnormality time period are recorded based on the identification results and used as the fourth output result.

6. The method according to claim 5, characterized in that, The multiple output results are aggregated, and cross-modal spatiotemporal alignment and fusion deduplication processing are performed to obtain the processing result, which specifically includes: The first output result, the second output result, the third output result, and the fourth output result are combined, and the quality abnormal time period, the sensitive time coordinate, the sensitive time period, and the audio abnormal time period are spatiotemporally aligned. For the second output result and the third output result, perform cross-modal deduplication processing.

7. The method according to claim 5, characterized in that, Perform no-reference video quality assessment on the standard image frame sequence and calculate actual multidimensional quality index data, specifically including: Extract the statistical features of each frame in the standard image frame sequence; The statistical features are input into a pre-trained no-reference video quality assessment model to obtain actual multidimensional quality index data.

8. A video quality assessment system based on multimodal parallel analysis, characterized in that, The system includes: The receiving module is used to receive course videos to be reviewed and load user-preset review parameter configurations. The unpacking and verification module is used to unpack the course video to be reviewed, obtain the initial video encoding stream and the initial audio encoding stream, and at the same time, perform audio track integrity verification. Based on the verification result, the audio analysis process is terminated and only video analysis is performed, or the video and audio analysis review process is performed simultaneously. The preprocessing module is used to preprocess the initial video encoded stream and the initial audio encoded stream to generate standard image frame sequences and standard audio data; The task execution module is used to concurrently execute multiple audit tasks on the standard image frame sequence and the standard audio data based on an asynchronous parallel architecture, and obtain multiple output results; The processing module is used to aggregate the multi-channel output results, perform cross-modal spatiotemporal alignment and fusion deduplication processing, and obtain the processing result; The report generation module is used to generate a structured audit report based on the processing results.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the video quality assessment method of multimodal parallel analysis according to any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements the steps of the video quality assessment method of multimodal parallel analysis according to any one of claims 1-7.

Images