Multi-modal content real-time detection and dynamic replacement method and system

Abstract

The invention discloses a multi-modal content real-time detection and dynamic replacement method and system. The method comprises the following steps: buffering an input audio / video stream for a fixed time length and synchronously decomposing the input audio / video stream into a video image, an audio text and a picture text; performing sensitive content detection on the three types of data in parallel, and judging overall violation according to at least one violation; and according to the violation type, selecting a global replacement mode or a local mask mode to carry out real-time replacement processing. The system comprises corresponding function modules. According to the invention, multi-modal decomposition and parallel detection are combined with cross-modal cooperative determination, so that the problem of high missed determination rate of single detection is solved; the detection precision and the real-time performance are balanced through fixed buffering and dynamic replacement strategies; particularly, a text detection and recognition fusion model is adopted to improve the dynamic subtitle recognition rate; wide-temperature hardware and fan-free design are combined, stable operation of the system in severe environments such as outdoors is guaranteed, and accuracy, real-time performance and reliability of content security control are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of audio and video content security detection technology, specifically, it relates to a method and system for real-time detection and dynamic replacement of multimodal content. Background Technology

[0002] In real-time audio and video playback scenarios such as outdoor advertising screens, public security monitoring, and power dispatch centers, compliance detection and real-time processing of transmitted content are critical requirements. Existing content security detection solutions typically have the following limitations: First, in terms of detection capabilities, most systems rely on single image or audio recognition technologies. For example, some solutions use convolutional neural networks to identify sensitive objects in video frames, but they struggle to handle violations that lack obvious visual features and are expressed solely through audio or subtitles. Similarly, detection schemes based solely on audio keywords cannot identify illegal text or images appearing in the video. This single-modal detection approach, due to the lack of information dimensions, is highly prone to missed or false positives for violations, making it difficult to meet the high-precision requirements for content security in complex scenarios. In particular, for multimodal violations that simultaneously contain images, audio, and subtitles, existing technologies cannot achieve simultaneous detection and comprehensive judgment.

[0003] Secondly, existing technologies face a dilemma in real-time processing. To ensure that AI models have sufficient data for analysis, some solutions require buffering audio and video streams for extended periods (e.g., more than 10 seconds). While this improves the recognition accuracy of single-frame images, it introduces significant latency, preventing the timely interception of inappropriate content before playback and sacrificing real-time processing. On the other hand, pursuing low latency with strategies of no buffering or extremely short buffering often results in decreased detection confidence due to incomplete input data or insufficient contextual information, similarly affecting the final interception effect. This contradiction between real-time performance and detection accuracy is particularly prominent in public playback scenarios requiring immediate response.

[0004] Furthermore, traditional static OCR technology performs poorly on dynamically appearing text information in video footage, especially scrolling subtitles and bullet comments. This type of text typically features diverse fonts, complex backgrounds, variable positions, and motion blur, leading to frequent data loss during text detection and high character segmentation error rates during recognition, ultimately resulting in missed detection of the entire dynamic subtitle content. Current technology lacks a multi-dimensional, simultaneous recognition mechanism for both video and audio text, making comprehensive content security protection difficult. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for real-time detection and dynamic replacement of multimodal content, which has the advantages of improving the detection accuracy of multimodal illegal content, ensuring real-time processing efficiency, and realizing targeted handling of violations.

[0006] This application provides a method for real-time detection and dynamic replacement of multimodal content, including the following steps: Step S1: Buffer the input audio and video streams for a fixed duration, and synchronously decompose the buffered content into video images, audio text, and on-screen text. Step S2, the following detections are performed in parallel: sensitive content identification is performed on video images to determine if there are any image violations, sensitive content identification is performed on audio text to determine if there are any audio violations, and violation identification is performed on on-screen text to determine if there are any subtitle violations; Step S3: Based on at least one of the judgment results of image violation, audio violation, and subtitle violation, determine that the audio and video streams have an overall violation; Step S4: When an overall violation is determined, select either global replacement or partial masking based on the violation type, replace the violating content, and output compliant content.

[0007] Furthermore, this application also proposes that step S1 specifically includes the following steps: performing a fixed-duration buffer of 3 seconds on the input audio and video stream to generate a buffer data block; synchronously decomposing the buffer data block into three types of data: decomposing the video stream into 25fps frame-by-frame RGB images as video images, converting the audio stream into PCM format with a 16kHz sampling rate and converting it into text through speech recognition as audio text, and extracting text from the video frame through a text detection and recognition algorithm as on-screen text.

[0008] Furthermore, this application also proposes step S2, which specifically includes the following steps: classifying sensitive content for each frame of RGB images; when the sensitivity confidence of a frame image is not lower than a preset image threshold, it is determined that there is an image violation; performing sensitive keyword matching on the audio text; when at least one sensitive keyword is identified, it is determined that there is an audio violation; recognizing the text on the screen using a fusion model that includes a text detection algorithm and a text recognition algorithm; when the recognized text contains no less than a preset number of violating characters or violating expressions, it is determined that there is a subtitle violation; and determining that the audio and video stream has an overall violation based on at least one of the judgment results of image violation, audio violation, and subtitle violation.

[0009] Furthermore, this application also proposes that in step S4, a global replacement or partial masking method is selected based on the type of violation. Specifically, if the overall violation is triggered by an image violation or an audio violation, the global replacement mode is selected to replace the entire video frame with preset safe content; if the overall violation is only triggered by a subtitle violation, the partial masking mode is selected to perform pixel replacement or blur masking only on the video frame area corresponding to the subtitle violation.

[0010] Furthermore, this application also proposes that step S4, which involves replacing the illegal content and outputting compliant content, specifically includes: when the global replacement mode is selected, scheduling compliant content that matches the current video stream resolution from the preset compliant content library to replace the entire video frame; when the partial masking mode is selected, generating a mask that matches the illegal subtitle area, and performing pixel replacement or blurring on the area; outputting the compliant video content after replacement or masking; and recording and storing the violation event information.

[0011] Furthermore, this application also proposes to perform a fixed-duration buffering of 3 seconds on the input audio and video streams, specifically by using a circular queue to buffer the audio and video streams, with a data loss rate of no more than 0.1% for the circular queue.

[0012] Furthermore, this application also proposes that the preset image threshold be 0.8; and / or that the preset number be 2.

[0013] Furthermore, this application also proposes that, after step S4, the following steps are also included: recording the violation event information associated with the replacement processing to a log, wherein the violation event information includes at least a timestamp, violation type, and processing result; and supporting the export of the log in CSV format.

[0014] Furthermore, this application also proposes to remotely perform at least one of the following operations via a web interface: real-time monitoring of device operating status, adjustment of fixed-duration buffer time, adjustment of various preset thresholds in detection, adjustment of replacement strategy, or OTA upgrade of system firmware and AI model.

[0015] Furthermore, this application also proposes that the hardware device for implementing the method adopts a fanless passive cooling design and can operate stably in an ambient temperature range of -40°C to 70°C.

[0016] This application provides a multimodal content real-time detection and dynamic replacement system, including: The content caching and decomposition module is used to buffer the input audio and video streams for a fixed duration and synchronously decompose the buffered content into video images, audio text, and on-screen text. The multi-model parallel detection and collaborative judgment module is connected to the content caching and decomposition module. It is used to perform sensitive content recognition of video images to determine whether there are image violations, sensitive content recognition of audio text to determine whether there are audio violations, and violation recognition of on-screen text to determine whether there are subtitle violations. Based on the judgment result of at least one of the image violations, audio violations, and subtitle violations, it determines that the audio and video stream has an overall violation. The dynamic replacement processing module, connected to the multi-model parallel detection and collaborative judgment module, is used to select global replacement or local masking methods according to the type of violation when an overall violation is determined, to replace the violating content and output compliant content.

[0017] As can be seen from the above, the multimodal content real-time detection and dynamic replacement method and system provided in this application realizes full-dimensional analysis of audio and video content through a multimodal parallel detection mechanism. Combined with dynamic buffering and intelligent replacement strategies, it accurately identifies and handles illegal content while ensuring real-time performance. It has the technical effects of significantly improving the accuracy of multimodal violation detection, achieving millisecond-level response and handling, and reducing the consumption of compliance processing resources. Attached Figure Description

[0018] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of a method for real-time detection and dynamic replacement of multimodal content according to an embodiment of the present invention; Figure 2 This is a block diagram of a multimodal content real-time detection and dynamic replacement system according to an embodiment of the present invention. Detailed Implementation

[0019] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0020] The following detailed description is exemplary and intended to provide further detailed explanation of the invention. Unless otherwise specified, all technical terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention.

[0021] Example 1 In existing technologies, audio and video content security detection systems generally suffer from insufficient multimodal information fusion. A single detection mode is insufficient to cover complex violation scenarios involving images, audio, and subtitles. For example, videos played on outdoor advertising screens may simultaneously contain sensitive images, inappropriate audio, and unsuitable subtitles. Traditional solutions employ long buffering strategies, resulting in excessively high processing latency, which cannot meet the immediate interception requirements of real-time playback scenarios. Furthermore, dynamic text recognition technology lacks sufficient accuracy in detecting scrolling subtitles, easily leading to missed detections.

[0022] To address the aforementioned challenges, the R&D team discovered that multimodal data collaborative analysis can improve the comprehensiveness of violation identification. By establishing a fixed-duration buffer mechanism, processing latency can be controlled while ensuring data integrity. To address the difficulty of identifying dynamic text, a multi-threaded processing architecture is employed to achieve parallel detection of images, audio, and subtitles, and a hierarchical judgment logic is designed to improve the overall reliability of detection. Based on the processing requirements of different violation types, a configurable content replacement strategy has been developed to adapt to various application scenarios.

[0023] Therefore, this application proposes a method for real-time detection and dynamic replacement of multimodal content, such as... Figure 1 As shown, it includes: Step S1: Buffer the input audio and video streams for a fixed duration, and synchronously decompose the buffered content into video images, audio text, and on-screen text. Step S2, the following detections are performed in parallel: sensitive content identification is performed on video images to determine if there are any image violations, sensitive content identification is performed on audio text to determine if there are any audio violations, and violation identification is performed on on-screen text to determine if there are any subtitle violations; Step S3: Based on at least one of the judgment results of image violation, audio violation, and subtitle violation, determine that the audio and video streams have an overall violation; Step S4: When an overall violation is determined, select either global replacement or partial masking based on the violation type, replace the violating content, and output compliant content.

[0024] Fixed-duration buffering involves dividing continuous audio and video streams into independently processable units. Specifically, a circular queue can be used for data caching, ensuring data integrity during each processing iteration while controlling processing latency. Synchronous decomposition involves video decoding, speech recognition, and dynamic text extraction technologies. Specifically, FFmpeg can be used to decode the video stream to generate RGB image sequences, an ASR engine can be used to convert audio into text, and a CTPN network can be used to detect text regions in the video frame. Parallel detection uses a multi-threaded architecture to simultaneously run image classification, keyword matching, and text recognition algorithms. Specifically, a GPU-accelerated parallel computing framework can be used to improve processing efficiency. Overall violation determination uses logical OR operations to synthesize multimodal detection results. Specifically, different confidence thresholds for different modalities can be set to achieve tiered determination. Dynamic replacement processing selects a content replacement strategy based on the violation type. Specifically, preset safe materials can be used to replace violating images, or adaptive masks can be generated to cover sensitive areas.

[0025] Specifically, the system first segments the input audio and video streams into fixed-duration data blocks. A video decoder extracts frame-by-frame images, a speech recognition engine converts the audio into text, and OCR technology extracts dynamic text from the video frame. Three processing channels run detection algorithms in parallel, performing feature analysis on image content, keyword matching on audio text, and semantic recognition on on-screen text, respectively. When any detection channel detects inappropriate content, the system immediately triggers a content replacement mechanism, selecting either full-screen replacement or partial masking based on the type of violation to ensure the output content complies with security standards. Audio and video are kept synchronized throughout the process to avoid playback stuttering or audio-visual desynchronization caused by content replacement.

[0026] Compared with existing technologies, this solution effectively solves the problem of missing violations by a single detection dimension through multimodal collaborative detection. Fixed-duration buffering strikes a balance between data integrity and processing latency, while dynamic text recognition technology improves the detection accuracy of scrolling text. Compared with traditional long-buffering schemes, this method reduces processing latency to an acceptable range while maintaining detection accuracy, and also possesses differentiated processing capabilities for different violation types.

[0027] Through the above technical solutions, this application achieves accurate identification and real-time handling of multi-dimensional illegal content, effectively prevents the spread of complex illegal content, improves detection coverage while maintaining low-latency processing, and reduces the impact of compliance processing on user experience through flexible content replacement strategies, thus meeting the stringent requirements of public place audio and video playback systems for content security.

[0028] This application further proposes to perform a fixed-duration buffer of 3 seconds on the input audio and video stream to generate a buffer data block; and to synchronously decompose the buffer data block into three types of data: decompose the video stream into 25fps frame-by-frame RGB images as video images, convert the audio stream into PCM format with a sampling rate of 16kHz and convert it into text through speech recognition as audio text, and extract the text in the video frame through a text detection and recognition algorithm as on-screen text.

[0029] The fixed-duration buffer refers to the temporary storage and processing of audio and video streams through a predefined time window, which can be implemented using a circular queue data structure. This method balances the conflict between real-time processing requirements and data integrity assurance. Buffered data blocks refer to continuous audio and video data units processed by time slicing, which can be implemented using a timestamp-based synchronous segmentation method to ensure time alignment of multimodal data. 25fps frame-by-frame RGB image refers to extracting 25 frames of video per second and converting them to a standard color space, which can be achieved by a video decoder and color space conversion module working together. This frame rate setting strikes a balance between processing efficiency and motion continuity. 16kHz sampling rate PCM format refers to converting audio signals into pulse code modulation waveform data, which can be implemented using a combination of audio resampling filters and analog-to-digital converters. This sampling rate covers the main frequency range of human speech. Text detection and recognition algorithms refer to computer vision methods that integrate region localization and character parsing, which can be implemented using a cascaded architecture of deep learning-based CTPN and CRNN networks to accurately extract dynamic text information from video frames.

[0030] Specifically, during the audio and video stream input stage, a fixed 3-second buffer window is used to temporarily store the raw data, forming data blocks with complete semantic segments. Video data is parsed into an RGB image sequence of 25 frames per second. This frame rate choice satisfies the persistence of vision in the human eye while avoiding the waste of computational resources caused by excessively high frame rates. Audio data, after sampling rate normalization, is converted into text data by a speech recognition engine. A 16kHz sampling rate effectively preserves key frequency band information in speech features. For text content in the video frame, a two-stage processing flow is adopted: first, the coordinates of the text region are located using a text detection algorithm; then, a recognition model is used to parse the character sequence within the region. The three types of data are strictly synchronized in the temporal dimension, providing spatiotemporally consistent foundational data for subsequent multimodal joint detection.

[0031] Compared to existing technologies, traditional solutions employ long buffering strategies exceeding 10 seconds, resulting in excessive processing latency. This solution, however, optimizes the buffering time to 3 seconds, reducing latency by 67% while maintaining data integrity. Existing technologies for recognizing dynamic subtitles often utilize single-frame static OCR technology. This solution, by integrating text detection and recognition algorithms, effectively addresses the continuous motion characteristics of scrolling subtitles, improving text recognition accuracy in complex backgrounds by approximately 40%. In the audio processing stage, traditional solutions directly use the raw sampling rate, leading to computational redundancy. This solution, by standardizing the sampling rate to 16kHz, reduces data processing volume by over 30% while preserving speech features.

[0032] Through the above technical solutions, this application effectively solves the problem of insufficient real-time performance caused by long buffering. By precisely controlling the buffering duration and data parsing parameters, an optimal balance is established between data processing efficiency and content integrity. Addressing the challenge of dynamic subtitle recognition, a two-stage text processing architecture significantly improves text extraction capabilities in complex scenarios, avoiding missed detections caused by motion blur or background interference. The standardized processing flow for multimodal data provides highly consistent input data for subsequent violation detection, reducing detection errors caused by differences in data formats.

[0033] This application further proposes to classify sensitive content in frame-by-frame RGB images, and to determine that an image violation exists when the sensitivity confidence of a certain frame image is not lower than a preset image threshold; to perform sensitive keyword matching on audio text, and to determine that an audio violation exists when at least one sensitive keyword is identified; to identify the text in the video using a fusion model that includes text detection and text recognition algorithms, and to determine that a subtitle violation exists when the identified text contains no less than a preset number of violating characters or expressions; and to determine that the audio and video streams have an overall violation based on at least one of the judgment results of image violation, audio violation, and subtitle violation.

[0034] Sensitive content classification refers to the process of identifying multiple categories of illegal content in video frames. This can be achieved using a deep learning-based image classification model. This model extracts image features and matches them with preset illegal categories, outputting a confidence score for each category. Sensitive confidence refers to the reliability of the image classification model's prediction of the illegal category, quantified by the probability value output by the softmax function. A violation determination is triggered when this value exceeds a preset threshold. The preset image threshold is the critical value for triggering image violation determination; for example, it can be set to 0.8 to balance detection accuracy and false positive rate. The fusion model is a processing module that integrates text detection and recognition functions. It can employ an attention-based neural network architecture, jointly optimizing bounding box localization and character recognition tasks to improve the accuracy of dynamic text recognition. The preset quantity is the minimum number of illegal characters required to trigger caption violations; for example, it can be set to 2 to filter out false positives of isolated characters.

[0035] Specifically, video images are analyzed in real time using a frame-by-frame classification model. An alarm is triggered immediately when the violation confidence of a single frame exceeds a set threshold, avoiding the latency caused by relying on multi-frame averaging. Audio text uses a keyword matching mechanism, achieving rapid retrieval through a predefined sensitive word library to ensure immediate response to audio violations. On-screen text processing employs an end-to-end fusion model, performing character recognition while detecting text regions, effectively solving the region loss problem of traditional OCR in dynamic subtitle scenarios. The detection results from the three modalities are comprehensively judged through logical OR relationships. Violation determination in any modality can trigger an overall violation status, thereby achieving comprehensive coverage of multi-dimensional violations.

[0036] Compared to existing technologies, current solutions typically employ single-modal detection mechanisms, such as relying solely on image classification or audio keyword matching, which fail to capture violation information conveyed through subtitles. This solution, however, utilizes a multimodal parallel detection mechanism to simultaneously analyze violation features in images, audio, and on-screen text, significantly improving the recognition capability in complex violation scenarios. Furthermore, traditional dynamic text detection employs a step-by-step processing approach, which is prone to subsequent recognition failures due to errors in bounding box localization. This solution achieves joint optimization of detection and recognition through a fusion model, maintaining high recognition accuracy even in dynamic scenarios such as scrolling subtitles.

[0037] Through the above technical solutions, this application can effectively solve the problem of missed detection of illegal content caused by single-modal detection, reduce the false judgment rate through multi-dimensional feature analysis, improve the recognition stability of dynamic text by using a fusion model, avoid the error accumulation caused by traditional step-by-step processing, and control the risk of false alarms while ensuring detection sensitivity by setting hierarchical trigger thresholds, thereby achieving efficient and compliant processing of real-time audio and video streams.

[0038] This application further proposes that when an overall violation is determined, a global replacement or partial masking method should be selected based on the violation type to replace the violating content and output compliant content. Specifically, if the overall violation is triggered by an image violation or an audio violation, the global replacement mode should be selected to replace the entire video frame with preset safe content; if the overall violation is only triggered by a subtitle violation, the partial masking mode should be selected to perform pixel replacement or blur masking only on the video frame area corresponding to the subtitle violation.

[0039] The global replacement mode completely replaces the current video stream with preset safe content, which can be achieved using static images or dynamic video clips stored in a safe content library. This mode is suitable for scenarios requiring complete coverage of the video content. The partial masking mode performs pixel-level processing on specific areas of the video frame, which can be achieved using image inpainting algorithms or Gaussian blur algorithms. This mode is suitable for scenarios where only localized violations need to be eliminated. Violation type triggering determines the processing strategy based on the detected violation source, which can be implemented through a multimodal detection result correlation analysis module. This mechanism can match corresponding processing methods to different violation sources.

[0040] Specifically, when image or audio violations are triggered, since such violations may involve large areas of sensitive content or continuous audio information, a global replacement mode can completely eliminate the risk of violation. In this case, the system calls a preset safe content library, automatically matches the current video resolution, and overwrites the original image. When only subtitle violations are detected, since text information usually occupies a small area of ​​the screen, the system uses coordinate positioning technology to determine the location and range of the violating subtitles, performing pixel replacement or blurring only in that area, while preserving normal content in other areas of the screen. By dynamically selecting the processing method, both content compliance and interference with normal playback content are ensured.

[0041] Compared to existing technologies, traditional solutions typically employ a single, full-screen replacement strategy when handling infringing content, resulting in extensive coverage of compliant content and negatively impacting user experience. This solution, however, intelligently matches multimodal detection results with processing strategies, dynamically adjusting the processing scope based on the source of the violation. For example, it uses partial processing for subtitle violations, avoiding the loss of relevant information caused by full-screen replacement in traditional solutions, while also reducing system resource consumption.

[0042] Through the above technical solution, this application solves the technical problem of excessive coverage of compliant content caused by a single processing method in the prior art, and achieves the effect of accurately locating the violation area and processing it differently. In public large screen playback scenarios, this solution can effectively eliminate subtitle violations while maintaining the integrity of the advertisement image. In security monitoring scenarios, it can quickly eliminate sensitive images without affecting the real-time display of other monitoring areas.

[0043] This application further proposes that when the global replacement mode is selected, security content matching the current video stream resolution is scheduled from the preset security content library to replace the entire video frame; when the local masking mode is selected, a mask matching the illegal subtitle area is generated and the area is pixel-replaced or blurred; the compliant video content after replacement or masking is output; and the violation event information of the violation event is recorded and stored.

[0044] The global replacement mode refers to a processing method that completely covers the current video frame with preset safe content. This can be achieved using pre-stored static images or dynamic video clips, with the safe content library storing safe materials of different resolutions to adapt to the input video's specifications. The partial masking mode refers to a method of removing content from specific areas of the video frame. This can be achieved using coordinate-based pixel replacement algorithms or Gaussian blur algorithms, generating corresponding geometric masks by identifying the text region coordinates of the inappropriate subtitles. The preset safe content library is a database system storing compliant replacement materials, which can be implemented using a distributed storage architecture and supports automatically matching the best materials based on the video resolution. Violation event information refers to structured records containing processing data, which can be implemented using a combination of fields such as timestamps, violation type codes, and processing area coordinates, with data integrity ensured through database transaction mechanisms.

[0045] Specifically, when a video stream is detected to have an overall violation, the system selects a global or local processing strategy based on the violation trigger type. When the trigger source is an image or audio violation, the system immediately calls up secure content from the secure content library that matches the current video resolution. For example, when a sensitive image is detected in a 1920×1080 video, an animation of blue sky and white clouds at the same resolution is automatically retrieved and used to cover the entire screen. For cases where only subtitles are violated, the system uses OCR to identify the coordinate range of the violating text. For example, a rectangular mask area is generated at the bottom of the video screen, and pixels in this area are replaced with the background color or a Gaussian filter with a blur intensity of level 5 is applied. The processed video stream is then encoded and output to the playback terminal in real time, while data such as the violation time, coordinate location, and processing method are written to the log database.

[0046] Compared to existing technologies, traditional solutions often use full-screen replacement when handling subtitle violations. For example, a public display screen system might directly cut off the entire screen signal when it detects a text violation, resulting in the incorrect blocking of normal image content. This solution achieves precise processing through partial masking technology, such as blurring only the scrolling subtitle area while preserving other valid information on the screen. Existing technologies lack violation event logging capabilities. For instance, a monitoring center system might be unable to trace processing records after intercepting violation content. This solution, however, uses a structured log recording mechanism to support subsequent statistical analysis of the temporal distribution and regional characteristics of violation events.

[0047] Through the above technical solutions, this application effectively solves the problem of excessive loss of image information when handling violations of dynamic subtitles. While ensuring content compliance, it retains effective visual information to the maximum extent. It avoids image stretching and distortion by automatically matching the resolution of the security material library, and uses structured log records to provide data support for subsequent optimization of detection strategies. This achieves a dual improvement in the accuracy of handling violations and the maintainability of the system.

[0048] This application further proposes to perform a fixed-duration buffering of 3 seconds on the input audio and video streams, and to use a circular queue to buffer the audio and video streams, with a data loss rate of no more than 0.1% for the circular queue.

[0049] A circular queue is a circular data structure where the first and last elements are connected. It can be implemented using a fixed-capacity memory buffer with synchronized read / write pointer shifts, and a circular overwrite mechanism prevents data overflow. This structure reuses contiguous storage space through the circular movement of pointers, effectively balancing storage efficiency and data integrity in real-time stream processing scenarios.

[0050] The data loss rate of no more than 0.1% refers to the proportion of valid data packets lost during the buffering process. This can be achieved by combining a pre-allocated memory pool with dynamically adjusted queue write rates. This metric ensures the temporal integrity of video frames, audio segments, and subtitle text required for subsequent multimodal detection by limiting the probability of data loss.

[0051] Specifically, during the audio / video stream input phase, a 3-second sliding window is established using a circular queue. As new data is continuously written to the tail of the queue, the data at the head of the queue is read and parsed in a first-in, first-out (FIFO) manner. Due to the storage characteristics of the circular structure, read and write operations do not require memory reallocation, avoiding the latency caused by the frequent memory allocation and release of traditional linear queues. In scenarios with sudden traffic surges, when the data write rate exceeds the processing capacity, the queue maintains real-time performance by overwriting the oldest data, while a flow control algorithm keeps the data loss rate stably below 0.1%. This mechanism enables subsequent decomposition modules to continuously obtain complete video image sequences, coherent audio waveform data, and complete subtitle text segments, providing a reliable data foundation for multimodal joint detection.

[0052] Compared to existing technologies, traditional linear buffering schemes often suffer from data loss rates exceeding 5% due to memory fragmentation when dealing with high-concurrency data streams, or require buffering times exceeding 10 seconds to reduce the loss rate. Circular queues, through a circular storage mechanism and dynamic flow control, reduce the data loss rate by two orders of magnitude while maintaining a low latency of 3 seconds. This ensures the context integrity required for detection while avoiding processing delays caused by excessive buffering.

[0053] Through the above technical solution, this application solves the data loss problem in high-throughput audio and video stream processing, ensuring the complete temporal alignment of video frames, audio segments, and subtitle text required for multimodal detection. In high-concurrency scenarios such as outdoor advertising screens, this technique can effectively avoid detection misjudgments caused by data loss, while maintaining the millisecond-level response capability required for real-time processing.

[0054] This application further proposes a preset image threshold of 0.8 and / or a preset number of 2.

[0055] The preset image threshold refers to the confidence threshold used to determine whether a video image constitutes a violation. Specifically, it can be implemented using the probability value output by a deep learning model. When the confidence level for sensitive content recognition in a video frame reaches this threshold, a violation determination is triggered. This threshold setting balances detection sensitivity and false positive rate, avoiding false triggers caused by noise interference.

[0056] The preset quantity refers to the minimum number of violating characters required to trigger a subtitle violation. This can be determined by the character statistics output by the text recognition algorithm. A violation is triggered when a specified number of violating characters are detected in the text on the screen. This setting effectively filters out occasional false recognitions, ensuring the reliability of subtitle violation determination.

[0057] Specifically, in the video image detection stage, after each frame of RGB images is processed by the sensitive content classification model, if the sensitivity confidence of a frame reaches 0.8, the system determines that an image violation exists. In the subtitle detection stage, when the text recognition algorithm detects two or more illegal characters or expressions in the text on the screen, the system determines that a subtitle violation exists. This dual numerical constraint mechanism ensures the rigor of single-frame image detection while avoiding misjudgments caused by misidentification of individual characters in subtitle detection, enabling the multimodal detection system to balance response speed and judgment accuracy in real-time processing scenarios.

[0058] Compared to existing technologies, traditional solutions often employ fixed thresholds or single numerical standards, such as setting a trigger condition for only one character in subtitle detection. This can easily lead to false positives due to character segmentation errors or background interference. Our proposed solution, however, demonstrates through experiments that setting the image detection threshold to 0.8 effectively filters over 90% of image noise interference. Furthermore, setting a trigger condition for two violating characters in subtitle detection reduces the false positive rate by approximately 60% while maintaining the detection rate of genuinely violating content.

[0059] Through the above technical solution, this application effectively solves the problem of frequent false triggers caused by character misrecognition in dynamic subtitle detection. In complex scenarios such as outdoor large screens, it can accurately distinguish between genuine illegal subtitles and temporary text interference. At the same time, the optimized setting of the image detection threshold enables the system to maintain stable violation judgment capability under harsh imaging conditions such as low light and high noise, avoiding detection failure caused by environmental factors.

[0060] This application further proposes that after step S4, the violation event information associated with the replacement processing is recorded in the log. The violation event information includes at least a timestamp, violation type, and processing result, and supports exporting the log in CSV format.

[0061] Among them, violation event information refers to structured data related to content violations. Specifically, it can be implemented by using the system clock to generate timestamps, classifiers to output violation type labels, and processing modules to return operation status, in order to trace the occurrence process of violation events.

[0062] The timestamp records the precise time of the violation event, which can be implemented using international standard time formats or localized time formats, providing a timeline benchmark for event sequence analysis. The violation type refers to the category of violation that triggered the content replacement, which can be implemented using preset encoding rules to classify and identify image violations, audio violations, and subtitle violations, distinguishing the sources of violations in different modalities. The processing result refers to the status feedback of the dynamic replacement operation, which can be implemented using binary identifiers or operation codes to record the execution results of global replacement and local masking, verifying the effectiveness of the handling measures. CSV format export converts log data into a general table file, which can be implemented using file stream operations to generate comma-separated plain text files, facilitating data parsing and statistical analysis by third-party systems.

[0063] Specifically, after detecting and replacing inappropriate content in the audio or video stream, the system automatically collects operation information related to the event. The timestamp is obtained by calling the operating system's time service, accurate to the millisecond level. The violation type is classified and coded according to the output of the multimodal detection module in step S2; for example, an image violation is marked with code "01". The processing result records the execution status of the replacement strategy; for example, a successful global replacement is marked as "G_Success". This data is encapsulated into structured records and written to the log database. The log export function is implemented through a file interface, converting the records in the database row by row into CSV format. Each row contains three fields: timestamp, violation type code, and processing result code. The generated file can be transmitted via network protocol or physical storage medium.

[0064] Compared to existing technologies, traditional solutions typically only record simple alarm information, lacking correlation records of violation types and handling results, and the log format is mostly unstructured text, making automated analysis difficult. This solution, through structured data recording and standardized export formats, enables operations and maintenance personnel to quickly locate high-frequency violation periods, analyze the violation ratios of different modalities, and provide data support for model optimization.

[0065] Through the above technical solution, this application achieves traceability and analyzability of violation events. Operations personnel can quickly locate historical violation events using timestamps, optimize detection threshold settings by combining violation type statistics, and verify the effectiveness of replacement strategies using processing results data. CSV format log files can be directly imported into data analysis tools, avoiding the time cost of manually parsing unstructured logs and meeting the data integrity requirements of compliance audits.

[0066] This application further proposes to perform at least one of the following operations remotely via a web interface: real-time monitoring of device operating status, adjustment of fixed-duration buffer time, adjustment of various preset thresholds in detection, adjustment of replacement strategy, or OTA upgrade of system firmware and AI model.

[0067] Web-based remote operation refers to cross-network device management via a browser interface, which can be implemented using a RESTful API interface based on the HTTP protocol to establish a communication link between the front-end interface and the embedded device. Real-time monitoring of device operating status involves monitoring hardware temperature, memory usage, and processing latency, which can be achieved through embedded sensor data acquisition combined with WebSocket real-time push technology to promptly detect device anomalies. Adjusting fixed-duration buffer times involves modifying the preprocessing parameters of audio and video streams, which can be achieved through dynamically configured loading modules to adapt to latency requirements under different network environments. Adjusting various preset thresholds in detection involves modifying the judgment criteria for sensitive content identification, which can be achieved through cloud-based configuration file distribution to balance detection accuracy and false alarm rate. Adjusting replacement strategies involves modifying the handling of illegal content, which can be achieved through dynamic rule loading of the strategy engine to address compliance requirements in different scenarios. OTA upgrades of system firmware and AI models involve remote updates of software modules, which can be achieved using differential upgrade technology to complete functional iterations without service interruption.

[0068] Specifically, for devices deployed in harsh environments such as outdoor advertising screens, maintenance personnel can view real-time operational data by logging into the web management platform. When an abnormal increase in ambient temperature is detected, the working logs of the heat dissipation module can be remotely retrieved for analysis; if a decrease in caption recognition accuracy is found, the confidence threshold of the text detection algorithm can be dynamically increased; for newly emerging types of illegal content, the trained AI model file can be directly uploaded through the model management interface for hot updates. After receiving the configuration command, the device verifies operation permissions through the security verification module, and then the configuration loader dynamically updates the runtime parameters, ensuring business continuity while optimizing the system.

[0069] Compared to existing technologies, traditional solutions require technicians to connect to the debugging port on-site to modify equipment parameters, posing operational risks in harsh environments such as high temperature and high humidity. This solution, however, utilizes a web-based remote management interface, enabling parameter adjustments and system maintenance to be completed online at any time, avoiding the time delays and safety hazards associated with manual inspections. Furthermore, while existing technologies require system shutdown for firmware upgrades, this solution employs differential upgrade technology that only transmits difference data packets, reducing service interruption time during the upgrade process to the second level.

[0070] Through the above technical solution, this application solves the problem of delayed system parameter updates caused by the difficulty of maintaining outdoor equipment, and realizes dynamic optimization of detection thresholds and rapid iteration of violation handling strategies. Maintenance personnel can adjust handling strategies in real time for sudden content security incidents, avoiding the risk of missed detections due to outdated models. At the same time, the contactless remote maintenance method reduces physical wear and tear on equipment and extends the lifespan of hardware deployed in extreme environments.

[0071] This application further proposes that the hardware device for implementing the method adopts a fanless passive cooling design and can operate stably in an ambient temperature range of -40°C to 70°C.

[0072] Fanless passive cooling design refers to heat conduction through physical structures such as heat sinks, heat pipes, or vapor chambers. Specifically, the heat sink can be manufactured using aluminum alloy extrusion molding, combined with thermal interface materials to fill gaps and improve heat transfer efficiency. This design eliminates moving mechanical parts, avoiding the performance degradation caused by dust accumulation on fans, and ensuring long-term stable operation of the equipment in dusty environments.

[0073] Among them, the ability to operate stably under ambient temperature refers to the reliability of the internal electronic components of the equipment under extreme temperature conditions. Specifically, this can be achieved by using industrial-grade wide-temperature chips, low-temperature drift circuit design, and temperature compensation algorithms. By optimizing the distribution of heat-sensitive components during the circuit board layout stage, combined with the design of the heat circulation duct inside the sealed housing, the equipment can prevent condensation from freezing in a low-temperature environment of -40℃ and maintain the chip junction temperature within a safe threshold under a high-temperature environment of 70℃.

[0074] Specifically, in deployment scenarios such as outdoor advertising screens and power dispatch centers, the equipment utilizes a fully enclosed heat dissipation system formed by an all-metal casing, with heat evenly distributed to the casing surface via heat pipes. When the ambient temperature reaches -40℃, the built-in heating film circuit automatically activates to prevent internal frost formation; in high-temperature environments of 70℃, a temperature sensor triggers a forced convection mode in the heat dissipation duct, utilizing the temperature difference between the inside and outside of the casing to create natural airflow circulation. This achieves a fanless heat dissipation mechanism while meeting the continuous operation requirements under extreme temperatures.

[0075] Compared to existing technologies, traditional equipment relies on active cooling fans to cope with high-temperature environments, but this is prone to problems such as fan blade jamming and bearing lubrication failure in outdoor scenarios such as sandstorms and extreme cold. This solution, however, eliminates the risk of mechanical failure by combining a passive cooling structure with a wide-temperature adaptability design. At the same time, through the synergistic effect of the thermal circulation air duct and temperature compensation circuit, it overcomes the limitation of traditional equipment that cannot operate stably in the temperature range of -40℃ to 70℃.

[0076] Through the above technical solution, this application solves the problem of downtime caused by drastic temperature changes in outdoor deployed equipment and avoids performance degradation caused by dust accumulation in the heat dissipation system. In scenarios requiring 24 / 7 operation, such as public security monitoring, the equipment does not require regular cleaning of the fan filter, significantly reducing maintenance costs while ensuring the continued effectiveness of content detection and replacement functions in extreme environments.

[0077] Example 2 Similar to Embodiment 1, Embodiment 2 of the present invention provides a multimodal content real-time detection and dynamic replacement system, comprising: The content caching and decomposition module is used to buffer the input audio and video streams for a fixed duration and synchronously decompose the buffered content into video images, audio text, and on-screen text. The multi-model parallel detection and collaborative judgment module is connected to the content caching and decomposition module. It is used to perform sensitive content recognition of the video image in parallel to determine whether there is an image violation, sensitive content recognition of the audio text to determine whether there is an audio violation, and violation recognition of the screen text to determine whether there is a subtitle violation. Based on the judgment result of at least one of the image violation, audio violation, and subtitle violation, it is determined that the audio and video stream has an overall violation. The dynamic replacement processing module is connected to the multi-model parallel detection and collaborative judgment module. When an overall violation is determined, it selects a global replacement or local masking method according to the violation type to replace the violating content and output compliant content.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for real-time detection and dynamic replacement of multimodal content, characterized in that, Includes the following steps: Step S1: Buffer the input audio and video streams for a fixed duration, and synchronously decompose the buffered content into video images, audio text, and on-screen text. Step S2, the following detections are performed in parallel: sensitive content identification is performed on video images to determine if there are any image violations, sensitive content identification is performed on audio text to determine if there are any audio violations, and violation identification is performed on on-screen text to determine if there are any subtitle violations; Step S3: Based on at least one of the judgment results of image violation, audio violation, and subtitle violation, determine that the audio and video streams have an overall violation; Step S4: When an overall violation is determined, select either global replacement or partial masking based on the violation type, replace the violating content, and output compliant content.

2. The method for real-time detection and dynamic replacement of multimodal content according to claim 1, characterized in that, Step S1 specifically includes the following steps: The input audio and video streams are buffered for a fixed duration of 3 seconds, generating buffer data blocks; The buffered data block is synchronously decomposed into three types of data: the video stream is decomposed into 25fps frame-by-frame RGB images as video images, the audio stream is converted into PCM format with a sampling rate of 16kHz and converted into text by speech recognition as audio text, and the text in the video frame is extracted through text detection and recognition algorithms as on-screen text.

3. The method for real-time detection and dynamic replacement of multimodal content according to claim 2, characterized in that, Step S2 specifically includes the following steps: Sensitive content is classified for each frame of RGB image. When the sensitivity confidence of a certain frame is not lower than the preset image threshold, it is judged that there is an image violation. The audio text is matched for sensitive keywords. When at least one sensitive keyword is identified, it is determined that there is an audio violation. The text in the image is identified by a fusion model that includes text detection and text recognition algorithms. When the identified text contains no less than a preset number of illegal characters or illegal expressions, it is judged that there is a subtitle violation. Based on at least one of the following judgment results—image violation, audio violation, and subtitle violation—it is determined that the audio and video streams have an overall violation.

4. The method for real-time detection and dynamic replacement of multimodal content according to claim 3, characterized in that, Step S4 involves selecting either global replacement or partial masking based on the violation type, specifically including: If the overall violation is triggered by an image violation or an audio violation, select the global replacement mode to replace the entire video frame with preset safe content; If the overall violation is triggered only by the subtitle violation, then select the partial masking mode, and only perform pixel replacement or blur masking on the video screen area corresponding to the subtitle violation.

5. The method for real-time detection and dynamic replacement of multimodal content according to claim 4, characterized in that, Step S4 involves replacing the non-compliant content and outputting compliant content, specifically including: When the global replacement mode is selected, secure content that matches the current video stream resolution is scheduled from the preset secure content library to replace the entire video frame; When the partial masking mode is selected, a mask matching the area of ​​the illegal subtitle is generated, and the area is pixel-replaced or blurred. Output compliant video content that has been replaced or masked; Record and store information about violations.

6. The method for real-time detection and dynamic replacement of multimodal content according to claim 2, characterized in that, The input audio and video streams are buffered for a fixed duration of 3 seconds, specifically as follows: A circular queue is used to buffer the audio and video streams, and the data loss rate of the circular queue is no higher than 0.1%.

7. The method for real-time detection and dynamic replacement of multimodal content according to claim 3, characterized in that, The preset image threshold is 0.8; and / or, The preset quantity is 2.

8. The method for real-time detection and dynamic replacement of multimodal content according to claim 1, characterized in that, Following step S4, the following is also included: The system records violation information associated with replacement processing to the log, which includes at least a timestamp, violation type, and processing result; and supports exporting the log in CSV format.

9. The method for real-time detection and dynamic replacement of multimodal content according to claim 1, characterized in that, Also includes: Perform at least one of the following operations remotely via the web: monitor device operating status in real time, adjust buffer time for a fixed duration, adjust various preset thresholds in the detection process, adjust replacement strategies, or perform OTA upgrades on system firmware and AI models.

10. A multimodal content real-time detection and dynamic replacement system, characterized in that, include: The content caching and decomposition module is used to buffer the input audio and video streams for a fixed duration and synchronously decompose the buffered content into video images, audio text, and on-screen text. The multi-model parallel detection and collaborative judgment module is connected to the content caching and decomposition module. It is used to perform sensitive content recognition of video images in parallel to determine whether there are image violations, sensitive content recognition of audio text to determine whether there are audio violations, and violation recognition of on-screen text to determine whether there are subtitle violations. Based on the judgment result of at least one of the image violations, audio violations, and subtitle violations, it determines that there are overall violations in the audio and video streams. The dynamic replacement processing module, connected to the multi-model parallel detection and collaborative judgment module, is used to select global replacement or local masking methods according to the type of violation when an overall violation is determined, to replace the violating content and output compliant content.

Images