Yellow storm terrorism content identification and filtering method and system

Abstract

The invention discloses a yellow storm terrorism content identification and filtering method and system, and relates to the technical field of digital content auditing, and the method comprises the steps: carrying out the modal decomposition and preprocessing of a to-be-detected content, including the calculation of a single-modal sensitive probability and confidence, and recording the coordinates and confidence of a local image region; calculating a dynamic weight and performing normalization processing based on the confidence and the prior credibility of each mode; calculating a final fusion score by combining the sensitive probability and the weight of each mode, and performing content auditing decision according to a preset threshold value; and the modal priori credibility is updated through manual feedback data. According to the method, the detection rate of cross-modal violation contents is remarkably improved through a multi-modal collaborative analysis and confidence-driven dynamic fusion technology; and the system adopts a three-level decision-making mechanism to automatically allocate auditing resources, so that the manual rechecking cost is greatly reduced, meanwhile, an online learning function is introduced to continuously optimize the model, the sensitivity to novel violation contents is kept, and the processing efficiency and detail identification are both considered.

Description

Technical Field

[0001] This invention relates to the field of digital content moderation technology, and in particular to a method and system for identifying and filtering pornographic, violent, and terrorist content. Background Technology

[0002] With the rapid development of internet and mobile communication technologies, digital content is experiencing explosive growth, making content security review a core task for ensuring a healthy online environment. Especially in large-scale communication service scenarios, the multimodal content (text, images, audio, etc.) generated by users through apps, websites, social media, and other channels urgently requires efficient and accurate mechanisms for identifying inappropriate content.

[0003] Existing technologies in the field of content security review have significant shortcomings: First, traditional manual review is inefficient and highly subjective, making it difficult to handle the real-time processing demands of massive amounts of data. Second, single-modal review systems can only process single types of content, such as text, images, or audio / video, lacking multimodal collaborative analysis capabilities, resulting in a high rate of missed detections for cross-modal violations (such as combined text and images containing pornography, or audio containing sensitive words). Furthermore, existing fusion technologies do not achieve dynamic weight allocation; for example, low-confidence audio-to-text conversion is still prioritized, or sensitive areas in images are not linked to the global results, affecting overall accuracy. In addition, systems generally lack online learning mechanisms and cannot adapt to the rapid evolution of violating content. Therefore, a multimodal dynamic fusion, automated standard detection, and real-time feedback optimization technology is urgently needed to solve these problems. Summary of the Invention

[0004] In view of the problems that existing content security review technologies have, such as low efficiency and high subjectivity of manual review; inability of single-modal systems to handle cross-modal violations; lack of dynamic weight allocation mechanism for multimodal fusion and insufficient utilization of confidence; and lack of adaptive capability, making it difficult to respond to new types of violations in a timely manner, this invention is proposed.

[0005] Therefore, the problem to be solved by this invention is to provide a method and system for identifying and filtering pornographic, violent, and terrorist content.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, embodiments of the present invention provide a method for identifying and filtering pornographic, violent, and terrorist content, comprising: performing modal decomposition and preprocessing on the content to be detected, including calculating the sensitivity probability and confidence level of a single modality, and recording the coordinates and confidence level of local image regions; calculating dynamic weights and performing normalization processing based on the confidence level and prior trust level of each modality to determine the contribution ratio of different modalities in the fusion score; calculating the final fusion score by combining the sensitivity probability and weight of each modality, and making content review decisions based on a preset threshold; and updating the prior trust level of the modality through manual feedback data to achieve adaptive optimization of the system.

[0007] As a preferred embodiment of the method for identifying and filtering pornographic, violent, and terrorist content described in this invention, the content to be detected includes text, images, and audio, and the specific steps for modal decomposition and preprocessing of the content to be detected include: The text content is denoised, segmented into sentences and words. A Transformer-based text classifier is used to calculate the sensitivity probability and confidence score. The sensitivity probability is obtained by combining the probabilities of each subclass with the multi-label sigmoid output. The confidence score is calculated by weighted combination of calibrated probabilities. The image content is processed by multi-scale scaling. A lightweight model is used for full-image initial screening and a precise model is used for local region detection. The sensitivity probability and confidence of local regions are calculated respectively. The overall sensitivity probability and confidence of the image are obtained by region area weighting or maximum value strategy. The audio content is sampled and features are extracted. The sensitivity probability and confidence are calculated through the text path and acoustic feature path transcribed by ASR, respectively. The weights of the two paths are dynamically allocated according to the ASR confidence, and then the overall sensitivity probability and confidence of the audio are obtained by merging them.

[0008] As a preferred embodiment of the method for identifying and filtering pornographic, violent, and terrorist content described in this invention, the specific steps of calculating dynamic weights and performing normalization processing include: Based on the confidence level of each modality and the preset prior confidence level, unnormalized weights are calculated, where the prior confidence level reflects the long-term reliability of the modality, and the confidence level strengthens the weight of high-confidence samples through a mapping function. The unnormalized weights of all modalities are normalized to obtain the final weights of each modality, ensuring that the sum of all weights is 1, which is used for subsequent fusion scoring calculation.

[0009] As a preferred embodiment of the method for identifying and filtering pornographic, violent, and terrorist content described in this invention, the content review decision includes automatic interception, manual review, or direct approval. The specific steps for calculating the final fusion score and making the content review decision include: The final fusion score is obtained by multiplying the sensitivity probabilities of each modality by the normalized weights and then summing the results. The fusion score is used to make tiered decisions based on preset interception thresholds and manual review thresholds: if the fusion score is greater than or equal to the interception threshold, the content is automatically intercepted; if the fusion score is between the manual review threshold and the interception threshold, it is pushed to the manual review queue; if the fusion score is less than the manual review threshold, the content is directly allowed.

[0010] As a preferred embodiment of the method for identifying and filtering pornographic, violent, and terrorist content described in this invention, the specific steps of updating the prior trust level through manual feedback data include: Collect data from manual review results, user reports, or regulatory feedback, and count the number of true positive, false positive, true negative, and false negative samples for each modality within the sliding time window; The actual accuracy estimate of each mode is calculated based on the statistical results, and the prior confidence is updated by the exponential moving average method to ensure that the system can adaptively optimize the mode weight allocation strategy.

[0011] As a preferred embodiment of the method for identifying and filtering pornographic, violent, and terrorist content described in this invention, the calculation of the sensitivity probability and confidence level of the local image region further includes: The category confidence of each local region is output by the object detector, and the local confidence is calculated by combining the model ensemble consistency factor, where the model ensemble consistency factor is the ratio of prediction consistency across multiple models or scales. The overall sensitivity probability of the image can be calculated based on the sensitivity probability of local regions and area weighting, or the maximum value can be directly taken as the overall sensitivity probability of the image.

[0012] As a preferred embodiment of the method for identifying and filtering pornographic, violent, and terrorist content described in this invention, the calculation of the sensitivity probability and confidence level of the audio content further includes: In the ASR-transcribed text path, the ASR confidence is calculated using the geometric mean token probability, and the transcribed text is fed into a text classifier to obtain the sensitivity probability and confidence of the text path. In the acoustic feature path, the acoustic subclass confidence is calculated based on the Mel spectrum features using a sound classifier, and the sensitivity probability and confidence of the acoustic path are obtained by combining them. The weights of the text path and the acoustic path are dynamically assigned based on the ASR confidence level, and the results of the two paths are merged to obtain the overall audio sensitivity probability and confidence level.

[0013] Secondly, in order to further solve the problems existing in digital content review, the present invention provides a pornographic, violent and terrorist content identification and filtering system, which includes: a modality decomposition and preprocessing module, used to receive the content to be detected, perform data normalization processing on text, images and audio, generate original data and preprocessed copies, calculate the sensitivity probability and confidence of each modality respectively, and record the coordinates and confidence of local areas of the image. The dynamic weight calculation module is used to calculate dynamic weights and perform normalization processing based on the confidence level of each modality and the preset prior confidence level, so as to determine the contribution ratio of different modalities in the fusion score. The fusion scoring and decision module is used to calculate the final fusion score by combining the sensitivity probability and normalized weight of each modality, and to make content review decisions based on preset interception thresholds and manual review thresholds. The adaptive optimization module is used to update the prior confidence level of each modality through human feedback data, thereby achieving continuous optimization of the system.

[0014] Thirdly, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements any step of the method for identifying and filtering pornographic, violent, and terrorist content as described in the first aspect of the present invention.

[0015] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the method for identifying and filtering pornographic, violent, and terrorist content as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: 1. This invention significantly improves the ability to identify complex violations through multimodal collaborative review. It can not only process text, image and audio data simultaneously, but also intelligently deal with highly concealed violation forms such as the combination of text and images or the nesting of audio and video. For example, when the confidence of speech-to-text is insufficient, it will automatically switch to acoustic feature analysis, which effectively solves the problem that traditional single-modal systems are prone to missing cross-modal violation content. 2. The present invention innovatively adopts a dynamic fusion mechanism to comprehensively consider the inherent reliability and real-time confidence of different modalities. For example, text and images are usually given higher weights, while audio is appropriately reduced in weight due to the risk of transcription error. This flexible adjustment strategy avoids low-quality data dragging down the overall judgment accuracy. 3. The system adopts a three-level decision-making mechanism to intelligently allocate review resources. High-confidence violations are automatically blocked, medium-risk content is pushed to manual review, and low-risk content is directly approved. In practical applications, this can significantly reduce invalid manual reviews. 4. This invention uses an online learning function to automatically optimize the trust level of each modality by continuously analyzing the results of manual review, so that the system can keep up with the evolution trend of new violation methods; 5. This invention supports diverse input formats and hierarchical calculation strategies, ensuring both processing efficiency and detailed recognition capabilities. In particular, the local area focusing technology ensures that small-sized prohibited items are not missed. In actual testing, the entire system significantly improves the cross-modal violation detection rate while greatly reducing the false interception rate, providing an efficient and reliable solution for massive content review. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1This is a flowchart illustrating the implementation of the method of the present invention in Example 1.

[0018] Figure 2 This is a flowchart illustrating the implementation of the modal decomposition and preprocessing module of the present invention in Example 1.

[0019] Figure 3 This is a system structure diagram of the present invention in Example 2. Detailed Implementation

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0023] Example 1 Reference Figure 1 , Figure 2 This is the first embodiment of the present invention, which provides a method for identifying and filtering pornographic, violent, and terrorist content, including the following steps: Step S1: Perform modal decomposition and preprocessing on the content to be detected. The content to be detected includes text, images, and audio, and includes the following sub-steps: S1-1: Standardize data input from different sources, including text, images, and audio, and save original and preprocessed copies according to a predetermined sampling / scaling strategy to ensure a traceable data link for subsequent processing; Specifically, the supported input types are: text (short text, long text, comments), images (static images, keyframes), audio clips (mono or stereo, sampling rate 16 kHz or 48 kHz), and video (frame-by-frame extraction). For example, the sampling / preprocessing parameters are: Image: Maximum side length 1536 px to ensure detail; Image copies are fed into the lightweight and precise models at two scales: 224×224 and 512×512.

[0024] Audio: Sampling rate 16 kHz, frame length 25 ms, frame shift 10 ms; Speech and audio feature extraction can be performed using the industry-standard Short Time Fourier Transform (STFT); Specifically, when extracting speech and audio features using Short Time Fourier Transform (STFT), the audio signal is first processed by dividing it into frames, each typically 25 milliseconds long, with a frame shift of 10 milliseconds to balance time and frequency resolution. After windowing (e.g., Hamming windowing) each frame to reduce spectral leakage, a Fast Fourier Transform (FFT) is performed to convert the time-domain signal into a frequency-domain representation, obtaining the spectrum of that frame. The power spectrum is obtained by calculating the square of the spectral amplitude. A Mel filter bank (e.g., 40 filters) is then applied to filter and logarithmically compress the power spectrum, ultimately yielding feature vectors such as Mel frequency cepstral coefficients (MFCC). Through this process of framing, windowing, transforming, and filtering, STFT effectively captures the time-frequency local characteristics of the audio signal, providing crucial features for subsequent speech recognition or audio classification.

[0025] Text: The original text and the text recognized by OCR and ASR are uniformly denoised, and then segmented into sentences and words; Specifically, to achieve unified denoising, sentence segmentation, and word segmentation of the original text and the text recognized by OCR and ASR, the following steps can be taken: First, denoise the text by removing control characters (such as newline characters, tab characters, etc.) and HTML tags (such as... 、 First, the text is preserved in its plain text state. Second, it is split into independent sentences based on punctuation marks (such as periods, question marks, exclamation marks, etc.) to ensure semantic integrity. Finally, for Chinese text, the Jieba word segmenter is used for word segmentation. This process ensures the standardization and structuring of the text, providing clean input data for subsequent multimodal content analysis and sensitivity probability calculation.

[0026] S1-2: Output the sensitivity probability for each mode using a dedicated model. With confidence level And the coordinates and confidence scores of the local image regions are recorded together; Specifically, define the modality set. , respectively represent (text), (image), (Audio), the single-modal output sensitivity probabilities are denoted as follows: The confidence levels are denoted as follows: If a local region of the image exists Then the local probability and confidence level are: , The area of ​​the region pixels is The sensitivity probability and confidence level for each modality are then output in the following ways: Text modality: Sensitivity probability calculation: Text classifiers based on Transformer models (such as BERT-family) use multi-label sigmoid output, which is suitable for scenarios where text may contain multiple types of violation features simultaneously; the output layer is the probability distribution for each sensitive subclass. For example, "pornography / violence / terrorism / involving minors," etc., all have a The class probability is obtained through the sigmoid function:

[0027] In the formula, It is the sigmoid activation function; It is the base of the natural logarithm (approximately 2.71828). This indicates that the text classification network represents the first... The original logit value output by each violating subclass; This represents a probability estimate of a single subclass after applying the sigmoid function. Furthermore, the final text modality probability of "any violation" is defined as the probability of "at least one subclass being true":

[0028] In the formula, This indicates the probability that the text contains any of the prohibited content categories. If the user prefers "By Highest Category," it can be replaced with... ; Confidence score calculation: The original confidence score can be taken as the confidence score of the highest category, and to ensure interpretability and comparability, the maximum sigmoid value or the average value is used as the confidence score. : First, apply temperature scaling to logits: In the formula The temperature scaling factor representing logits can be optimally obtained by minimizing the negative log-likelihood (NLL) on the calibration set. Specifically, this can be represented by the following formula:

[0029] In the formula, The label (0 / 1) indicates the first (0 / 1)th ... The sample at the th Manually labeled sensitive subclasses, where 1 indicates that the subclass is true (exists) and 0 indicates that it does not exist; To verify the sample size; The model represents the samples Subclass The original logit output (i.e., the real number fraction without sigmoid / softmax); Furthermore, using the calibrated probability ,definition The weighted combination of class average probabilities can be represented by the following formula:

[0030] In the formula, confidence level This represents the calibration estimate of the text classifier's "self-confidence" on this sample.

[0031] Image modality: Lightweight and fast models, such as MobileNet / ResNet-18 (for initial screening of the entire image, outputting top-K candidate suspicious regions), and precise models, such as Mask R-CNN / Cascade R-CNN / ViT + segmentationheads (for human body detection, naked area identification, and object recognition), are employed, and an object detector is used to evaluate each candidate region. Output category confidence ; Local sensitivity probability Calculation: If a single-label softmax detector is used, the probability of the corresponding violation class is directly taken. As If a multi-class sigmoid detector is used, for each violation subclass... get Define the probability of any violation in a local area:

[0032] Local confidence The calculation can be expressed by the following formula:

[0033] In the formula, This represents the detector's original box score value; The model integration consistency factor, defined as the ratio of prediction consistency among multiple models (or the same model at multiple scales), can be expressed as:

[0034] In the formula, The value is in the range [0,1], and is 1 if all models agree; Overall image sensitivity probability The calculation formula is: ; Overall confidence of the image The calculation formula is: .

[0035] The audio modality consists of two parts: text determination based on ASR. Acoustic models based on direct determination of audio features , merge and obtain and Specifically: Text path: Text transcribed from ASR and word-by-word / frame-by-frame probability ASR confidence level The token probability, defined as the geometric mean, can be expressed by the following formula:

[0036] In the formula, The number of tokens represents the overall confidence level of the transcription link; the closer it is to 1, the more reliable it is. Provided by ASR; if ASR does not provide word-by-word probabilities, the same scale can be obtained by exponentializing the log-likelihood average provided by ASR. Furthermore, the ASR text is fed into the aforementioned text classifier to obtain... and ; Pure acoustic approach: Sound classification based on a commonly used CNN model in the industry (input is Mel spectrum (number of Mel filter banks = 40)); output is the confidence score for each acoustic subclass. ,in The activation function is Sigmoid, and the category probability calculation formula is referred to in the text classifier section above; Specifically, CNN model frameworks such as ResNet receive 40-channel Mel spectrograms as input. The network includes convolutional layers (extracting local frequency / temporal features), pooling layers (dimensionality reduction), and fully connected layers (classification). The output layer uses the sigmoid activation function to output independent probability values ​​for each acoustic subclass (e.g., "violent audio," "sensitive speech," etc.). ; The audio acoustic probability is defined as:

[0037] Audio confidence The maximum value can be obtained by using the classifier softmax. Overall audio sensitivity probability The calculation formula is: ; Overall audio confidence The calculation formula is: ; In the formula, Weights can be adaptively assigned based on ASR confidence levels, for example:

[0038] Specifically, when ASR is reliable, text-based judgment is preferred; when ASR is unreliable, acoustic features are relied upon.

[0039] It should be noted that this step involves modal decomposition and preprocessing of the content to be detected. First, text, image, and audio data from different sources are standardized to ensure consistent data format and traceability. Next, a dedicated model is used for each modality to calculate its sensitivity probability and confidence level. For example, text is analyzed for sensitive words using a classifier, images are identified for local sensitive regions using a detection model, and audio is assessed using a combination of speech-to-text conversion and acoustic features. Simultaneously, the coordinates and confidence levels of local image regions are recorded to provide detailed information for subsequent fusion analysis.

[0040] Step S2: Calculate the confidence scores and weights for each modality, which includes the following sub-steps: S2-1: For each mode Define unnormalized weights: ; In the formula, The original measure representing the "confidence of this modality's contribution" under the current sample conditions, taking into account both the model's own confidence and the long-term reliability of this modality; The modal prior trust level, determined by system configuration or offline evaluation, reflects the prior engineering knowledge of the overall reliability of that modality (e.g., textual information). ,image Audio ); Here is the confidence mapping function, where Strengthen the weight of high-confidence samples. (Commonly used) (linear), or (Amplifying high-confidence differences).

[0041] S2-2: Normalize the unnormalized weights to the final weights. This can be reflected by the following formula:

[0042] In the formula, This represents the sum of all modal unnormalized weights; and .

[0043] It should be noted that this step dynamically assigns weights and performs normalization based on the confidence level of each modality and a preset prior confidence level. By comprehensively considering the real-time confidence level of the modality (e.g., text is generally more reliable) and historical performance (e.g., audio may be weighted less), the contribution ratio of each modality in the final score is calculated. This step ensures that high-confidence modalities have a greater impact on the results, thereby improving the accuracy of the overall judgment.

[0044] Step S3: Fusion score calculation and local synthesis, including the following sub-steps: S3-1: Sensitivity probability based on output of each modality and weight Calculate the final fusion score Specifically, this can be represented by the following formula:

[0045] In the formula, This indicates the "overall confidence level of being judged as pornography / violence / terrorism".

[0046] S3-2: Calculate the local area normalization weights This can be reflected by the following formula:

[0047] In the formula, This represents the sum of the areas of all candidate local regions; At this point, the overall probability of the image It can be calculated using an area-weighted average:

[0048] It should be noted that, at this point, the area-weighted average focuses more on the "percentage of violations".

[0049] S3-3: Define the automatic interception threshold as... (e.g., 0.75), the threshold for manual review is (e.g., 0.40), then the decision strategy is as follows: If the final fusion score If so, it will be automatically blocked and reported; If the final fusion score If so, it will enter the manual review queue; If the final fusion score If so, the application will be approved directly.

[0050] For example: Let the output of the sample model be: text: , ;image: , Audio: , ; Set prior trust level : , , ; use ,Right now The unnormalized weights are: ; ; ; Summation: ; Normalized weights: ; ; ; Calculate fusion score : + ; Set threshold , : because Therefore, it enters the manual review queue.

[0051] It should be noted that this step combines the sensitivity probability of each modality with dynamic weights to calculate the final fusion score, and makes decisions based on preset thresholds. Content with scores higher than the interception threshold is automatically blocked, content with medium scores enters the manual review queue, and content with low scores is directly allowed. In addition, area-weighted processing is applied to local areas of the image to ensure that small-sized violations are not missed, achieving accurate hierarchical review.

[0052] Step S4: Online feedback and adaptive weight update, including the following sub-steps: S4-1: Feedback Data Collection: Record manual review, user reports, and regulatory feedback as log items, including: Sample ID, modal raw data pointer, manual tag (violation / non-violation), reviewer ID, and review timestamp; within a sliding time window. (For example, the most recent 30 days or the most recent 10,000 samples) Calculate the true positives for each modality. False positive True Yin False negative ; S4-2: Define the actual accuracy estimation of the modality:

[0053] In the formula, Representing modes The number of samples that are automatically identified as violations within the statistical window and are also identified as violations after manual review (the number of true positives). Representing modes The number of samples that are automatically judged as violations within the window but are judged as non-violations after manual review (the number of false positives); and if the denominator is 0 (e.g., no judgment record), then no update is triggered directly; Furthermore, the prior confidence level is updated using an exponential moving average (EMA). :

[0054] In the formula, This represents the updated prior trust level. This indicates the prior trust level before the update; This represents the decay factor (e.g., 0.9), indicating a slow update rate. Furthermore, the updated version will be used in the next fusion. calculate ; It is important to note that, to avoid drastic fluctuations caused by a small sample size, a minimum sample size threshold can be set, for example, only when... Update when ≥100 Store and periodically (e.g., daily / weekly) rollback / verification Update the strategy and record the change history for auditing purposes.

[0055] It should be noted that this step dynamically updates the prior trust level of each modality by collecting feedback data such as manual review and user reports. The system periodically calculates indicators such as true positives and false positives for each modality and adjusts the weight benchmark using an exponential moving average algorithm, enabling the model to adaptively optimize and continuously improve its ability to identify new types of illegal content. This mechanism ensures the long-term effectiveness and adaptability of the system.

[0056] In summary, this invention provides a highly efficient and accurate method for identifying and filtering pornographic, violent, and terrorist content. It achieves collaborative analysis and comprehensive judgment of text, image, and audio content through multimodal dynamic fusion technology. The method first performs standardized preprocessing and feature extraction on the data from each modality, using a dedicated model to output sensitivity probabilities and confidence levels. Then, it dynamically allocates weights based on confidence levels and modal prior trust levels to achieve adaptive fusion of cross-modal information. Simultaneously, it introduces an online feedback mechanism to continuously optimize model performance and combines it with hierarchical decision-making strategies, such as automatic interception, manual review, and direct release, to achieve efficient review. This method effectively solves the technical challenges of low efficiency in traditional manual review, high false negative rates in single-modal systems, and insufficient multimodal fusion, significantly improving the accuracy of identifying illegal content and the system's adaptive capabilities, providing reliable technical support for large-scale digital content review.

[0057] Example 2 Example 2 is the second embodiment of the present invention. This embodiment differs from the first embodiment in that it also provides a system for identifying and filtering pornographic, violent, and terrorist content, including: Modal decomposition and preprocessing module This module is used to receive the multimodal content to be detected and perform the following operations: Text processing unit: performs noise reduction, sentence segmentation and word segmentation on the input text data, and uses a Transformer-based text classifier to calculate the text sensitivity probability and confidence level; Image processing unit: performs multi-scale scaling on the input image, and feeds it into the MobileNet lightweight model and the MaskR-CNN precise model respectively for full-image initial screening and local region detection, outputting the sensitivity probability, confidence level and coordinate information of the local region; Audio processing unit: Samples and extracts features from the input audio, calculates the sensitivity probability and confidence level through the ASR-transcribed text path and the acoustic feature path based on the CNN classifier, and dynamically fuses the results of the two paths based on the ASR confidence level.

[0058] Dynamic weight calculation module This module is used to calculate and normalize dynamic weights based on the confidence level of each modality and a preset prior confidence level. Specifically, it includes: Weight calculation unit: Calculates unnormalized weights using a confidence mapping function and prior confidence. Normalization unit: Normalizes the unnormalized weights of all modalities to ensure that the sum of the weights of all modalities is 1, which is used for subsequent fusion scoring calculation.

[0059] Integration of scoring and decision-making modules This module is used to integrate the sensitivity probabilities and weights of each modality, generate the final fusion score, and execute the review decision. Specifically, it includes: Scoring calculation unit: The fusion score is obtained by multiplying the sensitivity probability of each modality by the normalized weight and then summing the results. Decision execution unit: Automatically executes interception, pushes for manual review, or directly releases the application based on preset thresholds.

[0060] Adaptive optimization module This module dynamically updates system parameters based on user-submitted feedback data, specifically including: Data collection unit: Records the number of true positives in manually reviewed samples. Number of false positives , number of true negatives Number of false negatives Statistical information; Trust update unit: Calculates the actual accuracy of each modality based on the statistical results within the sliding time window, and updates the prior trust through exponential moving average to optimize the subsequent weight allocation strategy.

[0061] This system significantly improves the accuracy of identifying cross-modal violations through multimodal collaborative analysis and dynamic weight allocation; the three-level decision-making mechanism, such as automatic interception, manual review, and direct release, effectively reduces the cost of manual review; and the online learning function enables the system to continuously optimize and adapt to the evolving trend of new types of violations.

[0062] This embodiment provides a computer device applicable to a method for identifying and filtering pornographic, violent, and terrorist content, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the method for identifying and filtering pornographic, violent, and terrorist content as proposed in the above embodiment.

[0063] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0064] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements a method for identifying and filtering pornographic, violent, and terrorist content as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0065] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for identifying and filtering yellow, violent and terror content, characterized in that, The method comprises the following steps: modal decomposition and preprocessing of the content to be detected, including multiple modal sensitivity probability and confidence calculation, and recording the coordinates and confidence of the local image area; based on the confidence and prior trust of each modal, calculate the dynamic weight and normalize it, which is used to determine the contribution proportion of different modal in the fusion score; combine the sensitivity probability and weight of each modal to calculate the final fusion score, and make content review decision according to the preset threshold; update the modal prior trust through artificial feedback data, which is used to realize the adaptive optimization of the system.

2. The method of claim 1, wherein the yellow, violent, and terrorist content recognition and filtering method is characterized by, The content to be detected includes text, image and audio, and the specific steps of modal decomposition and preprocessing of the content to be detected include: denoising, sentence segmentation and word segmentation processing of text content, using Transformer-based text classifier to calculate sensitivity probability and confidence, wherein sensitivity probability is output by multi-label sigmoid and the overall violation probability is obtained by combining each sub-class probability, and confidence is calculated by weighted combination of calibrated probability; multi-scale scaling processing is performed on the image content, a lightweight model is used for full image preliminary screening, and an accurate model is used for local area detection, the sensitivity probability and confidence of local area are calculated respectively, and the image overall sensitivity probability and confidence are obtained through area weighting or maximum value strategy; audio content is sampled and feature extracted, sensitivity probability and confidence are calculated through text path and acoustic feature path converted by speech recognition ASR respectively, and the weight of the two paths is adaptively allocated according to the ASR confidence, and the overall sensitivity probability and confidence of audio are obtained by merging.

3. The method of claim 1, wherein the step of identifying the yellow / brown / red content comprises the steps of: determining whether the content is a yellow content; and determining whether the content is a brown content. The specific steps of calculating the dynamic weight and normalizing it include: according to the confidence of each modal and the preset prior trust, calculate the unnormalized weight, wherein the prior trust reflects the long-term reliability of the modal, and the confidence strengthens the weight of high confidence samples through mapping function; normalize the unnormalized weight of all modes to get the final weight of each modal, ensure that the sum of all weights is 1, and use it for subsequent fusion score calculation.

4. The method of claim 1, wherein the step of identifying the yellow / brown / red content comprises the steps of: The content review decision includes automatic interception, manual review or direct release, and the specific steps of calculating the final fusion score and making the content review decision include: ​ multiply the sensitivity probability of each modal by the normalized weight and sum them up to get the final fusion score; according to the preset interception threshold and manual review threshold, the fusion score is classified and decided: if the fusion score is greater than or equal to the interception threshold, the content is automatically intercepted; if the fusion score is between the manual review threshold and the interception threshold, it is pushed to the manual review queue; if the fusion score is less than the manual review threshold, the content is directly released.

5. The method of claim 1, wherein the step of identifying the yellow / brown / red content comprises the steps of: determining whether the content is a yellow content; and determining whether the content is a brown content. The specific steps of updating the prior trust through artificial feedback data include: collect manual review results, user reports or regulatory feedback data, and count the number of true positive, false positive, true negative and false negative samples of each modal in the sliding time window; calculate the actual precision estimate of each modal according to the statistical results, and update the prior trust by exponential moving average method, which is used to ensure that the system can adaptively optimize the modal weight allocation strategy.

6. The method of claim 2, wherein the step of identifying the yellow / brown / red content comprises the steps of: The sensitivity probability and confidence calculation of the image local area also includes: ​ The object detector outputs a class confidence of each local region, and a local confidence is calculated in combination with a model ensemble consistency factor, which is a ratio of prediction consistency under multiple models or scales; The overall sensitive probability of the image is calculated according to the sensitive probability and area weight of the local region, or the maximum value is directly taken as the overall sensitive probability of the image.

7. The method of claim 2, wherein the step of identifying the yellow / brown / red content comprises the steps of: The sensitive probability and confidence of the audio content further include: ​ In the ASR transcription text path, the ASR confidence is calculated by geometrically averaging the token probability, and the transcription text is sent into the text classifier to obtain the sensitive probability and confidence of the text path; In the acoustic feature path, the acoustic sub-class confidence is calculated by the sound classifier based on the mel spectrum feature, and the sensitive probability and confidence of the acoustic path are obtained by integration; The weights of the text path and the acoustic path are dynamically allocated according to the ASR confidence, and the overall sensitive probability and confidence of the audio are obtained by merging the results of the two paths.

8. A system for identifying and filtering pornographic, violent and terrorist content, based on the method for identifying and filtering pornographic, violent and terrorist content according to any one of claims 1 to 7, characterized in that, It includes: A modal decomposition and preprocessing module is configured to receive content to be detected, perform data normalization processing on text, image and audio, generate original data and preprocessed copies, and calculate the sensitive probability and confidence of each modality, while recording the coordinates and confidence of the local region of the image; A dynamic weight calculation module is configured to calculate dynamic weights based on the confidence of each modality and a preset prior trust degree, and perform normalization processing to determine the contribution proportion of different modalities in the fusion score; A fusion score and decision module is configured to calculate the final fusion score in combination with the sensitive probability and normalized weight of each modality, and make content review decisions according to a preset interception threshold and a manual review threshold; An adaptive optimization module is configured to update the prior trust degree of each modality through manual feedback data to realize continuous optimization of the system. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is characterized in that: The processor executes the computer program to realize the steps of the yellow, violent and terrorist content identification and filtering method of any one of claims 1-7.

10. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by the processor to realize the steps of the yellow, violent and terrorist content identification and filtering method of any one of claims 1-7.

Images