Intelligent identification method for juvenile bad network content based on cross-modal feature fusion

Abstract

The application provides an underage bad network content intelligent identification method based on cross-modal feature fusion, relates to the technical field of network content security and underage protection, and comprises the following steps: acquiring multi-modal data corresponding to network content to be identified, wherein the multi-modal data comprises video data, audio data and network text information, and a multi-modal sequence is constructed according to a unified time axis; and through the fusion of multi-modal data and the alignment of cross-modal features, the underage bad network content intelligent identification method realizes accurate identification and classification of network content, a multi-modal sequence is constructed according to a unified time axis and image, audio and text features are extracted, cross-modal alignment is realized through a shared semantic feature space, semantic complementation between different modal data is realized, and the identification accuracy and classification efficiency for underage bad network content are improved.

Description

Technical Field

[0001] This invention relates to the field of network content security and minor protection technology, specifically to an intelligent identification method for inappropriate online content for minors based on cross-modal feature fusion. Background Technology

[0002] In existing technologies, the identification and review of inappropriate online content for minors is usually achieved by combining rule-based strategies with machine learning or deep learning: one type is rule-based filtering based on sensitive word databases, blacklists, regular expression matching, and tag interception; another type is single-modal intelligent recognition, such as semantic classification of text, detection of pornography or violence by extracting frames from images or videos, and speech recognition of audio before reviewing the text; there is also a more common multi-stage pipeline, which summarizes the results of modules such as frame extraction detection, OCR subtitle recognition, and ASR speech transcription, and then gives a conclusion on whether inappropriate content has been detected by threshold or weighted rules, in order to meet the platform's need for automated review of massive amounts of multimedia content.

[0003] However, the above solutions generally suffer from cross-modal semantic fragmentation and rough integration: when malicious intent is scattered across different modalities or time segments such as caption stickers, cryptic text, background audio dialogue, comment guidance, and visual cues, single-modal or simple weighting is difficult to form a consistent overall understanding, which can easily lead to false positives and false negatives. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an intelligent identification method for inappropriate online content for minors based on cross-modal feature fusion. The technical problem this invention aims to solve is: how to address the issues of cross-modal semantic fragmentation and coarse fusion through cross-modal feature fusion and temporal contribution weight calculation methods.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an intelligent identification method for inappropriate online content for minors based on cross-modal feature fusion, comprising: S1. Obtain multimodal data corresponding to the network content to be identified. The multimodal data includes video data, audio data, and network text information. Construct a multimodal sequence according to a unified timeline. The construction includes: dividing the video data into multiple time segments and extracting image frame sequences from each time segment; performing optical character recognition on the image frame sequences to obtain subtitle or sticker text sequences; performing speech recognition on the audio data according to the time segments to obtain dialogue text sequences. The network text information includes title text, introduction text, tag text, and comment text. The network text information, the subtitle or sticker text sequences, and the dialogue text sequences are combined to form text data. The network text information is used as a global context and associated with each time segment. S2. For each time segment, feature extraction is performed on the image frame sequence, audio data and text data to obtain corresponding visual features, audio features and text features. The visual features, audio features and text features are mapped to the same shared semantic feature space through projection transformation to obtain the visual shared representation, audio shared representation and text shared representation and global context shared representation of each time segment. S3. For each time segment, within the shared semantic feature space, cross-modal alignment weights are determined based on the similarity between the text shared representation and the visual shared representation, and between the text shared representation and the audio shared representation. The cross-modal alignment weights are obtained by normalizing the similarity. Under the constraint of the cross-modal alignment weights, cross-attention fusion is performed using the text shared representation as the query vector and the visual shared representation and the audio shared representation as the key vectors to obtain the cross-modal fused feature representation of the same time segment. The cross-modal fused feature representation is then made to interact with the global context shared representation to introduce comment guidance, title summary and tag context, thereby achieving alignment and aggregation of information scattered in subtitle or sticker text, veiled text, background audio dialogue and visual cues in the same semantic space. S4. Introduce temporal location information into the cross-modal fusion feature representation of each time segment, calculate the temporal contribution weight of each time segment, and perform weighted aggregation on the cross-modal fusion feature representation of each time segment based on the temporal contribution weight to obtain a network content-level risk characterization. Perform linear mapping and normalization calculation on the network content-level risk characterization to obtain the category of inappropriate content and the corresponding risk score, and determine whether the network content to be identified belongs to inappropriate network content for minors based on the comparison result of the risk score and the preset threshold.

[0006] Preferably, the corresponding association includes: copying and associating the title text, description text, and tag text into the global context text of the same time segment within each time segment; when the comment text contains a time identifier, mapping the comment text to the corresponding time segment in which the time identifier falls according to the time identifier; when the comment text does not contain a time identifier, copying and associating the comment text to each time segment.

[0007] Preferably, the division of time segments includes: setting the duration of each time segment to 0.5s-5s, and setting a 10%-50% time overlap between adjacent time segments to ensure the continuous representation of malicious intent when it occurs across segments.

[0008] Preferably, the extraction of the image frame sequence includes: uniformly extracting frames from the time segment at a preset extraction frequency to obtain a basic frame set, and selecting key frames from the basic frame set based on the scene variability of adjacent frames. The scene variability is calculated from the color histogram difference of adjacent frames. When the scene variability is greater than a preset threshold, the corresponding frame is determined as a key frame.

[0009] Preferably, when the optical character recognition is performed, the confidence level and text region location information of each subtitle or sticker text are obtained simultaneously, and subtitles or sticker texts with a confidence level less than a first threshold are removed. When the speech recognition is performed, the start and end times and confidence levels of each dialogue text are obtained simultaneously, and dialogue texts with a confidence level less than a second threshold are removed, so that the text data entering subsequent fusion has a time alignment basis and reliability constraints.

[0010] Preferably, the projection transformation mapping includes: performing linear transformations on visual features, audio features, and text features respectively to unify the feature dimension as a preset dimension, and normalizing the transformed features to make the visual shared representation, audio shared representation, and text shared representation under the same metric scale, so that the cross-modal alignment based on similarity is comparable.

[0011] Preferably, the similarity is calculated using cosine similarity, and the cross-modal alignment weight is obtained as follows: cosine similarity is calculated between the text shared representation and the visual shared representation and the audio shared representation within the same time segment, and the cosine similarity is normalized using the Softmax function to obtain the cross-modal alignment weight.

[0012] Preferably, the calculation of the temporal contribution weight includes: concatenating the cross-modal fusion feature representation and the global context-shared representation of each time segment to obtain the joint representation of the time segment; inputting the joint representation into a gating calculation unit to obtain the original weight score of the time segment; and the gating calculation unit using the Sigmoid function to perform Softmax normalization on the original weight scores of all time segments to obtain the temporal contribution weight of each time segment.

[0013] This invention provides an intelligent method for identifying inappropriate online content for minors based on cross-modal feature fusion. It has the following beneficial effects: This intelligent identification method for inappropriate online content targeting minors, based on cross-modal feature fusion, achieves accurate identification and classification of online content through the fusion of multimodal data and cross-modal feature alignment. It constructs multimodal sequences along a unified timeline and extracts image, audio, and text features. By sharing a semantic feature space for cross-modal alignment, it achieves semantic complementarity between different modal data, thereby improving the identification accuracy and classification efficiency of inappropriate online content targeting minors.

[0014] By employing a cosine similarity-based cross-modal alignment method and a time-series weighted convergence strategy, the accuracy and robustness of the identification are improved. By incorporating temporal location information and cross-modal fusion features, the system considers the temporal and contextual aspects of the information, ensuring contextual consistency during the identification process and achieving accurate classification and risk assessment of complex network content. Attached Figure Description

[0015] Figure 1 This is an overall flowchart of the method of the present invention; Figure 2 This is a schematic diagram of cross-modal fusion according to the present invention; Figure 3 This is a schematic diagram of the timing convergence and identification of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1 like Figure 1-3 As shown, this embodiment of the invention provides an intelligent identification method for inappropriate online content for minors based on cross-modal feature fusion, including: S1. Obtain multimodal data corresponding to the network content to be identified. Multimodal data includes video data, audio data, and network text information. Construct a multimodal sequence along a unified timeline. This construction includes: dividing the video data into multiple time segments and extracting image frame sequences from each time segment; performing optical character recognition (OCR) on the image frame sequences to obtain subtitle or sticker text sequences; performing speech recognition on the audio data according to time segments to obtain dialogue text sequences; and including title text, description text, tag text, and comment text. The network text information, subtitle or sticker text sequences, and dialogue text sequences together constitute text data. The network text information is used as a global context and associated with each time segment. The association includes: copying and associating title text, description text, and tag text within each time segment as global context text for the same time segment; mapping comment text to the corresponding time segment where the time marker falls when it contains a time marker; and copying and associating comment text with each time segment when it does not contain a time marker. The time segment division includes: setting the time segment duration to 0.5s-5s, and setting a 10%-50% time overlap between adjacent time segments to ensure continuous representation when malicious intent occurs across segments. Image frame sequence extraction includes: uniformly extracting frames from the time segments at a preset extraction frequency to obtain a base frame set, and selecting keyframes from the base frame set based on the scene variability of adjacent frames. The scene variability is calculated from the color histogram differences between adjacent frames; when the scene variability exceeds a preset threshold, the corresponding frame is identified as a keyframe. During optical character recognition, the confidence level and text region location information of each subtitle or sticker text are simultaneously obtained, and subtitles or sticker texts with a confidence level less than a first threshold are removed. During speech recognition, the start and end times and confidence levels of each dialogue text are simultaneously obtained, and dialogue texts with a confidence level less than a second threshold are removed, ensuring that the text data entering subsequent fusion has a time alignment basis and reliability constraints.

[0018] S2. For each time segment, feature extraction is performed on the image frame sequence, audio data, and text data to obtain corresponding visual features, audio features, and text features. These features are then mapped to the same shared semantic feature space via projection transformation, resulting in shared visual, audio, and text representations for each time segment, as well as a shared global context representation. The projection transformation mapping includes: performing linear transformations on the visual, audio, and text features respectively, using a unified feature dimension as a preset dimension, and normalizing the transformed features to ensure that the shared visual, audio, and text representations are on the same metric scale, making cross-modal alignment based on similarity comparable.

[0019] S3. For each time segment, within the shared semantic feature space, cross-modal alignment weights are determined based on the similarity between the text shared representation and the visual shared representation, and between the text shared representation and the audio shared representation. These cross-modal alignment weights are obtained by normalizing the similarity. Under the constraint of these weights, cross-attention fusion is performed using the text shared representation as the query vector and the visual and audio shared representations as key vectors to obtain the cross-modal fused feature representation for the same time segment. This cross-modal fused feature representation then interacts with the global context shared representation to introduce comment guidance, title summaries, and tag context, achieving alignment and aggregation of information scattered across subtitles or sticker text, implicit copy, background audio dialogue, and visual cues within the same semantic space. Similarity is calculated using cosine similarity. The cross-modal alignment weights are obtained as follows: cosine similarity is calculated between the text shared representation and the visual and audio shared representations within the same time segment, and the cosine similarity is normalized using the Softmax function to obtain the cross-modal alignment weights.

[0020] S4. Incorporate temporal location information into the cross-modal fusion feature representations of each time segment, calculate the temporal contribution weight of each time segment, and perform weighted aggregation of the cross-modal fusion feature representations of each time segment based on the temporal contribution weight to obtain a network content-level risk characterization. Perform linear mapping and normalization calculations on the network content-level risk characterization to obtain the category of inappropriate content and its corresponding risk score. Based on the comparison result of the risk score and a preset threshold, determine whether the network content to be identified belongs to inappropriate network content for minors. The calculation of the temporal contribution weight includes: concatenating the cross-modal fusion feature representation of each time segment with the global context-shared representation to obtain the joint representation of that time segment; inputting the joint representation into the gating calculation unit to obtain the original weight score of the time segment; the gating calculation unit uses the Sigmoid function to perform Softmax normalization on the original weight scores of all time segments to obtain the temporal contribution weight of each time segment.

[0021] This invention improves the accuracy and robustness of identifying inappropriate online content for minors by fusing multimodal data such as video, audio, and text. Precise temporal segmentation and overlapping design ensure continuous identification of inappropriate information across segments. Feature extraction and projection transformation map different data into a shared semantic space, improving intermodal alignment accuracy. Confidence threshold filtering effectively eliminates low-quality data, preventing interference with identification results. Optimized cross-modal alignment weights and the introduction of temporal contribution weights ensure the accuracy of information fusion and risk assessment. The system can automatically assess the risks of online content, is widely applicable to various online platforms, and provides effective online safety protection for minors.

[0022] Example 2 This embodiment is a method for intelligent identification of inappropriate online content for minors based on cross-modal feature fusion. Through multimodal data fusion, it achieves intelligent identification of inappropriate online content for minors based on user-uploaded video, audio, and online text information. The specific implementation method is as follows: 1. Video data The input network content is a 120-second user-uploaded video, which is obtained from the short video category of a video platform. The content involves behaviors that occur in primary and secondary school settings and are mainly committed by minors.

[0023] The video data was divided into 24 time segments, each lasting 5 seconds.

[0024] 24 frames are uniformly extracted from the video. Video frame extraction is performed using a fast tool for processing MPEG standard media files. The scene variability is assessed based on the color histogram differences between adjacent frames to determine whether to select keyframes.

[0025] Use the selection command of the quick tool for processing MPEG standard media files to extract one frame every 5 seconds. If the difference between adjacent frames exceeds a preset threshold, it is considered a keyframe.

[0026] 2. Audio data The audio data comes from background music and dialogue that are synchronized with the video content.

[0027] The total duration of the audio data is 120 seconds. The audio is divided into 24 time segments synchronized with the video data, each time segment lasting 5 seconds.

[0028] The audio was converted to text using the URL1 speech recognition interface. The following is a portion of the dialogue text after the audio data was converted: Time segment 1: 0-5 seconds "I'm warning you, you'd better do as I say." Time Segment 2: 5-10 seconds, "If you don't do as I say, you won't have a good time at school." Time Segment 3: 10-15 seconds, "No one will help you, you'd better be sensible." " Data for other time segments follow the same pattern.

[0029] 3. Online text information Title text: The title of the video is: "A clip of verbal threats and humiliation against minors on campus".

[0030] Description text: The video description reads: "This video contains content related to school bullying, such as verbal threats, humiliation, and harassment of minors."

[0031] Tag text: The video's tags include: "school bullying", "minors", and "inappropriate content".

[0032] Comment text: Comment text comes from the video platform, and comment data is collected from the video's comment section. For comments without time markers, sentiment analysis and keyword matching methods are used to associate them with the theme of the video content. If a specific time period cannot be determined, the comment will be copied and associated with all time segments to avoid information loss.

[0033] The following are some of the comments and their timestamps: Comment 1: "This kind of person deserves to be taught a lesson; he's too cowardly." (Time stamp: 10 seconds)

[0034] Comment 2: "If you don't listen, don't even think about getting by in school." (Time stamp: 30 seconds)

[0035] Comment 3: "Let's all isolate him and see if he dares to do it again." (Time stamp: 60 seconds)

[0036] Comment 4: "This kind of student just needs to be disciplined." (No time stamp)

[0037] Comment 5: "He deserves to be bullied; who told him to be so weak?" (Time stamp: 90 seconds)

[0038] Data association: Global text information will be copied and associated with each time segment.

[0039] Comments with time stamps will be mapped to the corresponding time segments. Comments without time stamps will be associated with the most relevant time segments based on the theme of the video content, sentiment analysis results, or keywords. If a specific time segment cannot be determined, it will be copied and associated with all time segments.

[0040] 4. Extraction of image frame sequences When extracting image features, a dynamic keyframe selection strategy based on scene variability was adopted. A convolutional neural network model was used to perform preliminary processing on video frames, and the scene variability was re-evaluated by combining image features and audio information.

[0041] One frame is extracted from each video time segment. Using the selection command of the fast tool for processing MPEG standard media files, one frame is extracted every 5 seconds, resulting in 24 image frames. These image frames are then fed into an image processing model for feature extraction.

[0042] The scene variability threshold, determined through testing in various video scenarios, is 0.5, making it suitable for most video scenarios. Scene variability is calculated based on the difference in color histograms between adjacent frames; if the variability exceeds the set threshold, it is selected as a keyframe.

[0043] Keyframes are selected based on the degree of scene change between adjacent frames. If the degree of scene change exceeds a set threshold, it is considered a keyframe.

[0044] 5. Audio Feature Extraction Audio feature extraction employs an audio emotion recognition model based on sentiment analysis, which analyzes intonation, speech rate, and emotion distribution in the audio to establish emotional consistency between audio data and visual content.

[0045] After the audio data is converted into text data through the speech recognition interface at URL 1, a speech sentiment analysis tool is used to extract features from the audio at each time segment. The audio text of each time segment is represented as a feature vector: The audio feature vector for time segment 1 is [0.85, 0.78, 0.92], with each dimension representing emotional intensity, speech rate, and intonation variation.

[0046] The audio feature vector for time segment 2 is [0.88, 0.80, 0.94].

[0047] 6. Text Feature Extraction In the process of text feature extraction, in order to ensure efficient integration of comments and subtitles in the video with video images and audio data, a text model based on a self-attention mechanism is adopted. Threat semantic modeling is achieved by assigning higher attention weights to threatening words, imperative sentences and humiliating expressions.

[0048] Optical character recognition (OCR) and natural language processing are performed on the caption text, sticker text, and comment text in each time segment to extract text features. Each comment, caption text, etc., is converted into a high-dimensional feature vector.

[0049] Subtitle text for time segment 1: I'm warning you, you'd better do as I say. "Transformed into an eigenvector [0.78, 0.80, 0.72].

[0050] Subtitle text for time segment 2: If you don't do as I say, you won't have a good time at school from now on. "Transformed into feature vector [0.75, 0.82, 0.79].

[0051] The feature vector of comment 1 is [0.82, 0.75, 0.88], which is mapped to time segment 1.

[0052] 7. Feature Mapping and Normalization Image, audio, and text features are unified into a 128-dimensional vector through linear transformation. Feature mapping uses principal component analysis (PCA) or an autoencoder for dimensionality reduction; PCA is suitable for low-dimensional data, while autoencoders are suitable for high-dimensional data. Dimensionality reduction and mapping are used to eliminate modal differences to support cross-modal feature alignment.

[0053] For all features, Min-Max normalization is used to ensure that the feature data are on the same metric scale, providing effective support for subsequent cross-modal alignment and feature fusion. Based on the same time-slice index, image, audio, and text features within the same time slice are aligned across modally.

[0054] When the fusion features meet the preset criteria for determining inappropriate content, the corresponding time segment is output as inappropriate online content for minors.

[0055] This embodiment extracts and fuses features from video, audio, and text data through comprehensive processing, employing cross-modal alignment technology to improve the accuracy of identifying potentially harmful online content. By using data from publicly available video platforms and combining technologies such as speech recognition, optical character recognition, and sentiment analysis, it identifies and analyzes content unsuitable for minors, improving the accuracy and reliability of identifying harmful online content related to school bullying involving minors.

[0056] Example 3 This embodiment is an intelligent identification method for inappropriate online content targeting minors based on cross-modal feature fusion. Through cross-modal feature fusion and intelligent identification, it accurately identifies and classifies violent, terrifying, or inappropriate content in videos, ensuring minors are protected from harmful information. The specific implementation method is as follows: 1. Data Sources and Basis Video data: Dataset source: The video data comes from publicly available videos on a certain platform. After screening, clips containing violent, terrifying, or inappropriate content that may be viewed by minors were selected.

[0057] Short video clips related to violence, terrorism, or inappropriate content are obtained through platform interfaces. The selected videos are manually tagged to ensure they contain potentially harmful content relevant to the protection of minors.

[0058] The dataset contains 1000 video clips, of which 40% contain violent content, 30% contain terrorist content, and 30% contain other inappropriate content. The dataset covers multiple countries and regions.

[0059] Video length: Each video segment is 10 seconds long.

[0060] The video clips were manually labeled as containing violence or other inappropriate content. The labeling process was carried out by professionals with backgrounds in psychology or sociology, following the standards of the Law on the Protection of Minors and other relevant laws and regulations, and in conjunction with industry standards.

[0061] Annotators need to undergo regular training and use standardized annotation tools to ensure consistency and accuracy in annotation. The annotation standards cover multiple categories in the video, such as violence, horror, and parody, to ensure that the video content is accurately categorized.

[0062] Each video segment is divided into 5 time segments, each lasting 2 seconds.

[0063] Audio data: Audio data was extracted from the same video, and audio tracks were extracted using sound processing tools. The audio for each video segment was divided into three segments, each corresponding to a time segment.

[0064] Open-source speech recognition tools are used to perform speech recognition on the audio of each time segment, generating corresponding text data.

[0065] Subtitles and text data: The video subtitles can be obtained directly through the platform interface, or the subtitle text can be extracted from the video frames using optical character recognition (OCR) technology. The OCR technology is configured to use an English language model with an accuracy of 98%, and can recognize clear subtitle text in the video.

[0066] To improve the accuracy of optical character recognition (OCR), this system preprocesses video frames before OCR, including image denoising, contrast enhancement, and text region localization, ensuring efficient extraction of subtitle text even in low-quality videos or against complex backgrounds. For videos without subtitles, text is generated from OCR, achieving a recognition accuracy of over 95%.

[0067] Comment data is captured via platform API and used as global context input to analyze the potential risks of the video. The captured comments cover a period of 30 days after the video's release and include all publicly available comments related to the video.

[0068] 2. Data Processing and Preprocessing Time segment division: The video is 10 seconds long and is divided into 5 time segments, each lasting 2 seconds.

[0069] Each time segment consists of 5 image frames and 3 audio segments. The subtitle text is extracted using optical characters or existing subtitles.

[0070] Feature extraction and mapping: Visual features are extracted using a residual network model, resulting in a 2048-dimensional image feature vector for each time segment.

[0071] Audio features are extracted using an audio feature extraction model, resulting in a 128-dimensional audio feature vector for each time segment.

[0072] Text features were extracted using a deep learning model, resulting in a 768-dimensional text feature vector for each time segment.

[0073] Time segment 1: Seconds 1-2.

[0074] Image features: [0.923, 0.899, 0.945, ..., 0.732], audio features: [0.845, 0.911, 0.876, ..., 0.543], text features extracted from the subtitle text "Violence Warning": [0.863, 0.722, 0.902, ..., 0.563] 3. Cosine similarity calculation Cosine similarity is used to measure the similarity between two vectors and is suitable for calculating the similarity between image features and text features.

[0075] Calculate the cosine similarity between text and visual representations: Text sharing representation: Visual sharing representation: .

[0076] Cosine similarity calculation formula:

[0077] Dot product:

[0078] Modulus Calculation:

[0079] Cosine similarity: Calculate the cosine similarity between the text and the audio: Audio sharing means: .

[0080] The formula for calculating cosine similarity is the same as above: Dot product:

[0081] Modulus calculation: Cosine similarity: 4. Softmax Normalization Softmax normalization transforms cosine similarity into a probability distribution, ensuring that the weights of each feature sum to 1. This allows for a clearer comparison and fusion of similarities between textual, visual, and audio features.

[0082] The Softmax function is used to convert cosine similarity into cross-modal alignment weights, and the formula is:

[0083] For time segment 1, the cosine similarity calculation result is: The cosine similarity between text and visual data is 0.9991, and the cosine similarity between text and audio data is 0.9984.

[0084] Calculate Softmax normalization: Index calculation: , .

[0085] Softmax normalization:

[0086] 5. Time-series contribution weights and weighted aggregation Time-series contribution weights: The fused features of each time segment are concatenated, and the raw score is calculated using the Sigmoid function. The Sigmoid function maps the input real values ​​to between 0 and 1, and is used to measure the importance and relevance of each time segment. The raw score for time segment 1 is 0.75, the raw score for time segment 2 is 0.70, the raw score for time segment 3 is 0.60, the raw score for time segment 4 is 0.50, and the raw score for time segment 5 is 0.80.

[0087] Time-series weight normalization: The original scores are Softmax normalized to obtain the time-series contribution weights: Time segment 1 is 0.30, time segment 2 is 0.25, time segment 3 is 0.15, time segment 4 is 0.10, and time segment 5 is 0.20.

[0088] Weighted aggregation: Based on the temporal contribution weights, the cross-modal fusion features of each time segment are weighted and aggregated to obtain the final network content risk characterization.

[0089] 6. Risk Assessment and Classification Risk score calculation: The extracted cross-modal features are linearly mapped and normalized using a linear regression model to obtain the final risk score.

[0090] The linear regression model was trained using labeled data in the training set and evaluated using K-fold cross-validation to ensure accurate assessment of the risk of video content. The calculated risk score was 0.88.

[0091] Compared with the preset threshold: Preset threshold: A threshold of 0.7 was determined through cross-validation experiments on the validation set to ensure 95% accuracy in identifying violent and inappropriate content. The threshold can be further adjusted to optimize system performance depending on the specific scenario.

[0092] The calculated risk score is compared with a preset threshold. Since 0.88 > 0.7, the video content is judged as inappropriate content.

[0093] Output result: The system outputs a video containing violent content and marks it as potentially having a negative impact on minors.

[0094] Through the above steps, effective fusion of cross-modal features was achieved. A linear regression model was used to assess the risk of the extracted cross-modal features, calculating the risk score for each video content and classifying it according to a preset threshold. Experimental results show that the intelligent identification method for inappropriate online content for minors based on cross-modal feature fusion can efficiently and accurately identify violent and inappropriate content, and has strong flexibility, allowing for threshold adjustment based on different scenarios to optimize identification performance. It has broad application prospects in the protection of minors.

[0095] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for intelligent identification of harmful online content for minors based on cross-modal feature fusion, characterized in that, include: S1. Obtain multimodal data corresponding to the network content to be identified. The multimodal data includes video data, audio data, and network text information. Construct a multimodal sequence according to a unified timeline. The construction includes: dividing the video data into multiple time segments and extracting image frame sequences from each time segment; performing optical character recognition on the image frame sequences to obtain subtitle or sticker text sequences; performing speech recognition on the audio data according to the time segments to obtain dialogue text sequences. The network text information includes title text, introduction text, tag text, and comment text. The network text information, the subtitle or sticker text sequences, and the dialogue text sequences are combined to form text data. The network text information is used as a global context and associated with each time segment. S2. For each time segment, feature extraction is performed on the image frame sequence, audio data and text data to obtain corresponding visual features, audio features and text features. The visual features, audio features and text features are mapped to the same shared semantic feature space through projection transformation to obtain the visual shared representation, audio shared representation and text shared representation and global context shared representation of each time segment. S3. For each time segment, in the shared semantic feature space, determine the cross-modal alignment weight based on the similarity between the text shared representation and the visual shared representation, and between the text shared representation and the audio shared representation, wherein the cross-modal alignment weight is obtained by normalizing the similarity. Under the constraint of the cross-modal alignment weight, perform cross-attention fusion with the text shared representation as the query vector and the visual shared representation and the audio shared representation as the key vector to obtain the cross-modal fusion feature representation of the same time segment; S4. Introduce temporal location information into the cross-modal fusion feature representation of each time segment, calculate the temporal contribution weight of each time segment, and perform weighted aggregation on the cross-modal fusion feature representation of each time segment based on the temporal contribution weight to obtain a network content-level risk characterization. Perform linear mapping and normalization calculation on the network content-level risk characterization to obtain the category of inappropriate content and the corresponding risk score, and determine whether the network content to be identified belongs to inappropriate network content for minors based on the comparison result of the risk score and the preset threshold.

2. The intelligent identification method for harmful online content on minors based on cross-modal feature fusion according to claim 1, characterized in that: The corresponding association includes: copying and associating the title text, introduction text, and tag text into the global context text of the same time segment within each time segment; when the comment text contains a time identifier, mapping the comment text to the corresponding time segment in which the time identifier falls according to the time identifier; when the comment text does not contain a time identifier, copying and associating the comment text to each time segment.

3. The intelligent identification method for harmful online content on minors based on cross-modal feature fusion according to claim 1, characterized in that: The division of time segments includes: the duration of each time segment is set to 0.5s-5s, and there is a 10%-50% time overlap between adjacent time segments.

4. The intelligent identification method for harmful online content on minors based on cross-modal feature fusion according to claim 1, characterized in that: The extraction of the image frame sequence includes: uniformly extracting frames from the time segment at a preset extraction frequency to obtain a basic frame set, and selecting key frames from the basic frame set based on the scene variability of adjacent frames. The scene variability is calculated from the color histogram difference of adjacent frames. When the scene variability is greater than a preset threshold, the corresponding frame is determined as a key frame.

5. The intelligent identification method for harmful online content on minors based on cross-modal feature fusion according to claim 1, characterized in that: When the optical character recognition is performed, the confidence level and text region location information of each subtitle or sticker text are obtained simultaneously, and subtitles or sticker texts with a confidence level less than the first threshold are removed. When the speech recognition is performed, the start and end times and confidence level of each dialogue text are obtained simultaneously, and dialogue texts with a confidence level less than the second threshold are removed.

6. The intelligent identification method for harmful online content on minors based on cross-modal feature fusion according to claim 1, characterized in that: The projection transformation mapping includes: performing linear transformations on visual features, audio features, and text features respectively, using a unified feature dimension as a preset dimension, and normalizing the transformed features to make the visual shared representation, audio shared representation, and text shared representation under the same metric scale.

7. The intelligent identification method for harmful online content on minors based on cross-modal feature fusion according to claim 1, characterized in that: The similarity is calculated using cosine similarity, and the cross-modal alignment weights are obtained as follows: cosine similarity is calculated between the text shared representation and the visual shared representation and the audio shared representation within the same time segment, and the cosine similarity is normalized using the Softmax function to obtain the cross-modal alignment weights.

8. The intelligent identification method for harmful online content on minors based on cross-modal feature fusion according to claim 1, characterized in that: The calculation of the temporal contribution weight includes: concatenating the cross-modal fusion feature representation and the global context-shared representation of each time segment to obtain the joint representation of the time segment; inputting the joint representation into the gating calculation unit to obtain the original weight score of the time segment; the gating calculation unit uses the Sigmoid function to perform Softmax normalization on the original weight scores of all time segments to obtain the temporal contribution weight of each time segment.

Images