Multi-modal hate speech detection method based on rationality-enhanced dynamic anchor points

Abstract

The application discloses a multi-modal hate speech detection method based on rational enhancement dynamic anchor points, relates to the technical field of multi-modal content detection, and comprises the following steps: collecting multi-modal social platform data streams, extracting text and visual modal segments, and constructing a sample set; generating event triples by semantic role labeling and parsing the text, and establishing a cross-modal entity alignment mapping table in combination with the position of the visual face detection frame; extracting a text deep semantic vector and a visual area feature vector, and fusing to generate a multi-modal joint representation tensor; calculating the attention weight between modes by a dynamic anchor point generator, determining a rational enhancement dynamic anchor point set, and re-weighting the joint representation tensor to obtain a corrected feature matrix; and inputting the feature matrix into a pre-trained classification decision forest to output a hate speech detection result. The method realizes accurate alignment of cross-modal entities, dynamically optimizes feature weights, and improves the multi-modal hate speech detection effect.

Description

Technical Field

[0001] This invention belongs to the field of multimodal content detection technology, specifically a multimodal hate speech detection method based on rationality-enhanced dynamic anchor points. Background Technology

[0002] Existing multimodal hate speech detection technologies directly collect data streams from multimodal social media platforms, extract text and visual modal fragments to construct a multimodal sample set, directly extract global semantic features from the text, and extract overall visual features from the visual data. However, they do not use semantic role labeling tools to parse the subject and object of events in the text, do not form a set of event triples, and do not combine the face detection box position information in the visual modality to carry out cross-modal entity association. They only rely on conventional feature similarity to complete simple matching between modalities.

[0003] Existing multimodal hate speech detection schemes lack structured cross-modal entity association criteria. Text event semantics and visual entity features cannot form a precise correspondence, resulting in significant semantic misalignment in cross-modal feature matching and limited modality fusion effectiveness. After multimodal feature fusion forms a joint representation, targeted dynamic weight adjustment is not performed. It is impossible to calculate the intermodal attention weight distribution through a dedicated module, generate a set of rationality-enhancing dynamic anchor points, or reweight the multimodal joint representation tensor. Irrational semantic features continuously interfere with the detection process, and rational semantics in multimodal features cannot be effectively strengthened. This invention aims to establish a cross-modal entity alignment mapping table corresponding to text event triples and visual face detection box positions. It also requires a dynamic anchor point generator to determine a set of rationality-enhancing dynamic anchor points and uses this set to reweight the multimodal joint representation tensor, thereby meeting the practical needs of accurate multimodal hate speech detection. Summary of the Invention

[0004] This invention aims to solve at least one of the technical problems existing in the prior art; To this end, this invention proposes a multimodal hate speech detection method based on rationality-enhanced dynamic anchors, including: Collect data streams from multimodal social platforms, extract text modality fragments and visual modality fragments, and construct a multimodal sample set; Based on the multimodal sample set, the event subjects and objects in the text modal segments are parsed using a semantic role labeling tool to generate a set of event triples; Read the event triple set and combine it with the face detection box position information in the visual modality segment to establish a cross-modal entity alignment mapping table; Based on the cross-modal entity alignment mapping table, the deep semantic vector of the text modality segment is extracted by the context semantic encoder, and the region feature vector of the visual modality segment is extracted by the visual feature extractor. By fusing the deep semantic vector with the region feature vector, a multimodal joint representation tensor is generated; Based on the multimodal joint representation tensor, the intermodal attention weight distribution is calculated through a dynamic anchor generator to determine the set of rationality enhancement dynamic anchors; The multimodal joint representation tensor is reweighted using the rational enhancement dynamic anchor point set to generate a modified multimodal feature matrix; The corrected multimodal feature matrix is ​​input into a pre-trained classification decision forest, and the hate speech detection results are output.

[0005] Furthermore, a multimodal sample set is constructed, including: The data stream from the multimodal social platform is subjected to noise filtering to remove invalid data entries with missing text or images. The retained data entries are timestamped to ensure that the text modal fragments and the visual modal fragments originate from the same publication event; The text modal fragments are segmented into sentences to generate text fragment units; The visual modal segments are segmented into scene units to generate visual frame units; Establish a one-to-one correspondence between the text fragment units and the visual frame units, and collect all correspondences to form a multimodal sample set.

[0006] Furthermore, generate a set of event triples, including: Load the text modality fragment from the multimodal sample set; The dependency parser is invoked to identify the predicate verbs in the text modality fragment; Taking the predicate verb as the core, extract its corresponding subject phrase as the subject of the event, and extract its corresponding object phrase as the object of the event; The event subject, the predicate verb, and the event object are combined to form an event triple; Iterate through all predicate verbs in the text modal fragment, collect all generated event triples, and form an event triple set.

[0007] Furthermore, a cross-modal entity alignment mapping table is established, including: Read the event subject and event object from the event triple set as the text entities to be aligned; Run a face detection algorithm on the visual modal segment to obtain the coordinate information of all face detection boxes and the corresponding face feature descriptors; Calculate the semantic similarity score between the text entity to be aligned and the face feature descriptor; Pairs with semantic similarity scores exceeding a preset threshold are filtered out, and the corresponding text entities to be aligned are bound to the face detection boxes; Record the binding relationships, including text entity identifiers, face detection box identifiers, and similarity scores, and generate a cross-modal entity alignment mapping table.

[0008] Furthermore, the multimodal joint representation tensor is generated, including: The deep semantic vector is transformed in dimension and expanded into semantic feature blocks that match the preset number of channels; The region feature vector is spatially flattened to generate a visual feature column vector. The semantic feature block and the visual feature column vector are combined by the outer product operation to generate an initial multimodal tensor; The initial multimodal tensor is subjected to intermodal normalization to eliminate modal scale differences, resulting in a multimodal joint representation tensor.

[0009] Furthermore, the set of dynamic anchor points for enhancing rationality is determined, including: The multimodal joint representation tensor is input into the multi-head attention computation layer; Calculate the correlation coefficient matrix between the deep semantic vector corresponding to the text modality segment and the region feature vector corresponding to the visual modality segment; A nonlinear transformation is performed on the correlation coefficient matrix to generate the intermodal attention weight distribution; Based on the cross-modal entity alignment mapping table, retrieve the weight value of the face detection box bound to the text entity to be aligned in the correlation coefficient matrix; Connection points whose weight values ​​exceed the dynamic threshold are selected and marked as rationality-enhancing dynamic anchor points; All the aforementioned rationality-enhancing dynamic anchor points are summarized to form a set of rationality-enhancing dynamic anchor points.

[0010] Furthermore, the corrected multimodal feature matrix is ​​generated, including: Iterate through each element in the multimodal joint representation tensor; Query the set of rationality-enhancing dynamic anchors and determine whether the current element's position belongs to the set of rationality-enhancing dynamic anchors; If it belongs to the category, multiply the value of the element by a preset enhancement coefficient; If it does not belong to the category, keep the value of the element unchanged; After traversing all elements, the adjusted multimodal joint representation tensor is reshaped into a two-dimensional matrix structure to obtain the corrected multimodal feature matrix.

[0011] Furthermore, the classification decision forest consists of multiple decision trees, each of which is constructed through the following steps: Randomly select a portion of the feature dimensions from the modified multimodal feature matrix as candidate splitting nodes; Calculate the Gini impurity decrease value corresponding to the candidate splitting node; The candidate splitting node with the largest decrease in Gini impurity is selected as the splitting rule for the current node; The samples are divided into left and right subtrees according to the splitting rules. The steps of selecting candidate splitting nodes and selecting splitting rules are recursively executed until the maximum depth is reached or the number of leaf node samples is less than the preset value.

[0012] Furthermore, the hate speech detection results are output, including: The corrected multimodal feature matrix is ​​input in parallel into each decision tree in the classification decision forest; Each decision tree outputs a category prediction result, which includes a hate category and a non-hate category; Count the number of hate categories output by all the decision trees; The proportion of the number of hate categories to the total number of decision trees is calculated as the hate confidence score; If the hate confidence score exceeds a preset judgment threshold, the current multimodal sample is determined to be hate speech; otherwise, it is determined to be non-hate speech.

[0013] Furthermore, the dynamic threshold is adjusted through an online learning mechanism, including: Maintain a historical detection result cache to store the detection results of recently labeled multimodal samples; Periodically calculate the mean distribution of the hate confidence scores of misjudged samples in the historical detection result cache; The distribution mean is set as a new dynamic threshold and updated in the detection logic of the classification decision forest.

[0014] Compared with the prior art, the beneficial effects of the present invention are: This paper analyzes the subject and object of events in text modal segments using a semantic role labeling tool, generating a set of event triples. A cross-modal entity alignment mapping table is then established by combining this table with the face detection bounding box location information from visual modal segments. Based on this mapping table, a contextual semantic encoder extracts deep semantic vectors from the text modal segments, and a visual feature extractor extracts regional feature vectors from the visual modal segments. This results in a directional mapping relationship between entities corresponding to text events and visual face entities. The deep semantic vectors of the text align with the semantic logic of the events, while the visual regional feature vectors focus on the local regions corresponding to the face entities. This improves the correlation and matching degree between text and visual features, eliminates entity correspondence bias during cross-modal feature fusion, and enhances the semantic coherence of feature fusion.

[0015] Based on the multimodal joint representation tensor, a dynamic anchor generator is used to calculate the intermodal attention weight distribution, determine the rationality-enhancing dynamic anchor set, and reweight the multimodal joint representation tensor using the rationality-enhancing dynamic anchor set to generate a modified multimodal feature matrix. This modified multimodal feature matrix is ​​then input into a pre-trained classification decision forest to output hate speech detection results. The intermodal attention weights can adapt to the dynamic changes in the semantic state of the multimodal joint representation tensor; the feature weights corresponding to irrational semantics are weakened, while the feature weights corresponding to rational semantics are strengthened. The modified multimodal feature matrix eliminates redundant irrational semantic interference, and the classification decision forest can use this pure feature matrix to complete the hate speech discrimination, with the discrimination results conforming to the rationality semantic standard. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the steps of the multimodal hate speech detection method based on rationality-enhanced dynamic anchors described in this invention. Figure 2 A flowchart for constructing a multimodal sample set; Figure 3 A flowchart for establishing a cross-modal entity alignment mapping table; Figure 4 A line graph showing the relationship between decision tree depth and the decrease in Gini impurity; Figure 5 This is a diagram showing the dynamic anchor point generation and attention weight distribution. Detailed Implementation

[0017] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] See Figure 1This invention provides a multimodal hate speech detection method based on rationality-enhanced dynamic anchors. The method includes: collecting data streams from multimodal social platforms and extracting text modal and visual modal segments. The collected data is cleaned and aligned, invalid entries are removed, and temporal consistency between text and visual information is ensured to construct a multimodal sample set. Based on this multimodal sample set, a semantic role labeling tool is used to parse the text modal segments, identifying the event subject and event object, and generating a structured event triple set. This event triple set is read and combined with face detection box location information obtained from analyzing the visual modal segments to establish a correlation between text entities and visual entities, forming a cross-modal entity alignment mapping table. Based on this mapping table, a contextual semantic encoder extracts deep semantic vectors from the text modal segments, and a visual feature extractor extracts regional feature vectors from the visual modal segments. The extracted deep semantic vectors and regional feature vectors are fused to generate a unified multimodal joint representation tensor. Based on this multimodal joint representation tensor, an intermodal attention weight distribution is calculated using a dynamic anchor generator, and key connection points are selected according to a cross-modal entity alignment mapping table to determine the rationality-enhancing dynamic anchor set. This rationality-enhancing dynamic anchor set is then used to reweight the multimodal joint representation tensor, highlighting key cross-modal association features to generate a revised multimodal feature matrix. This revised multimodal feature matrix is ​​then input into a pre-trained classification decision forest model, which outputs the final hate speech detection result.

[0019] In one embodiment of the present invention, during the construction of the multimodal sample set, refer to... Figure 2 First, noise filtering is performed on the collected multimodal social platform data stream. This step removes invalid data entries lacking text content or visual images, ensuring the integrity of the data foundation. For the retained valid data entries, alignment is performed based on their accompanying timestamp information to ensure that the text modal fragments and visual modal fragments in each data entry originate from the same posting event, maintaining temporal consistency across modal information. For text modal fragments, sentence segmentation is performed, dividing them into multiple semantically independent text segment units. For visual modal fragments, scene segmentation is performed, dividing them into a series of temporally or content-continuous visual frame units. A one-to-one mapping relationship is established between text segment units and visual frame units; this relationship can be determined based on time synchronization or content thematic relevance. All text-visual unit pairs with established mapping relationships are aggregated to form the final multimodal sample set for subsequent processing.

[0020] In its implementation, the multimodal hate speech detection method based on rationality-enhanced dynamic anchors collects multimodal social media platform data streams during the multimodal sample set construction phase. These data streams may include posts from platforms like Weibo or Twitter, containing both text descriptions and accompanying images or videos. Noise filtering is performed on the raw data stream. The system scans each data entry, identifying and removing invalid entries with completely missing text content or visual attachments, retaining only valid entries containing both text and visual modal fragments. For these valid entries, the system reads the timestamp attached to each entry and compares the creation time of the text modal fragment with the upload time of the visual modal fragment. Only text and visual modal fragments with a time difference within a preset synchronization window are considered to originate from the same posting event and are retained for subsequent processing. In some embodiments, timestamp alignment ensures the consistency of multimodal information. For each data entry that passes the alignment check, the system performs sentence segmentation on the text modal fragment, using sentence boundary recognition technology from natural language processing to cut a complete text modal fragment into multiple syntactically and semantically independent text segment units. For example, a comment text "This person is despicable. His behavior is despicable." will be segmented into two text segment units. On the visual modal segment processing side, for image-type visual modal segments, scene segmentation operations may involve using image segmentation algorithms to identify different salient regions; for video-type visual modal segments, scene segmentation operations divide the video stream into a series of temporally continuous and relatively complete visual frame units by analyzing inter-frame differences or using content understanding-based methods.

[0021] Optionally, after generating text fragment units and visual frame units, the system establishes a one-to-one correspondence between them. This correspondence can be based on temporal alignment, pairing the first text fragment unit with the first visual frame unit in chronological order, the second with the second, and so on. Alternatively, the correspondence can be matched based on shallow semantic associations, such as by analyzing keywords in text fragment units and associating them with object labels in visual frame units. All successfully paired text fragment units and visual frame units are collected to form a structured multimodal sample set. Each sample entry in the multimodal sample set explicitly records the mapping relationship between the text fragment unit identifier and the visual frame unit identifier. It can be understood that the process of constructing the multimodal sample set is the foundation for all subsequent processing. In specific implementations, noise filtering effectively improves the quality of input data, timestamp alignment ensures the inherent correlation of cross-modal information, and sentence segmentation and scene segmentation discretize continuous and complex multimodal information into more easily processed units. Establishing a one-to-one correspondence between text fragment units and visual frame units creates a structured data foundation for subsequent cross-modal feature alignment and fusion. The quality of the multimodal sample set construction directly affects the performance of subsequent event triple extraction, entity alignment, and final detection. The entire construction process can be expressed by the following formulaic rules:

[0022] in: This represents the final constructed multimodal sample set. To represent a multimodal sample pair, It is the first Each text segment unit It is the first Each visual frame unit. and These represent the sets of all text fragment units and all visual frame units obtained after sentence segmentation and scene segmentation, respectively. (Function) Predicate used to extract timestamp information from units Used to determine two timestamps and Whether the preset synchronization conditions are met, only unit pairs that simultaneously meet time synchronization and come from the corresponding set will be included in the multimodal sample set. .

[0023] In one embodiment of the present invention, during the process of generating the event triplet set, processed text modal fragments from a multimodal sample set are loaded. A dependency parsing tool is invoked to perform syntactic structure analysis on the loaded text modal fragments, identifying the predicate verb components in the sentences. Using each identified predicate verb as the core, based on the syntactic analysis results, a subject phrase with grammatical association with it is extracted and used as the event subject describing the event. An object phrase with grammatical association with the same predicate verb is extracted and used as the event object describing the event. The event subject, predicate verb, and event object are combined to form a complete event triplet. All identified predicate verbs in the text modal fragment are traversed, and the steps of extracting the event subject and object and combining triples are repeated for each predicate verb. All generated event triples are collected to constitute the event triplet set corresponding to the text fragment. For the process of establishing the cross-modal entity alignment mapping table, refer to [reference needed]. Figure 3 The process begins by reading all event subjects and objects contained in the event triple set, treating them as text entities to be aligned across modalities. A face detection algorithm is run on the visual modal segment within the same multimodal sample to obtain the coordinates of all face detection boxes in the image or video frame, and the corresponding face region feature descriptor is extracted for each detection box. A semantic similarity score is calculated between the semantic representation of each text entity to be aligned and each face feature descriptor, reflecting the semantic association between the text entity and the visual face entity. A preset similarity threshold is set, and text entities with semantic similarity scores exceeding this threshold are paired with face detection boxes. The paired text entities are then bound to their corresponding face detection boxes, treating them as representations of the same real-world entity in different modalities. Detailed information about all binding relationships is recorded, including the identifiers of the text entities, the identifiers of the face detection boxes, and the calculated similarity scores. These records are then organized into a structured cross-modal entity alignment mapping table.

[0024] In practice, the process of generating event triples takes processed text modal fragments from a multimodal sample set as input. The system loads a text modal fragment, such as a user comment, "That tall man insulted the lady next to him." A dependency parser is invoked to perform syntactic parsing on this text modal fragment, constructing a grammatical dependency tree between words. The dependency parser identifies the core predicate verb in this tree, such as "insult." Using the identified predicate verb "insult" as the core, the dependency parser extracts the subject phrase "that tall man," directly associated with the predicate verb "insult," as the event subject, and the object phrase "the lady next to him," as the event object, based on grammatical relationships. The event subject "that tall man," the predicate verb "insult," and the event object "the lady next to him" are combined to form a structured event triple. The system iterates through all predicate verbs in the text modality fragment and repeats the above extraction and combination operations for each identified predicate verb. For example, the text "They maliciously damaged public property and fled quickly" may generate two event triples: "(they, damaged, public property)" and "(they, fled)". The latter triple may lack an explicit object. Collecting all generated event triples constitutes the event triple set corresponding to the text modality fragment.

[0025] In some embodiments, the process of establishing a cross-modal entity alignment mapping table is closely linked to the generation of the event triple set. The system reads all event subjects and event objects recorded in the event triple set and defines them uniformly as text entities to be aligned. A face detection algorithm is run on the corresponding visual modal segment in the same multimodal sample. The face detection algorithm outputs the coordinate information of each face region in the visual modal segment and the high-dimensional face feature descriptor vector extracted from that region. In a specific implementation, a text entity to be aligned is "tall man". Two different face detection boxes A and B are detected in the visual modal segment. The system calculates a cosine similarity score of 0.92 between the semantic embedding vector of the text entity "tall man" and the face feature descriptor of face detection box A, and simultaneously calculates a similarity score of 0.45 between it and the face feature descriptor of face detection box B. A preset similarity threshold is set to 0.8, and the system selects pairs with a score exceeding the threshold of 0.8 ("tall man", face detection box A). The system binds the text entity "tall man" to be aligned with the face detection box A, and records the binding relationship information including the text entity identifier, the face detection box identifier, and the similarity score of 0.92. After calculating and filtering all text entities to be aligned and all face detection boxes, the binding relationship of all records is used to generate a structured cross-modal entity alignment mapping table.

[0026] Optionally, the process of calculating semantic similarity scores can be formalized. In a specific implementation, given a text entity to be aligned... The semantic vector is represented as and a face detection box The facial feature descriptor vector is represented as The semantic similarity score between the two Calculated using the following formula:

[0027] in: This represents the semantic similarity score between the text entity to be aligned and the face detection box. Its value range is [-1, 1]. The higher the score, the stronger the semantic correlation. It is the semantic vector representation of the text entity to be aligned. This is the face feature descriptor vector corresponding to the face detection bounding box. "·" indicates the dot product operation of the vectors. and Representing vectors respectively sum vector The Euclidean norm of the equation is used to quantify the strength of cross-modal associations, providing a numerical basis for establishing reliable binding relationships.

[0028] The generation of event triples extracts structured semantic relation units from unstructured text, providing a foundation for understanding interactive behaviors within the text. The establishment of the cross-modal entity alignment mapping table, through quantified similarity calculation and threshold filtering, establishes a clear and traceable correspondence between entities described in the text and facial entities in the image. These two steps work together to provide precise guidance for subsequent steps in achieving deep cross-modal information fusion based on rationality-enhanced dynamic anchors.

[0029] In one embodiment of the present invention, during the generation of the multimodal joint representation tensor, the deep semantic vectors extracted from text modal segments undergo dimensionality transformation to expand them into semantic feature blocks matching a preset number of channels. The regional feature vectors extracted from visual modal segments are spatially flattened to convert them into a visual feature column vector. The semantic feature blocks and the visual feature column vector are combined through an outer product operation to generate an initial multimodal tensor containing cross-modal feature interaction information. The initial multimodal tensor undergoes intermodal normalization to eliminate potential distributional differences between text and visual features due to variations in source and scale, resulting in a scale-uniform, modally balanced multimodal joint representation tensor. During the determination of the rational enhancement dynamic anchor set, the multimodal joint representation tensor is input into a multi-head attention computation layer. In this layer, the correlation coefficient matrix between the deep semantic vectors corresponding to text modal segments and the regional feature vectors corresponding to visual modal segments is calculated. This matrix quantifies the association strength between different text units and different visual regions. A nonlinear transformation is applied to the calculated correlation coefficient matrix to generate the intermodal attention weight distribution. Based on the previously established cross-modal entity alignment mapping table, face detection boxes that are bound to the text entities to be aligned are retrieved, and the attention weight values ​​corresponding to these detection boxes are located in the correlation coefficient matrix. A dynamic threshold is set, and cross-modal connection points whose attention weight values ​​exceed this dynamic threshold are selected and marked as rationality enhancement dynamic anchors. All marked rationality enhancement dynamic anchors are summarized to form a rationality enhancement dynamic anchor set. During the generation of the modified multimodal feature matrix, each element in the multimodal joint representation tensor is traversed. The rationality enhancement dynamic anchor set is queried to determine whether the coordinates of the currently traversed element belong to the set. If the current element position is confirmed as a rationality enhancement dynamic anchor, the value of the element is multiplied by a preset enhancement coefficient to amplify its contribution. If the current element position does not belong to a rationality enhancement dynamic anchor, the original value of the element remains unchanged. After traversing and conditionally correcting all elements of the entire multimodal joint representation tensor, the adjusted tensor data is reshaped into a two-dimensional matrix structure, and the resulting matrix is ​​the corrected multimodal feature matrix.

[0030] In practice, the process of generating the multimodal joint representation tensor begins with the acquisition of deep semantic vectors and region feature vectors. Deep semantic vectors are extracted from text modalities using a contextual semantic encoder, while region feature vectors are extracted from visual modalities using a visual feature extractor. In practice, the deep semantic vector is a vector with a dimension of... The system performs a dimension transformation operation on the deep semantic vector, expanding its dimensions to a preset number of channels through a linear projection layer. , forming a shape The semantic feature blocks. A region feature vector is a two-dimensional feature map extracted from a visual modality fragment. The system performs a spatial flattening operation on the region feature vector, concatenating the feature vectors at each spatial location of the two-dimensional feature map into a long visual feature column vector. The dimension of the visual feature column vector is... ,in and It refers to the spatial height and width of the feature map. This represents the dimension of each location feature. The outer product operation is used to transform the shape into... The semantic feature blocks and shapes are The visual feature column vectors are combined to generate an initial multimodal tensor, which is a three-dimensional tensor structure. The initial multimodal tensor undergoes intermodal normalization. Layer normalization is used to scale the feature dimensions from both the text and visual modalities, eliminating feature scale differences caused by modal differences, resulting in a scale-uniform multimodal joint representation tensor. The shape of the multimodal joint representation tensor is... Or the shape adjusted according to the calculation.

[0031] In some embodiments, the process of determining the set of rationality-enhancing dynamic anchors takes a multimodal joint representation tensor as input. The multimodal joint representation tensor is input into a multi-head attention computation layer, which contains multiple parallel attention heads. In each attention head, a correlation coefficient matrix is ​​calculated between the deep semantic vector corresponding to the text modality segment and the region feature vector corresponding to the visual modality segment. Each element of the correlation coefficient matrix represents the association strength between a text semantic unit and a visual region feature. A nonlinear transformation is applied to the calculated correlation coefficient matrix, using the Softmax function to normalize the correlation coefficients along a specific dimension, generating an intermodal attention weight distribution, which is a matrix with values ​​ranging from 0 to 1. Based on the cross-modal entity alignment mapping table, the system retrieves face detection boxes bound to the text entities to be aligned. These face detection boxes correspond to spatial location indices in the visual feature map, and the corresponding weight values ​​are found in the intermodal attention weight distribution matrix based on the spatial location indices. A dynamic threshold is used to filter important connection points. The dynamic threshold can be a preset fixed scalar or a value derived from weight distribution statistics. Connection points whose weight values ​​exceed the dynamic threshold are selected and marked as rationality-enhancing dynamic anchor points. All marked rationality-enhancing dynamic anchor points are then aggregated to form a set of rationality-enhancing dynamic anchor points, which can be recorded as a list of coordinates or indices.

[0032] Optionally, during the generation of the modified multimodal feature matrix, the system iterates through each element in the multimodal joint representation tensor. It queries the set of rational enhancement dynamic anchors to determine if the current element's three-dimensional coordinates in the tensor exist in the coordinate list recorded in the set. If the current element's coordinates belong to a rational enhancement dynamic anchor, the system multiplies the current element's value by a preset enhancement coefficient. Enhancement coefficient It is a real number greater than 1. If the current element's coordinates do not belong to the rationality-enhancing dynamic anchor point, the system keeps the original value of the current element unchanged. After completing the traversal and condition correction operations of all elements of the multimodal joint representation tensor, the system reshapes the corrected three-dimensional multimodal joint representation tensor into a two-dimensional matrix structure. The reshaping operation is usually achieved by flattening the multiple dimensions of the tensor into one dimension. The final two-dimensional matrix is ​​the corrected multimodal feature matrix. The number of rows in the corrected multimodal feature matrix corresponds to the number of samples, and the number of columns corresponds to the total dimension of the features. The set of rationality-enhancing dynamic anchor points is shown in Table 1.

[0033] Table 1: Set of Dynamic Anchor Points for Enhancing Rationality

[0034] In some embodiments, the determination of the dynamic threshold can be related to a global statistic of the intermodal attention weight distribution. Dynamic Threshold It can be calculated using the following formula:

[0035] in: This represents a dynamic threshold used to filter rationality-enhancing dynamic anchor points. This represents the mean of all weights in the intermodal attention weight distribution matrix. This represents the standard deviation of all weight values ​​in the intermodal attention weight distribution matrix. This is an adjustable control coefficient used to control the degree to which the threshold deviates from the distribution mean. The dynamic threshold calculated using this formula can adapt to the distribution characteristics of the attention weights.

[0036] Understandably, generating a multimodal joint representation tensor achieves a deep fusion of textual and visual features, providing a unified representation for subsequent analysis. Determining the rationality-enhanced dynamic anchor set involves filtering key interaction nodes based on cross-modal alignment information, focusing on feature connections strongly correlated with entities. Generating a corrected multimodal feature matrix selectively enhances features at anchor point locations, highlighting cross-modal signals that may be more important for the detection task. These steps work together to improve the multimodal representation's ability to distinguish semantics related to hate speech.

[0037] In one embodiment of the present invention, during the construction of a classification decision forest, the classification decision forest is integrated from multiple decision tree models, and each decision tree is constructed independently through the following steps: From all feature dimensions contained in the modified multimodal feature matrix, a subset of feature dimensions are randomly selected as candidate splitting nodes for the current decision tree node. For each candidate splitting node, the Gini impurity reduction value resulting from splitting the current dataset based on that node's features is calculated; this value is used to measure the effectiveness of the split. From all candidate splitting nodes, the node feature that brings the largest Gini impurity reduction value is selected and determined as the final splitting rule for the current node. According to the determined splitting rule, the sample dataset at the current node is divided into a left subtree or a right subtree. For the left and right subtree nodes generated after the split, the steps of randomly selecting candidate splitting nodes, calculating and selecting the best splitting rule are recursively repeated until the decision tree growth reaches a preset maximum depth limit, or the number of samples contained in a leaf node is less than a certain preset value, at which point the splitting stops. In the process of outputting hate speech detection results, the corrected multimodal feature matrix of the sample to be detected is input in parallel into each of the completed decision trees in the classification decision forest. Each decision tree judges the feature matrix according to its own tree structure and independently outputs a category prediction result, which includes hate speech or non-hate speech. The number of results classified as hate speech in all decision tree outputs is counted. The proportion of the number of hate speech cases to the total number of decision trees is calculated, and this proportion is used as the model's hate confidence score for that sample. A preset judgment threshold is set, and the hate confidence score is compared with this threshold. If the hate confidence score exceeds the threshold, the current multimodal sample is judged as hate speech; otherwise, it is judged as non-hate speech.

[0038] In its implementation, the classification decision forest is constructed using a modified multimodal feature matrix as input data, and is formed by the parallel integration of multiple decision tree models. When constructing a single decision tree, a subset is randomly selected from all feature dimensions of the modified multimodal feature matrix. For example, a subset containing 64 feature dimensions is randomly selected from a total of 512 feature dimensions. This selected feature dimension serves as a candidate splitting node for the current tree node. For each candidate splitting node, the system calculates the Gini impurity reduction value corresponding to that node. This calculation requires dividing the samples at the current node into left and right subsets based on the value of the feature dimension and comparing the Gini impurities of the parent and child nodes. From all candidate splitting nodes, the system selects the candidate splitting node with the largest Gini impurity reduction value as the final splitting rule for the current node, such as a rule based on whether the value of feature dimension 127 is greater than 0.5. According to the determined splitting rule, the system assigns all sample data contained at the current node to either the left or right subtree node. For newly generated left and right subtree nodes, the system recursively repeats the steps of randomly selecting candidate split nodes from the corrected multimodal feature matrix and calculating and selecting the candidate split node with the largest decrease in Gini impurity as the splitting rule. The growth of the decision tree is recursively executed until the preset maximum depth limit is reached, for example, the maximum depth is 10, or until the number of samples contained in the leaf node is less than the preset value, for example, the minimum number of samples in the leaf node is 5.

[0039] In some embodiments, calculating the Gini impurity decrease value is central to decision tree construction. For a node containing N samples, its Gini impurity... The calculation formula is as follows:

[0040] in: This represents the Gini impurity of the current node, used to measure the degree of confounding between sample classes within the node. K represents the total number of categories. In the hate speech detection task, K is 2, corresponding to the hate category and the non-hate category, respectively. This represents the proportion of samples of class k in the current node. Assuming a node contains 50 samples, of which 30 are hate-type samples and 20 are non-hate-type samples, then the Gini impurity of this node is: If a candidate splitting rule divides the node into a left child (30 samples, 20 of which are hate-related) and a right child (20 samples, 10 of which are hate-related), then the weighted Gini impurity after the split is: Gini impurity decrease value The system evaluates all candidate split nodes and selects... The biggest rule.

[0041] Optionally, during the output of hate speech detection results, the system inputs the corrected multimodal feature matrix of the sample to be detected into each of the completed decision trees in the classification decision forest in parallel. Each decision tree starts from the root node and judges the feature values ​​in the corrected multimodal feature matrix according to the splitting rules of each node within it, guiding the sample feature vector downwards along the tree structure until it reaches a leaf node. This leaf node outputs a category prediction result, which is either hate or non-hate. The system collects and statistically analyzes the category prediction results output by all decision trees in the classification decision forest, calculating the number of decision trees classified as hate. The hate confidence score is calculated by dividing the number of hate-class decision trees by the total number of decision trees in the classification decision forest. The preset judgment threshold is a pre-defined scalar, for example, 0.6. The system compares the calculated hate confidence score with the preset judgment threshold. If the hate confidence score is greater than the judgment threshold, the system determines the current multimodal sample as hate speech; otherwise, the system determines the current multimodal sample as non-hate speech (see Table 2).

[0042] Table 2: Statistical Table of Output Results of Classification Decision Forest

[0043] In some embodiments, the classification decision forest contains 10 decision trees, as shown in the example above. Five decision trees (001, 003, 004, 007, and 009) output the hate category, resulting in a total of 5 decision trees for the hate category. The hate confidence score is 5 divided by 10, which equals 0.5. The preset judgment threshold is 0.6. Since the hate confidence score of 0.5 is less than the judgment threshold of 0.6, the system determines that the current sample is non-hate speech. Different feature dimensions of the modified multimodal feature matrix have different importance in decision tree splitting, and the mechanism of randomly selecting candidate split nodes enhances the diversity of the classification decision forest. It can be understood that by integrating the prediction results of multiple decision trees, the classification decision forest effectively reduces the overfitting risk that a single decision tree might bring, and improves the overall generalization ability and robustness of the model. The process of outputting hate speech detection results integrates the opinions of all decision trees in the forest. It makes a final decision by calculating a hate confidence score and comparing it to a threshold. This method is more robust than single-model voting, smoothing out the predictions of individual anomalous trees and thus providing more reliable detection conclusions. Feature selection and sample partitioning are recursively performed when constructing the decision tree, automatically learning the complex nonlinear mapping relationship from the corrected multimodal feature matrix to the hate category.

[0044] See Figure 4This is a line graph showing the relationship between decision tree depth and the decrease in Gini impurity. Gini impurity measures the degree of class mixing among samples within a node. The decrease value equals the parent node's Gini value minus the weighted Gini value of the child nodes after the split. A larger value indicates a more significant improvement in sample purity after the split. During decision tree growth, the root node (depth 1) has the highest sample mixing, thus the first split yields the greatest benefit. As the depth increases, the samples within the node become increasingly pure, and the space for purity improvement in subsequent splits continuously shrinks, resulting in a gradual decrease in the decrease value. This graph visually demonstrates that decision tree depth is not always better the deeper it is. Splitting at depths of 1-5 yields significant benefits and is the core stage for the model to learn key features; however, the benefits at depths of 6-10 are extremely low, and further growth only increases model complexity and easily leads to overfitting. Therefore, a reasonable maximum depth can be set to ensure model performance while avoiding overfitting and improving inference efficiency.

[0045] In one embodiment of the present invention, during the implementation of adjusting the dynamic threshold through an online learning mechanism, the system maintains a historical detection result cache during operation. This cache stores recently detected multimodal samples that have obtained real labels and their detection results. The system periodically performs calculations and analyses on the data in the historical detection result cache, identifying samples that were misclassified by the model, including samples that misclassify hate speech as non-hate speech, or vice versa. The mean distribution of the hate confidence scores obtained by these misclassified samples in previous detection processes is calculated. This mean reflects the model's tendency to judge difficult samples under the current data distribution. The mean distribution of the calculated hate confidence scores of the misclassified samples is set as the new dynamic threshold that the classification decision forest needs to use in subsequent detections. This updated dynamic threshold is synchronized to the detection logic of the classification decision forest, replacing the original threshold parameters, so that the model's judgment criteria can adapt to changes in the data flow.

[0046] In practice, the dynamic threshold is adjusted through an online learning mechanism. During system operation, a historical detection result cache is maintained. This cache stores recently tested multimodal samples that have received manual review or other reliable labeling methods, along with their detection process data. Entries stored in the multimodal samples and their detection process data typically include the sample's unique identifier, the corrected multimodal feature matrix, the hate confidence score output by the classification decision forest, the model's predicted class, and the verified true class label. The historical detection result cache has a fixed capacity. When new data arrives, the system updates the cache content according to a first-in, first-out (FIFO) principle, ensuring that the cache always reflects recent data distribution and model performance.

[0047] In some embodiments, the system periodically calculates and analyzes data in the historical detection result cache to update the dynamic threshold. The system iterates through all entries in the historical detection result cache, identifying samples misclassified by the model by comparing the model's predicted category with the true category label. Misclassified samples include false positives (samples whose true label is non-hate speech but are predicted as hate speech by the model) and false negatives (samples whose true label is hate speech but are predicted as non-hate speech by the model). The system extracts the hate confidence score output by the model during detection from the stored records corresponding to these misclassified samples; the hate confidence scores of all misclassified samples constitute a dataset. An online learning mechanism calculates the mean distribution of this dataset, which reflects the central tendency of the confidence level given by the model for boundary samples that are difficult to classify correctly at the current stage. The system sets the calculated mean distribution as the new dynamic threshold to be used by the classification decision forest in subsequent detections; the new dynamic threshold replaces the previously used old dynamic threshold parameters. After the update operation is completed, the new dynamic threshold is synchronized to the detection logic of the classification decision forest. When processing new multimodal samples, the classification decision forest will compare this updated dynamic threshold with the calculated hate confidence score to make a final hate speech determination. Optionally, the formula for calculating the updated dynamic threshold can be expressed as:

[0048] in: This represents the new dynamic threshold calculated through an online learning mechanism. This represents the set of all misclassified samples identified from the historical detection results cache during a periodic analysis cycle. Represents a set The number of misclassified samples, i.e., the cardinality of the set. Indicates misjudged samples The hate confidence score output by the classification decision forest during the previous detection process. Represents a set Hate confidence score of all misjudged samples Perform summation. Calculate the sum. This is the arithmetic mean of the hate confidence scores of the misjudged samples, which is set as the dynamic threshold for the next stage.

[0049] In some embodiments, updates to the historical detection result cache and threshold calculations are triggered at fixed time intervals, such as after processing 1000 new samples. The historical detection result cache stores data from the most recent 5000 labeled samples. In a periodic analysis, the system identifies 120 misclassified samples from the cache, and the set of hate confidence scores for these 120 misclassified samples is as follows: The mean value is 0.55. The online learning mechanism updates the new dynamic threshold from the original 0.6 to 0.55. When the classification decision forest subsequently detects input samples, the judgment rule is adjusted from comparing whether the hate confidence score is greater than 0.6 to whether it is greater than 0.55. It can be understood that the online learning mechanism of the dynamic threshold enables the model to adapt to possible changes in data distribution over time. By analyzing the confidence performance of recent misjudged samples and adjusting the decision threshold accordingly, the system can dynamically calibrate its judgment boundary, thereby maintaining more stable detection performance under the condition of data flow concept drift. Maintaining a cache of historical detection results provides the data foundation for threshold adjustment, while periodic calculation and updates enable adaptive optimization of model decision parameters.

[0050] See Figure 5 This is a dynamic anchor point generation and attention weight distribution chart, visually illustrating the changes in attention weights for text and visual modalities across different feature dimensions. Dimensions 1-7 show high weights for both text and visual modalities, representing the core feature areas the model focuses on. Weights decrease steadily from dimensions 8-15, reaching their lowest point near dimension 12, indicating a weak feature area. Weights rapidly rebound from dimensions 16-20, finally reaching a high attention level of around 0.6 at dimension 20. The global peak for the text modal occurs at dimension 4, while the peak for the visual modal occurs at dimension 2, suggesting a difference in the core attention features of the two modalities, resulting in complementarity. The two curves intersect near dimension 3, indicating comparable attention levels for both modalities at that dimension. Between dimensions 17-20, the two curves almost overlap, suggesting that the importance of high-dimensional features for both modalities tends to be consistent. The differentiated distribution of the two curves demonstrates the different roles played by text and visual modalities in hate speech detection, highlighting the necessity of multimodal fusion.

[0051] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A method for multi-modal hate speech detection based on rationality-enhanced dynamic anchor points, characterized in that, include: Collect data streams from multimodal social platforms, extract text modality fragments and visual modality fragments, and construct a multimodal sample set; Based on the multimodal sample set, the event subjects and objects in the text modal segments are parsed using a semantic role labeling tool to generate a set of event triples; Read the event triple set and combine it with the face detection box position information in the visual modality segment to establish a cross-modal entity alignment mapping table; Based on the cross-modal entity alignment mapping table, the deep semantic vector of the text modality segment is extracted by the context semantic encoder, and the region feature vector of the visual modality segment is extracted by the visual feature extractor. By fusing the deep semantic vector with the region feature vector, a multimodal joint representation tensor is generated; Based on the multimodal joint representation tensor, the intermodal attention weight distribution is calculated through a dynamic anchor generator to determine the set of rationality enhancement dynamic anchors; The multimodal joint representation tensor is reweighted using the rational enhancement dynamic anchor point set to generate a modified multimodal feature matrix; The corrected multimodal feature matrix is ​​input into a pre-trained classification decision forest, and the hate speech detection results are output.

2. The multi-modal hate speech detection method based on rationality-enhanced dynamic anchor points according to claim 1, wherein, Constructing a multimodal sample set includes: The data stream from the multimodal social platform is subjected to noise filtering to remove invalid data entries that are missing text or images. The retained data entries are timestamped to ensure that the text modal fragments and the visual modal fragments originate from the same publication event; The text modal fragments are segmented into sentences to generate text fragment units; The visual modal segments are segmented into scene units to generate visual frame units; Establish a one-to-one correspondence between the text fragment units and the visual frame units, and collect all correspondences to form a multimodal sample set.

3. The multi-modal hate speech detection method based on rationality-enhanced dynamic anchor points according to claim 2, characterized in that, Generate a set of event triples, including: Load the text modality fragment from the multimodal sample set; The dependency parser is invoked to identify the predicate verbs in the text modality fragment; Taking the predicate verb as the core, extract its corresponding subject phrase as the subject of the event, and extract its corresponding object phrase as the object of the event; The event subject, the predicate verb, and the event object are combined to form an event triple; Iterate through all predicate verbs in the text modal fragment, collect all generated event triples, and form an event triple set.

4. The multi-modal hate speech detection method based on rationality-enhanced dynamic anchor points according to claim 3, characterized in that, Establish a cross-modal entity alignment mapping table, including: Read the event subject and event object from the event triple set as the text entities to be aligned; Run a face detection algorithm on the visual modal segment to obtain the coordinate information of all face detection boxes and the corresponding face feature descriptors; Calculate the semantic similarity score between the text entity to be aligned and the face feature descriptor; Pairs with semantic similarity scores exceeding a preset threshold are filtered out, and the corresponding text entities to be aligned are bound to the face detection boxes; Record the binding relationships, including text entity identifiers, face detection box identifiers, and similarity scores, and generate a cross-modal entity alignment mapping table.

5. The multi-modal hate speech detection method based on rationality-enhanced dynamic anchor points according to claim 4, characterized in that, Generate a multimodal joint representation tensor, including: The deep semantic vector is transformed in dimension and expanded into semantic feature blocks that match the preset number of channels; The region feature vector is spatially flattened to generate a visual feature column vector. The semantic feature block and the visual feature column vector are combined by the outer product operation to generate an initial multimodal tensor; The initial multimodal tensor is subjected to intermodal normalization to eliminate modal scale differences, resulting in a multimodal joint representation tensor.

6. The multi-modal hate speech detection method based on rationality-enhanced dynamic anchor points according to claim 5, characterized in that, Determine the set of dynamic anchor points for enhancing rationality, including: The multimodal joint representation tensor is input into the multi-head attention computation layer; Calculate the correlation coefficient matrix between the deep semantic vector corresponding to the text modality segment and the region feature vector corresponding to the visual modality segment; A nonlinear transformation is performed on the correlation coefficient matrix to generate the intermodal attention weight distribution; Based on the cross-modal entity alignment mapping table, retrieve the weight value of the face detection box bound to the text entity to be aligned in the correlation coefficient matrix; Connection points whose weight values ​​exceed the dynamic threshold are selected and marked as rationality-enhancing dynamic anchor points; All the aforementioned rationality-enhancing dynamic anchor points are summarized to form a set of rationality-enhancing dynamic anchor points.

7. The multi-modal hate speech detection method based on rationality-enhanced dynamic anchor points according to claim 6, characterized in that, Generate the corrected multimodal feature matrix, including: Iterate through each element in the multimodal joint representation tensor; Query the set of rationality-enhancing dynamic anchors and determine whether the current element's position belongs to the set of rationality-enhancing dynamic anchors; If it belongs to the category, multiply the value of the element by a preset enhancement coefficient; If it does not belong to the category, keep the value of the element unchanged; After traversing all elements, the adjusted multimodal joint representation tensor is reshaped into a two-dimensional matrix structure to obtain the corrected multimodal feature matrix.

8. The multi-modal hate speech detection method based on rationality-enhanced dynamic anchor points according to claim 7, characterized in that, The classification decision forest consists of multiple decision trees, each of which is constructed through the following steps: Randomly select a portion of the feature dimensions from the modified multimodal feature matrix as candidate splitting nodes; Calculate the Gini impurity decrease value corresponding to the candidate splitting node; The candidate splitting node with the largest decrease in Gini impurity is selected as the splitting rule for the current node; The samples are divided into left and right subtrees according to the splitting rules. The steps of selecting candidate splitting nodes and selecting splitting rules are recursively executed until the maximum depth is reached or the number of leaf node samples is less than the preset value.

9. The multi-modal hate speech detection method based on rationality-enhanced dynamic anchor points according to claim 8, characterized in that, Output hate speech detection results, including: The corrected multimodal feature matrix is ​​input in parallel into each decision tree in the classification decision forest; Each decision tree outputs a category prediction result, which includes a hate category and a non-hate category; Count the number of hate categories output by all the decision trees; The proportion of the number of hate categories to the total number of decision trees is calculated as the hate confidence score; If the hate confidence score exceeds a preset judgment threshold, the current multimodal sample is determined to be hate speech; otherwise, it is determined to be non-hate speech.

10. The multi-modal hate speech detection method based on rationality-enhanced dynamic anchor points according to claim 9, wherein, The dynamic threshold is adjusted through an online learning mechanism, including: Maintain a historical detection result cache to store the detection results of recently labeled multimodal samples; Periodically calculate the mean distribution of the hate confidence scores of misjudged samples in the historical detection result cache; The distribution mean is set as a new dynamic threshold and updated in the detection logic of the classification decision forest.

Images