Harmful information detection method, system and equipment based on cross-modal fusion characterization technology and medium
By using a cross-modal cross-attention network and spatial relationship feature computation, the shallow semantic alignment problem in cross-modal harmful information detection in existing technologies is solved. This enables fine-grained interaction between image regions and text words and fusion of high-level semantic features, thereby improving the accuracy and adaptability of detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-10
- Publication Date
- 2026-03-31
AI Technical Summary
Existing technologies, when detecting cross-modal harmful information, only achieve shallow semantic alignment, lack fine-grained interactions between modalities, have difficulty capturing the local correspondence between image regions and text words, and ignore the semantic gap between modalities. This results in limited detection capabilities for highly disguised hidden harmful information, a high false alarm rate, and weak generalization ability.
By introducing a cross-modal cross-attention network and spatial relationship feature calculation, the feature vectors of multi-source detection data are mapped to the same coordinate space, spatial distance is calculated, and fusion is performed using connected layers and fully connected layers. Combined with weighted concatenation by the cross-modal cross-attention network, the final fused feature vector is generated.
It enables fine-grained interaction between image regions and text words, enhances the ability to detect highly disguised harmful information, reduces the false alarm rate, and improves the recognition accuracy and generalization ability in complex social scenarios.
Smart Images
Figure CN121765243A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of harmful information detection technology, and in particular to a harmful information detection method, system, device and medium based on cross-modal fusion characterization technology. Background Technology
[0002] In existing technologies, harmful information detection methods rely on single-modal data (such as plain text or plain images) for identification, which is insufficient to handle cross-modal harmful information with mixed text and images and strong metaphors in current online content. In recent years, some solutions have attempted to introduce multimodal analysis, such as image-text matching technology based on the CLIP model, which maps text and images to a unified semantic space through contrastive learning, achieving preliminary cross-modal semantic alignment.
[0003] However, such methods only achieve shallow semantic alignment, lack fine-grained interactions between modalities, and struggle to capture local correspondences between image regions and text words. Their ability to detect highly disguised, implicitly harmful information (such as irony and metaphors) is also limited. Furthermore, existing technologies generally neglect the semantic gap between modalities, failing to effectively integrate contextual information with high-level semantic features. This results in high false positive rates and weak generalization in complex social scenarios, failing to meet the practical needs for accurate identification of cross-modal harmful information. Summary of the Invention
[0004] This application provides a method, system, device, and medium for detecting harmful information based on cross-modal fusion representation technology, in order to solve the problems of existing solutions that only achieve shallow semantic alignment, lack fine-grained interaction between modalities, have difficulty capturing the local correspondence between image regions and text words, have limited ability to detect highly disguised hidden harmful information (such as irony and metaphor), ignore the semantic gap between modalities, and fail to effectively integrate contextual information and high-level semantic features.
[0005] Firstly, this application provides a method for detecting harmful information based on cross-modal fusion characterization technology, the method comprising: Acquire multi-source detection data; the multi-source detection data includes at least: text, image, and audio; determine the corresponding parsing program based on the specific type of multi-source detection data, and then obtain the corresponding feature vector; The eigenvectors of various types are concatenated, and the concatenated eigenvectors are centered to obtain the eigenma corresponding to the covariance matrix; the first part of the eigenma is then... p The columns are used as transformation matrices to perform dimensionality transformations on feature vectors of different types, mapping them to the same coordinate space; the spatial distances between feature vectors of different types in the same coordinate space are calculated; and the spatial distances are concatenated using the connect function. Layers and fully connected layers are used to obtain the final output spatial relationship features; Based on the feature vectors of each type, a cross-modal cross-attention network is used to finally weight and concatenate them to obtain a cross-modal fusion representation; By utilizing spatial relationship features and cross-modal fusion representation, a final fusion feature vector is obtained, and the result of harmful information detection is obtained using the final fusion feature vector.
[0006] In one implementation of this application, a corresponding parsing procedure is determined based on the specific type of multi-source detection data, thereby obtaining the corresponding feature vector, specifically including: The ALBERT encoder is used to extract fixed-dimensional contextual semantic feature representations from the text; The ConvNeXt encoder, based on a pure convolutional network structure, extracts local texture and high-level semantic features of the image layer by layer, and obtains multi-scale feature maps from the image. The audio waveform is converted into a preset high-dimensional vector sequence using the wav2vec 2.0 encoder, and then semantic information feature vectors in speech are extracted from the preset high-dimensional vector sequence.
[0007] In one implementation of this application, the feature vectors of various types are concatenated, and the concatenated feature vectors are centered to obtain the feature matrix corresponding to the covariance matrix, specifically including: Through the formula: ; ; ; ; The concatenated feature vector X is calculated; where n represents the total number of feature vectors of each type, and T represents the text representation vector after feature extraction. This represents the feature vector corresponding to the j-th text. This represents the representation vector of the image after feature extraction. This represents the feature vector corresponding to the j-th image. This represents the representation vector of the audio after feature extraction. This represents the feature vector corresponding to the j-th audio, and ⊕ represents the concatenation operator; Through the formula: The covariance matrix S is calculated. Then the characteristic matrix corresponding to the covariance matrix is obtained.
[0008] In one implementation of this application, the first part of the feature matrix is... p The column serves as a transformation matrix, used to perform dimensionality transformations on eigenvectors of various types, mapping them to the same coordinate space. Specifically, this includes: Through the formula: ; Obtain the feature vectors of text, images, and audio after dimensionality transformation; in, , , These are preset matrices used to convert the representation vectors of text, images, and audio into matrices of the same dimension as the transformation matrix; , , Let represent the feature vectors of text, image, and audio after dimensional transformation, respectively, and P represent the transformation matrix.
[0009] In one implementation of this application, the spatial distance between feature vectors of different types in the same coordinate space is calculated; the spatial distances are then concatenated, and a connection is used... Layers and fully connected layers are used to obtain the final output spatial relationship features, specifically including: Through the formula: ; ; ; Calculate the spatial distance between text and image, text and audio, and image and audio. , , ; Through the formula: ; ; ; Obtain the final output spatial relationship features ; in, express Activation function express layer, Indicates a fully connected layer. This represents the concatenation operator.
[0010] In one implementation of this application, a cross-modal cross-attention network is used to finally weight and concatenate the feature vectors of each type to obtain a cross-modal fusion representation, specifically including: A cross-modal attention network is used to obtain the fusion features of the baseline layer for feature vectors of various types; Through the formula: Finally, weighted splicing is used to obtain the cross-modal fusion representation. ; in, , , To preset learnable weights, Indicates text-based fusion features, Represents image-based fusion features, This represents the image-audio fusion features.
[0011] In one implementation of this application, spatial relationship features and cross-modal fusion representations are used to obtain a final fused feature vector, and the final fused feature vector is used to obtain the harmful information detection result, specifically including: Through the formula: F= Foutput1+(1- Foutput2, obtain the final fused feature vector F; Through the formula: Result = Softmax(MLP(F)) to obtain the harmful information detection result Result; Where MLP stands for Multilayer Perceptron and Softmax stands for Normalization Function.
[0012] Secondly, this application provides a harmful information detection system based on cross-modal fusion characterization technology, the system comprising: The acquisition module is used to acquire multi-source detection data; the multi-source detection data includes at least: text, image, and audio; the corresponding parsing program is determined according to the specific type of multi-source detection data, and then the corresponding feature vector is obtained; The spatial module is used to concatenate feature vectors of various types, center the concatenated feature vectors to obtain the feature matrix corresponding to the covariance matrix, use the first P columns of the feature matrix as the transformation matrix to perform dimensionality transformation on the feature vectors of various types, mapping them to the same coordinate space, calculate the spatial distance between feature vectors of different types in the same coordinate space, and concatenate the spatial distances using the connect function. Layers and fully connected layers are used to obtain the final output spatial relationship features; The fusion module is used to obtain cross-modal fusion representations by weighted concatenation of feature vectors of various types using a cross-modal cross-attention network. The results module is used to obtain the final fused feature vector by utilizing spatial relationship features and cross-modal fusion representation, and then use the final fused feature vector to obtain the harmful information detection results.
[0013] Thirdly, this application provides a harmful information detection device based on cross-modal fusion characterization technology, the device comprising: processor; And a memory that stores executable code, which, when executed, causes the processor to execute a harmful information detection method based on cross-modal fusion characterization technology, as described above.
[0014] Fourthly, this application provides a non-volatile computer storage medium storing computer instructions thereon, which, when executed, implement a harmful information detection method based on cross-modal fusion characterization technology as described above.
[0015] As can be seen from the above technical solutions, this application has the following advantages: By introducing a cross-modal cross-attention network and spatial relation feature computation, the problem of achieving only shallow semantic alignment is directly solved. Specifically, in the feature vector processing stage, features of various types are mapped to the same coordinate space, and the spatial distance between them is calculated. This enables fine-grained interactions between image regions and local elements such as text words through distance metrics. For example, when parsing multi-source data, the concatenation and centering of feature vectors ensures consistent alignment between modalities, while the dimensionality transformation of the transformation matrix facilitates the capture of implicit local correspondences, thereby effectively identifying highly disguised implicit harmful information, such as irony or metaphor. This design avoids the limitations of existing technologies that rely on simple alignment. By integrating spatial relation features, it enhances the detection system's sensitivity to complex semantic cues, demonstrating its direct advantage in enhancing local interactions without additional experiments.
[0016] To address the issues of intermodal semantic gaps and insufficient contextual fusion, a final fused feature vector generation mechanism achieves deep fusion of high-level semantic features and contextual context. The technical solution utilizes connected and fully connected layers to process the spatial distance after concatenation, and combines this with cross-modal fusion representation to form the final fused feature vector. This directly reduces false positives caused by semantic gaps. For example, in complex social scenarios, covariance matrix analysis and unified mapping of multi-source data ensure intermodal semantic consistency, while the weighted concatenation of cross-modal attention networks strengthens the integration of contextual information, enabling the system to more accurately identify harmful cross-modal information. Its improved generalization ability can be demonstrated without relying on hypothetical experiments. This structural design avoids the shortcomings of existing technologies that neglect high-level semantics, directly reducing the false positive rate through multi-stage fusion, and enhancing adaptability and accuracy in practical applications. Attached Figure Description
[0017] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a flowchart of a harmful information detection method based on cross-modal fusion characterization technology provided in an embodiment of this application.
[0019] Figure 2 This is a schematic diagram of the internal structure of a harmful information detection system based on cross-modal fusion characterization technology provided in an embodiment of this application.
[0020] Figure 3 This is a schematic diagram of the internal structure of a harmful information detection device based on cross-modal fusion characterization technology provided in an embodiment of this application. Detailed Implementation
[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0022] Those skilled in the art should understand that the embodiments described below are merely preferred embodiments of this disclosure and do not imply that this disclosure can only be implemented through these preferred embodiments. These preferred embodiments are merely used to explain the technical principles of this disclosure and are not intended to limit the scope of protection of this disclosure. Based on the preferred embodiments provided by this disclosure, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of this disclosure.
[0023] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0024] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0025] The embodiment provides a method for detecting harmful information based on cross-modal fusion characterization technology, such as Figure 1 As shown in the embodiments of this application, the method mainly includes the following steps: Step 110: Obtain multi-source detection data; determine the corresponding parsing program based on the specific type of multi-source detection data, and then obtain the corresponding feature vector.
[0026] The multi-source detection data includes at least: text, images, and audio.
[0027] In some embodiments, the corresponding parsing procedure is determined based on the specific type of multi-source detection data, thereby obtaining the corresponding feature vector, specifically including: The ALBERT encoder is used to extract fixed-dimensional contextual semantic feature representations from the text; The ConvNeXt encoder, based on a pure convolutional network structure, extracts local texture and high-level semantic features of the image layer by layer, and obtains multi-scale feature maps from the image. The audio waveform is converted into a preset high-dimensional vector sequence using the wav2vec 2.0 encoder, and then semantic information feature vectors in speech are extracted from the preset high-dimensional vector sequence.
[0028] It should be noted that in step 110, acquiring multi-source detection data and extracting corresponding feature vectors is the foundation for constructing cross-modal fusion representations. For text data, the ALBERT encoder is used. This encoder generates fixed-dimensional contextual semantic feature representations through parameter sharing and factorization techniques, effectively capturing the dependencies between words. For image data, the ConvNeXt encoder is applied. This pure convolutional network extracts local texture features (such as edges and contours) and high-level semantic features (such as object structure and scene context) of the image through layer-by-layer convolution operations, and outputs multi-scale feature maps to adapt to different granularities of analysis. For audio data, the wav2vec 2.0 encoder is used to convert the original audio waveform into a preset high-dimensional vector sequence, and then extracts feature vectors containing speech content and semantic information through a self-supervised learning mechanism. The selection of these three encoders is based on their efficient representation capabilities for their respective modal data, ensuring the consistency and repeatability of feature extraction.
[0029] During feature extraction, each encoder uses standard pre-trained model parameters, requiring no additional custom layers or data augmentation steps. When processing text, the ALBERT encoder maintains a fixed output dimension of 768 dimensions to preserve the integrity of the contextual semantics. The ConvNeXt encoder generates multi-scale feature maps with resolution decreasing according to network layers, thus covering visual information from fine-grained to coarse-grained. The wav2vec 2.0 encoder maps audio waveforms to a high-dimensional space through a quantization module, with the sequence length proportional to the input audio duration. The entire process does not rely on specific datasets or optimization techniques, achieving feature transformation solely through forward inference, ensuring the versatility and feasibility of the technical solution and providing a consistent and interpretable input foundation for subsequent cross-modal fusion.
[0030] Step 120: Concatenate the feature vectors of each type, center the concatenated feature vectors to obtain the feature matrix corresponding to the covariance matrix; use the first p columns of the feature matrix as the transformation matrix, and use the transformation matrix to perform dimensionality transformation on the feature vectors of each type, mapping them to the same coordinate space; calculate the spatial distance between the feature vectors of each type in the same coordinate space; concatenate the spatial distances, and use the connect... Layers and fully connected layers are used to obtain the final output spatial relationship features.
[0031] It should be noted that because text, images, and audio use different encoders, their feature vectors are not in the same coordinate system. Therefore, directly calculating their feature vectors and their similarity will be difficult to guarantee accurate results. Thus, a spatial dimension transformation of the feature vectors of multimodal data is necessary before semantic representation.
[0032] This involves concatenating the feature vectors of various types, centering the concatenated feature vectors, and obtaining the feature matrix corresponding to the covariance matrix. Specifically, this includes: Through the formula: ; ; ; ; The concatenated feature vector X is calculated; where n represents the total number of feature vectors of each type, and T represents the text representation vector after feature extraction. This represents the feature vector corresponding to the j-th text. This represents the representation vector of the image after feature extraction. This represents the feature vector corresponding to the j-th image. This represents the representation vector of the audio after feature extraction. This represents the feature vector corresponding to the j-th audio, and ⊕ represents the concatenation operator; Through the formula: The covariance matrix S is calculated. Then the characteristic matrix corresponding to the covariance matrix is obtained.
[0033] It should be noted that, based on the covariance matrix S, the eigenvalues and corresponding eigenvectors are solved to obtain λ and W, where λ is a diagonal matrix with the eigenvalues on the main diagonal, and W is a matrix composed of the eigenvectors corresponding to the eigenvalues. The first... p The column, as a transformation matrix P, performs dimensional transformation on the representation vectors of the three data types, mapping them to the same coordinate space.
[0034] Among them, the first part of the feature matrix p The column serves as a transformation matrix, used to perform dimensionality transformations on eigenvectors of various types, mapping them to the same coordinate space. Specifically, this includes: Through the formula: ; Obtain the feature vectors of text, images, and audio after dimensionality transformation; in, , , These are preset matrices used to convert the representation vectors of text, images, and audio into matrices of the same dimension as the transformation matrix; , , Let represent the feature vectors of text, image, and audio after dimensional transformation, respectively, and P represent the transformation matrix.
[0035] This involves calculating the spatial distance between feature vectors of different types within the same coordinate space; concatenating these spatial distances; and utilizing the connect... Layers and fully connected layers are used to obtain the final output spatial relationship features, specifically including: Through the formula: ; ; ; Calculate the spatial distance between text and image, text and audio, and image and audio. , , ; Through the formula: ; ; ; Obtain the final output spatial relationship features ; in, express Activation function express layer, Indicates a fully connected layer. This represents the concatenation operator.
[0036] It should be noted that calculating the spatial distance between feature vectors of different modalities within the same spatial dimension... This is used to represent the semantic similarity between multimodal data, thereby measuring the semantic relationships between the data. , , These represent the spatial distances between images and text, audio and text, and audio and images, respectively. Finally, the spatial distance matrices are concatenated and connected. Layers and fully connected layers are used to ultimately output spatial relationship features. Layers are used to reduce the risk of overfitting in the network, and connecting multiple fully connected layers is used to further fuse multimodal features.
[0037] Step 130: Based on the feature vectors of each type, a cross-modal cross-attention network is used to finally weight and concatenate them to obtain the cross-modal fusion representation.
[0038] It should be noted that the feature vectors of each type are extracted from text, image and audio data features in step 110, and are jointly modeled using a cross-modal cross-attention network. The feature vectors learn the features within the modality through a self-attention mechanism, emphasizing the alignment and fusion of cross-modal data.
[0039] Specifically, based on the feature vectors of each type, a cross-modal cross-attention network is used to finally weight and concatenate them to obtain a cross-modal fusion representation, which includes: A cross-modal attention network is used to obtain the fusion features of the baseline layer for feature vectors of various types; Through the formula: Finally, weighted splicing is used to obtain the cross-modal fusion representation. ; in, , , To preset learnable weights, Indicates text-based fusion features, Represents image-based fusion features, This represents the image-audio fusion features.
[0040] Furthermore, as an example, the fusion features of the baseline layer for obtaining the feature vector of the text, using a cross-modal cross-attention network, can be specifically as follows: Network based The architecture employs an attention mechanism that enhances the weights in some parts of the neural network's input data while weakening the weights in others, thus enabling the neural network to focus on the key components of the data. The calculation of the attention mechanism is as follows: ; in Each refers to a query ( ,key( ) and value ( The lookup matrix contains data representing the information to be focused on, the key matrix contains data used to calculate the similarity between information (i.e., attention score), and the value matrix contains data used to calculate the final output based on the similarity. In a cross-modal attention network, any modality ( From the other two modalities The relevant information is dynamically extracted from the data, and the calculation formula is as follows: ; in, Represents the query matrix for mode A. The bond matrix representing the B mode. This represents the value matrix for mode B. For the three modes in this patent, a three-channel cross-attention network is established, consisting of a text baseline layer, an image baseline layer, and an audio baseline layer. Taking the text baseline as an example, firstly, word-level semantic features are learned based on the text's self-attention mechanism, and then, using the text as... Images and audio are , By generating text-guided attention within the visual and audio information streams, the model simultaneously focuses on corresponding features of the image and audio while extracting text features, thereby associating and fusing data from different modalities. The final fused features of the text baseline layer are as follows: .
[0041] Step 140: Utilize spatial relationship features and cross-modal fusion representation to obtain the final fusion feature vector, and use the final fusion feature vector to obtain the harmful information detection result.
[0042] In some embodiments, spatial relationship features and cross-modal fusion representations are used to obtain a final fused feature vector, and the final fused feature vector is used to obtain harmful information detection results, specifically including: Through the formula: F= Foutput1+(1- Foutput2, obtain the final fused feature vector F; Through the formula: Result = Softmax(MLP(F)) to obtain the harmful information detection result Result; Where MLP stands for Multilayer Perceptron and Softmax stands for Normalization Function.
[0043] As described above, this embodiment proposes for the first time a dual-channel fusion architecture that combines a spatially aligned relation-aware fusion path with a dynamic interaction fusion path based on cross-attention. The fusion features output by the two paths are weighted and fused using learnable weights, which preserves global semantic consistency while incorporating local semantic details, significantly improving the model's discriminative ability in complex multimodal scenarios.
[0044] This design employs an explicit cross-modal semantic relationship measurement mechanism. After spatial alignment, it explicitly calculates spatial distances (such as Euclidean distance) between text-image, audio-text, and audio-image pairs, and uses these distance matrices as a direct measure of semantic similarity. This design makes the model's decision-making process traceable and visualizeable, making it particularly suitable for identifying anomalous semantics such as "image-text mismatch" and "audio-visual disconnect" in harmful information detection. It provides an interpretable multimodal consistency verification mechanism for the content security field, outperforming traditional end-to-end black-box models.
[0045] A three-channel cross-modal cross-attention network is constructed to achieve dynamic semantic alignment from multiple perspectives. A three-baseline cross-attention architecture (text baseline, image baseline, and audio baseline) is designed, with one modality as the query and the other two modalities providing the key and value, to achieve cross-modal information guidance. This breaks through the limitations of traditional unidirectional attention, supports multi-perspective semantic focusing, and enhances the model's ability to capture key semantic segments.
[0046] In addition, this application Figure 2 This application provides a harmful information detection system based on cross-modal fusion characterization technology. For example... Figure 2 As shown in the embodiments of this application, the system mainly includes: The module 210 is used to acquire multi-source detection data; wherein, the multi-source detection data includes at least: text, image, and audio; the corresponding parsing program is determined according to the specific type of multi-source detection data, and then the corresponding feature vector is obtained; Spatial module 220 is used to concatenate feature vectors of various types, center the concatenated feature vectors to obtain the feature matrix corresponding to the covariance matrix; use the first P columns of the feature matrix as a transformation matrix to perform dimensionality transformation on the feature vectors of various types, mapping them to the same coordinate space; calculate the spatial distance between feature vectors of various types in the same coordinate space; concatenate the spatial distances, and use the connection... Layers and fully connected layers are used to obtain the final output spatial relationship features; The fusion module 230 is used to obtain a cross-modal fusion representation by weighted concatenation of feature vectors of various types using a cross-modal cross-attention network. The results module 240 is used to obtain the final fused feature vector by utilizing spatial relationship features and cross-modal fusion representation, and to obtain the harmful information detection results by using the final fused feature vector.
[0047] The above are method embodiments of this application. Based on the same inventive concept, embodiments of this application also provide a harmful information detection device based on cross-modal fusion characterization technology. Figure 3 As shown, the device includes: a processor; and a memory storing executable code thereon, which, when executed, causes the processor to perform a harmful information detection method based on cross-modal fusion characterization technology as described in the above embodiments.
[0048] Specifically, the server acquires multi-source detection data; this multi-source detection data includes at least text, images, and audio; based on the specific type of multi-source detection data, a corresponding parsing program is determined to obtain the corresponding feature vectors; the feature vectors of each type are concatenated, and the concatenated feature vectors are centered to obtain the feature matrix corresponding to the covariance matrix; the first... p The columns are used as transformation matrices to perform dimensionality transformations on feature vectors of different types, mapping them to the same coordinate space; the spatial distances between feature vectors of different types in the same coordinate space are calculated; and the spatial distances are concatenated using the connect function. The system uses a multilayer and fully connected layer to obtain the final output spatial relationship features. Based on the feature vectors of each type, a cross-modal cross-attention network is used to finally weight and concatenate them to obtain a cross-modal fusion representation. Using the spatial relationship features and the cross-modal fusion representation, the final fusion feature vector is obtained, and the result of harmful information detection is obtained using the final fusion feature vector.
[0049] In addition, embodiments of this application also provide a non-volatile computer storage medium storing executable instructions, which, when executed, implement a harmful information detection method based on cross-modal fusion characterization technology as described above.
[0050] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for detecting harmful information based on cross-modal fusion characterization technology, characterized in that, The method includes: Acquire multi-source detection data; the multi-source detection data includes at least: text, image, and audio; determine the corresponding parsing program based on the specific type of multi-source detection data, and then obtain the corresponding feature vector; The eigenvectors of various types are concatenated, and the concatenated eigenvectors are centered to obtain the eigenma corresponding to the covariance matrix; the first part of the eigenma is then... p The columns are used as transformation matrices to perform dimensionality transformations on feature vectors of different types, mapping them to the same coordinate space; the spatial distances between feature vectors of different types in the same coordinate space are calculated; and the spatial distances are concatenated using the connect function. Layers and fully connected layers are used to obtain the final output spatial relationship features; Based on the feature vectors of each type, a cross-modal cross-attention network is used to finally weight and concatenate them to obtain a cross-modal fusion representation; By utilizing spatial relationship features and cross-modal fusion representation, a final fusion feature vector is obtained, and the result of harmful information detection is obtained using the final fusion feature vector.
2. The harmful information detection method based on cross-modal fusion characterization technology according to claim 1, characterized in that, Based on the specific type of multi-source detection data, the corresponding parsing procedure is determined, and then the corresponding feature vector is obtained, specifically including: The ALBERT encoder is used to extract fixed-dimensional contextual semantic feature representations from the text; The ConvNeXt encoder, based on a pure convolutional network structure, extracts local texture and high-level semantic features of the image layer by layer, and obtains multi-scale feature maps from the image. The audio waveform is converted into a preset high-dimensional vector sequence using the wav2vec 2.0 encoder, and then semantic information feature vectors in speech are extracted from the preset high-dimensional vector sequence.
3. The harmful information detection method based on cross-modal fusion characterization technology according to claim 1, characterized in that, The eigenvectors of various types are concatenated, and the concatenated eigenvectors are centered to obtain the eigenma corresponding to the covariance matrix. Specifically, this includes: Through the formula: ； ； ； ； The concatenated feature vector X is calculated; where n represents the total number of feature vectors of each type, and T represents the text representation vector after feature extraction. This represents the feature vector corresponding to the j-th text. This represents the representation vector of the image after feature extraction. This represents the feature vector corresponding to the j-th image. This represents the representation vector of the audio after feature extraction. This represents the feature vector corresponding to the j-th audio, and ⊕ represents the concatenation operator; Through the formula: The covariance matrix S is calculated. Then the characteristic matrix corresponding to the covariance matrix is obtained.
4. The method for detecting harmful information based on cross-modal fusion characterization technology according to claim 1, characterized in that, The first part of the feature matrix p The column serves as a transformation matrix, used to perform dimensionality transformations on eigenvectors of various types, mapping them to the same coordinate space. Specifically, this includes: Through the formula: ； Obtain the feature vectors of text, images, and audio after dimensionality transformation; in, , , These are preset matrices used to convert the representation vectors of text, images, and audio into matrices of the same dimension as the transformation matrix; , , Let represent the feature vectors of text, image, and audio after dimensional transformation, respectively, and P represent the transformation matrix.
5. The harmful information detection method based on cross-modal fusion characterization technology according to claim 1, characterized in that, Calculate the spatial distance between feature vectors of different types in the same coordinate space; concatenate the spatial distances and use the connect... Layers and fully connected layers are used to obtain the final output spatial relationship features, specifically including: Through the formula: ； ； ； Calculate the spatial distance between text and image, text and audio, and image and audio. , , ; Through the formula: ； ； ； Obtain the final output spatial relationship features ; in, express Activation function express layer, Indicates a fully connected layer. This represents the concatenation operator.
6. The harmful information detection method based on cross-modal fusion characterization technology according to claim 1, characterized in that, Based on the feature vectors of each type, a cross-modal cross-attention network is used to finally weight and concatenate them to obtain a cross-modal fusion representation, which specifically includes: A cross-modal attention network is used to obtain the fusion features of the baseline layer for feature vectors of various types; Through the formula: Finally, weighted splicing is used to obtain the cross-modal fusion representation. ; in, , , To preset learnable weights, Indicates text-based fusion features, Represents image-based fusion features, This represents the image-audio fusion features.
7. The harmful information detection method based on cross-modal fusion characterization technology according to claim 1, characterized in that, By utilizing spatial relationship features and cross-modal fusion representation, a final fused feature vector is obtained. This final fused feature vector is then used to obtain the harmful information detection results, specifically including: Through the formula: F= Foutput1+(1- Foutput2, obtain the final fused feature vector F; Through the formula: Result = Softmax(MLP(F)) to obtain the harmful information detection result Result; Where MLP stands for Multilayer Perceptron and Softmax stands for Normalization Function.
8. A harmful information detection system based on cross-modal fusion characterization technology, characterized in that, The system includes: The acquisition module is used to acquire multi-source detection data; the multi-source detection data includes at least: text, image, and audio; the corresponding parsing program is determined according to the specific type of multi-source detection data, and then the corresponding feature vector is obtained; The spatial module is used to concatenate feature vectors of various types, center the concatenated feature vectors to obtain the feature matrix corresponding to the covariance matrix, use the first P columns of the feature matrix as the transformation matrix to perform dimensionality transformation on the feature vectors of various types, mapping them to the same coordinate space, calculate the spatial distance between feature vectors of different types in the same coordinate space, and concatenate the spatial distances using the connect function. Layers and fully connected layers are used to obtain the final output spatial relationship features; The fusion module is used to obtain cross-modal fusion representations by weighted concatenation of feature vectors of various types using a cross-modal cross-attention network. The results module is used to obtain the final fused feature vector by utilizing spatial relationship features and cross-modal fusion representation, and then use the final fused feature vector to obtain the harmful information detection results.
9. A harmful information detection device based on cross-modal fusion characterization technology, characterized in that, The device includes: processor; And a memory having executable code stored thereon, which, when executed, causes the processor to perform a harmful information detection method based on cross-modal fusion characterization technology as described in any one of claims 1-7.
10. A non-volatile computer storage medium, characterized in that, It stores computer instructions, which, when executed, implement a harmful information detection method based on cross-modal fusion characterization technology as described in any one of claims 1-7.