Multi-modal fusion content fine-grained classification system, method, equipment and medium

Abstract

The invention discloses a multi-modal fused content fine-grained classification system, method and equipment and a medium. The system comprises a multi-modal data cleaning module, a multi-modal data processing module and a multi-modal data processing module, wherein the multi-modal data cleaning module is used for carrying out special cleaning on text data, image data and video data respectively; the data enhancement processing module is used for sampling by adopting an oversampling strategy based on the number of labels, and performing text enhancement processing and image enhancement processing on sampled samples respectively; the feature extraction processing module performs feature extraction on the text features to output a final text vector, and performs feature extraction on the image features to obtain a final image vector; the multi-modal coding module carries out coding processing on the text vector, the image vector and the mixed vector; the fine-grained classification module outputs a final classification result according to the output vector of the multi-modal coding; according to the method, the deep learning technology can be applied, multi-modal information is effectively combined, the classification accuracy is improved by combining multi-modal data design, and the problems of data imbalance and modal association missing are effectively solved.

Description

Technical Field

[0001] This invention relates to the fields of information flow content processing and deep learning technology. Specifically, this invention is applied to the field of multimodal content classification in social media, news aggregation, and video platforms, and particularly relates to a multimodal fusion fine-grained content classification system, method, device, and medium. Background Technology

[0002] Existing information flow classification technologies have significant drawbacks:

[0003] (1) Manual marking is inefficient and inconsistent;

[0004] (2) Rule engines and traditional machine learning have difficulty handling complex content, and subjectivity and label noise are prominent.

[0005] (3) Original multimodal fusion ignores modal correlation, and feature splicing causes the curse of dimensionality;

[0006] (4) Data imbalance leads to poor minority class performance, insufficient ambiguity and domain adaptability;

[0007] The aforementioned issues result in low accuracy in "content-user" matching, failing to meet the platform's content management and monetization needs. Summary of the Invention

[0008] The purpose of this invention is to provide a multimodal fusion content fine-grained classification system, method, device and medium to address the above-mentioned problems in the prior art, thereby solving all or one of the above-mentioned problems in the prior art.

[0009] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:

[0010] On one hand, the present invention provides a multimodal fusion-based fine-grained content classification system, comprising:

[0011] The multimodal data cleaning module is used to perform specialized cleaning for text data, image data, and video data respectively.

[0012] The data augmentation processing module is used to: adopt an oversampling strategy based on the number of labels, sample labels with a sample number lower than the mean according to the first sampling formula, and perform text augmentation processing based on synonym replacement and image augmentation processing based on multiple operation queues on the sampled samples respectively.

[0013] The feature extraction processing module is used to: extract text features using a text feature extraction model trained in a vertical domain, and process the output vector through multi-layer average pooling to obtain the final text vector; and extract image features using an image feature extraction model trained in a vertical domain, and calculate the final image vector based on the flag bit vector and the mean.

[0014] The multimodal encoding module is used to encode text vectors, image vectors, and text-image hybrid vectors by integrating multi-head self-attention mechanism, feedforward neural network and LayerNorm technology, and output feature vectors of corresponding dimensions.

[0015] The fine-grained classification module is used to: pass the multimodal encoded output vector through a three-layer fully connected perceptron containing a GELU activation function and a Dropout layer to output the final classification result; during the training phase, cross-entropy loss or weighted cross-entropy loss is used according to the data balance, the learning rate is set in layers and the F1 score is used as the early stopping index.

[0016] As an improved solution, the multimodal data cleaning module is further used to: filter out redundant symbols, escape characters, empty characters, emoticons and links from the text data; divide the filtered text data into blocks by reverse searching for punctuation marks according to the first character length; and associate blocks with the same content through an ID list.

[0017] As an improved solution, the multimodal data cleaning module is further used to: adjust the image to a uniform size by resampling based on pixel area relationships; perform normalization processing on pixel data; and associate multiple images with the same content through an ID list.

[0018] As an improved solution, the multimodal data cleaning module is further configured to: decompose the video data into audio data and video data; convert the audio data into text data; sort the video data by the importance of video frames and retain key frames.

[0019] As an improved approach, the first sampling formula is: [(maximum number of labels - average number of labels) / log(current number of labels)];

[0020] The data augmentation processing module is further configured to perform the text augmentation processing: segment the text into words, calculate the high cosine similarity synonyms for each word and construct a dictionary; adjust the number of replacement words according to the length of the segmented text to ensure that the semantics of the augmented text remain unchanged;

[0021] The data enhancement processing module is further used to perform the image enhancement processing: performing random cropping, horizontal / left / right flipping, brightness / contrast / saturation adjustment, and random masking of rectangular regions in sequence according to a preset queue.

[0022] As an improved solution, the multimodal coding module is further configured to: segment each input vector into equal lengths based on the multi-head self-attention mechanism; copy each segmented vector to three copies; pass the copied vectors through fully connected layers with the same input and output dimensions, calculate according to the self-attention formula, concatenate them, and then pass them through fully connected layers with the same input and output dimensions again to obtain the multi-head self-attention layer output; and add the multi-head self-attention layer output to the corresponding input vector through a residual connection to obtain the final output vector of the attention layer.

[0023] The multimodal coding module is further configured to: pass the final output vector of the attention layer sequentially through the LayerNorm layer, the first fully connected layer, the LayerNorm layer, the GELU activation function, and the second fully connected layer, and then connect the resulting output vector with the final output vector of the attention layer through a residual connection to obtain the encoded vector output by the feedforward neural network.

[0024] As an improved solution, the fine-grained classification module is further configured to: concatenate the three-class output vectors of the multimodal encoding module using a multilayer perceptron neural network to obtain a 3072-dimensional vector; pass the 3072-dimensional vector through a first fully connected layer followed by a GELU activation function and a Dropout layer with a first parameter; pass the output vector of the first fully connected layer to a second fully connected layer, which is then followed by a GELU activation function and a Dropout layer with a second parameter, and pass the output vector to a third fully connected layer; and use the output of the third fully connected layer as the final classification result.

[0025] On the other hand, the present invention also provides a multimodal fusion method for fine-grained content classification, comprising the following steps:

[0026] Multimodal data cleaning:

[0027] Specialized cleaning methods were applied to text data, image data, and video data respectively.

[0028] Data augmentation processing:

[0029] An oversampling strategy based on the number of labels is adopted. Labels with a sample number lower than the mean are sampled according to the first sampling formula. Text enhancement processing based on synonym replacement and image enhancement processing based on multiple operation queues are performed on the sampled samples respectively.

[0030] Feature extraction processing:

[0031] A text feature extraction model trained in a vertical domain is used to extract text features, and the output vector is processed by multi-layer average pooling to obtain the final text vector.

[0032] An image feature extraction model trained in a vertical domain is used to extract image features, and the final image vector is calculated based on the flag bit vector and the mean.

[0033] Multimodal coding:

[0034] A multimodal coding module integrating multi-head self-attention mechanism, feedforward neural network and LayerNorm technology is used to encode text vectors, image vectors and text-image hybrid vectors respectively, and output feature vectors of corresponding dimensions.

[0035] Fine-grained classification:

[0036] The multimodal encoded output vector is passed through a three-layer fully connected perceptron containing a GELU activation function and a Dropout layer to output the final classification result. During the training phase, cross-entropy loss or weighted cross-entropy loss is used according to the data balance, the learning rate is set in layers, and the F1 score is used as the early stopping index.

[0037] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the multimodal fusion content fine-grained classification method.

[0038] On the other hand, the present invention also provides a computer device, the computer device including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein:

[0039] The memory is used to store computer programs;

[0040] The processor is configured to execute the steps of the multimodal fusion content fine-grained classification method by running a program stored in the memory.

[0041] The beneficial effects of the technical solution of this invention are:

[0042] 1. The multimodal fusion content fine-grained classification system described in this invention can effectively combine multimodal information through the cooperation of system modules and the application of deep learning technology to solve the semantic missing problem. By combining multimodal data design, it enriches the semantic expression of content, alleviates human error labeling, greatly improves review efficiency, improves classification accuracy, and effectively solves the problems of data imbalance and missing modal associations.

[0043] 2. The multimodal fusion content fine-grained classification method of the present invention can call system modules in an orderly manner, thereby realizing the system logic of the multimodal fusion content fine-grained classification system of the present invention.

[0044] 3. The computer-readable storage medium of the present invention can enable the guidance system module to cooperate in order to realize the multimodal fusion content fine-grained classification method of the present invention. Furthermore, the computer-readable storage medium of the present invention also effectively improves the operability of the multimodal fusion content fine-grained classification method.

[0045] 4. The computer device described in this invention can store and execute the computer-readable storage medium, thereby realizing the multimodal fusion content fine-grained classification method described in this invention. Attached Figure Description

[0046] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram of the architecture of the multimodal fusion content fine-grained classification system described in Embodiment 1 of the present invention;

[0048] Figure 2 This is a schematic diagram of the logical flow of the multimodal fusion content fine-grained classification system described in Embodiment 1 of the present invention;

[0049] Figure 3 This is a flowchart illustrating the multimodal fusion content fine-grained classification method described in Embodiment 2 of the present invention;

[0050] Figure 4 This is a schematic diagram of the structure of the computer device described in Embodiment 4 of the present invention;

[0051] The markings in the attached diagram are explained as follows:

[0052] 1501. Processor; 1502. Communication interface; 1503. Memory; 1504. Communication bus. Detailed Implementation

[0053] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0054] In the description of this invention, it should be noted that the embodiments described in this invention are only some embodiments of this invention, not all embodiments; all other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0055] The terms "first," "second," etc., used in this specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0056] In the description of this invention, it should be noted that the existing technical solutions for fine-grained classification systems of information flow content encompass a variety of methods and tools. These methods and tools can be used to classify and label text data, and common solutions are as follows:

[0057] (1) Labeling based on rule engine and statistical learning: This method uses a predefined set of rules to automatically label content, uses regular expressions to match content keywords, and uses hard coding to match the corresponding tags of the content. This method is simple and direct, the rules are transparent and there is no training cost, but it is usually not suitable for processing complex content because it requires manual writing and maintenance of rules.

[0058] (2) Traditional machine learning approach: implemented through manual feature extraction + shallow classifier; manual feature extraction mainly includes various natural language processing schemes, as follows:

[0059] TF-IDF (Term Frequency-Inverse Document Frequency) is a commonly used text feature representation method that encodes text information based on the frequency of word occurrence and its importance in the entire text set.

[0060] The Bag-of-Words (BOW) model represents text as a set of word frequency statistics, without considering word order and grammatical structure, and uses the frequency of each word appearing in the document as a feature representing the text.

[0061] Word embeddings, such as Word2Vec, GloVe, and FastText, can map words to a low-dimensional vector space to capture semantic relationships between words.

[0062] Image feature extraction schemes mainly include:

[0063] SIFT (Scale Invariant Feature Transform) is a machine vision algorithm used to detect and describe local features in images. It finds extreme points in spatial scale and extracts their position, scale, and rotation invariants.

[0064] SURF (Accelerated Robust Features) is an efficient variant of SIFT that extracts scale-invariant features. It uses the determinant of the Hessian matrix as feature points and accelerates computation using integral images.

[0065] ORB combines FAST keypoint detection and an improved BRIEF descriptor, resulting in an efficient and rotation-invariant feature extraction algorithm.

[0066] The extracted features are then combined with machine learning models such as Naive Bayes, Logistic Regression, Support Vector Machine, and Random Forest to perform unimodal classification.

[0067] (3) Original scheme for multimodal fusion:

[0068] Decision-level fusion: A classifier is trained for each modality, and the prediction results of the classifiers are weighted and fused to obtain the final prediction result.

[0069] Feature-level fusion: Features extracted from different modalities are concatenated and then used to make predictions using machine learning models such as Naive Bayes, Logistic Regression, Support Vector Machine, and Random Forest.

[0070] In the description of this invention, it should be noted that although the fine-grained classification system for information flow content has shown high value in many applications, it still has some existing drawbacks and challenges, some of which include:

[0071] (1) Subjectivity: The interpretation and labeling of text content are often influenced by subjective factors; different people may give the same text different labels. This makes automatic labeling systems perform poorly when handling subjective content.

[0072] (2) Label noise: Automatic labeling systems sometimes assign incorrect labels to text, especially when the text is ambiguous or unclear; this may lead to incorrect text classification or label assignment.

[0073] (3) Lack of modal association: Different modal representations may have different categories. A single modal representation cannot fully represent the content. A crude weighted summation ignores the association and leads to misjudgment.

[0074] (4) Imbalanced data: Text data is often imbalanced, meaning that the number of samples for some labels is much greater than that for other labels; this may lead to a decrease in the performance of the model on a few categories because they do not have enough training examples.

[0075] (5) Curse of Dimensions: Directly splicing features for prediction can lead to excessively high feature dimensionality, thus causing the curse of dimensionality and reducing accuracy.

[0076] (6) Domain specificity: Automatic labeling systems usually require a large amount of training data and may not be universal across different domains; building a universal system applicable to multiple domains may be more challenging.

[0077] (7) Handling ambiguity: Natural language texts are often full of ambiguity, and the same word or phrase may have different meanings in different contexts; automatic labeling systems need to solve this ambiguity problem to ensure accuracy.

[0078] (8) Difficulty in designing a tag system: Designing a suitable tag system requires domain expertise and a deep understanding of user needs; an incorrect tag system design may result in inaccurate or unmet tags.

[0079] Example 1

[0080] This embodiment provides a multimodal fusion-based fine-grained content classification system, such as... Figure 1 and Figure 2 As shown, it includes:

[0081] The core technology of this invention lies in its focus on multimodal collaboration, data adaptation, and precise encoding. It ensures data quality through data cleaning and enhancement, extracts high-quality features using pre-trained models in vertical domains, achieves semantic fusion through multimodal encoding, and finally outputs results through fine-grained classification, forming a closed-loop process. The details are as follows:

[0082] (1) A multimodal data cleaning module, used to perform multimodal data cleaning steps, including:

[0083] S101, Text Cleaning:

[0084] Use regular expressions to filter out redundant symbols, escape characters, empty characters, emojis, and HTTP / HTTPS links from the text.

[0085] According to the rule of length ≤ 512, punctuation marks are searched backwards from 512 characters (i.e. the length of the first character) to form blocks. Blocks with the same content are associated through an ID list, and the average value of text features with the same ID is then calculated and merged.

[0086] It should be noted that since the number of text blocks after dividing a single piece of content may be greater than one, an additional set of variables is passed in during training. This set is a list of content IDs corresponding to the text blocks. For example, if content a, b, and c have text blocks 4, 2, and 1 respectively, then this variable would be [0,0,0,0,1,1,2]).

[0087] S102, Image Cleaning:

[0088] The image is resampled to a uniform size by pixel area relationship resampling, and the pixel data is normalized so that the mean of each channel pixel is 0 and the variance is 1.

[0089] Multiple image contents are associated through an ID list, and the average feature of images with the same ID is subsequently calculated.

[0090] It should be noted that, similarly, the number of images for a content may be greater than one. In this case, another set of variables is passed in during training, which is a list of content IDs corresponding to the images. For example, if content a, b, and c have 3, 1, and 2 images respectively, then this variable is [0,0,0,1,2,2].

[0091] S103, Video Cleaning:

[0092] The video with sound is broken down into audio data and video data;

[0093] Audio data is converted into text data using the Whisper model;

[0094] The video data uses a ResNet+LSTM model to sort the importance of video frames, and retains the Top-K frames as video keyframes.

[0095] Finally, each piece of content is output as text data and image data. If any data is empty, it is filled with a vector of all zeros.

[0096] (2) A data augmentation processing module, used to perform data augmentation steps, including:

[0097] S201, Oversampling Processing:

[0098] Count the number of samples for each label;

[0099] Labels with a sample size lower than the mean are sampled using the preset first sampling formula ((maximum number of labels - mean number of labels) / log(current number of labels)) to avoid duplicate sample selection.

[0100] Randomly select text enhancement and image enhancement from the extracted samples;

[0101] It should be noted that, due to the limited number of samples in the domain and the differences in the amount of content between different labels, this imbalance of samples can lead to overfitting on some labels with large amounts of data in the final training results, resulting in artificially high metrics and the loss of some label information in the model. Therefore, the above-mentioned oversampling process is performed to solve this problem.

[0102] S202, Text Enhancement:

[0103] The text is segmented using Jieba, and the Top 5 cosine similarity synonyms (similarity ≥ 0.95) of each word are calculated using the BERT model and a dictionary is constructed.

[0104] The number of replacement words is adjusted according to the length of the segmented text to ensure that the semantics of the enhanced text remain unchanged.

[0105] As an alternative example: for samples with 20 or more words, randomly select 2 to 6 words and randomly select one word from the top 5 words with the highest similarity to replace them; for samples with less than 20 words, randomly select 1 or 2 words and randomly select one word from the top 5 words with the highest similarity to replace them.

[0106] S203, Image Enhancement:

[0107] Perform operations such as random cropping, horizontal / left / right flipping, brightness / contrast / saturation adjustment, and random masking of rectangular areas in sequence according to a preset queue to enhance the diversity of image data;

[0108] The preset queue is: [Random cropping → Random horizontal image flipping → Random horizontal image flipping → Modify brightness, contrast, and saturation → Randomly mask a rectangular area].

[0109] (3) Model training module, used to execute model training steps, including:

[0110] The feature extraction and processing module is used to perform:

[0111] Text feature extraction:

[0112] The MacBERT model is used to convert the cleaned text into 512-length tokens, with padding for any length less than 512.

[0113] The output vectors of the first and last layers of the model are retained, and average pooling is performed on each layer and then concatenated. After concatenation, average pooling is performed again to obtain the text vector of each text segment.

[0114] The average of the text vectors of multiple text blocks with the same content is used as the final text vector.

[0115] Image feature extraction:

[0116] The ViT model is used to convert the image into a set of vectors, and its [cls] flag vector is extracted, which represents the global features of the image;

[0117] The average value of image vectors with the same content and multiple image features is used as the final image vector.

[0118] The multimodal coding module is used to perform:

[0119] Attention layer:

[0120] The number of attention heads is set to 12, and each input vector is split into equal lengths.

[0121] Each of the split vectors is copied three times and labeled as q, k, and v respectively;

[0122] q, k, and v are passed through fully connected layers with the same input and output dimensions, calculated according to the self-attention formula, and then concatenated. After concatenation, they are passed through another fully connected layer with the same input and output dimensions to obtain the output of the multi-head self-attention layer.

[0123] The vector output from the multi-head self-attention layer is passed through the residual connection module and added to the corresponding input vector to obtain the final output vector;

[0124] The formula for self-attention is as follows:

[0125] ;

[0126] In the formula, is the dot product of the transposes of q and k, divided by the square root of the dimension of k, then softmaxed, passed through a dropout layer, and finally the dot product with vector v.

[0127] Feedforward layer:

[0128] The final output vector is passed through LayerNorm, fully connected layer 1 (i.e., the first fully connected layer), LayerNorm, GELU activation function and fully connected layer 2 (i.e., the second fully connected layer) in sequence.

[0129] The vector obtained from the above operations is concatenated with the final output vector through a residual connection to obtain the encoded vector output by the feedforward neural network.

[0130] In this case, the input dimension of fully connected layer 1 and the output dimension of fully connected layer 2 are the same, and the output dimension of fully connected layer 1 and the input dimension of fully connected layer 2 are set to 256.

[0131] In the modal coding process, three types of coding modules are used to adapt to text vector input, image vector input, and hybrid vector input after concatenation of text vector and image vector;

[0132] The output dimensions of the three types of encoding modules are 768 dimensions for text encoding, 768 dimensions for image encoding, and 1536 dimensions for hybrid encoding.

[0133] The fine-grained classification module is used to perform:

[0134] A 3072-dimensional vector is obtained by concatenating three types of encoded output vectors using a multilayer perceptron neural network.

[0135] The 3072-dimensional vector is then processed through three fully connected layers;

[0136] The first fully connected layer has an output dimension of 768. After the first fully connected layer, a GELU activation function and a Dropout layer with a parameter of 0.3 are applied. The output vector of the first fully connected layer is then passed to the second fully connected layer, which has an output dimension of 128. After the second fully connected layer, a GELU activation function and a Dropout layer with a parameter of 0.2 are applied, and the output vector is then passed to the third fully connected layer. The output dimension of the third fully connected layer is the number of categories to be classified, thus obtaining the corresponding classification result.

[0137] The training optimization module is used to perform:

[0138] The data produced by the multimodal data cleaning and data augmentation steps are fed into the model of this S300 step for model training.

[0139] Among them, cross-entropy loss or weighted cross-entropy loss is selected according to the data balance (i.e., cross-entropy loss is used normally; if the maximum number of labels after data augmentation is more than 10 times the minimum number of labels, then weighted cross-entropy loss is used, with the weight being the -1 power of the number of samples respectively).

[0140] The learning rate is set in layers (1 / 5 of the global learning rate for the feature extraction layer), and the F1 value is verified every 1000 steps. If it does not improve after five steps, early stopping is triggered.

[0141] In the online prediction, the top 5 F1 score models are retained, and two models are selected for deployment based on business feedback.

[0142] It should be noted that the above examples are merely for explaining the present invention and should not be construed as limiting the scope of protection of the present invention.

[0143] Example 2

[0144] This embodiment, based on the same inventive concept as the multimodal fusion content fine-grained classification system described in Embodiment 1, provides a multimodal fusion content fine-grained classification method, such as... Figure 3 As shown, it includes the following steps:

[0145] S100, Multimodal Data Cleaning:

[0146] Specialized cleaning methods were applied to text data, image data, and video data respectively.

[0147] S200, Data Augmentation Processing:

[0148] An oversampling strategy based on the number of labels is adopted. Labels with a sample number lower than the mean are sampled according to the first sampling formula. Text enhancement processing based on synonym replacement and image enhancement processing based on multiple operation queues are performed on the sampled samples respectively.

[0149] S300, Feature Extraction Processing:

[0150] A text feature extraction model trained in a vertical domain is used to extract text features, and the output vector is processed by multi-layer average pooling to obtain the final text vector.

[0151] An image feature extraction model trained in a vertical domain is used to extract image features, and the final image vector is calculated based on the flag bit vector and the mean.

[0152] S400, Multimodal Coding:

[0153] A multimodal coding module integrating multi-head self-attention mechanism, feedforward neural network and LayerNorm technology is used to encode text vectors, image vectors and text-image hybrid vectors respectively, and output feature vectors of corresponding dimensions.

[0154] S500, fine-grained classification:

[0155] The multimodal encoded output vector is passed through a three-layer fully connected perceptron containing a GELU activation function and a Dropout layer to output the final classification result. During the training phase, cross-entropy loss or weighted cross-entropy loss is used according to the data balance, the learning rate is set in layers, and the F1 score is used as the early stopping index.

[0156] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific logical process of the method described above can be referred to the corresponding working process of the system, device and unit in the foregoing method embodiments, and will not be repeated here.

[0157] Example 3

[0158] This embodiment provides a computer-readable storage medium, including:

[0159] The storage medium is used to store computer software instructions used to implement the multimodal fusion content fine-grained classification method described in Embodiment 2 above. It includes a program for executing the program set up for the multimodal fusion content fine-grained classification method. Specifically, the executable program can be built into the multimodal fusion content fine-grained classification system described in Embodiment 1. In this way, the multimodal fusion content fine-grained classification system can implement the multimodal fusion content fine-grained classification method described in Embodiment 2 by executing the built-in executable program.

[0160] Furthermore, the computer-readable storage medium in this embodiment can be any combination of one or more readable storage media, wherein the readable storage medium includes an electrical, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.

[0161] Example 4

[0162] This embodiment provides an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 1501, a communication interface 1502, a memory 1503, and a communication bus 1504, wherein the processor 1501, the communication interface 1502, and the memory 1503 communicate with each other through the communication bus 1504.

[0163] Memory 1503 is used to store computer programs;

[0164] When the processor 1501 executes the computer program stored in the memory 1503, it implements the steps of the multimodal fusion content fine-grained classification method described in Embodiment 1 above.

[0165] As one embodiment of the present invention, the communication bus mentioned in the terminal above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0166] As one embodiment of the present invention, the communication interface is used for communication between the aforementioned terminal and other devices.

[0167] In one embodiment of the present invention, the memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0168] As one embodiment of the present invention, the processor described above may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0169] It should be understood that in the various embodiments of this document, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0170] It should also be understood that, in the embodiments herein, the term "and / or" is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following associated objects have an "or" relationship.

[0171] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this document.

[0172] In the embodiments provided herein, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.

[0173] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described herein, depending on actual needs.

[0174] Furthermore, the functional units in the various embodiments of this document can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0175] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this paper, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this paper. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0176] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A multimodal fusion-based fine-grained content classification system, characterized in that, include: The multimodal data cleaning module is used to perform specialized cleaning for text data, image data, and video data respectively. The data augmentation processing module is used to: adopt an oversampling strategy based on the number of labels, sample labels with a sample number lower than the mean according to the first sampling formula, and perform text augmentation processing based on synonym replacement and image augmentation processing based on multiple operation queues on the sampled samples respectively. The feature extraction processing module is used to: extract text features using a text feature extraction model trained in a vertical domain, and process the output vector through multi-layer average pooling to obtain the final text vector; and extract image features using an image feature extraction model trained in a vertical domain, and calculate the final image vector based on the flag bit vector and the mean. The multimodal encoding module is used to encode text vectors, image vectors, and text-image hybrid vectors by integrating multi-head self-attention mechanism, feedforward neural network and LayerNorm technology, and output feature vectors of corresponding dimensions. The fine-grained classification module is used to: pass the multimodal encoded output vector through a three-layer fully connected perceptron containing a GELU activation function and a Dropout layer to output the final classification result; during the training phase, cross-entropy loss or weighted cross-entropy loss is used according to the data balance, the learning rate is set in layers and the F1 score is used as the early stopping index.

2. The multimodal fusion content fine-grained classification system according to claim 1, characterized in that: The multimodal data cleaning module is further configured to: filter out redundant symbols, escape characters, empty characters, emoticons, and links from the text data; and divide the filtered text data into blocks by reverse searching for punctuation marks according to the first character length. Link content blocks together using a list of IDs.

3. The multimodal fusion content fine-grained classification system according to claim 1, characterized in that: The multimodal data cleaning module is further configured to: adjust the image to a uniform size by resampling based on pixel area relationships for the image data; perform normalization processing on the pixel data; and associate multiple images with the same content through an ID list.

4. The multimodal fusion content fine-grained classification system according to claim 1, characterized in that: The multimodal data cleaning module is further configured to: for the video data, decompose the video with sound into audio data and video data; convert the audio data into text data; sort the video data by the importance of video frames and retain the video keyframes.

5. The multimodal fusion content fine-grained classification system according to claim 1, characterized in that: The first sampling formula is: [(maximum number of labels - average number of labels) / log(current number of labels)]; The data augmentation processing module is further configured to perform the text augmentation processing: segment the text into words, calculate the high cosine similarity synonyms for each word and construct a dictionary; adjust the number of replacement words according to the length of the segmented text to ensure that the semantics of the augmented text remain unchanged; The data enhancement processing module is further used to perform the image enhancement processing: performing random cropping, horizontal / left / right flipping, brightness / contrast / saturation adjustment, and random masking of rectangular regions in sequence according to a preset queue.

6. The multimodal fusion content fine-grained classification system according to claim 1, characterized in that: The multimodal coding module is further configured to: segment each input vector into equal lengths based on the multi-head self-attention mechanism; copy each segmented vector to three copies; pass the copied vectors through fully connected layers with the same input and output dimensions, calculate according to the self-attention formula, concatenate them, and then pass them through fully connected layers with the same input and output dimensions again to obtain the multi-head self-attention layer output; and add the multi-head self-attention layer output to the corresponding input vector through a residual connection to obtain the final output vector of the attention layer. The multimodal coding module is further configured to: pass the final output vector of the attention layer sequentially through the LayerNorm layer, the first fully connected layer, the LayerNorm layer, the GELU activation function, and the second fully connected layer, and then connect the resulting output vector with the final output vector of the attention layer through a residual connection to obtain the encoded vector output by the feedforward neural network.

7. The multimodal fusion content fine-grained classification system according to claim 1, characterized in that: The fine-grained classification module is further configured to: concatenate the three-class output vectors of the multimodal encoding module using a multilayer perceptron neural network to obtain a 3072-dimensional vector; pass the 3072-dimensional vector through a first fully connected layer followed by a GELU activation function and a Dropout layer with a first parameter; pass the output vector of the first fully connected layer to a second fully connected layer, which is then followed by a GELU activation function and a Dropout layer with a second parameter, and pass the output vector to a third fully connected layer; and use the output of the third fully connected layer as the final classification result.

8. A method for fine-grained content classification based on a multimodal fusion content fine-grained classification system according to any one of claims 1 to 7, characterized in that, The method includes the following steps: Multimodal data cleaning: Specialized cleaning methods were applied to text data, image data, and video data respectively. Data augmentation processing: An oversampling strategy based on the number of labels is adopted. Labels with a sample number lower than the mean are sampled according to the first sampling formula. Text enhancement processing based on synonym replacement and image enhancement processing based on multiple operation queues are performed on the sampled samples respectively. Feature extraction processing: A text feature extraction model trained in a vertical domain is used to extract text features, and the output vector is processed by multi-layer average pooling to obtain the final text vector. An image feature extraction model trained in a vertical domain is used to extract image features, and the final image vector is calculated based on the flag bit vector and the mean. Multimodal coding: A multimodal coding module integrating multi-head self-attention mechanism, feedforward neural network and LayerNorm technology is used to encode text vectors, image vectors and text-image hybrid vectors respectively, and output feature vectors of corresponding dimensions. Fine-grained classification: The multimodal encoded output vector is passed through a three-layer fully connected perceptron containing a GELU activation function and a Dropout layer to output the final classification result. During the training phase, cross-entropy loss or weighted cross-entropy loss is used according to the data balance, the learning rate is set in layers, and the F1 score is used as the early stopping index.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the multimodal fusion content fine-grained classification method of claim 8.

10. A computer device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein: The memory is used to store computer programs; The processor is configured to execute the steps of the content fine-grained classification method of multimodal fusion as described in claim 8 by running a program stored in the memory.

Images