Content classification method and device for multi-modal data and storage medium

Abstract

The embodiment of the invention provides a multi-modal data content classification method and device and a storage medium, and the method comprises the steps: obtaining multi-modal data; splitting the multi-modal data to obtain text data, picture data and an image-text arrangement sequence; obtaining a text feature vector of the text data and an image feature vector of the picture data by utilizing an embedding technology; according to the image-text arrangement sequence, splicing the text feature vector and the image feature vector to obtain a matrix; and based on a convolutional neural network (CNN), determining the category of the multi-modal data according to the matrix, a preset category number and a preset category. According to the method, the key features of the multi-modal data can be captured more comprehensively, so that the classification accuracy and robustness are improved.

Description

Technical Field

[0001] This document relates to the technical field of data classification, and in particular to a content classification method, device and storage medium for multimodal data. Background Art

[0002] Image-text multimodal feature recognition combines information from two different modalities, image and text, and can more accurately classify the target object by complementing and verifying each other.

[0003] Existing technologies often only focus on information of a single modality, such as text or images, and perform classification based on the features of a single modality vector.

[0004] However, in the prior art, due to the different features extracted from images and texts, multimodal features are difficult to combine, and the spatial arrangement of images and texts is complex and the quantity is different, which increases the difficulty of feature fusion and makes it difficult to quickly and accurately classify articles. Summary of the invention

[0005] In view of the above scheme, the present application aims to propose a content classification method, device and storage medium for multimodal data to solve at least one of the above technical problems.

[0006] In a first aspect, one or more embodiments of this specification provide a method for content classification of multimodal data, including:

[0007] Acquire multimodal data;

[0008] Split the multimodal data to obtain text data, image data, and the order of text and image data;

[0009] Using embedding technology to obtain a text feature vector of the text data and an image feature vector of the picture data;

[0010] According to the arrangement order of the image and text, concatenate the text feature vector and the image feature vector to obtain a matrix; and

[0011] Based on a convolutional neural network (CNN), the category of the multimodal data is determined according to the matrix, the preset number of categories and the preset categories.

[0012] Furthermore, the text feature vector and the image feature vector are concatenated according to the arrangement order of the image and text to obtain a matrix including:

[0013] According to the text feature vector, determine the number of text feature vector rows m;

[0014] Determine the number of rows n of the image feature vector according to the image feature vector; and

[0015] According to the arrangement order of the image and text, the number of rows m, the number of rows n and the preset vector dimension d, the text feature vectors and the image feature vectors are alternately spliced ​​to generate a matrix M with (m+n) rows and d columns.

[0016] Further, the determining the category of the multimodal data based on the convolutional neural network (CNN) according to the matrix, the preset number of categories and the preset category includes:

[0017] Based on a convolutional neural network, a convolution dimension reduction process is performed on the matrix to obtain a text image feature vector;

[0018] Based on a feedforward neural network (FNN), a fully connected layer parameter matrix is ​​set according to the text image feature vector and a preset number of categories to map the feature vector into a category vector;

[0019] Based on a multi-class classification algorithm (softmax), the probability of each class is obtained according to the class vector; and

[0020] The category of the multimodal data is determined according to the probability of each category.

[0021] Furthermore, the step of performing convolution dimensionality reduction processing on the matrix based on a convolutional neural network (CNN) to obtain a text image feature vector comprises:

[0022] Based on the convolutional neural network, determining the convolution kernel according to the preset vector dimension and hyperparameters; and

[0023] The matrix is ​​traversed using a convolution kernel and a preset step size to obtain a text image feature vector.

[0024] Furthermore, the dimension calculation formula of the text image feature vector is as follows:

[0025]

[0026] Among them, j represents the number of convolution kernels;

[0027] i represents the i-th, i∈(1,2,...,j);

[0028] k i Represents the width of the i-th convolution kernel;

[0029] m represents the number of rows of text feature vectors; and

[0030] n represents the number of rows of the image feature vector.

[0031] Furthermore, the multi-category classification algorithm (softmax) is used to obtain the probability of each category according to the category vector, including:

[0032] Based on a multi-category classification algorithm, according to a preset number of categories, the category vectors are converted into category probabilities whose sum is 1, and each of the category probabilities is output.

[0033] In a second aspect, an embodiment of the present application provides a content classification device for multimodal data, including:

[0034] An acquisition module, used to acquire multimodal data;

[0035] A splitting module is used to split the multimodal data to obtain text data, image data and the order of image and text;

[0036] An obtaining module, used for obtaining a text feature vector of the text data and an image feature vector of the picture data by using an embedding technology;

[0037] an arrangement module, used to concatenate the text feature vector and the image feature vector according to the arrangement order of the image and text to obtain a matrix; and

[0038] A determination module is used to determine the category of the multimodal data based on a convolutional neural network (CNN) according to the matrix, a preset number of categories and a preset category.

[0039] Furthermore, the arrangement module is configured to:

[0040] According to the text feature vector, determine the number of text feature vector rows m;

[0041] Determine the number of rows n of the image feature vector according to the image feature vector; and

[0042] According to the arrangement order of the image and text, the number of rows m, the number of rows n and the preset vector dimension d, the text feature vectors and the image feature vectors are alternately spliced ​​to generate a matrix M with (m+n) rows and d columns.

[0043] Further, the determination module is configured to:

[0044] Based on a convolutional neural network, a convolution dimension reduction process is performed on the matrix to obtain a text image feature vector;

[0045] Based on a feedforward neural network (FNN), a fully connected layer parameter matrix is ​​set according to the text image feature vector and a preset number of categories to map the feature vector into a category vector;

[0046] Based on a multi-class classification algorithm (softmax), the probability of each class is obtained according to the class vector; and

[0047] The category of the multimodal data is determined according to the probability of each category.

[0048] In a third aspect, an embodiment of the present application provides a storage medium for storing computer-executable instructions, which, when executed, implement the steps of the method for content classification of multimodal data described in any one of the first aspects.

[0049] Compared with the prior art, this application can at least achieve the following technical effects:

[0050] This application can process multimodal data, ensure that the dimensions of all vectors are the same through Embedding technology, and ensure the consistency of data processing; and splice the processed vector data so that the convolutional neural network can better understand the logical relationship between images and texts; and then use the convolutional neural network to determine the category of multimodal data, which can comprehensively capture the key features of multimodal data, thereby improving the accuracy and robustness of classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0052] Figure 1 A flow chart of a method for content classification of multimodal data provided for one or more embodiments of this specification;

[0053] Figure 2 A schematic diagram of matrix splicing provided for one or more embodiments of this specification;

[0054] Figure 3 A schematic diagram of a neural network structure provided for one or more embodiments of this specification;

[0055] Figure 4 A schematic diagram of processing a splicing matrix using a convolution kernel provided in one or more embodiments of this specification;

[0056] Figure 5 A schematic diagram of the structure of a content classification device for multimodal data provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0058] With the rapid development of information technology, multimedia content on the Internet is becoming increasingly rich, and the combination of text and images has become the mainstream of information dissemination. Multimodal content of text and images not only contains rich visual information (such as pictures and videos), but also integrates detailed text descriptions, providing users with a comprehensive and intuitive information experience. However, how to effectively classify such multimodal content for better management and utilization has become an urgent problem to be solved. Traditional classification methods are mostly based on single-modal data, such as only classifying text or images, ignoring the intrinsic connection and complementarity between text and images.

[0059] In view of the above technical problems, this application proposes a content classification method for multimodal data, such as Figure 1 As shown, the specific steps are as follows:

[0060] Step S1, obtaining multimodal data.

[0061] In the embodiment of the present application, multimodal data that the user needs to classify is obtained. Multimodal data refers to a data set containing multiple different types of information, such as pictures and texts, which come from different sensory channels and describe the same object or event together, and can provide more comprehensive and rich information input for the artificial intelligence system.

[0062] Step S2, split the multimodal data to obtain text data, image data and the order of text and image.

[0063] In an embodiment of the present application, the multimodal data of images and texts are split, images and texts are obtained from the data respectively, and the document is recognized, and the arrangement order of each paragraph of text and image is recorded.

[0064] For example: Use Python code to split text and image multimodal data. According to the original format of text and image multimodal data, parse and store them separately. When the original format of text and image multimodal data is a word document, use Python code to parse the word document, obtain the content type paragraph by paragraph, store the text and pictures separately, and record the corresponding text and image arrangement order; when text and image multimodal data is stored in a formatted database such as MySQL or Oracle, there is no order problem between text and image content, and the pictures can be placed uniformly after the text; or the structured database has already formatted and stored the order, and only needs to be read and used.

[0065] Step S3, using embedding technology to obtain the text feature vector of the text data and the image feature vector of the picture data.

[0066] In an embodiment of the present application, a cross-modal pre-training model is used, and the Embedding technology therein is called to process the text data and the image data to obtain a text feature vector and an image feature vector, respectively. For example: a cross-modal pre-training model CLIP (Contrastive Language-Image Pre-training, a text-image multimodal pre-training model developed by OpenAI) in which both text data and image data are available, or a unimodal pre-training model of a processing model BERT (Bidirectional Encoder Representations from Transformers) for text data and a processing model ViT (Vision Transformer) for image data.

[0067] Specifically, when the CLIP model is used to process text data and image data, the CLIP model mainly consists of two parts: an image encoder and a text encoder.

[0068] Image encoder: Responsible for converting image data into image feature vectors. Image encoders can be convolutional neural networks (such as ResNet) or Transformer models (such as ViT). These encoders can capture key visual features in images and convert them into high-dimensional vector representations.

[0069] Text encoder: responsible for converting text data into text feature vectors. The text encoder is usually a Transformer model that can understand the semantic information of the text and convert it into a vector representation of the same dimension as the image feature vector.

[0070] The working principle of the CLIP model is based on contrastive learning. In the pre-training stage, CLIP receives a batch of image-text pairs as input and calculates the feature vectors of the images and texts. The goal of the model is to bring the matching image and text vectors closer in a common semantic space and push the mismatching vectors away. In other words, the CLIP model uses a contrastive loss function to optimize the model parameters so that the similarity between the feature vectors of matching image-text pairs is maximized, while the similarity between mismatching image-text pairs is minimized.

[0071] The training process of the CLIP model is as follows: (1) Obtaining the dataset.

[0072] Step 1.1: Collect a large-scale image-text pair dataset. This part of the training process requires a dataset containing a large number of images and their corresponding text descriptions, such as OpenAI's WIT (WebImage Text) dataset, which contains more than 400 million image-text pairs.

[0073] Step 1.2: Data cleaning and preprocessing. Clean the collected data set to remove noise and erroneous data, and perform necessary preprocessing on images and texts, such as image cropping, scaling, normalization, etc., as well as text segmentation and stop word removal.

[0074] (2) Model structure design.

[0075] Step 2.1: Design an image encoder. The image encoder can be a convolutional neural network (such as ResNet) or a Transformer model (such as ViT) to convert the input image into a high-dimensional feature vector.

[0076] Step 2.2: Design a text encoder. The text encoder is a Transformer model that converts the input text into a feature vector of the same dimension as the image vector feature.

[0077] (3) Contrastive learning training.

[0078] Step 3.1: Initialize model parameters. Assign random initial weights to the image encoder and text encoder.

[0079] Step 3.2: Construct training batches. Randomly sample a batch of image-text pairs from the dataset as training input.

[0080] Step 3.3: Feature extraction. Use the image encoder to convert the images in the training batch into image feature vectors, and use the text encoder to convert the text in the training batch into text feature vectors.

[0081] Step 3.4: Define positive and negative samples. In the same batch, paired image-text pairs are considered positive samples, and unpaired image-text pairs are considered negative samples.

[0082] Step 3.5: Calculate similarity. Use cosine similarity or inner product to calculate the similarity between each image feature vector and the text feature vector, and construct a similarity matrix.

[0083] Step 3.6: Calculate the contrast loss. According to the similarity matrix, a contrast loss function (such as InfoNCE loss) is used to calculate the loss value, which reflects the difference between the similarity of positive samples and the similarity of negative samples.

[0084] Step 3.7: Parameter update. Use optimization methods such as back-propagation and gradient descent to update the weights of the image encoder and text encoder to minimize the contrast loss.

[0085] (4) Iterative training.

[0086] Step 4.1: Repeat steps 3.2 to 3.7 until the performance of the model on the validation set reaches the preset standard or the number of training rounds reaches the preset upper limit.

[0087] Step 4.2: During the training process, strategies such as early stopping and learning rate adjustment can be used to prevent overfitting and improve the generalization ability of the model.

[0088] (5) Model evaluation and tuning.

[0089] Step 5.1: Evaluate the performance of the model on the test set, including accuracy, recall and other indicators for tasks such as image classification, image retrieval, and text generation.

[0090] Step 5.2: Tune the model based on the evaluation results, including adjusting the model structure, optimizer settings, regularization methods, etc.

[0091] After completing the training of the cross-modal pre-trained model, the CLIP model is output and applied to different scenarios to output vectors of fixed dimensions. For example, both image data and text data output vectors of 512 dimensions.

[0092] By using a cross-modal pre-training model (such as CLIP) or a unimodal pre-training model that uses text (such as BERT) and images (such as ViT), the text and image content is converted into a vector representation of a unified dimension, and the semantic association between the text and image is maintained. Different from the traditional unimodal classification method, the present invention improves the accuracy and generalization ability of classification by integrating the information of the two modalities of text and image.

[0093] Step S4, according to the arrangement order of the image and text, concatenate the text feature vector and the image feature vector to obtain a matrix.

[0094] In an embodiment of the present application, the order in which text and images are arranged is used to determine the total number of text and image pairs, and to initialize an empty matrix; the number of rows in this matrix corresponds to the amount of text data and image data, and the number of columns corresponds to the vector dimension output by the pre-trained model; and then, according to the order in which text and images are arranged, the text feature vectors and image feature vectors are filled into the matrix in turn.

[0095] Specifically, according to the order of the recorded text and images, the corresponding text feature vectors and image feature vectors are concatenated in sequence. Figure 2 As shown, the multimodal data includes m segments of text data and n images. The pre-trained model is used to process the m segments of text data and n images, and d-dimensional vector text features and image features are output. The vectors are then concatenated to generate a matrix M with m+n rows and d columns.

[0096] By recording the order of each text and picture, the contextual information of the text and picture content can be retained in the subsequent processing, thereby improving the accuracy of multimodal data classification.

[0097] Step S5, based on a convolutional neural network (CNN), determining the category of the multimodal data according to the matrix, the preset number of categories and the preset category.

[0098] In an embodiment of the present application, a convolutional neural network is based on a matrix as input; multiple convolutional layers are used to extract features, each convolutional layer contains multiple convolution kernels for capturing different features; after convolution, a pooling layer is used to reduce the dimension of the features while retaining important information; after feature extraction, a fully connected layer is used to map the features to the category space; according to the preset number of categories, the number of neurons in the output layer is set, and the output is converted into a probability distribution using a softmax function; the preset category with the highest probability is output as the category of the multimodal data.

[0099] Specifically, drawing on the idea of ​​convolutional neural networks, the trained convolutional neural network will set j dimensions to k. i *d, convolution kernel with a step size of 1. Wherein, j is the number of convolution kernels, which can be selected from 1 to multiple, and can be set and adjusted according to the complexity and richness of the image and text content to be classified. k iis the width of the i-th convolution kernel, i∈(1,2,...,j), which can be adjusted according to the size of the upper and lower associated windows of the image and text content. d is the dimension of the image and text vector, which is related to the selected vectorization model and cannot be modified. Then use j convolution kernels to perform convolution processing on the (m+n)*d-order matrix M to extract the feature parameter values ​​and perform data dimensionality reduction; then multiply the feature parameter values ​​in the convolution kernel with the values ​​of the corresponding positions on the matrix M, and then sum them, and the result is used as the value of the corresponding position in the output vector. According to the set step size, the convolution kernel slides down and repeats the above weighted summation operation until all areas of the matrix M are traversed. After processing, a k-dimensional vector will be obtained, and the dimension calculation formula is as follows:

[0100]

[0101] Among them, j represents the number of convolution kernels;

[0102] i represents the i-th, i∈(1,2,...,j);

[0103] k i Represents the width of the i-th convolution kernel;

[0104] m represents the number of rows of text feature vector;

[0105] n represents the number of rows of the image feature vector.

[0106] Reset The parameter matrix with columns and rows c, where c is the preset number of categories for classification, is the fully connected layer in the neural network. It is multiplied by the text image feature vector to obtain the category vector. The softmax method is then used to convert the category vector into the probability of c categories with a sum of 1, such as Figure 3 As shown; then output the classification result according to the maximum probability.

[0107] Specifically, the convolutional neural network selects a certain amount of multimodal content of pictures and texts, labels their categories, and obtains a training data set. The multimodal neural network is trained by combining the training data set with the gradient descent method used in neural network training. After the parameters converge, the model is saved and used for the prediction of the category of multimodal content of pictures and texts. The specific training process is as follows:

[0108] (1) Data preprocessing

[0109] Data collection and division. Collect a large amount of annotated multimodal data of images and texts, and divide them into training set, validation set and test set. The training set is used for model training, the validation set is used to adjust model parameters, and the test set is used to evaluate model performance. Clean the training data to remove noise, outliers and irrelevant information.

[0110] (2) Weight initialization:

[0111] The network weights are initialized with small random numbers to ensure that the network can start learning from a random state and avoid falling into a local optimum. Initialize the weights and biases for each convolutional layer and fully connected layer.

[0112] (2) Forward propagation:

[0113] After the input data is vectorized by the pre-trained model, it passes through the convolution layer. After the convolution operation, an activation function (such as ReLU, Sigmoid, Tanh, etc.) is applied to perform nonlinear transformation on the feature map to increase the expressive power of the model. The feature map is then reduced in dimension by the pooling layer to reduce the amount of calculation and the number of parameters. Common pooling methods include maximum pooling and average pooling. The present invention mainly adopts the maximum pooling method. Finally, the fully connected layer integrates and classifies the features extracted by the convolution layer and the pooling layer, and outputs the final prediction result.

[0114] (3) Loss calculation:

[0115] By comparing the difference between the network output and the true label, a loss function such as cross entropy loss or mean squared error loss is calculated to measure the difference between the model prediction result and the true label.

[0116] (4) Back propagation and optimization:

[0117] The back-propagation algorithm is used to calculate the gradient and pass the gradient from the output layer to the input layer. And the optimization algorithm (such as SGD, Adam, etc.) is used to update the network weights to minimize the loss function.

[0118] (5) Regularization and dropout:

[0119] L1, L2 regularization and dropout techniques are introduced during the training process to further improve the generalization ability of the model.

[0120] (6) Model evaluation and tuning:

[0121] Evaluate the performance of the model on the validation set and calculate indicators such as accuracy, precision, and recall. Tune the model based on the evaluation results, including adjusting the network structure, optimizer settings, and learning rate.

[0122] (7) Iterative training

[0123] Repeat the above steps until the model performance reaches the expected level or the preset number of training rounds is reached.

[0124] This application designs a specific CNN network structure by drawing on the CNN idea, including multiple convolutional layers, pooling layers and fully connected layers. The convolutional layer uses convolution kernels of different sizes to capture features of different scales. The size and number of convolution kernels can be flexibly adjusted according to the complexity of the graphic content to meet the classification needs in different scenarios. Through the convolution operation, this application can automatically learn the high-level feature representation in the graphic content, reduce the number of model parameters through local connection and parameter sharing, improve computational efficiency, and improve the robustness and generalization ability of the classification model. In the pooling layer, maximum pooling or average pooling is used to reduce feature dimensions and reduce overfitting. In the fully connected layer, the predicted probability of each category is obtained by multiplying the parameter matrix with the vector after convolution processing. The softmax function is used to normalize the probability to ensure the rationality and interpretability of the classification results. The design of the fully connected layer enables the model to output intuitive classification results, which is convenient for subsequent applications and decision-making.

[0125] In the embodiments of the present application, by comparing whether an event reported by a user on a certain platform on a certain day is a small event and comparing the experimental data of the present application using multimodal data to determine whether it is a small event, it is proved that the recall rate and accuracy of the multimodal classification method of the present application are improved, and the comprehensive graphic information is of great help in the classification of multimodal data.

[0126] When a user reports an event on a certain platform on a certain day and only uses text data for binary classification, the result data is shown in Table 1:

[0127] Table 1 Experiment on whether the events reported by users on a certain platform on a certain day are small events

[0128]

[0129] The recall rate of this experiment = 38 / (10+38) = 79%;

[0130] Accuracy = 38 / (16+38) = 70%;

[0131] Accuracy = (38+45) / (38+10+16+45) = 76%.

[0132] When the multimodal technology of this application is used for binary classification, the result data is shown in Table 2:

[0133] Table 2 Multimodal data to determine whether it is a small event

[0134]

[0135] The recall rate of this application experiment = 40 / (8+40) = 83%;

[0136] Accuracy = 40 / (13+40) = 75%;

[0137] Accuracy = (40 + 48) / (40 + 13 + 8 + 48) = 80.73%.

[0138] Furthermore, based on the text feature vector, the number of rows m of the text feature vector is determined; based on the image feature vector, the number of rows n of the image feature vector is determined; and based on the arrangement order of the image and text, the number of rows m, the number of rows n and the preset vector dimension d, the text feature vector and the image feature vector are alternately spliced ​​to generate a matrix M with (m+n) rows and d columns.

[0139] Specifically, according to the number of rows of text feature vectors, the number of rows of image feature vectors and the preset vector dimensions, an empty matrix is ​​created; this matrix will be used to store the alternately concatenated feature vectors. According to the order of text and image arrangement, vectors are extracted from the text feature vectors and image feature vectors in turn, and they are alternately placed in the matrix to generate a matrix M with (m+n) rows and d columns.

[0140] Furthermore, based on a convolutional neural network, the matrix is ​​subjected to convolution dimensionality reduction processing to obtain a text image feature vector; based on a feedforward neural network (FNN), a fully connected layer parameter matrix is ​​set according to the text image feature vector and a preset number of categories, and the feature vector is mapped to a category vector; based on a multi-category classification algorithm (softmax), the probability of each category is obtained according to the category vector; and according to the probability of each category, the category of the multimodal data is determined.

[0141] Specifically, first, convolution is used for dimensionality reduction, and the matrix containing text and image feature vectors is used as the input of CNN. The dimension of this matrix is ​​(m+n) rows and d columns, where m is the number of text feature vectors, n is the number of image feature vectors, and d is the dimension of the feature vector. One or more convolutional layers are applied to extract features. These convolutional layers learn how to capture local patterns in the matrix and generate feature maps. After the convolutional layer, a pooling layer is added to reduce the dimension of the feature map while retaining important information. A fully connected layer is then used to map the feature map to a fixed-size vector, which is the text image feature vector. The dimension of this vector is smaller than that of the input matrix, thus achieving the effect of dimensionality reduction. Secondly, the reduced-dimensional text image feature vector is used as input, and the feature vector is mapped to a category vector through the calculation of the fully connected layer; then the category vector is mapped to a probability distribution using softmax, and it is converted into multiple category probabilities with a sum of 1 according to the preset number of categories and preset categories. Finally, the category with the highest probability is selected as the predicted category of the multimodal data, and the predicted category is output as the final result.

[0142] Furthermore, based on the convolutional neural network, the convolution kernel is determined according to the preset vector dimension and hyperparameters; and the convolution kernel and the preset step size are used to traverse the matrix to obtain the text image feature vector.

[0143] For example, assume that the dimensions d of the image and text vectors are both 512. Set j = 2 convolution kernels, k 1 =1, k 2 =2, that is, the sizes of the two convolution kernels are 1*512 and 2*512 respectively. At this time, convolution kernel 1 only performs convolution processing on a single paragraph of text or a single picture at a time, and processes it downward in the order of arrangement; convolution kernel 2 processes two parts of content each time. Taking the splicing matrix M in the picture in the third step as an example, the contents processed by the convolution kernel are: text-text, text-image, image-text, text-image, and image-image. As shown in the figure, the convolution kernel performs convolution processing with the corresponding row vectors on the left (the corresponding position values ​​are multiplied and then summed). After each processing is completed, it moves down one step and repeats the operation until the entire matrix M is traversed to determine the feature vector of the text image. Convolution kernel 1 and convolution kernel 2 are processed in the same way, such as Figure 4 shown.

[0144] Furthermore, the dimension calculation formula of the text image feature vector is as follows:

[0145]

[0146] Among them, j represents the number of convolution kernels;

[0147] i represents the i-th, i∈(1,2,...,j);

[0148] k i Represents the width of the i-th convolution kernel;

[0149] m represents the number of rows of text feature vectors; and

[0150] n represents the number of rows of the image feature vector.

[0151] Furthermore, based on a multi-category classification algorithm, according to a preset number of categories, the category vectors are converted into category probabilities whose sum is 1, and each of the category probabilities is output.

[0152] The present application embodiment provides a content classification device for multimodal data, such as Figure 5 As shown, including:

[0153] An acquisition module 101 is used to acquire multimodal data;

[0154] A splitting module 102 is used to split the multimodal data to obtain text data, image data and the order of image and text;

[0155] An obtaining module 103 is used to obtain a text feature vector of the text data and an image feature vector of the picture data by using an embedding technology;

[0156] An arrangement module 104, configured to concatenate the text feature vector and the image feature vector according to the arrangement order of the image and text to obtain a matrix; and

[0157] The determination module 105 is used to determine the category of the multimodal data based on a convolutional neural network (CNN) according to the matrix, the preset number of categories and the preset category.

[0158] Furthermore, the arrangement module is configured to:

[0159] According to the text feature vector, determine the number of text feature vector rows m;

[0160] Determine the number of rows n of the image feature vector according to the image feature vector; and

[0161] According to the arrangement order of the image and text, the number of rows m, the number of rows n and the preset vector dimension d, the text feature vectors and the image feature vectors are alternately spliced ​​to generate a matrix M with (m+n) rows and d columns.

[0162] Furthermore, the module configuration is determined as follows:

[0163] Based on a convolutional neural network, a convolution dimension reduction process is performed on the matrix to obtain a text image feature vector;

[0164] Based on a feedforward neural network (FNN), a fully connected layer parameter matrix is ​​set according to the text image feature vector and a preset number of categories to map the feature vector into a category vector;

[0165] Based on a multi-class classification algorithm (softmax), the probability of each class is obtained according to the class vector; and

[0166] The category of the multimodal data is determined according to the probability of each category.

[0167] An embodiment of the present application provides a storage medium for storing computer-executable instructions, and when the computer-executable instructions are executed, the steps of the method for content classification of multimodal data described in any one of the above embodiments are implemented.

[0168] It should be noted that the embodiments of the storage medium in this specification and the embodiments of the blockchain-based service provision method in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the aforementioned corresponding implementation of the blockchain-based service provision method, and the repeated parts will not be repeated.

[0169] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0170] In the 1930s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0171] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0172] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0173] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0174] It should be understood by those skilled in the art that one or more embodiments of this specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0176] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0178] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0179] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0180] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0181] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0182] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0183] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0184] The above description is only an embodiment of this document and is not intended to limit this document. For those skilled in the art, this document may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this document should be included in the scope of the claims of this document.

Claims

1. A content classification method for multimodal data, characterized in that include: Acquire multimodal data; Split the multimodal data to obtain text data, image data, and the order of text and image data; Using embedding technology to obtain a text feature vector of the text data and an image feature vector of the picture data; According to the arrangement order of the image and text, concatenate the text feature vector and the image feature vector to obtain a matrix; and Based on a convolutional neural network (CNN), the category of the multimodal data is determined according to the matrix, the preset number of categories and the preset categories.

2. The method according to claim 1, characterized in that The text feature vector and the image feature vector are concatenated according to the arrangement order of the image and text to obtain a matrix including: According to the text feature vector, determine the number of text feature vector rows m; Determine the number of rows n of the image feature vector according to the image feature vector; and According to the arrangement order of the image and text, the number of rows m, the number of rows n and the preset vector dimension d, the text feature vectors and the image feature vectors are alternately spliced ​​to generate a matrix M with (m+n) rows and d columns.

3. The method according to claim 1, characterized in that The determining the category of the multimodal data based on a convolutional neural network (CNN) according to the matrix, the preset number of categories and the preset category includes: Based on a convolutional neural network, a convolution dimension reduction process is performed on the matrix to obtain a text image feature vector; Based on a feedforward neural network (FNN), a fully connected layer parameter matrix is ​​set according to the text image feature vector and a preset number of categories to map the feature vector into a category vector; Based on a multi-class classification algorithm (softmax), the probability of each class is obtained according to the class vector; and The category of the multimodal data is determined according to the probability of each category.

4. The method according to claim 3, characterized in that The step of performing convolution dimensionality reduction processing on the matrix based on a convolutional neural network (CNN) to obtain a text image feature vector comprises: Based on the convolutional neural network, determining the convolution kernel according to the preset vector dimension and hyperparameters; and The matrix is ​​traversed using a convolution kernel and a preset step size to obtain a text image feature vector.

5. The method according to claim 4, characterized in that The dimension calculation formula of the text image feature vector is as follows: Among them, j represents the number of convolution kernels; i represents the i-th, i∈(1,2,...,j); k i Represents the width of the i-th convolution kernel; m represents the number of rows of text feature vectors; and n represents the number of rows of the image feature vector.

6. The method according to claim 3, characterized in that: The multi-category classification algorithm (softmax) is used to obtain the probability of each category according to the category vector, including: Based on a multi-category classification algorithm, according to a preset number of categories, the category vectors are converted into category probabilities whose sum is 1, and each of the category probabilities is output.

7. A content classification device for multimodal data, characterized in that include: An acquisition module, used to acquire multimodal data; A splitting module is used to split the multimodal data to obtain text data, image data and the order of image and text; An obtaining module, used for obtaining a text feature vector of the text data and an image feature vector of the picture data by using an embedding technology; An arrangement module, used for concatenating the text feature vector and the image feature vector according to the arrangement order of the image and text to obtain a matrix; as well as A determination module is used to determine the category of the multimodal data based on a convolutional neural network (CNN) according to the matrix, a preset number of categories and a preset category.

8. The device according to claim 7, characterized in that The arrangement module is configured as follows: According to the text feature vector, determine the number of text feature vector rows m; Determine the number of rows n of the image feature vector according to the image feature vector; and According to the arrangement order of the image and text, the number of rows m, the number of rows n and the preset vector dimension d, the text feature vectors and the image feature vectors are alternately spliced ​​to generate a matrix M with (m+n) rows and d columns.

9. The device according to claim 7, characterized in that The determination module is configured to: Based on a convolutional neural network, a convolution dimension reduction process is performed on the matrix to obtain a text image feature vector; Based on a feedforward neural network (FNN), a fully connected layer parameter matrix is ​​set according to the text image feature vector and a preset number of categories to map the feature vector into a category vector; Based on a multi-class classification algorithm (softmax), the probability of each class is obtained according to the class vector; The category of the multimodal data is determined according to the probability of each category.

10. A storage medium for storing computer executable instructions, characterized in that: When the computer executable instructions are executed, the steps of the method for content classification of multimodal data described in any one of claims 1 to 6 are implemented.

Images