A content classification method and device of multi-modal data and a storage medium

Abstract

The embodiment of the specification provides a content classification method and device of multi-modal data and a storage medium, wherein the method comprises the following steps: acquiring multi-modal data; splitting the multi-modal data to obtain text data, picture data and a text-picture arrangement order; obtaining a text feature vector of the text data and an image feature vector of the picture data by using an embedding technology; splicing the text feature vector and the image feature vector according to the text-picture arrangement order to obtain a matrix; and determining a category of the multi-modal data based on a convolutional neural network (CNN), the matrix, a preset category number and a preset category. The application can more comprehensively capture key features of multi-modal data, thereby improving the accuracy and robustness of classification.

Description

TECHNICAL FIELD

[0001] The present document relates to the technical field of data classification, and particularly relates to a content classification method and device for multi-modal data and a storage medium. BACKGROUND

[0002] The image-text multi-modal feature recognition combines information of two different modalities of images and texts, and can more accurately classify target objects by not supplementing and verifying each other.

[0003] The prior art often only focuses on information of a single modality of text or pictures, and classifies according to a single modality vector feature.

[0004] However, in the prior art, the features extracted from pictures and texts are different, the multi-modal features are difficult to combine, and the arrangement of pictures and texts in space is complex and the number is different, which increases the difficulty of feature fusion, and it is difficult to quickly and accurately classify articles. SUMMARY

[0005] In view of the above scheme, the present application aims to provide a content classification method and device for multi-modal data and a storage medium to solve at least one of the above technical problems.

[0006] In a first aspect, one or more embodiments of the present specification provide a content classification method for multi-modal data, comprising:

[0007] obtaining multi-modal data;

[0008] splitting the multi-modal data to obtain text data, picture data, and an image-text arrangement order;

[0009] obtaining a text feature vector of the text data and an image feature vector of the picture data by using embedding technology;

[0010] splicing the text feature vector and the image feature vector according to the image-text arrangement order to obtain a matrix; and

[0011] determining a category of the multi-modal data based on a convolutional neural network (CNN), the matrix, a preset category number, and a preset category.

[0012] Further, the splicing the text feature vector and the image feature vector according to the image-text arrangement order to obtain a matrix comprises:

[0013] determining a text feature vector row number m according to the text feature vector;

[0014] determining an image feature vector row number n according to the image feature vector; and

[0015] According to the text-image arrangement order, the number of rows m, the number of rows n, and the preset vector dimension d, the text feature vector and the image feature vector are alternately spliced to generate a matrix M of (m+n) rows and d columns.

[0016] Further, the convolutional neural network (CNN) is used to determine the category of the multi-modal data according to the matrix, the preset number of categories, and the preset categories.

[0017] The convolutional neural network is used to perform convolutional dimension reduction processing on the matrix to obtain a text-image feature vector.

[0018] The feedforward neural network (FNN) is used to set a fully connected layer parameter matrix according to the text-image feature vector and the preset number of categories, and map the feature vector to a category vector.

[0019] The multi-category classification algorithm (softmax) is used to obtain the probability of each category according to the category vector.

[0020] The category of the multi-modal data is determined according to the probability of each category.

[0021] Further, the convolutional neural network (CNN) is used to perform convolutional dimension reduction processing on the matrix to obtain a text-image feature vector.

[0022] The convolutional neural network is used to determine a convolution kernel according to a preset vector dimension and a hyperparameter.

[0023] The convolution kernel and a preset step size are used to traverse the matrix to obtain a text-image feature vector.

[0024] Further, the dimension of the text-image feature vector is calculated according to the following formula:

[0025]

[0026] where j represents the number of convolution kernels.

[0027] i represents the i-th, i∈(1, 2,..., j);

[0028] k i represents the width of the i-th convolution kernel.

[0029] m represents the number of rows of the text feature vector; and

[0030] n represents the number of rows of the image feature vector.

[0031] Further, the multi-category classification algorithm (softmax) is used to obtain the probability of each category according to the category vector.

[0032] Based on a multi-class classification algorithm, the category vectors are converted into category probabilities with a sum of 1 according to a preset number of categories, and each category probability is output.

[0033] In a second aspect, the embodiments of the present application provide a content classification device for multi-modal data, comprising:

[0034] An acquisition module is configured to acquire multi-modal data.

[0035] A splitting module is configured to split the multi-modal data to obtain text data, picture data, and a text-picture arrangement order.

[0036] An obtaining module is configured to obtain a text feature vector of the text data and an image feature vector of the picture data by using an embedding technique.

[0037] An arrangement module is configured to splice the text feature vector and the image feature vector according to the text-picture arrangement order to obtain a matrix.

[0038] A determination module is configured to determine a category of the multi-modal data based on a convolutional neural network (CNN), the matrix, a preset number of categories, and a preset category.

[0039] Further, the arrangement module is configured to:

[0040] determine a number of rows m of the text feature vector according to the text feature vector;

[0041] determine a number of rows n of the image feature vector according to the image feature vector; and

[0042] splice the text feature vector and the image feature vector alternately according to the text-picture arrangement order, the number of rows m, the number of rows n, and a preset vector dimension d to generate a matrix M with (m+n) rows and d columns.

[0043] Further, the determination module is configured to:

[0044] perform convolution dimension reduction processing on the matrix based on a convolutional neural network to obtain a text-image feature vector;

[0045] set a fully connected layer parameter matrix based on a feedforward neural network (FNN) according to the text-image feature vector and a preset number of categories, map the feature vector to a category vector, and

[0046] obtain a probability of each category based on a multi-class classification algorithm (softmax) according to the category vector; and

[0047] According to the probabilities of each category, the category of the multi-modal data is determined.

[0048] In a third aspect, the embodiments of the present application provide a storage medium for storing computer executable instructions, which, when executed, implement the steps of the multi-modal data content classification method in any one of the first aspect.

[0049] Compared with the prior art, the present application can at least achieve the following technical effects:

[0050] The present application can process multi-modal data, ensure the consistency of data processing by ensuring that the dimensions of all vectors are the same through Embedding technology, and splice the processed vector data so that the convolutional neural network can better understand the logical relationship between the text and the picture; and then determine the category of the multi-modal data by using the convolutional neural network, which can comprehensively capture the key features of the multi-modal data, thereby improving the accuracy and robustness of classification. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the one or more embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present specification, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0052] Figure 1 A multi-modal data content classification method flowchart is provided for one or more embodiments of the present specification.

[0053] Figure 2 A matrix splicing schematic diagram is provided for one or more embodiments of the present specification.

[0054] Figure 3 A neural network structure schematic diagram is provided for one or more embodiments of the present specification.

[0055] Figure 4 A schematic diagram of processing spliced matrix using convolution kernel is provided for one or more embodiments of the present specification.

[0056] Figure 5 A multi-modal data content classification device structure schematic diagram is provided for one or more embodiments of the present specification. DETAILED DESCRIPTION

[0057] In order for those skilled in the art to better understand the technical solutions in the one or more embodiments of the present specification, the technical solutions in the one or more embodiments of the present specification will be clearly and completely described in the following with reference to the drawings in the one or more embodiments of the present specification. Obviously, the described embodiments are only part of the embodiments of the present specification, rather than all the embodiments. Based on the one or more embodiments of the present specification, all other embodiments obtained by those of ordinary skill in the art without creative labor should belong to the protection scope of the present document.

[0058] With the rapid development of information technology, multimedia content on the Internet is increasingly rich, and the form of combination of text and pictures has become the mainstream of information dissemination. The multimodal content of text and pictures not only contains rich visual information (such as pictures, videos), but also integrates detailed text description, providing users with comprehensive and intuitive information experience. However, how to effectively classify such multi-modal content in order to better manage and utilize it has become a problem to be solved. Traditional classification methods are mostly based on single-modal data, such as classifying only text or image, ignoring the internal relationship and complementarity between text and pictures.

[0059] In view of the above technical problems, the present application provides a content classification method for multi-modal data, as shown in Figure 1 The specific steps are as follows:

[0060] Step S1, obtaining multi-modal data.

[0061] In the embodiments of the present application, multi-modal data required by the user to be classified is obtained. The multi-modal data refers to a data set containing multiple types of information, such as pictures and text. These data come from different sensory channels and describe the same object or event, which can provide more comprehensive and rich information input for artificial intelligence systems.

[0062] Step S2, splitting the multi-modal data to obtain text data, picture data and picture-text arrangement order.

[0063] In the embodiments of the present application, the multi-modal data of text and pictures is split to obtain pictures and text from the data, and the document is identified to record the arrangement order of each piece of text and picture.

[0064] For example, use python code to split the image-text multi-modal data. According to the original format of the image-text multi-modal data, parse and store respectively. When the original format of the image-text multi-modal data is a word document, use python code to parse the word document, get the content type by section, store the text and picture respectively, and record the corresponding image-text arrangement order. When the image-text multi-modal data is stored in a formatted database such as MySQL or Oracle, there is no order problem between the text and picture content at this time, and the pictures can be placed uniformly after the text. Or the structured database has formatted the order for storage, which can be read and used directly.

[0065] Step S3, using embedding technology to obtain the text feature vector of the text data and the image feature vector of the picture data.

[0066] In the embodiments of the present application, the cross-modal pre-training model is used to call the Embedding technology to process the text data and the picture data, and obtain the text feature vector and the image feature vector respectively. For example, the cross-modal pre-training model CLIP (Contrastive Language-Image Pre-training, a text-image multi-modal pre-training model developed by OpenAI) or the single-modal pre-training model of the processing model BERT (Bidirectional Encoder Representations from Transformers, a bidirectional encoder representation from Transformers) of the text data and the processing model ViT (Vision Transformer, vision Transformer) of the picture data.

[0067] Specifically, if the CLIP model is used to process the text data and the picture data, the CLIP model mainly consists of two parts: image encoder and text encoder.

[0068] Image encoder: responsible for converting picture data into image feature vector. The image encoder can be a convolutional neural network (such as ResNet) or a Transformer model (such as ViT). These encoders can capture key visual features in images and convert them into high-dimensional vector representations.

[0069] Text encoder: responsible for converting text data into text feature vector. The text encoder is usually a Transformer model, which can understand the semantic information of the text and convert it into a vector representation of the same dimension as the image feature vector.

[0070] The working principle of the CLIP model is based on contrastive learning. In the pre-training phase, CLIP receives a batch of image-text pairs as input and calculates the feature vectors of the images and texts. The goal of the model is to bring the matching image and text vectors closer in the common semantic space, while pushing the mismatched vectors further apart. That is, the CLIP model uses a contrastive loss function to optimize the model parameters, maximizing the similarity between the feature vectors of matching image-text pairs, and minimizing the similarity between mismatched image-text pairs.

[0071] The training process of the CLIP model is as follows: (1) Obtain the dataset.

[0072] Step 1.1: Collect a large-scale image-text pair dataset. The training process of this part requires a dataset containing a large number of images and their corresponding text descriptions, such as the WIT (WebImage Text) dataset of OpenAI, which contains more than 400 million image-text pairs.

[0073] Step 1.2: Data cleaning and preprocessing. Clean the collected dataset, remove noise and error data, and perform necessary preprocessing on images and texts, such as image cropping, scaling, normalization, etc., and text segmentation, stop word removal, etc.

[0074] (2) Model structure design.

[0075] Step 2.1: Design the image encoder. The image encoder can be a convolutional neural network (such as ResNet) or a Transformer model (such as ViT), which is used to convert the input image into a high-dimensional feature vector.

[0076] Step 2.2: Design the text encoder. The text encoder is a Transformer model that converts the input text into a feature vector of the same dimension as the image vector feature.

[0077] (3) Contrastive learning training.

[0078] Step 3.1: Initialize model parameters. Assign random initial weights to the image encoder and text encoder.

[0079] Step 3.2: Build training batches. Randomly select a batch of image-text pairs from the dataset as training input.

[0080] Step 3.3: Feature extraction. Use the image encoder to convert the images in the training batch to image feature vectors, and use the text encoder to convert the texts in the training batch to text feature vectors.

[0081] Step 3.4: Define positive and negative samples. In the same batch, paired image-text pairs are considered as positive samples, and unpaired image-text pairs are considered as negative samples.

[0082] Step 3.5: Calculate similarity. Calculate the similarity between each image feature vector and text feature vector using metrics such as cosine similarity or inner product, and construct a similarity matrix.

[0083] Step 3.6: Calculate contrastive loss. According to the similarity matrix, use the contrastive loss function (such as InfoNCE loss) to calculate the loss value, which reflects the difference between the similarity of positive samples and the similarity of negative samples.

[0084] Step 3.7: Parameter update. Use optimization methods such as backpropagation algorithm and gradient descent to update the weights of image encoder and text encoder to minimize the contrastive loss.

[0085] (4) Iterative training.

[0086] Step 4.1: Repeat steps 3.2 to 3.7 until the performance of the model on the validation set reaches the preset standard or the training round reaches the preset upper limit.

[0087] Step 4.2: During training, strategies such as early stopping and learning rate adjustment can be used to prevent overfitting and improve the generalization ability of the model.

[0088] (5) Model evaluation and optimization.

[0089] Step 5.1: Evaluate the performance of the model on the test set, including accuracy, recall rate and other indicators of image classification, image retrieval, text generation and other tasks.

[0090] Step 5.2: Optimize the model according to the evaluation results, including adjusting the model structure, optimizer settings, regularization methods, etc.

[0091] After completing the training of the cross-modal pre-training model, the CLIP model is output and applied to different scenarios to output fixed-dimensional vectors. For example: both picture data and text data output vectors with a dimension of 512.

[0092] By using cross-modal pre-training models (such as CLIP) or single-modal pre-training models for text (such as BERT) and pictures (such as ViT), the image-text content is converted into a unified dimensional vector representation, and the semantic association between images and texts is maintained. Unlike traditional single-modal classification methods, the present invention improves the accuracy and generalization ability of classification by fusing information from both image and text modalities.

[0093] Step S4, according to the image-text arrangement order, splicing the text feature vector and the image feature vector to obtain a matrix.

[0094] In the embodiment of the present application, the total number of text and image pairs is determined by the arrangement order of the text and image, and an empty matrix is initialized. The number of rows of the matrix corresponds to the number of text data and image data, and the number of columns corresponds to the dimension of the vector output by the pre-trained model. Then, according to the arrangement order of the text and image, the text feature vector and the image feature vector are sequentially filled into the matrix.

[0095] Specifically, according to the recorded arrangement order of the text and image, the corresponding text feature vector and image feature vector are sequentially spliced. For example, as shown in Figure 2 The multi-modal data includes m pieces of text data and n pieces of image data. The pre-trained model is used to process the m pieces of text data and the n pieces of image data, and outputs d-dimensional text features and image features. Then, the vectors are spliced to generate a matrix M with m+n rows and d columns.

[0096] By recording the arrangement order of each piece of text and image, the context information of the text and image content can be preserved in the subsequent processing process, thereby improving the accuracy of multi-modal data classification.

[0097] In step S5, the class of the multi-modal data is determined based on a convolutional neural network (CNN) according to the matrix, a preset number of classes, and a preset class.

[0098] In the embodiment of the present application, the convolutional neural network is used as the input of the matrix. Multiple convolutional layers are used to extract features, each convolutional layer including multiple convolutional kernels for capturing different features. After convolution, a pooling layer is used to reduce the dimension of the features while preserving important information. After feature extraction, a fully connected layer is used to map the features to a class space. According to the preset number of classes, the number of neurons in the output layer is set, and the softmax function is used to convert the output to a probability distribution. The preset class with the highest probability is the class of the multi-modal data.

[0099] Specifically, by referring to the idea of convolutional neural network, the trained convolutional neural network will set j convolutional kernels with size k i *d and a step size of 1. Wherein, j is the number of convolutional kernels, which can be selected from 1 to multiple, and can be adjusted according to the complexity and richness of the text and image content to be classified. k i`i` represents the width of the `i`-th convolutional kernel, where `i` ∈ (1, 2, ..., j), and can be adjusted according to the size of the window related to the text and image content. `d` represents the dimension of the text and image vector, which is related to the selected vectorization model and cannot be modified. Then, `j` convolutional kernels are used to convolve the (m+n)*d matrix `M` to extract feature parameter values ​​and perform dimensionality reduction. The feature parameter values ​​in the convolutional kernels are then multiplied by the corresponding values ​​in matrix `M`, and the sum is used as the value at the corresponding position in the output vector. Based on the set stride, the convolutional kernels are slid downwards, and the above weighted summation operation is repeated until all regions of matrix `M` have been traversed. After processing, a k-dimensional vector is obtained. The dimension calculation formula is as follows:

[0100]

[0101] Where j represents the number of convolution kernels;

[0102] i represents the i-th element, i∈(1,2,...,j);

[0103] k i This represents the width of the i-th convolutional kernel;

[0104] m represents the number of rows in the text feature vector;

[0105] n represents the number of rows in the image feature vector.

[0106] Reset The parameter matrix, with c columns and c rows, where c is the preset number of categories (i.e., the fully connected layer in the neural network), is multiplied by the text image feature vector to obtain the category vector. Then, the softmax method is used to transform this category vector into c probabilities of each category, summing to 1, as shown below. Figure 3 As shown; then output the classification result based on the highest probability.

[0107] Specifically, the convolutional neural network selects a certain amount of multimodal text and image content, labels its category, and obtains a training dataset. The multimodal neural network is then trained using the gradient descent method used for neural network training, combined with the training dataset. After the parameters converge, the model is saved and used for multimodal text and image content category prediction. The specific training process is as follows:

[0108] (1) Data preprocessing

[0109] Data Collection and Partitioning. A large amount of labeled text and image multimodal data was collected and divided into training, validation, and test sets. The training set was used for model training, the validation set for adjusting model parameters, and the test set for evaluating model performance. The training data was cleaned to remove noise, outliers, and irrelevant information.

[0110] (2) Weight initialization:

[0111] The weights of the network are initialized by small random numbers to ensure that the network can learn from a random state and avoid getting stuck in local optima. The weights and biases are initialized for each convolutional layer and fully connected layer.

[0112] (2) Forward propagation:

[0113] After the input data is vectorized by the pre-trained model, it passes through the convolutional layer. After the convolution operation, an activation function (such as ReLU, Sigmoid, Tanh, etc.) is applied to the feature map for non-linear transformation, increasing the model's expression ability. The pooling layer then reduces the dimension of the feature map, reducing the amount of calculation and the number of parameters. Common pooling methods include max pooling and average pooling, and the invention mainly uses the max pooling method. Finally, the fully connected layer integrates and classifies the features extracted by the convolutional layer and the pooling layer, outputting the final prediction result.

[0114] (3) Loss calculation:

[0115] By comparing the difference between the network output and the true label, the loss function such as cross-entropy loss or mean square error loss is calculated to measure the difference between the model's prediction result and the true label.

[0116] (4) Backpropagation and optimization:

[0117] The backpropagation algorithm is used to calculate the gradient, which is passed from the output layer to the input layer. The optimization algorithm (such as SGD, Adam, etc.) is used to update the network weights to minimize the loss function.

[0118] (5) Regularization and dropout:

[0119] L1, L2 regularization and dropout techniques are introduced during training to further improve the model's generalization ability.

[0120] (6) Model evaluation and tuning:

[0121] Evaluate the model's performance on the validation set, calculate accuracy, precision, recall, etc. According to the evaluation results, the model is tuned, including adjusting the network structure, optimizer settings, learning rate adjustment, etc.

[0122] (7) Iterative training

[0123] Repeat the above steps until the model's performance meets expectations or reaches the preset number of training rounds.

[0124] The application designs a specific CNN network structure by drawing on the idea of CNN, including multiple convolution layers, pooling layers and fully connected layers. The convolution layer uses convolution kernels of different sizes to capture features of different scales. The size and number of convolution kernels can be flexibly adjusted according to the complexity of the graphic text content to adapt to the classification needs in different scenarios. Through convolution operation, the application can automatically learn the high-level feature representation in the graphic text content, reduce the parameter quantity of the model through local connection and parameter sharing, improve the calculation efficiency, and improve the robustness and generalization ability of the classification model. In the pooling layer, maximum pooling or average pooling is used to reduce the feature dimension and reduce overfitting. In the fully connected layer, the prediction probability of each class is obtained by multiplying the parameter matrix with the vector after convolution processing. The softmax function is used to normalize the probability to ensure the rationality and interpretability of the classification result. The design of the fully connected layer enables the model to output intuitive classification results, which is convenient for subsequent application and decision-making.

[0125] In the embodiment of the application, by comparing the experimental data of whether the user reported event of a platform on a certain day is a small micro event and whether the small micro event is determined by the multi-modal data of the application, it is proved that the recall rate and accuracy of the multi-modal classification method of the application are improved, and the graphic text information is helpful for the classification of multi-modal data.

[0126] When the user reported event of a platform on a certain day is only classified by using text data, the result data is shown in Table 1:

[0127] Table 1 Experiment of whether the user reported event of a platform on a certain day is a small micro event

[0128]

[0129] The recall rate of this experiment = 38 / (10+38) = 79%;

[0130] The precision = 38 / (16+38) = 70%;

[0131] The accuracy = (38+45) / (38+10+16+45) = 76%.

[0132] When the multi-modal technology of the application is used for binary classification, the result data is shown in Table 2:

[0133] Table 2 Multi-modal data to determine whether it is a small micro event

[0134]

[0135] The recall rate of the experiment of the application = 40 / (8+40) = 83%;

[0136] The precision = 40 / (13+40) = 75%;

[0137] Accuracy = (40 + 48) / (40 + 13 + 8 + 48) = 80.73%.

[0138] Further, according to the text feature vector, the number of rows m of the text feature vector is determined; according to the image feature vector, the number of rows n of the image feature vector is determined; and according to the image-text arrangement order, the number of rows m, the number of rows n, and the preset vector dimension d, the text feature vector and the image feature vector are alternately spliced to generate a (m+n) row d column matrix M.

[0139] Specifically, according to the number of rows of the text feature vector, the number of rows of the image feature vector, and the preset vector dimension, an empty matrix is created; this matrix will be used to store the alternately spliced feature vectors. According to the image-text arrangement order, vectors are extracted from the text feature vector and the image feature vector in turn, and they are alternately placed in the matrix to generate a (m+n) row d column matrix M.

[0140] Further, based on the convolutional neural network, the matrix is subjected to convolutional dimension reduction processing to obtain a text image feature vector; based on the feedforward neural network (FNN), according to the text image feature vector and the preset number of categories, a fully connected layer parameter matrix is set to map the feature vector to a category vector; based on the multi-category classification algorithm (softmax), according to the category vector, the probability of each category is obtained; and according to the probability of each category, the category of the multi-modal data is determined.

[0141] Specifically, first, use convolution for dimension reduction processing, and use the matrix containing the text and image feature vectors as the input of CNN. The dimension of this matrix is (m+n) rows d columns, where m is the number of text feature vectors, n is the number of image feature vectors, and d is the dimension of the feature vector. One or more convolution layers are applied to extract features. These convolution layers learn how to capture local patterns in the matrix and generate feature maps. After the convolution layer, a pooling layer is added to reduce the dimension of the feature map while preserving important information. Then use the fully connected layer to map the feature map to a fixed size vector, which is the text image feature vector. The dimension of this vector is smaller than the dimension of the input matrix, thus achieving the effect of dimension reduction. Second, use the dimension-reduced text image feature vector as input, and through the calculation of the fully connected layer, the feature vector is mapped to a category vector; then use softmax to map the category vector to a probability distribution, and according to the preset number of categories and the preset categories, it is converted into multiple category probabilities with a sum of 1. Finally, select the category with the highest probability as the predicted category of the multi-modal data, and output the predicted category as the final result.

[0142] Further, based on the convolutional neural network, a convolution kernel is determined according to a preset vector dimension and a hyperparameter, and the matrix is traversed by using the convolution kernel and a preset step length to obtain a text image feature vector.

[0143] For example, assuming that the dimensions d of the picture and the text vector are both 512. Two convolution kernels are set, k1=1 and k2=2, that is, the sizes of the two convolution kernels are 1*512 and 2*512 respectively. At this time, the convolution kernel 1 only performs convolution processing on a single text or a single picture each time, and sequentially processes down according to the arrangement order; the convolution kernel 2 processes two parts of content each time. Taking the picture in the third step as an example, the content processed by the convolution kernel is: text-text, text-picture, picture-text, text-picture, and picture-picture in turn. As shown in the figure, the convolution kernel is convolved with the corresponding row vector on the left side (the product of the corresponding positions is summed), and after each processing is completed, it is moved down by one step, and the operation is repeated until the entire matrix M is traversed, and the text image feature vector is determined. The processing manner of the convolution kernel 1 is the same as that of the convolution kernel 2, as shown in the figure. Figure 4

[0144] Further, the dimension calculation formula of the text image feature vector is as follows:

[0145]

[0146] wherein j represents the number of convolution kernels;

[0147] i represents the i-th, i∈(1, 2,..., j);

[0148] k i represents the width of the i-th convolution kernel;

[0149] m represents the number of text feature vector rows; and

[0150] n represents the number of image feature vector rows.

[0151] Further, based on a multi-class classification algorithm, the category vector is converted into a category probability with a sum of 1 according to a preset number of categories, and each category probability is output.

[0152] An embodiment of the present application provides a content classification device for multi-modal data, as shown in the figure, which comprises: Figure 5

[0153] An acquisition module 101 is configured to acquire multi-modal data.

[0154] A splitting module 102 is configured to split the multi-modal data to obtain text data, picture data, and a text-picture arrangement order.

[0155] ​​The obtaining module 103 is configured to obtain a text feature vector of the text data and an image feature vector of the picture data by using an embedding technology.

[0156] The arranging module 104 is configured to splice the text feature vector and the image feature vector according to the graphic-text arrangement order to obtain a matrix.

[0157] The determining module 105 is configured to determine the category of the multi-modal data based on a convolutional neural network (CNN) according to the matrix, a preset category number and a preset category.

[0158] Further, the arranging module is configured to:

[0159] determine a text feature vector row number m according to the text feature vector;

[0160] determine an image feature vector row number n according to the image feature vector; and

[0161] splice the text feature vector and the image feature vector alternately according to the graphic-text arrangement order, the row number m, the row number n and a preset vector dimension d to generate a matrix M of (m+n) rows and d columns.

[0162] Further, the determining module is configured to:

[0163] perform convolution dimension reduction processing on the matrix based on a convolutional neural network to obtain a text image feature vector;

[0164] set a full connection layer parameter matrix according to the text image feature vector and a preset category number based on a feedforward neural network (FNN) to map the feature vector into a category vector;

[0165] obtain a probability of each category according to the category vector based on a multi-category classification algorithm (softmax); and

[0166] determine the category of the multi-modal data according to the probability of each category.

[0167] Embodiments of the present application provide a storage medium for storing computer executable instructions, which, when executed, implement the steps of the content classification method of multi-modal data according to any one of the above embodiments.

[0168] It should be noted that the embodiments of the storage medium in the present specification and the embodiments of the service providing method based on the blockchain in the present specification are based on the same inventive concept, and therefore the specific implementation of this embodiment can be referred to the foregoing implementation of the corresponding service providing method based on the blockchain, and the repeated parts will not be described herein.

[0169] The above describes particular embodiments of the present specification. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than the order in which they are recited and still achieve desirable results. Additionally, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or necessary.

[0170] In the 1930s, it was clear to distinguish whether an improvement in a technology was in hardware (e.g., improvement in circuit structure of diodes, transistors, switches, etc.) or in software (e.g., improvement in method flow). However, as technology has evolved, many improvements in method flow today can be considered as direct improvements in hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented by a hardware entity module. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a PLD by the designer programming it, rather than by asking a chip manufacturer to design and fabricate a custom integrated circuit chip. Moreover, instead of manually fabricating an integrated circuit chip, this programming is now mostly implemented by "logic compiler" software, which is similar to software compilers used in program development, and the original code to be compiled is written in a specific programming language, called a hardware description language (HDL), of which there are many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., the most commonly used being VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. It should be clear to those skilled in the art that, by simply logically programming a method flow in one of the above hardware description languages and programming it into an integrated circuit, a hardware circuit implementing the logical method flow can be easily obtained.

[0171] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, e.g. software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code, the controller can equally well be implemented to perform the same functions using logic gates, switches, an application specific integrated circuit, a programmable logic controller and an embedded microcontroller, etc. by means of a logical programming of the method steps. The controller can thus be considered as a hardware component, and the means comprised therein for performing the various functions can be considered as structures within the hardware component. Alternatively, the means for performing the various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0172] The systems, apparatuses, modules or units illustrated by the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0173] For the sake of description, the above apparatuses are described in functional division and are described respectively as various units. Of course, the functions of the units can be implemented in the same or multiple software and / or hardware when implementing the embodiments of the specification.

[0174] Those skilled in the art will understand that one or more embodiments of the specification can be provided as a method, a system or a computer program product. Therefore, one or more embodiments of the specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] The specification is presented with reference to flow diagrams and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing element or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Although the flow diagrams and / or block diagrams can present a method, apparatus or computer program product in a particular, it is understood that the method, apparatus and computer program product can include one or more additional steps, operations, or functions, and the method, apparatus and computer program product can include fewer than all of the steps, operations, or functions shown in the figures. Additionally, the steps, operations, or functions need not be implemented in the order shown in the figures. Further, the steps, operations, or functions can be implemented using additional or different steps, operations, or functions. Figure 1 These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flow diagrams and / or block diagrams block or blocks.

[0176] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Although the flow diagrams and / or block diagrams can present a method, apparatus or computer program product in a particular, it is understood that the method, apparatus and computer program product can include one or more additional steps, operations, or functions, and the method, apparatus and computer program product can include fewer than all of the steps, operations, or functions shown in the figures. Additionally, the steps, operations, or functions need not be implemented in the order shown in the figures. Further, the steps, operations, or functions can be implemented using additional or different steps, operations, or functions. Figure 1 These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flow diagrams and / or block diagrams block or blocks.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Although the flow diagrams and / or block diagrams can present a method, apparatus or computer program product in a particular, it is understood that the method, apparatus and computer program product can include one or more additional steps, operations, or functions, and the method, apparatus and computer program product can include fewer than all of the steps, operations, or functions shown in the figures. Additionally, the steps, operations, or functions need not be implemented in the order shown in the figures. Further, the steps, operations, or functions can be implemented using additional or different steps, operations, or functions. Figure 1 These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flow diagrams and / or block diagrams block or blocks.

[0178] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0179] The memory can include non-persistent memory, random access memory (RAM), and / or non-volatile memory, etc. in the form of computer-readable media, such as read only memory (ROM) or flash memory (flash RAM). The memory is an example of computer-readable media.

[0180] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0181] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusion, such that processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed, or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0182] One or more embodiments of the present specification can be described in the general context of computer-executable instructions being executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform particular tasks or implement particular abstract data types. One or more embodiments of the present specification can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media, including storage devices.

[0183] Each embodiment in the present specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0184] The above merely provides the example of the present document and is not intended to limit the present document. For those skilled in the art, the present document can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present document shall be included in the scope of claims of the present document.

Claims

1. A method for classifying the content of multimodal data, characterized in that... include: Acquire multimodal data; The multimodal data is split into text data, image data, and an image-text arrangement order; The text feature vector of the text data and the image feature vector of the image data are obtained using embedding technology; Based on the order of the text and images, the text feature vector and the image feature vector are concatenated to obtain a matrix; Based on the text feature vector, determine the number of text feature vector rows m; Based on the image feature vector, determine the number of rows n of the image feature vector; Based on the text and image arrangement order, the number of rows m, the number of rows n, and the preset vector dimension d, the text feature vector and the image feature vector are alternately concatenated to generate a (m+n) row d column matrix M; Based on a convolutional neural network (CNN), the categories of the multimodal data are determined according to the matrix, the preset number of categories, and the preset categories.

2. The method according to claim 1, characterized in that, The method of determining the categories of the multimodal data based on the convolutional neural network (CNN), according to the matrix, the preset number of categories, and the preset categories, includes: Based on a convolutional neural network, the matrix is ​​subjected to convolutional dimensionality reduction processing to obtain the text image feature vector; Based on a feedforward neural network (FNN), the parameter matrix of the fully connected layer is set according to the text image feature vector and the preset number of categories, and the feature vector is mapped to the category vector. Based on the multi-class classification algorithm (softmax), the probabilities of each class are obtained according to the class vector; and The categories of the multimodal data are determined based on the probabilities of each category.

3. The method according to claim 2, characterized in that, The text image feature vector obtained by performing convolutional dimensionality reduction on the matrix based on a convolutional neural network (CNN) includes: Based on convolutional neural networks, the convolution kernel is determined according to the preset vector dimension and hyperparameters; and By using a convolution kernel and a preset stride, the matrix is ​​traversed to obtain the text image feature vector.

4. The method according to claim 3, characterized in that, The formula for calculating the dimension of the text image feature vector is as follows: Where j represents the number of convolution kernels; i represents the i-th element, i∈(1,2,...,j); k i This represents the width of the i-th convolutional kernel; m represents the number of rows in the text feature vector; and n represents the number of rows in the image feature vector.

5. The method according to claim 2, characterized in that, The probability of each category obtained from the category vector using the multi-class classification algorithm (softmax) includes: Based on a multi-class classification algorithm, the class vectors are converted into class probabilities that sum to 1 according to a preset number of classes, and the probabilities of each class are output.

6. A content classification device for multimodal data, characterized in that... include: The acquisition module is used to acquire multimodal data; The splitting module is used to split multimodal data to obtain text data, image data, and the order of text and image arrangement; The module is used to obtain the text feature vector of the text data and the image feature vector of the image data using embedding technology; The arrangement module is used to concatenate the text feature vector and the image feature vector according to the text and image arrangement order to obtain a matrix; Based on the text feature vector, determine the number of text feature vector rows m; Based on the image feature vector, determine the number of rows n of the image feature vector; Based on the text and image arrangement order, the number of rows m, the number of rows n, and the preset vector dimension d, the text feature vector and the image feature vector are alternately concatenated to generate a (m+n) row d column matrix M; The determination module is used to determine the category of the multimodal data based on the convolutional neural network (CNN), according to the matrix, the preset number of categories, and the preset categories.

7. The apparatus according to claim 6, characterized in that... The determining module is configured as follows: Based on a convolutional neural network, the matrix is ​​subjected to convolutional dimensionality reduction processing to obtain the text image feature vector; Based on a feedforward neural network (FNN), the parameter matrix of the fully connected layer is set according to the text image feature vector and the preset number of categories, and the feature vector is mapped to the category vector. Based on the multi-class classification algorithm (softmax), the probability of each class is obtained according to the class vector; The categories of the multimodal data are determined based on the probabilities of each category.

8. A storage medium for storing computer-executable instructions, characterized in that, When executed, the computer-executable instructions implement the steps of the content classification method for multimodal data as described in any one of claims 1-5.

Images