Multimodal data classification method and device

Abstract

The present invention provides a method and apparatus for classifying multimodal data, wherein the method comprises: generating a first matrix carrying position information of the target object for each target object in the multimodal data to be classified based on the type of the multimodal data to be classified; obtaining a first vector based on the first matrix and the eigenvector of the target object; and inputting each first vector into a classification model to obtain a classification result of the multimodal data to be classified; wherein the type of the multimodal data to be classified includes text, audio, image, or video; and the first matrix is a rotation matrix. The method and apparatus for classifying multimodal data provided by the present invention perform position embedding by carrying the position information of the target object in a rotation matrix, and select a more robust position information encoding and embedding method based on the type of the multimodal data. This method can reduce computational complexity, avoid dispersing position information, improve the performance of the classification model, and improve the efficiency and accuracy of multimodal data classification.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for classifying multimodal data. Background Art

[0002] The Transformer model is an attention-based model that uses a pre-training mechanism. It has excellent classification effects in scenarios such as natural language processing (NLP) and computer vision (CV), and has a strong ability to process multimodal data.

[0003] Multimodal data such as text, audio, images, and videos have positional attributes. In order for the Transformer model to capture the input order of the objects to be classified, positional embedding is required during the embedding processing stage of the feature vector.

[0004] Existing techniques typically achieve position embedding by adding a feature vector to a position vector. However, for models with a non-recurrent structure that utilizes an attention mechanism, such as the Transformer model, existing position embedding methods increase computational complexity and disperse position information, resulting in lower efficiency and accuracy in multimodal data classification using the Transformer model. Summary of the Invention

[0005] The present invention provides a method and device for classifying multimodal data, which are used to solve the defects of low efficiency and accuracy of multimodal data classification in the prior art and achieve high-efficiency and high-accuracy multimodal data classification.

[0006] The present invention provides a method for classifying multimodal data, comprising:

[0007] Based on the type of the multimodal data to be classified, for each target object in the multimodal data to be classified, generating a first matrix carrying position information of the target object;

[0008] Obtaining a first vector based on the first matrix and the eigenvector of the target object;

[0009] Inputting each of the first vectors into a classification model to obtain a classification result of the multimodal data to be classified;

[0010] The type of the multimodal data to be classified includes text, audio, picture or video; and the first matrix is a rotation matrix.

[0011] According to the present invention, a method for classifying multimodal data is provided, wherein based on the type of the multimodal data to be classified, for each target object in the multimodal data to be classified, a first matrix carrying the location information of the target object is generated, specifically comprising:

[0012] generating the first matrix of the first dimension according to the position information of the target object and a trigonometric function;

[0013] The first dimension is determined according to the type of the multimodal data to be classified.

[0014] According to a multimodal data classification method provided by the present invention, obtaining a first vector based on the first matrix and the eigenvector of the target object specifically includes:

[0015] Dividing the feature vector into a plurality of first sub-vectors of a second dimension;

[0016] multiplying the first matrix by each of the first sub-vectors to obtain a plurality of second sub-vectors;

[0017] concatenating the second sub-vectors to obtain the first vector;

[0018] The second dimension is determined according to the type of the multimodal data to be classified.

[0019] According to a multimodal data classification method provided by the present invention, generating the first matrix according to the position information of the target object and a trigonometric function specifically includes:

[0020] When the type of the multimodal data to be classified is text or audio, the first matrix is generated according to the following formula

[0021]

[0022] Among them, m represents the location information of the target object; R m represents the first matrix; θ represents the reference angle.

[0023] According to a multimodal data classification method provided by the present invention, generating the first matrix according to the position information of the target object and a trigonometric function specifically includes:

[0024] When the type of the multimodal data to be classified is an image, the first matrix is generated according to the following formula

[0025]

[0026] Among them, (m,n) represents the location information of the target object; R m,n represents the first matrix; θ represents the reference angle;

[0027] According to a multimodal data classification method provided by the present invention, generating the first matrix according to the position information of the target object and a trigonometric function specifically includes:

[0028] When the type of the multimodal data to be classified is video, the first matrix is generated according to the following formula

[0029]

[0030] Among them, (m,n) represents the spatial location information of the target object; k represents the temporal location information of the target object; R m,n,k represents the first matrix; θ represents the reference angle;

[0031] The present invention also provides a multimodal data classification device, comprising:

[0032] a matrix generating module, configured to generate, for each target object in the multimodal data to be classified, a first matrix carrying position information of the target object based on the type of the multimodal data to be classified;

[0033] a position embedding module, configured to obtain a first vector based on the first matrix and the eigenvector of the target object;

[0034] a data classification module, configured to input each of the first vectors into a classification model to obtain a classification result of the multimodal data to be classified;

[0035] The type of the multimodal data to be classified includes text, audio, picture or video; and the first matrix is a rotation matrix.

[0036] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the multimodal data classification method as described above are implemented.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the multimodal data classification methods described above.

[0038] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any of the above-described methods for classifying multimodal data.

[0039] The multimodal data classification method and device provided by the present invention perform position embedding by carrying the position information of the target object in a rotation matrix. Based on the type of multimodal data, a more powerful position information encoding and embedding method is selected, which can reduce the complexity of calculation, will not disperse the position information, can improve the performance of the classification model, and can improve the efficiency and accuracy of multimodal data classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 1 is a flow chart of the multimodal data classification method provided by the present invention;

[0042] Figure 2 Schematic diagram of the structure of the multimodal data classification device provided by the present invention;

[0043] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0045] In the description of the embodiments of the present invention, the terms "first", "second" and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance, and are not related to the order.

[0046] In the description of the embodiments of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the embodiments of the present invention based on the specific circumstances.

[0047] To facilitate understanding of the following embodiments of the present invention, the following first describes the relevant characteristics of the attention mechanism and multimodal data.

[0048] For models with non-cyclic structures that use the attention mechanism, such as the Transformer model, the pure Attention model based on the Transformer is fully symmetric, that is, for any position m, n,

[0049] f(...,x m ,...x n ,...)=f(...,x n ,...,x m ,...)

[0050] Therefore, due to the characteristics of multimodal data itself, the position information varies greatly. The above model does not have a cyclic structure similar to RNN (Recurrent Neural Network), cannot capture the order of input, and cannot distinguish the embeddings of different positions.

[0051] For full symmetry, more generally, there is the property that f(x,y)=f(y,x), so that [x,y] and [y,x] are indistinguishable.

[0052] In order to break the above symmetry, a straightforward approach is to add a different position vector at each position.

[0053]

[0054] Therefore, it is necessary to encode the location information through the location embedding method and properly integrate it with the embedding of the multimodal data itself.

[0055] The relevant calculation formula of the attention mechanism of the Transformer model is as follows:

[0056] q m =(x m +p m )*W q

[0057] k n =(x n +p n )*W k

[0058]

[0059] The final attention calculation depends on the inner product of q and k, so position embedding is performed in the calculation of q and k.

[0060] Expand

[0061]

[0062] Among them, the above The four terms in the expanded form represent input m - input n, input m - position n, position m - input n, and position m - position n, respectively. The result of combining the four terms constitutes the main body of attention. Therefore, traditional position embedding methods not only increase the computational complexity but also disperse the position information.

[0063] Multimodal data can be processed using a unified technology, namely embedding technology. Formally speaking, embedding uses a low-dimensional dense vector to represent an object. This object can be a sentence, an audio clip, an image, or a video. Objects of different modalities can be converted into vectors in the same vector space through embedding technology, and operations between vectors can include semantic relationships between words or images. This process provides a consistent technical foundation for universal multimodal text alignment.

[0064] The following combination Figures 1 to 3 The present invention describes a method and apparatus for classifying multimodal data.

[0065] Figure 1 This is a flow chart of the multimodal data classification method provided by this application. Figure 1 Describe the classification method of multimodal data in the embodiment of the present application. Figure 1 The method includes: step 101, based on the type of the multimodal data to be classified, for each target object in the multimodal data to be classified, generating a first matrix carrying the position information of the target object.

[0066] The types of multimodal data to be classified include text, audio, picture or video; and the first matrix is a rotation matrix.

[0067] Specifically, the execution subject of the multimodal data classification method provided by the embodiment of the present invention is a multimodal data classification device.

[0068] The multimodal data classification method provided by the embodiment of the present invention can be used to classify data of multiple modalities (ie, types) such as text, audio, picture, or video.

[0069] Optionally, the type of the multimodal data to be classified is used to determine the dimension of the first matrix. Each type of the multimodal data to be classified corresponds to a dimension value.

[0070] The first matrix is the rotation matrix. A rotation matrix is a matrix that, when multiplied by a vector, changes the direction of the vector without changing its magnitude and preserves its chirality.

[0071] Exemplarily, when the type of multimodal data to be classified is text, the dimension of the first matrix can be 2, that is, the first matrix is a 2-order square matrix; when the type of multimodal data to be classified is an image, the dimension of the first matrix can be 4, that is, the first matrix is a 4-order square matrix.

[0072] Optionally, the dimension of the first matrix may also be the same as the dimension of the feature vector of the target object.

[0073] The target object is the component of the multimodal data to be classified.

[0074] For multimodal data to be classified of type text, the target object is each word in the multimodal data to be classified; for multimodal data to be classified of type audio, the target object is each audio frame in the multimodal data to be classified; for multimodal data to be classified of type audio, the target object is each pixel in the multimodal data to be classified; for multimodal data to be classified of type video, the target object is each pixel in each video frame in the multimodal data to be classified.

[0075] For multimodal text data to be classified, text data is essentially one-dimensional sequence data. The location information of each target object can be represented by a one-dimensional array similar to a serial number. The position of a word can be represented by its order in the text.

[0076] For multimodal audio data to be classified, audio data is essentially one-dimensional time series data. The location information of each target object can be represented by a one-dimensional array. The location information of an audio frame can be represented by the frame number of the audio frame.

[0077] For multimodal image data to be classified, image data is essentially two-dimensional, and the location information of each target object can be represented by a two-dimensional array. The location information of a pixel can be represented by its coordinates in the image (the horizontal axis is the direction of the horizontal axis, and the vertical axis is the direction of the vertical axis).

[0078] For multimodal video data to be classified, video data is essentially two-dimensional time series data and can be viewed as a one-dimensional time series of two-dimensional images. The position information of a pixel can include both spatial and temporal information. Spatial position information can be represented by the pixel's coordinates within the video frame, while temporal position information can be represented by the frame number of the video.

[0079] For each target object, a rotation matrix may be constructed according to the position information of the target object to obtain a first matrix carrying the position information of the target object.

[0080] Optionally, a simulated cooling algorithm, a non-connected set combined with a covering design algorithm, a greedy algorithm, or an inducement algorithm may be used to construct a rotation matrix based on the position information of the target object. The embodiment of the present invention does not limit the specific algorithm used to construct the rotation matrix.

[0081] Step 102: Obtain a first vector based on the first matrix and the eigenvector of the target object.

[0082] Specifically, a matrix operation may be performed on the first matrix and the eigenvector of the target object to obtain the first vector.

[0083] Optionally, when the dimension of the first matrix is the same as the dimension of the eigenvector of the target object, the first matrix and the eigenvector of the target object can be multiplied to obtain a first vector, thereby realizing the fusion of the multimodal data to be classified and the position information.

[0084] Step 103: Input each first vector into a classification model to obtain a classification result of the multimodal data to be classified.

[0085] Specifically, a classification model is a trained model.

[0086] Optionally, the classification model is a model with a non-cyclic structure using an attention mechanism. Preferably, the classification model is a Transformer model.

[0087] It is understandable that the classification model is obtained after training based on the sample multimodal data. During the training process, the feature vectors of the objects in the sample multimodal data also undergo position embedding processing similar to steps 101 and 102 before being input into the classification model.

[0088] After each first vector is input into the classification model, the classification model may perform classification based on each first vector and output a classification result of the multimodal data to be classified.

[0089] The embodiment of the present invention performs position embedding by carrying the position information of the target object in a rotation matrix. Based on the type of multimodal data, a more powerful position information encoding and embedding method is selected, which can reduce the complexity of calculation, will not disperse the position information, can improve the performance of the classification model, and can improve the efficiency and accuracy of multimodal data classification.

[0090] Based on the content of any of the above embodiments and based on the type of the multimodal data to be classified, for each target object in the multimodal data to be classified, a first matrix carrying the position information of the target object is generated, specifically including: generating a first matrix of a first dimension based on the position information of the target object and a trigonometric function.

[0091] The first dimension is determined according to the type of multimodal data to be classified.

[0092] Optionally, when the type of multimodal data to be classified is text or audio, the first dimension can be 2, that is, the first matrix is a 2-order square matrix; when the type of multimodal data to be classified is picture or video, the first dimension can be 4, that is, the first matrix is a 4-order square matrix.

[0093] Optionally, when the type of multimodal data to be classified is text or audio, the dimension of the first matrix can be 3, that is, the first matrix is a 3-order square matrix; when the type of multimodal data to be classified is picture or video, the dimension of the first matrix can be 6, that is, the first matrix is a 6-order square matrix.

[0094] The sine function and the cosine function can be used to represent the position information of each target object; based on the position information of the target object represented by the sine function and the cosine function, a rotation matrix is constructed to obtain a first matrix.

[0095] The embodiment of the present invention generates a first matrix of a first dimension based on the position information of the target object and a trigonometric function. The steps for obtaining the first matrix are simpler and more efficient, and can reduce the complexity of the calculation, thereby improving the performance of the classification model and improving the efficiency and accuracy of multimodal data classification.

[0096] Based on the content of any of the above embodiments, obtaining the first vector based on the first matrix and the eigenvector of the target object specifically includes: dividing the eigenvector into multiple first sub-vectors of the second dimension.

[0097] The second dimension is determined according to the type of multimodal data to be classified.

[0098] Specifically, based on the second dimension, the feature vector of the target object is divided into multiple sub-vectors to obtain multiple first sub-vectors.

[0099] The second dimension is the same as the first dimension.

[0100] Optionally, when the type of the multimodal data to be classified is text or audio, the second dimension can be 2, that is, the first sub-vector is a 2-dimensional vector; when the type of the multimodal data to be classified is a picture or video, the second dimension can be 4, that is, the first sub-vector is a 4-dimensional vector.

[0101] Optionally, when the type of the multimodal data to be classified is text or audio, the second dimension can be 3, that is, the first sub-vector is a 3-dimensional vector. When the type of the multimodal data to be classified is a picture or video, the second dimension can be 6, that is, the first sub-vector is a 6-dimensional vector.

[0102] For example, in the case where the second dimension can be a, the d-dimensional feature vector x m , divided into The first subvector. Indicates rounding up. If d is not an integer multiple of a, the last first subvector can be padded with 0. Where m represents the index of the target object.

[0103] The first matrix is multiplied by each first sub-vector to obtain a plurality of second sub-vectors.

[0104] Specifically, the first matrix is multiplied by each first sub-vector to obtain a second sub-vector corresponding to each first sub-vector. The first sub-vectors correspond to the second sub-vectors in a one-to-one correspondence.

[0105] The second sub-vectors are concatenated to obtain the first vector.

[0106] Specifically, based on the division order of the first sub-vectors, the second sub-vectors are concatenated to obtain the first vector.

[0107] The first vector is the vector obtained by position embedding the feature vector.

[0108] The embodiment of the present invention divides the feature vector into multiple first sub-vectors of the second dimension, multiplies the first matrix by each first sub-vector to obtain multiple second sub-vectors, and concatenates the second sub-vectors to obtain the first vector, thereby achieving more efficient and better position embedding.

[0109] Based on the content of any of the above embodiments, a first matrix is generated according to the position information of the target object and the trigonometric function, specifically including: when the type of the multimodal data to be classified is text or audio, the first matrix is generated according to the following formula

[0110]

[0111] Among them, m represents the location information of the target object; R m represents the first matrix; θ represents the reference angle.

[0112] Specifically, for text sequence data, the order of text is an important position information, and text position information is a simple one-dimensional sequence information.

[0113] For multimodal data to be classified of type text, for the mth target object (i.e. the target object at position m), the first subvector v m Incorporating one-dimensional position information p m , get the first vector q m , so there is

[0114] q m =f(v m ,p m )=v m e imθ

[0115] Converting the polar coordinates into matrix form in the complex plane, we can get the first matrix of the target object as

[0116]

[0117] in, s represents the total number of target objects included in the multimodal data to be classified.

[0118] exist In the case of , the calculation formula of the second sub-vector is

[0119]

[0120] Through the above steps, the absolute position (i.e. the position of a single target object) information can be given. From the expanded formula, we can see that the calculation of the attention mechanism takes into account the combination of all position information, including both absolute position information and relative position information (i.e., the relative position between two target objects).

[0121] The first subvector v of the target object at position n n Multiply by the first matrix R of the target object n , the calculation formula of the attention mechanism becomes

[0122]

[0123] Because the rotation matrix R can be accumulated, the relative position matrix R can be obtained n-m , it is possible to embed the absolute position information of a single target object, and the relative position information can be easily obtained during the calculation process of the attention mechanism, greatly enhancing the effect of the Transformer-like model.

[0124] Optionally, the first matrix R m It can also be generated by the following formula

[0125]

[0126] It should be noted that audio sequence data is also one-dimensional, with precise time intervals between elements. Furthermore, the continuity of audio sequence data is far greater than that of other discrete sequence information, which implies a strong correlation between sequence data and audio values. Therefore, position information can also be incorporated using the first matrix multiplication method. However, due to the data density of audio data, the absolute position information becomes extremely large as the audio duration increases. Therefore, methods such as segmenting to share the same position information and downsampling the audio can be used to enhance data features.

[0127] The embodiment of the present invention generates a first matrix of a first dimension based on the position information of the target object and a trigonometric function. The steps for obtaining the first matrix are simpler and more efficient, and can reduce the complexity of the calculation, thereby improving the performance of the classification model and improving the efficiency and accuracy of multimodal data classification.

[0128] Based on the content of any of the above embodiments, the first matrix is generated according to the position information of the target object and the trigonometric function, specifically including: when the type of the multimodal data to be classified is an image, the first matrix is generated according to the following formula

[0129]

[0130] Among them, (m,n) represents the location information of the target object; R m,n represents the first matrix; θ represents the reference angle;

[0131] Specifically, for images, the pixel location information is (m, n), which is the coordinate of the pixel in the image, and is two-dimensional. Image data also exhibits continuity and strong positional correlation between pixels, and is a denser matrix data. Data or model enhancement methods such as downsampling or model dropout can be used to effectively enhance data features.

[0132] The above method of processing the one-dimensional position information of the text can be extended to two dimensions. When the first matrix of the multimodal data to be classified is of the second order, the fourth-order first matrix R is obtained. m,n .

[0133]

[0134] in, h and l represent the height and width of the multimodal data (i.e., image) to be classified, respectively.

[0135] exist In the case of , the calculation formula of the second sub-vector is

[0136]

[0137] It is understandable that For images, it is also possible to embed the absolute position information of a single target object, and the relative position information can be easily obtained during the calculation process of the attention mechanism.

[0138] The embodiment of the present invention generates a first matrix of a first dimension based on the position information of the target object and a trigonometric function. The steps for obtaining the first matrix are simpler and more efficient, and can reduce the complexity of the calculation, thereby improving the performance of the classification model and improving the efficiency and accuracy of multimodal data classification.

[0139] Based on the content of any of the above embodiments, the first matrix is generated according to the position information of the target object and the trigonometric function, specifically including: when the type of the multimodal data to be classified is video, the first matrix is generated according to the following formula

[0140]

[0141] Among them, (m,n) represents the spatial location information of the target object; k represents the temporal location information of the target object; R m,n,k represents the first matrix; θ represents the reference angle;

[0142] Specifically, video data can be regarded as a time series of pictures. The picture position matrix R can be transformed into m,n and the time position matrix R k Multiply them to get the first matrix R of the multimodal data to be classified as video m,n,k .

[0143] It should be noted that, due to R m,n It is a fourth-order matrix, so the time position matrix R k It also needs to be expanded to a fourth-order matrix.

[0144]

[0145]

[0146] The first subvector of the pixel at position (m,n) in the k-th frame of video In the case of , the calculation formula of the second sub-vector is

[0147]

[0148] The embodiment of the present invention generates a first matrix of a first dimension based on the position information of the target object and a trigonometric function. The steps for obtaining the first matrix are simpler and more efficient, and can reduce the complexity of the calculation, thereby improving the performance of the classification model and improving the efficiency and accuracy of multimodal data classification.

[0149] The multimodal data classification device provided by the present invention is described below. The multimodal data classification device described below and the multimodal data classification method described above can be referenced to each other.

[0150] Figure 2 A schematic diagram of the structure of a multimodal data classification device according to an embodiment of the present invention. Based on the content of any of the above embodiments, such as Figure 2 As shown, the device includes a matrix generation module 201, a position embedding module 202 and a data classification module 203, wherein:

[0151] The matrix generation module 201 is configured to generate, for each target object in the multimodal data to be classified, a first matrix carrying position information of the target object based on the type of the multimodal data to be classified;

[0152] A position embedding module 202 is configured to obtain a first vector based on the first matrix and the eigenvector of the target object;

[0153] A data classification module 203 is configured to input each first vector into a classification model to obtain a classification result of the multimodal data to be classified;

[0154] The types of multimodal data to be classified include text, audio, picture or video; and the first matrix is a rotation matrix.

[0155] Specifically, the matrix generation module 201, the position embedding module 202 and the data classification module 203 are electrically connected in sequence.

[0156] For each target object, the matrix generation module 201 may construct a rotation matrix according to the position information of the target object to obtain a first matrix carrying the position information of the target object.

[0157] The position embedding module 202 may perform a matrix operation on the first matrix and the eigenvector of the target object to obtain the first vector, thereby achieving fusion of the multimodal data to be classified and the position information.

[0158] After the data classification module 203 inputs each first vector into the classification model, the classification model may perform classification based on each first vector and output a classification result of the multimodal data to be classified.

[0159] Optionally, the matrix generating module 201 may be specifically configured to generate a first matrix of a first dimension according to the position information of the target object and a trigonometric function;

[0160] The first dimension is determined according to the type of multimodal data to be classified.

[0161] Optionally, the location embedding module 202 may specifically include:

[0162] a grouping unit, configured to divide the feature vector into a plurality of first sub-vectors of a second dimension;

[0163] an embedding unit, configured to multiply the first matrix by each first sub-vector to obtain a plurality of second sub-vectors;

[0164] a concatenation unit, configured to concatenate the second sub-vectors to obtain a first vector;

[0165] The second dimension is determined according to the type of multimodal data to be classified.

[0166] Optionally, the matrix generation module 201 may be specifically configured to generate a first matrix according to the following formula when the type of the multimodal data to be classified is text or audio:

[0167]

[0168] Among them, m represents the location information of the target object; R m represents the first matrix; θ represents the reference angle.

[0169] Optionally, the matrix generation module 201 can be specifically configured to generate a first matrix according to the following formula when the type of the multimodal data to be classified is an image:

[0170]

[0171] Among them, (m,n) represents the location information of the target object; R m,n represents the first matrix; θ represents the reference angle;

[0172] Optionally, the matrix generation module 201 may be specifically configured to generate a first matrix according to the following formula when the type of the multimodal data to be classified is video:

[0173]

[0174] Among them, (m,n) represents the spatial location information of the target object; k represents the temporal location information of the target object; R m,n,k represents the first matrix; θ represents the reference angle;

[0175] The multimodal data classification device provided in an embodiment of the present invention is used to execute the above-mentioned multimodal data classification method of the present invention. Its implementation method is consistent with the implementation method of the multimodal data classification method provided by the present invention and can achieve the same beneficial effects, which will not be repeated here.

[0176] The multimodal data classification device is used in the multimodal data classification method of each of the aforementioned embodiments. Therefore, the description and definition of the multimodal data classification method in each of the aforementioned embodiments can be used to understand each execution module in the embodiments of the present invention.

[0177] The embodiment of the present invention performs position embedding by carrying the position information of the target object in a rotation matrix. Based on the type of multimodal data, a more powerful position information encoding and embedding method is selected, which can reduce the complexity of calculation, will not disperse the position information, can improve the performance of the classification model, and can improve the efficiency and accuracy of multimodal data classification.

[0178] Figure 3 An example of a physical structure diagram of an electronic device is shown below. Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute a multimodal data classification method, the method comprising: based on the type of the multimodal data to be classified, for each target object in the multimodal data to be classified, generating a first matrix carrying the location information of the target object; obtaining a first vector based on the first matrix and the eigenvector of the target object; inputting each first vector into a classification model to obtain a classification result of the multimodal data to be classified; wherein the type of the multimodal data to be classified includes text, audio, picture, or video; and the first matrix is a rotation matrix.

[0179] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0180] The processor 310 in the electronic device provided in the embodiment of the present application can call the logic instructions in the memory 330. Its implementation method is consistent with the implementation method of the multimodal data classification method provided in the present application and can achieve the same beneficial effects, which will not be repeated here.

[0181] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the multimodal data classification method provided by the above methods, the method including: based on the type of multimodal data to be classified, for each target object in the multimodal data to be classified, generating a first matrix carrying the position information of the target object; based on the first matrix and the characteristic vector of the target object, obtaining a first vector; inputting each first vector into a classification model to obtain a classification result of the multimodal data to be classified; wherein the type of multimodal data to be classified includes text, audio, picture or video; the first matrix is a rotation matrix.

[0182] When the computer program product provided in the embodiment of the present application is executed, the above-mentioned multimodal data classification method is implemented. Its specific implementation method is consistent with the implementation method described in the embodiment of the aforementioned method and can achieve the same beneficial effects, which will not be repeated here.

[0183] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the above-mentioned classification method for multimodal data, the method comprising: based on the type of multimodal data to be classified, for each target object in the multimodal data to be classified, generating a first matrix carrying the position information of the target object; based on the first matrix and the characteristic vector of the target object, obtaining a first vector; inputting each first vector into a classification model to obtain a classification result of the multimodal data to be classified; wherein the type of multimodal data to be classified includes text, audio, picture or video; the first matrix is a rotation matrix.

[0184] When the computer program stored on the non-transitory computer-readable storage medium provided in the embodiment of the present application is executed, the above-mentioned multimodal data classification method is implemented. Its specific implementation method is consistent with the implementation method described in the embodiment of the aforementioned method and can achieve the same beneficial effects, which will not be repeated here.

[0185] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0186] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A classification method for multimodal data, characterized in that: include: Based on the type of the multimodal data to be classified, for each target object in the multimodal data to be classified, generating a first matrix carrying position information of the target object; Obtaining a first vector based on the first matrix and the eigenvector of the target object; Inputting each of the first vectors into a classification model to obtain a classification result of the multimodal data to be classified; The type of the multimodal data to be classified includes text, audio, picture or video; the first matrix is a rotation matrix; The step of generating, based on the type of the multimodal data to be classified, for each target object in the multimodal data to be classified, a first matrix carrying the position information of the target object specifically includes: generating the first matrix of a first dimension according to the position information of the target object and a trigonometric function; wherein the first dimension is determined according to the type of the multimodal data to be classified; Generating the first matrix according to the position information of the target object and a trigonometric function specifically includes: When the type of the multimodal data to be classified is video, the first matrix is generated according to the following formula: ； in, represents the spatial location information of the target object; k represents the temporal location information of the target object; R m,n,k represents the first matrix; θ represents the reference angle; .

2. The multimodal data classification method according to claim 1, characterized in that: The acquiring a first vector based on the first matrix and the eigenvector of the target object specifically includes: Dividing the feature vector into a plurality of first sub-vectors of a second dimension; multiplying the first matrix by each of the first sub-vectors to obtain a plurality of second sub-vectors; concatenating the second sub-vectors to obtain the first vector; The second dimension is determined according to the type of the multimodal data to be classified.

3. The multimodal data classification method according to claim 1, characterized in that: Generating the first matrix according to the position information of the target object and a trigonometric function specifically includes: When the type of the multimodal data to be classified is text or audio, the first matrix is generated according to the following formula ； Among them, m represents the location information of the target object; R m represents the first matrix; θ represents the reference angle.

4. The multimodal data classification method according to claim 1, characterized in that: Generating the first matrix according to the position information of the target object and a trigonometric function specifically includes: When the type of the multimodal data to be classified is an image, the first matrix is generated according to the following formula ； in, Represents the location information of the target object; R m,n represents the first matrix; θ represents the reference angle; .

5. A multimodal data classification device, characterized in that: include: a matrix generating module, configured to generate, for each target object in the multimodal data to be classified, a first matrix carrying position information of the target object based on the type of the multimodal data to be classified; a position embedding module, configured to obtain a first vector based on the first matrix and the eigenvector of the target object; a data classification module, configured to input each of the first vectors into a classification model to obtain a classification result of the multimodal data to be classified; The type of the multimodal data to be classified includes text, audio, picture or video; the first matrix is a rotation matrix; The matrix generation module is specifically configured to generate the first matrix of a first dimension based on the position information of the target object and a trigonometric function; wherein the first dimension is determined according to the type of the multimodal data to be classified; The generating of the first matrix according to the position information of the target object and the trigonometric function specifically includes: when the type of the multimodal data to be classified is video, the generating of the first matrix according to the following formula ； in, represents the spatial location information of the target object; k represents the temporal location information of the target object; R m,n,k represents the first matrix; θ represents the reference angle; .

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the multimodal data classification method according to any one of claims 1 to 4 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal data classification method according to any one of claims 1 to 4 are implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the multimodal data classification method according to any one of claims 1 to 4 are implemented.

Images