A tobacco industry data grading method based on multi-modal fusion

Abstract

The application discloses a tobacco industry data grading method based on multi-modal fusion, which comprises the following steps: 1, inputting multi-modal tobacco industry text data and image data into a grading network based on a transformer for processing; 2, inputting the processed results into a modal feature extractor to extract features in each modal; 3, inputting the extracted tobacco picture features and text features into a multi-modal feature fusion module to extract inter-modal features; 4, inputting the image combination features and text combination features into a grader to obtain a predicted category probability vector, and selecting the category corresponding to the maximum probability value as a predicted label; 5, constructing a cross-entropy loss by using the predicted label and a real label, and training and optimizing the grading network based on the transformer. The application can well explore the coupling relationship between multi-modal data, so that the security level of multi-modal tobacco data can be accurately judged, and the labor cost is reduced.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically a data classification method for the tobacco industry based on multimodal fusion. Background Technology

[0002] With strong support from national policies, digital applications are leading transformations across various vertical sectors, and this trend will generate increasing value from data flow. However, data use is a double-edged sword; as long as data is in flow, there is a risk of sensitive data leakage. With the rapid development of information technology, tobacco business data resources in the digital age are constantly growing, exhibiting characteristics of massive scale, diverse structures, and complex relationships. Tobacco industry data encompasses multiple aspects such as production, supply chain, sales, and marketing. Due to its special nature, much of this data has both commercial value and a background of strong national control, requiring high data sensitivity. Furthermore, as a strategic industry for the nation, the tobacco industry generates a large amount of important and sensitive data, including trade secrets, business decisions, and partner information. If this sensitive data is not effectively managed and protected, it could lead to the leakage of sensitive information, seriously threatening the tobacco company's own commercial competitiveness, reputation, and economic interests, and even affecting the normal operation of the tobacco supply chain, negatively impacting the national tobacco industry and social stability.

[0003] To enhance the value of data utilization and avoid adverse consequences from the leakage of sensitive data, tobacco digitalization enterprises need to identify the location of sensitive data while aggregating and sharing massive amounts of data, and classify and standardize data according to its sensitivity level and establish a data classification and hierarchical protection mechanism.

[0004] Currently, the identification and classification of sensitive tobacco data mainly fall into three categories: First, manual identification methods, typically performed by experienced professionals who classify and grade each piece of data based on their expertise. Due to the differences and correlations in the business operations of various departments within tobacco companies, employees often find it difficult to accurately determine the sensitivity of existing business data. Furthermore, individual differences in subjective judgment lead to inconsistencies and inaccuracies in the classification and grading of sensitive data.

[0005] Secondly, there's the rule-based matching method. This typically involves matching data against a pre-established keyword database. If the data contains these keywords, it can be marked as sensitive data. Regular expressions and metadata analysis are then used to assist in the classification of sensitive data. However, rule-based matching systems may produce false positives or false negatives due to inaccurate rule definitions or outdated data. Furthermore, identifying sensitive data requires consideration of complex contextual information, which rule-based matching systems sometimes struggle to understand and handle. For example, the same keyword may have different meanings in different contexts, potentially leading to inaccurate sensitive data classification.

[0006] Thirdly, there are artificial intelligence methods that use general algorithm models trained on large amounts of data to achieve the effect of identifying and classifying sensitive data. However, as the model becomes more complex, it is prone to overfitting, significantly reducing the accuracy of sensitive data classification. Therefore, some existing sensitive data classification and grading products based on artificial intelligence methods have limitations in terms of classification granularity and accuracy, and are also insufficient in meeting users' more advanced and customized needs. Furthermore, with the diversification of data types and formats, current sensitive data classification and grading products also have certain shortcomings in processing unstructured data. Current data exhibits a trend of multiple modalities such as text, images, videos, and audio being intertwined. Current products use different types of artificial intelligence models to process different types of unstructured data. However, single-modal models often ignore information contained in other modalities when processing data from one modality, failing to integrate information from multiple modalities or fully extract sensitive information. Summary of the Invention

[0007] The present invention addresses the shortcomings of the existing technology by proposing a multimodal fusion-based data classification method for the tobacco industry. This method aims to effectively uncover the coupling relationships between multimodal data, thereby accurately determining the sensitivity level of tobacco data and reducing labor costs.

[0008] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0009] The present invention provides a data classification method for the tobacco industry based on multimodal fusion, characterized by the following steps:

[0010] Step 1: Construct a multimodal tobacco industry dataset A = {T, P}, where T represents an unstructured tobacco text dataset and P represents a tobacco image dataset; denote any multimodal tobacco industry data as x = {t, p}, where t represents any tobacco text data and p represents any tobacco image; let the true label of x be y; y∈{1,2,…,C}, where C represents the sensitivity level;

[0011] Step 2: Construct a Transformer-based hierarchical network, including: an embedding layer, a modal feature extractor, a multimodal feature fusion module, and a classifier;

[0012] Step 2.1: The embedding layer consists of two word vector embedding layers and one position embedding layer, and is used to process tobacco text data t and tobacco image p, thereby obtaining the embedding features Pt of t and the embedding features Pp of p.

[0013] Step 2.2: The modal feature extractor consists of N identical modal feature extraction blocks stacked together; wherein each modal feature extraction block contains a multi-head self-attention layer and a feedforward network layer;

[0014] Step 2.2.1: The modal feature extractor processes Pp and outputs the intramodal feature vector R = {r1, r2...r...} of the tobacco image p. i ...r n}; thereby calculating the aggregation features of the tobacco image p. Where, r i Represents the features of the i-th image patch;

[0015] Step 2.2.2: The modal feature extractor processes Pt to obtain the intramodal feature vector E = {e1, e2, ... e} of the tobacco text data t. i , ...e n}, where e i Indicate the features of the i-th segment;

[0016] Step 2.2.3: Apply a convolution with a window size of l to the text embedding features E = {e1, e2, ... e}. i , ...e n The convolutional features p of text t are then processed to obtain the convolutional features p of text t. l，1 p l，2 , ..., p l，i , ..., p l，n Therefore, the max-pooling feature q with a window size of l is calculated. l =max(p l，1 p l，2 , ..., p l，n ); where p l，i Indicates e i The convolutional features obtained after performing a convolution operation with a window size of l;

[0017] Step 2.2.4: Change the window size l value L times, and return to step 2.2.3 for processing, thereby obtaining the multi-scale convolutional feature {q}. l |l=1,2,…L};

[0018] Step 2.2.5: Use equation (2) to obtain the final representation c0 of the intramodal features of the tobacco text data t:

[0019] c0 = LayerNorm(W e concat(q1, q2, ..., q) l ,...q L )+b e (2)

[0020] In equation (2), W e It is a convolutional filtering matrix, b e It's a deviation;

[0021] Step 2.3: The multimodal feature fusion module includes: a self-attention layer, an average pooling layer, a max pooling layer, and a fusion self-attention module, and processes the superimposed features. The process is performed to obtain the final embedding i1 of the tobacco image p and the final embedding c1 of the tobacco text data t;

[0022] Step 2.4: After the classifier processes the combined features i = i0 + i1 of the tobacco image p and the combined features c = c0 + c1 of the tobacco text data t, it obtains the predicted category probability vector of x, and selects the category corresponding to the maximum probability value as the predicted label y′ of x.

[0023] Step 3, based on y and y′ A cross-entropy loss is constructed, and the Adam optimizer is used to train the Transformer-based hierarchical network. The cross-entropy loss is calculated to update the network parameters until the maximum number of iterations is reached or the cross-entropy loss value reaches the minimum. Training is then stopped to obtain the optimal hierarchical network model after training, which is used to classify the sensitivity of the input multimodal tobacco industry dataset.

[0024] The characteristic of the tobacco industry data classification method based on multimodal fusion described in this invention is that step 2.1 includes:

[0025] Step 2.1.1: The first word embedding layer performs word transformation on the tobacco text data t to obtain the word vector representation of t (t cls ,t1,t2,...,t i , ..., t n , t scp ); where t cls The t indicates the beginning of a word transition. scp The t indicates the end of a word transition. i This represents the word vector of the i-th segment; n represents the number of word vectors.

[0026] The second word embedding layer divides the tobacco image p into n image patches, and then maps each image patch into a one-dimensional vector through linear mapping, thus obtaining the word vector representation of p (p1, p2, ..., p...). i ..., p n ); where p i The word vector representing the i-th image patch;

[0027] Step 2.1.2: The positional embedding layer performs positional encoding on the word vector representations of t and p, respectively, to obtain the embedding feature Pt = {w0 + t} of t. cls +w1+t1, w2+t2, ..., w i +t i , ..., w n +t n +w n+1 +t scp The embedding features of} and p are Pp={w1+p1,w2+p2,...,w i +p i , ..., w n +p n}; where w i This represents the positional encoding of the i-th position in the word vector.

[0028] Step 2.2.1 includes:

[0029] Step 2.2.1.1: When s = 1, define the input of the s-th modal feature extraction block as X. s =Pp;

[0030] Step 2.2.1.2: The j-th head self-attention layer in the s-th modality feature extraction block is subjected to different linear transformations on X. s The process is performed to obtain the j-th query vector of the s-th modal feature extraction block. The j-th key vector of the s-th modality feature extraction block The j-th value vector of the s-th modality feature extraction block in, These represent the weight matrices corresponding to the three linear transformations in the j-th head self-attention layer of the s-th modality feature extraction block, respectively;

[0031] The j-th self-attention layer in the s-th modality feature extractor obtains the j-th tobacco image feature head of the tobacco image p using equation (1). s，j :

[0032]

[0033] In equation (1), d represents the dimension of Pp, SparsityMask is a neural network used to output a sparsity mask, and sofmax represents the activation function; (K s，j )′ represents K s，j Transpose of;

[0034] Step 2.2.1.3: After aggregating the tobacco image features output by all self-attention heads in the s-th modality feature extraction block, the s-th aggregated tobacco image feature head is obtained. s, head s After processing in the feedforward neural network, the feature value f output by the s-th modality feature extraction block is obtained. s ;

[0035] Step 2.2.1.4: If s≥N, then extract the feature value f from the Nth modal feature extraction block. N The intramodal feature vector R = {r1, r2, ... r1} of the tobacco image p output by the modal feature extractor is... i ...r n Otherwise, assign s+1 to s and let X... s =f s-1 Then, return to step 2.2.1.2 and execute sequentially.

[0036] Step 2.3 includes:

[0037] Step 2.3.1: Overlay features The input is processed by the self-attention layer through three different linear transformations to obtain the key vector. Query vector value vector Thus, the multimodal fusion self-attention value is obtained using equation (3). Among them, K R K represents the key vector of tobacco text data t. E Q represents the key vector of the tobacco image p. R Q represents the query vector of tobacco text data t. E V represents the query vector for the tobacco image p. R V represents the value vector of tobacco text data t. E W represents the value vector of the tobacco image p. Q W K W V These are three different weight matrices, P c T represents the cross features of the tobacco image p. c The cross features representing tobacco text data t;

[0038]

[0039] In equation (3), sofmax represents the activation function;

[0040] Step 2.3.2, the average pooling layer for P C The image pooling feature V is obtained through processing.

[0041] Step 2.3.3, the maximum pooling layer for T C The text pooling feature M is obtained through processing.

[0042] Step 2.3.4: After the fusion self-attention module processes V and M respectively, it obtains the final embedding i1 of the tobacco image p and the final embedding c1 of the tobacco text data t.

[0043] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the tobacco industry data grading method, and the processor is configured to execute the program stored in the memory.

[0044] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, performs the steps of the tobacco industry data grading method.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0046] 1. This invention is based on the transformer model for feature representation learning, which can capture the global semantic information and local semantic features of text data; through model fine-tuning and prediction, it can efficiently and accurately classify tobacco text data; this classification method can improve the accuracy and efficiency of classification and provide reliable decision support for information security management of tobacco systems.

[0047] 2. This invention employs a sparse multi-head self-attention mechanism to capture the dependencies and couplings between corpora, significantly reducing the algorithm's time and space complexity. In the field of information processing within the tobacco industry, this optimization means the ability to handle larger datasets while maintaining high operating speed and low resource consumption. This represents a significant technological advancement for tobacco information systems that require real-time or near-real-time processing of large amounts of text data, substantially improving the system's response speed and processing capabilities.

[0048] 3. This invention transforms different types of data into feature vectors that can be processed by the model through preprocessing and feature processing. By integrating features from different modalities, it comprehensively utilizes data information from different modalities, improving the comprehensiveness and completeness of the classification. This enables the model to better understand and describe the background and characteristics of tobacco data, thereby improving the accuracy and robustness of tobacco classification. Attached Figure Description

[0049] Figure 1 This is a flowchart of a data classification method for the tobacco industry based on multimodal fusion in this invention;

[0050] Figure 2 This is a model structure diagram of the tobacco industry data classification method based on multimodal fusion in this invention. Detailed Implementation

[0051] In this embodiment, as Figure 1 As shown, a data classification method for the tobacco industry based on multimodal fusion is carried out according to the following steps:

[0052] Step 1: Construct a multimodal tobacco industry dataset A = {T, P}, where T represents an unstructured tobacco text dataset and P represents a tobacco image dataset; denote any multimodal tobacco industry data as x = {t, p}, where t represents any tobacco text data and p represents any tobacco image; let the true label of x be y; y∈{1,2,…,C}, where C represents the sensitivity level;

[0053] Step 2, as follows Figure 2 As shown, a Transformer-based hierarchical network is constructed, including: an embedding layer, a modal feature extractor, a multimodal feature fusion module, and a classifier;

[0054] Step 2.1: The embedding layer consists of two word vector embedding layers and one position embedding layer;

[0055] Step 2.1.1: The first word embedding layer performs word transformation on the tobacco text data t to obtain the word vector representation of t (t cls ,t1,t2,...,t i , ..., t n , t scp ); where t cls The t indicates the beginning of a word transition. scp The t indicates the end of a word transition. i This represents the word vector of the i-th segment; n represents the number of word vectors.

[0056] The second word embedding layer divides the tobacco image p into n image patches, and then maps each image patch into a one-dimensional vector through linear mapping, thus obtaining the word vector representation of p (p1, p2, ..., p...). i ..., p n ); where p i The word vector representing the i-th image patch;

[0057] Step 2.1.2: The positional embedding layer performs positional encoding on the word vector representations of t and p, respectively, to obtain the embedding feature Pt = {w0 + t} of t. cls +w1+t1, w2+t2, ..., w i +t i , ..., w n +t n +w n+1 +t scpThe embedding features of} and p are Pp={w1+p1,w2+p2,...,w i +p i , ..., w n +p n}; where w i This represents the positional encoding of the i-th position in the word vector.

[0058] Step 2.2: The modal feature extractor consists of N identical modal feature extraction blocks stacked together; each modal feature extraction block contains a multi-head self-attention layer and a feedforward network layer;

[0059] Step 2.2.1: The modal feature extractor processes Pp and outputs the intramodal feature vector R = {r1, r2...r...} of the tobacco image p. i ...r n}; where r i Represents the features of the i-th image patch;

[0060] Step 2.2.1.1: When s = 1, define the input of the s-th modal feature extraction block as X. s =Pp;

[0061] Step 2.2.1.2: The j-th head self-attention layer in the s-th modality feature extraction block is subjected to different linear transformations on X. s After processing, the j-th query vector of the s-th modal feature extraction block is obtained. The j-th key vector of the s-th modality feature extraction block The j-th value vector of the s-th modality feature extraction block in, These represent the weight matrices corresponding to the three linear transformations, respectively.

[0062] The j-th self-attention layer in the s-th modality feature extractor obtains the j-th tobacco image feature head of the tobacco image p using equation (1). s，j Sparse multi-head self-attention mechanisms help reduce time complexity and improve the ability to model long sequence dependencies by selectively focusing on key positions in the sequence rather than being limited to local contexts.

[0063]

[0064] In equation (1), d represents the dimension of Pp, and SparsityMask is a neural network used to output a sparsity mask. This network will determine the sparsity mask based on the input Q. s，j and (K) s，j )' calculates a sparse mask, where softmax represents the activation function; (K s，j )′ represents Ks，j Transpose of;

[0065] Step 2.2.1.3: Aggregate the outputs of all self-attention heads in the s-th modality feature extraction block to obtain the head. s Then head s The value f of the s-th modal feature extraction block is obtained after processing by the feedforward neural network. s ;

[0066] Step 2.2.1.4, if s≥N, then extract the feature value f from the Nth modal feature extraction block. N The intramodal feature vector R = {r1, r2, ... r1} of the tobacco image p output by the modal feature extractor is... i ...r n Otherwise, assign s+1 to s and let X... s =f s-1 Then, return to step 2.2.1.2 and execute sequentially;

[0067] Step 2.2.1.5: The feature vector R containing the relationships between regions in the image is aggregated using average pooling to obtain the feature representations of the image regions.

[0068] Step 2.2.2: Process the text data Pt according to the process in steps 2.2.1.1-2.2.1.4 to obtain the intra-modal feature vector E = {e1, e2, ... e} of the tobacco text. i , ...e n}, where e i Indicate the features of the i-th segment;

[0069] Step 2.2.3: Apply a convolution with a window size of l to the text embedding features E = {e1, e2, ... e}. i , ...e n The convolutional features p of text t are then processed to obtain the convolutional features p of text t. l，1 p l，2 , ..., p l，i , ..., p l，n ; where p l，i This represents the convolutional feature obtained by performing a convolution operation with a window size of 1 at the i-th position.

[0070] Calculate the max pooling feature q l =max(p l，1 p l，2 , ..., p l，n );

[0071] Step 2.2.4: Change the window size l value L times, and return to step 2.2.3 for processing, thereby obtaining the multi-scale convolutional feature {q}. l |l=1,2,…L};

[0072] Step 2.2.5: Use equation (2) to obtain the final representation c0 of the intramodal features of the text data:

[0073] c0 = LayerNorm(W e concat(q1, q2, q3) + b e (2)

[0074] In equation (2), W e It is a convolutional filtering matrix, b e It's a deviation;

[0075] Step 2.3: The multimodal feature fusion module includes: a self-attention layer, an average pooling layer, a max pooling layer, and a fusion self-attention module;

[0076] Step 2.3.1: Overlay features In the input multimodal feature fusion module, the self-attention layer obtains the key vector through three different linear transformations. Query vector value vector Thus, the multimodal fusion self-attention value is obtained using equation (3). Among them, K R K represents the key vector of text data. E Q represents the key vector of image data. R Q represents the query vector of text data. E V represents the query vector for image data. R V represents a value vector of text data. E W represents the value vector of image data. Q W K W V P represents different weight matrices. c T represents the cross features of tobacco images. c The text cross-features of tobacco are represented; the fusion self-attention module improves the model's performance and efficiency by associating and integrating information from different sources and dynamic attention mechanisms, and provides better interpretability.

[0077]

[0078] In equation (3), sofmax represents the activation function;

[0079] Step 2.3.2, Average pooling layer for P CThe image pooling feature V is obtained through processing.

[0080] Step 2.3.3, Max pooling layer on T C The text pooling feature M is obtained through processing.

[0081] Step 2.3.4: After the self-attention module processes V and M respectively, the final embedding i1 of the image and the final embedding c1 of the text are obtained.

[0082] Step 2.4: After the classifier processes the image-based combined feature i = i0 + i1 and the text-based combined feature c = c0 + c1, it obtains the predicted class probability vector of x, and selects the class corresponding to the highest probability value as the predicted label y′ of x.

[0083] Step 3: Construct cross-entropy loss based on y and y′, and use the Adam optimizer to train the Transformer-based hierarchical network. Calculate the cross-entropy loss to update the network parameters until the number of iterations reaches the maximum number of iterations or the cross-entropy loss value reaches the minimum. Stop training to obtain the optimal hierarchical network model after training, which is used to classify the sensitivity of the input multimodal tobacco industry dataset.

[0084] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0085] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

Claims

1. A data classification method for the tobacco industry based on multimodal fusion, characterized in that, The procedure is as follows: Step 1: Construct a multimodal tobacco industry dataset A = {T, P}, where T represents an unstructured tobacco text dataset and P represents a tobacco image dataset; denote any multimodal tobacco industry data as x = {t, p}, where t represents any tobacco text data and p represents any tobacco image; let the true label of x be y; y ∈ {1, 2, ..., C}, where C represents the sensitivity level; Step 2: Construct a Transformer-based hierarchical network, including: an embedding layer, a modal feature extractor, a multimodal feature fusion module, and a classifier; Step 2.1: The embedding layer consists of two word vector embedding layers and one position embedding layer, and is used to process tobacco text data t and tobacco image p, thereby obtaining the embedding features Pt of t and the embedding features Pp of p. Step 2.2: The modal feature extractor consists of N identical modal feature extraction blocks stacked together; wherein each modal feature extraction block contains a multi-head self-attention layer and a feedforward network layer; Step 2.2.1: The modal feature extractor processes Pp and outputs the intramodal feature vector R = {r1, r2...r} of the tobacco image p. i ...r n }; thereby calculating the aggregation features of the tobacco image p. Where ri represents the feature of the i-th image patch; Step 2.2.2: The modal feature extractor processes Pt to obtain the intramodal feature vector E = {e1, e2, ... e} of the tobacco text data t. i ,...e n }, where e i Indicate the features of the i-th segment; Step 2.2.3: Apply a convolution with a window size of l to the text embedding features E = {e1, e2, ... e}. i ,...e n The convolutional features p of text t are then processed to obtain the convolutional features p of text t. l,1 ,p l,2 ,…,p l,i ,…,p l,n Therefore, the max-pooling feature q with a window size of l is calculated. l =max(p l,1 ,p l,2 ,…,p l,n ); where p l,i Indicates e i The convolutional features obtained after performing a convolution operation with a window size of l; Step 2.2.4: Change the window size l value L times, and return to step 2.2.3 for processing, thereby obtaining the multi-scale convolutional feature {q}. l |l=1,2,…L}; Step 2.2.5: Use equation (2) to obtain the final representation c0 of the intramodal features of the tobacco text data t: c0=LayerNorm(W e concat(q1,q2,…,q l ,…q L )+b e ) (2) In equation (2), W e It is a convolutional filtering matrix, b e It's a deviation; Step 2.3: The multimodal feature fusion module includes: a self-attention layer, an average pooling layer, a max pooling layer, and a fusion self-attention module, and processes the superimposed features. The process is performed to obtain the final embedding i1 of the tobacco image p and the final embedding c1 of the tobacco text data t; Step 2.4: After the classifier processes the combined features i = i0 + i1 of the tobacco image p and the combined features c = c0 + c1 of the tobacco text data t, it obtains the predicted category probability vector of x, and selects the category corresponding to the maximum probability value as the predicted label y′ of x. Step 3: Construct cross-entropy loss based on y and y′, and train the Transformer-based hierarchical network using the Adam optimizer. Calculate the cross-entropy loss to update the network parameters until the maximum number of iterations is reached or the cross-entropy loss value reaches its minimum. Stop training to obtain the optimal hierarchical network model after training, which is used to classify the sensitivity of the input multimodal tobacco industry dataset.

2. The tobacco industry data classification method based on multimodal fusion according to claim 1, characterized in that, Step 2.1 includes: Step 2.1.1: The first word embedding layer performs word transformation on the tobacco text data t to obtain the word vector representation of t (t cls ,t1,t2,...,t i ,...,t n ,t scp ); where t cls The t indicates the beginning of a word transition. scp The t indicates the end of a word transition. i This represents the word vector of the i-th segment; n represents the number of word vectors. The second word embedding layer divides the tobacco image p into n image patches, and then maps each image patch into a one-dimensional vector through linear mapping, thus obtaining the word vector representation of p (p1, p2, ..., p...). i ...,p n ); where p i The word vector representing the i-th image patch; Step 2.1.2: The positional embedding layer performs positional encoding on the word vector representations of t and p, respectively, to obtain the embedding feature Pt = {w0 + t} of t. cls +w1+t1,w2+t2,...,w i +t i ,...,w n +t n +w n+1 +t scp The embedding features of} and p are Pp={w1+p1,w2+p2,...,w i +p i ,...,w n +p n }; where w i This represents the positional encoding of the i-th position in the word vector.

3. The tobacco industry data classification method based on multimodal fusion according to claim 1, characterized in that, Step 2.2.1 includes: Step 2.2.1.1: When s = 1, define the input of the s-th modal feature extraction block as X. s =Pp; Step 2.2.1.2: The j-th head self-attention layer in the s-th modality feature extraction block is subjected to different linear transformations on X. s After processing, the j-th query vector of the s-th modal feature extraction block is obtained. The j-th key vector of the s-th modality feature extraction block The j-th value vector of the s-th modality feature extraction block in, These represent the weight matrices corresponding to the three linear transformations in the j-th head self-attention layer of the s-th modality feature extraction block, respectively; The j-th self-attention layer in the s-th modality feature extractor obtains the j-th tobacco image feature head of the tobacco image p using equation (1). s,j : In equation (1), d represents the dimension of Pp, SparsityMask is a neural network used to output a sparsity mask, and sofmax represents the activation function; (K s,j )′ represents the transpose of Ks,j; Step 2.2.1.3: After aggregating the tobacco image features output by all self-attention heads in the s-th modality feature extraction block, the s-th aggregated tobacco image feature head is obtained. s , head s After processing in the feedforward neural network, the feature value f output by the s-th modality feature extraction block is obtained. s ; Step 2.2.1.4: If s≥N, then extract the feature value f from the Nth modal feature extraction block. N The intramodal feature vector R = {r1, r2, ... r1} of the tobacco image p output by the modal feature extractor is... i ...r n Otherwise, assign s+1 to s and let X... s =f s-1 Then, return to step 2.2.1.2 and execute sequentially.

4. The tobacco industry data classification method based on multimodal fusion according to claim 1, characterized in that, Step 2.3 includes: Step 2.3.1: Overlay features The input is processed by the self-attention layer through three different linear transformations to obtain the key vector. Query vector value vector Thus, the multimodal fusion self-attention value is obtained using equation (3). Among them, K R K represents the key vector of tobacco text data t. E Q represents the key vector of the tobacco image p. R Q represents the query vector of tobacco text data t. E V represents the query vector for the tobacco image p. R V represents the value vector of tobacco text data t. E W represents the value vector of the tobacco image p. Q W K W V These are three different weight matrices, P c T represents the cross features of the tobacco image p. c The cross features representing tobacco text data t; In equation (3), sofmax represents the activation function; Step 2.3.2, the average pooling layer for P C The image pooling feature V is obtained through processing. Step 2.3.3, the maximum pooling layer for T C The text pooling feature M is obtained through processing. Step 2.3.4: After the fusion self-attention module processes V and M respectively, it obtains the final embedding i1 of the tobacco image p and the final embedding c1 of the tobacco text data t.

5. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store programs that support the processor in executing any of the tobacco industry data grading methods of claims 1-4, and the processor is configured to execute the programs stored in the memory.

6. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program, when run by a processor, performs the steps of any of the tobacco industry data grading methods described in claims 1-4.

Images