Mobile application classification and recommendation method based on multi-modal feature fusion

Abstract

The application discloses a mobile application classification and recommendation method based on multi-modal feature fusion, comprising the following steps: (1) a mobile application feature extraction layer; (2) a mobile application classification layer; and (3) a mobile application recommendation layer. The application belongs to the technical field of computer networks, and specifically refers to a mobile application classification and recommendation method based on multi-modal feature fusion, which has better recommendation precision and quality, and is superior to other methods in Macro F1, Accurac, AUC and Logloss indexes.

Description

Technical Field

[0001] This invention belongs to the field of computer network technology, specifically referring to a mobile application classification and recommendation method based on multimodal feature fusion. Background Technology

[0002] According to Statista, as of October 2021, China had nearly 3.99 million mobile applications, ranking first globally. A wealth of applications, including e-commerce, online food delivery, games, and social media, are comprehensively impacting people's lives and changing their lifestyles. In recent years, the number of mobile applications on the internet has grown exponentially. Faced with this massive number of mobile applications, although a large amount of training data already exists, problems such as cold start and data sparsity still arise when new data is processed. The main challenge in training models using existing large-scale, well-classified data samples is selecting a suitable model. When new mobile applications emerge, the information they contain is mostly limited to images, descriptions, and publisher information. On the one hand, for industry professionals, it's difficult to conduct overall benchmarking and analysis of the mobile application market; therefore, precise classification of mobile applications is necessary to complete subsequent risk control and data analysis tasks. On the other hand, for users, it's difficult to choose mobile applications that suit their personalized preferences and needs. Therefore, it's necessary to provide a high-quality mobile application recommendation mechanism to improve the user experience.

[0003] Traditional mobile app classification methods, such as multilayer perceptrons and support vector machines, rely heavily on the quality of labeled datasets for performance, which requires significant manual input. These methods are also susceptible to human error due to their manual design, resulting in poor generalization ability; features that perform well in one domain may not perform well in others. In contrast, traditional mobile app recommendation methods, such as collaborative filtering and matrix factorization, typically transform the mobile app recommendation problem into a supervised learning problem. Essentially, these models first embed users and applications separately, then utilize the interaction information between them to optimize the model and perform recommendations. These methods perform well in many recommendation and ranking tasks. However, they also have limitations, such as sensitivity to sparse data, limited predictive ability for new users, and the fact that they only learn linear interactions between users and services.

[0004] With the growth of multimodal data on the internet, content information from different modalities (visual, auditory, etc.) has recently been used to provide complementary feature signals for traditional text features. Most existing research in this field focuses on emotion classification in dialogues. Specifically, Poria et al. proposed a multi-core learning method and an LSTM-based sequential architecture in 2015 and 2017, respectively, to fuse textual, visual, and audio features. Based on this work, Zadeh et al. and Zadeh et al. further designed tensor fusion networks and memory fusion networks to better capture the interactions between different modalities. However, these methods are designed for coarse-grained classification, which may not be very effective for our fine-grained, target-oriented mobile application classification. Summary of the Invention

[0005] To address the aforementioned challenges, this invention provides a mobile application classification and recommendation method based on multimodal feature fusion that offers superior recommendation accuracy and quality, outperforming other methods in metrics such as Macro F1, Accurac, AUC, and Logloss.

[0006] To achieve the above functions, the present invention adopts the following technical solution: a mobile application classification and recommendation method based on multimodal feature fusion, comprising the following steps:

[0007] (1) Mobile application feature extraction layer

[0008] A set of multimodal samples D is extracted from the mobile application dataset. For each sample c∈D, it contains a sentence S consisting of n mobile application description words (w1, ..., wn) and an associated mobile application image I. D is used as the training corpus to train a mobile application classifier to correctly predict the category label of the mobile application in samples that have not been trained. After initial normalization and autoencoder wordization preprocessing, the BERT model is used to extract the mobile application description features in the feature extraction layer, and the residual network (RedNet) of the involution module is used to extract image features.

[0009] (2) Mobile application classification layer

[0010] The self-attention and multi-head attention mechanisms in Transformer are used to distinguish the feature importance of different modalities and fuse them, and the Softmax classifier is used to classify mobile applications based on the fused feature information.

[0011] (3) Mobile application recommendation layer

[0012] After classification, the data is input into the FiBiNet model according to its category. The model dynamically learns the importance of features by fitting the relationship between features and samples through weights. More important features are assigned greater weights, while the weights of non-critical features are weakened. Bilinear operations are used to consider the importance of each dimension simultaneously to complete mobile application recommendations. The upper part of the FiBiNet model is the deep part, which mainly uses an MLP network to connect and integrate the outputs of the bilinear interaction layer into a dense vector through connection layers. The cross-combined features are then input into the neural network, and the prediction score is obtained in the prediction layer. The lower part, the shallow part, is the core of FiBiNet, which mainly processes the input features.

[0013] Furthermore, the feature extraction in step 1 includes the following steps:

[0014] A pre-trained dual-source BERT model was selected as the initial model, and its parameters were fine-tuned and learned. A multi-head self-attention layer was used to convert each position in the input sequence into a weighted sum of the input layers. Specifically, for the i-th head attention, the input layer X∈Rd×N was transformed based on a dot product attention mechanism.

[0015]

[0016] Among them, {W Qi W Ki W Vi}∈ R d / m×d These are learnable parameters corresponding to the query, key, and value, respectively; then, the outputs of the m attention mechanisms are concatenated and subjected to a linear transformation.

[0017] The description information of each mobile application is represented by an autoencoder and input into the pre-trained BERT. In addition to word tokens, a specific classification token ([CLS]) is inserted at the beginning of each input sequence. The output of the last Transformer layer corresponding to this classification token is used to aggregate the representation information of the entire sequence. The [CLS] vector and the extracted semantic vector are retained as output O to improve the model accuracy.

[0018] O = [ H 0 , H [ C L S ] ]

[0019] The output O is then linearly transformed using the Softmax function to obtain the final D*N dimensional text information representation vector of the mobile application I. .

[0020] Furthermore, the image feature extraction in step 1 includes the following steps:

[0021] Involution kernel H i，j ∈ Rv is generated by the function φ, conditioned on a single pixel at (i, j), and then the channels are rearranged into space. The involution multiplication and addition operation is decomposed into two steps: the product operation multiplies the tensors of the C channels by the involution kernel H respectively, and the addition operation adds the elements within the range of the involution kernel to the kernel core. The involution kernel is specifically located at the pixel X at the corresponding coordinate (i, j). i，j Customized, but shared across channels, G calculates the number of groups sharing the same involution kernel, and uses the involution kernel to perform multiplication and addition operations on the input to obtain the representation output of the involution module, defined as:

[0022]

[0023] Let the kernel generating function be denoted by φ, and abstract the function mapping at each position (i, j) as follows:

[0024]

[0025] Inputting the mobile application image I from the dataset into the RedNet-152 visual model yields the output of the last convolutional layer:

[0026]

[0027] The original mobile application image is segmented into 7×7=49 regions, each represented by a 2048-dimensional vector rj. The visual features of the mobile application are projected onto the same space of text features using a linear transformation function: G=WvResNet(I), where Wv∈Rd×2048 are learnable parameters. The output ResNet(I) is then linearly transformed through the Softmax function to obtain the final representation vector of the mobile application image information in D*2048 dimensions, G=WTResNet(I).

[0028] Furthermore, the Transformer formula in step 2 is as follows:

[0029] M E ( G , H S ) = B T L m ( [ G , H S ] )

[0030] Where Lm is the number of layers in the multimodal encoder, and the final hidden state marked with "[CLS]" is used for mobile application classification tasks to effectively capture dynamic attention within and between mobile application modalities.

[0031] The formula for the softmax generic function is as follows:

[0032]

[0033] Furthermore, the shallow portion of step 3 includes the following steps:

[0034] After classification, mobile applications are input into the initial embedding layer of FibiNet according to their categories. This embeds sparse features into low-dimensional continuous real-valued vectors, transforms the sparse matrix into a dense matrix through linear transformation, extracts the hidden features of the matrix, and improves the model's generalization ability. The output of the embedding layer is represented as follows:

[0035] E = [e1, e2, . . , ei, … , eƒ]

[0036] The SENET network is introduced for training to obtain embedding weights and output the final embedding result. Dimensionality reduction is performed on the embedding features obtained in the embedding layer to obtain global features. Then, a sigmoid activation operation is performed on these features to learn the relationships between each embedding and obtain the weights of embeddings in different domains. Finally, the original embedding results are multiplied to obtain the final embedding result.

[0037] Furthermore, the dimensionality reduction includes the following steps:

[0038] The original embedding E is compressed into a statistical vector Z=[z1,…,z] using average pooling. i , ..., Z ƒ In the middle, z i It can be calculated using the following formula:

[0039]

[0040] Among them, z i It represents global information about the representation of the i-th feature, and k is the embedding size.

[0041] Furthermore, the activation includes the following steps:

[0042] The embedding weights for each domain are learned based on the statistical vector Z, using two fully connected layers. The first fully connected layer is a dimensionality reduction layer with parameter W1, using σ1 as a non-linear function. The second fully connected layer re-stores the original dimensions by adding dimensions using parameter W2. Formally, the domain embedding weights can be calculated as follows:

[0043]

[0044] Where A ∈ ℝ ƒ Let σ1 and σ2 be the representation vectors, and let σ1 and σ2 be the activation functions.

[0045] Furthermore, the reweighting includes the following steps:

[0046] Each field in the embedding layer is multiplied by its corresponding weight to obtain the final embedding result V = {v1, ..., V}. ƒ The entire operation can be viewed as learning the weight coefficients of each domain embedding, which makes the model more discriminative of the features of each domain embedding. Utilizing the SENET mechanism, the weights of important features are increased, while the weights of features with insufficient information are decreased, resulting in the SENET layer output V, as shown below:

[0047] V = [ a 1 ⋅ e 1 , … , a f ⋅ e f ] = [ v 1 , … , v f ]

[0048] After obtaining the mobile application representation embeddings of the initial embedding layer and the SENET layer, second-order and higher-order feature interactions are performed on sparse and dense features.

[0049] The interaction vectors p and q between the output E of the embedding layer and the output V of the SENET layer are calculated:

[0050]

[0051] p = [ p 1 , … , p i , … , p n ]

[0052] q = [ q 1 , … , q i , … , q n ]

[0053] The two interaction vectors obtained above are concatenated and then input into the deeper part.

[0054] Furthermore, the calculation formula for the deep portion in step 3 is as follows:

[0055]

[0056] Where ŷ is the model mobile application recommendation prediction value in the interval ∈ (0,1), σ is the sigmoid function, m is the feature size, and the rest are the linear regression part;

[0057] Use Logloss as the objective function for model recommendation optimization:

[0058]

[0059] Where y is the true label of the i-th mobile application, ŷ corresponds to the predicted label of the i-th mobile application, and N is the total number of mobile applications.

[0060] The beneficial effects achieved by adopting the above structure in this invention are as follows:

[0061] 1. For the first time, a residual network with an involution module is introduced into image feature extraction for mobile applications. This helps to focus on local features in mobile application logo images and improves image feature extraction performance.

[0062] 2. By using attention mechanisms to learn the importance of dynamic features of different modalities, feature interactions can be learned in a fine-grained manner, thereby improving the accuracy of service classification and recommendation;

[0063] 3. The method proposed in this invention outperforms all comparable models in terms of Macro F1, Accurac, AUC, and Logloss. Attached Figure Description

[0064] Figure 1 A method framework diagram for the mobile application classification and recommendation method based on multimodal feature fusion provided by this invention;

[0065] Figure 2 A diagram of the FiBiNet model for the mobile application classification and recommendation method based on multimodal feature fusion provided by this invention;

[0066] Figure 3 The mobile application classification accuracy diagram for the mobile application classification and recommendation method based on multimodal feature fusion provided by this invention;

[0067] Figure 4 The mobile application classification Macro-F1 diagram for the mobile application classification and recommendation method based on multimodal feature fusion provided by this invention;

[0068] Figure 5 The mobile application recommendation Logloss diagram for the mobile application classification and recommendation method based on multimodal feature fusion provided by this invention;

[0069] Figure 6 The mobile application recommendation AUC diagram is provided for the mobile application classification and recommendation method based on multimodal feature fusion provided in this invention. Detailed Implementation

[0070] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The invention will be further described in detail below with reference to the accompanying drawings.

[0072] like Figure 1 As shown, the mobile application classification and recommendation method based on multimodal feature fusion provided by this invention mainly includes three parts: (1) a mobile application feature extraction layer, which realizes feature extraction of image and description information of mobile application nodes; (2) a mobile application classification layer, which uses the self-attention and multi-head attention mechanisms in Transformer to distinguish the importance of features of different modalities and fuse them, and uses the Softmax classifier to classify mobile applications according to the fused feature information; (3) a mobile application recommendation layer, which inputs the classified data into the FiBiNet model according to its category, and dynamically learns the importance of features by fitting the relationship between features and samples through weights. For more important features, greater weights will be assigned, and the weights of non-critical features will be weakened; the importance of each dimension will be considered simultaneously using bilinear operations to complete the mobile application recommendation.

[0073] like Figure 2 As shown, the upper part of the Bilinear Feature Interaction Model (FiBiNet) is the deep layer, where the MLP network integrates the outputs of the bilinear interaction layer into a dense vector through connection layers. The cross-combined features are then input into the neural network, and a prediction score is obtained in the prediction layer. The lower, shallower layer is the core of FiBiNet, primarily processing the input features. First, in the lower left part of the diagram, the high-dimensional sparse input features (APP features) are mapped to low-dimensional dense vector representations after passing through the initial embedding layer. These vector representation embeddings are then dynamically learned for feature importance by the SENET layer, resulting in SENET-Like embeddings. Next, the initial representation embeddings and SENET-Like embeddings are input into the bilinear interaction layer for feature cross-combination. The output cross-combined features are then input into the MLP to complete the mobile application recommendation. Specific Implementation Example 1:

[0075] 1. Mobile application classification experiment and analysis

[0076] The top 5, 10, 15, and 20 categories with the most mobile applications were selected as experimental data. The distribution of the top 20 categories with the most mobile applications is shown in Table 1. 60% of the experimental data was used as the training set, 20% as the validation set, and 20% as the test set.

[0077] Table 1. Kaggle Dataset Information

[0078]

[0079] 2. Mobile Application Classification Experiment and Analysis

[0080] (1) Evaluation indicators

[0081] To evaluate the effectiveness of mobile application classification, two commonly used evaluation metrics were used in the experiment: MacroF1 and Accuracy.

[0082] Accuracy: Represents the ratio of the number of correct decisions to the total number of decisions. The number of correct decisions is the sum of true positives (TP) and true negatives (TN). The total number of decisions is the sum of the four possible outcomes (false positives (FP), false negatives (FN), true positives (TP), and true negatives (TN). The formula for calculating accuracy is as follows:

[0083]

[0084] By calculating the recall rate for each category ( ) and accuracy ( ) to obtain the average recall rate for all N categories ( ) and average accuracy ( ), and finally calculated Among them, recall rate Describe the percentage of mobile apps that were correctly categorized out of all mobile apps in that category; accuracy. The proportion of mobile applications that actually belong to that category in the final classification result of the description model. It is the harmonic mean of recall and precision, calculated using the following formula:

[0085]

[0086] (2) Comparison method

[0087] TResBert: Its text representation is the text features and positional encoding extracted by BERT, while the image input is the image region features extracted by the original ResNet plus the corresponding positional encoding. The above two representation vectors are concatenated and then input into the Encoder layer in the Transformer. The attention mechanism is used to dynamically allocate the weights between multimodal features and between individual modalities. Finally, the Softmax classifier classifies the mobile application based on the final representation.

[0088] Res-BERT: Its text part is represented by the text features and positional encoding extracted by BERT, while the image part is input by the image region features extracted by the original ResNet plus the corresponding positional encoding. The two representation vectors are concatenated by vector only and directly input into the Softmax classifier to obtain the mobile application classification.

[0089] Red-BERT: Its text representation is represented by text features and positional encoding extracted by BERT, while the image input is the image region features extracted by the involute residual network RedNet plus the corresponding positional encoding. The two representation vectors are concatenated by vector only and directly input into the Softmax classifier to obtain the mobile application classification.

[0090] BERT: Classifies mobile applications solely based on mobile application description features extracted by BERT.

[0091] (3) Experimental results and analysis

[0092] The relevant parameter settings include: BatchSize of 32, learning rate of 5e-5, and warm-start rate of 0.1. Experimental results for all methods are shown in Tables 2 and 3. Figure 3 , 4 As shown, we can find that:

[0093] During data preprocessing, no anonymization was performed on the text data of the mobile application, and only the mobile application documents were tokenized and self-encoded, so the overall accuracy of the experiment was not high.

[0094] Of all the comparison methods, BERT alone performed the worst. This means that classifying mobile applications using only text information yielded the worst accuracy. Therefore, more refined mining of information between different modalities, such as image-text feature interactions, is needed to help the model establish correlations between words and objects. This allows multimodal pre-trained models to achieve better accuracy than unimodal models under the same experimental settings.

[0095] In most cases, the model using Involution to replace CNN for image feature extraction in mobile applications has higher accuracy, indicating that using the attention mechanism in multi-modal feature fusion can better distinguish the importance degree of different features.

[0096] Overall, TRedBert maintains good performance. In particular, when the number of classes is 20, TRedBert has improvements of 50.77%, 66.55%, 76.75%, and 83.6% in Accuracy compared to TResbert, Redbert, Resbert, and Bert respectively. The model using Transformer for feature fusion has higher accuracy than the model only using vector concatenation, indicating that using the attention mechanism in multi-modal feature fusion can better distinguish the importance degree of different features and make the representation granularity closer to the downstream tasks.

[0097] Table 2 Mobile Application Classification Accuracy

[0098]

[0099] Table 3 Mobile Application Classification Macro-F1

[0100]

[0101] 3. Mobile Application Recommendation Experiment and Analysis

[0102] (1) Evaluation Metrics

[0103] AUC: Usually, for binary classification problems, we can set a threshold to divide the samples into positive and negative classes. According to different thresholds, calculate the corresponding coordinate points in the ROC to form the ROC curve. AUC is the area under the ROC curve. When 0.5 < AUC < 1, the model is better than the random classifier. In particular, the closer AUC is to 1.0, the higher the authenticity; when it is equal to 0.5, the authenticity is the lowest. The calculation formula is as follows:

[0104]

[0105] where fpr represents the false positive rate and tpr represents the true positive rate. In the ROC space, the coordinate points describe the trade-off between FP (false positive cases) and TP (true positive cases).

[0106] Logloss: Measures the accuracy of the classifier by penalizing misclassifications. Minimizing the log loss is basically equivalent to maximizing the accuracy of the classifier. Logloss reflects the average deviation of the samples and is often used as the loss function of the model for optimization. The calculation formula is as follows:

[0107] .

[0108] (4) Comparison method

[0109] MLR: LR is a type of regression analysis that uses a least squares function called a linear regression equation to model the relationship between one or more independent and dependent variables. LR cannot fit nonlinear data, while MLR can fit nonlinear data using multiple variables.

[0110] FNN: The FNN model only includes a deep part for high-order feature extraction in mobile applications. It extracts features by splicing them together. However, due to the lack of a shallow part (machine learning model), it cannot fit low-order features and requires a pre-trained model.

[0111] AFM: AFM introduces an attention mechanism into the factorization machine model, which can assign weights to different combinations of features. The overall idea is to give different levels of attention to different combinations of mobile application features, and to handle cross-features more finely.

[0112] NFM: Neural Factor Decomposer is a neural network-based attempt to transform the FM model by using the second-order cross term of FM as input to the Deep model to enhance the model's expressive power.

[0113] DeepFM consists of two parts: Wide and Deep. The Wide part extracts low-order features using FM, while the Deep part extracts high-order features using DNN. In mobile application recommendation scenarios, both low-order and high-order feature combinations can influence the final recommendation result. Therefore, the most important aspect is learning the feature combinations implicitly behind user click behavior.

[0114] (5) Results of mobile application recommendation experiment

[0115] The relevant parameter settings include: Test_Size = 0.2, learning rate = 1e-5, and Batch_Size = 32. Experimental results for all methods are shown in Tables 4, 5, 6, and 7. Figure 5 , 6 As shown, we can find that:

[0116] As the number of categories in the dataset increases, with other experimental settings remaining unchanged, the overall recommendation performance decreases with the increase in the number of categories, especially for FM-like models. Due to the increase in the sparsity of the feature matrix, the feature interaction performance of the factorization machine also decreases. However, due to the large dataset size, the performance difference is not significant when the number of categories reaches more than 15.

[0117] Among all the compared methods, MLR and AFM performed poorly. This is because they cannot learn high-order interaction features, thus affecting the performance of mobile application recommendations. NFM and DeepFM models performed better overall, indicating that their learned low-order and high-order feature interactions contribute to improved recommendation quality.

[0118] Deep models such as FNN and DeepFM outperform MLR. When the input has 20 classes, FNN and DeepFM improve performance by 15.88% and 13.72%, respectively. The results show that deep models are better able to model and extract effective information when features are sparse.

[0119] Overall, TRedBert+FiBiNET maintains good performance. Specifically, when the number of categories is 20, FiBiNET shows improvements in AUC of 166.55%, 20.83%, 26.75%, and 113.6% compared to AFM, DeepFM, NFM, and MLR, respectively. This demonstrates that by using attention mechanisms to distinguish the importance of multi-dimensional mobile application features and by learning fine-grained high- and low-order feature interactions, models considering these interactions can achieve better recommendation performance under the same experimental settings.

[0120] Table 4. Mobile application recommendation results under five categories

[0121]

[0122] Table 5. Mobile application recommendation results under ten categories

[0123]

[0124] Table 6. Recommendation results of mobile applications under fifteen categories

[0125]

[0126] Table 7. Recommendation results of mobile applications under twenty categories

[0127]

[0128] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1.A method for mobile application classification and recommendation based on multi-modal feature fusion, characterized in that, Comprising the following steps: (1) Mobile application feature extraction layer A set of multi-modal samples D is extracted from a mobile application data set, for each sample c∈D, it contains a sentence S composed of n mobile application description information words (w1, …, wn) and an associated mobile application image I; D is used as a training corpus to train and learn in the mobile application classifier, correctly predict the class label of the mobile application in the sample not learned; after completing the initial normalization and self-encoding wordization preprocessing, the BERT model is used in the feature extraction layer to extract the mobile application description features, and the residual network (RedNet) of the inner convolution module is used for image feature extraction; (2) Mobile application classification layer The self-attention and multi-head attention mechanism in the Transformer are used to distinguish the importance of features of different modalities and fuse them, and the Softmax classifier is used to classify the mobile application according to the fused feature information; (3) Mobile application recommendation layer The data after classification is input into the FiBiNet model according to its category, the relationship between the feature and the sample is fitted through the weight, and the importance of the feature is dynamically learned; the more important the feature is, the greater the weight will be given, and the weight of the non-key feature will be weakened; the importance of each dimension is considered simultaneously by using the bilinear operation to complete the mobile application recommendation; the upper half of the FiBiNet model is the deep part, mainly the MLP network connects the output of the bilinear interaction layer to the dense vector through the connection layer, and then inputs the cross combination feature into the neural network to get the prediction score in the prediction layer; the lower half is the core of the FiBiNet, which mainly processes the input features; The shallow part in step (3) comprises the following steps: The mobile application after classification is input into the initial embedding layer in FibiNet, the sparse feature is embedded into a low-dimensional continuous real value vector, the sparse matrix is converted into a dense matrix through linear transformation, the hidden features of the matrix are extracted, and the generalization ability of the model is improved, and the output of the embedding layer is as follows: E = [e1, e2,.., ei, …, eƒ] SENET network is introduced for training and learning, embedding weights are obtained, and the final embedding result is output, the embedding features obtained in the embedding layer are dimensionality reduced to obtain global features, then the Sigmoid activation operation is performed on it, the relationship between each embedding is learned, the weights of different domain embeddings are obtained, and finally the original embedding result is multiplied to obtain the final embedding result; The dimensionality reduction comprises the following steps: The original embedding E is compressed into a statistical vector Z=[z1,…,z] using average pooling. i , ..., Z ƒ In the middle, z i It can be calculated using the following formula: ； where z i is global information about the i-th feature representation, k is the embedding dimension; The activation comprises the following steps: The embedding weights of each domain are learned based on the statistical vector Z, and two fully connected layers are used to learn the weights, the first fully connected layer is a dimensionality reduction layer of parameter W1, which uses σ1 as a nonlinear function, and the second fully connected layer increases the dimension by using parameter W2 to store the original dimension, formally, the weight calculation of the domain embedding is as follows: ； where A ∈ ℝ ƒ are the activation functions; The final embedding result comprises the following steps: Each field of the embedding layer is multiplied by a corresponding weight to obtain a final embedding result V = {v1,..., V ƒ} The entire operation is to learn the weight coefficients of each domain embedding, which makes the model more discriminative for the features of each domain embedding. Using the SENET mechanism, the weight of important features is increased, and the weight of those features with insufficient information is reduced to obtain the SENET layer output V, which is represented as follows: ； After obtaining the mobile application representation embedding of the initial embedding layer and the SENET layer, the sparse features and the dense features are subjected to second-order and high-order feature interaction; An interaction vector p and q of the output E of the embedding layer and the output V of the SENET layer is calculated: ； ； ； The two interaction vectors obtained above are connected and input to the deep layer part; The calculation formula of the deep layer part in the step (3) is: ； Wherein, ŷ is the model mobile application recommendation prediction value in the interval ∈ (0, 1), σ is a sigmoid function, m is a feature size, and the rest are linear regression parts; Logloss is used as the model recommendation optimization objective function: ； Wherein, y is the true label of the ith mobile application, ŷ is the predicted label of the ith mobile application, and N is the total number of mobile applications. 2.The mobile application classification and recommendation method based on multi-modal feature fusion according to claim 1, characterized in that, The feature extraction in the step (1) includes the following steps: A pre-trained double pre-embedding BERT is selected as an initial model, and the parameters thereof are adjusted and learned through Fine-Tune; the input sequence in each position is converted into a weighted sum of the input layer by using a multi-head self-attention layer; specifically, for the ith head attention, the input layer X ∈ Rd×N is converted based on a dot product attention mechanism: ； where {W Qi , W Ki , W Vi}∈ R d / m×d are learnable parameters corresponding to the query, key and value respectively; afterwards, the outputs of the m attention mechanisms are concatenated and linearly transformed; The description information of each mobile application is represented by a self-encoding manner and input to the pre-trained BERT; in addition to the token of the word, a specific classification token ([CLS]) is inserted at the beginning of each sequence in the input, and the last Transformer layer output corresponding to the classification token is used to aggregate the entire sequence representation information, and the [CLS] vector and the extracted semantic vector are reserved as the output O, so as to improve the model accuracy: After that, the output O is linearly changed by a Softmax function to obtain a D*N-dimensional text information final representation vector of the mobile application I .

Images