Product recommendation method based on multimodal data fusion

Abstract

The present invention provides a commodity recommendation method based on multimodal data fusion, which belongs to the field of deep learning technology. ALBERT-TextCNN is used to improve the representation ability of word vectors extracted from customer basic attribute data sets and to retain customer semantic information at different levels to the greatest extent, thereby improving the classification effect of different customer groups; AlBert-BiLSTM-CRF is used to effectively solve problems that cannot be solved by traditional language processing models, such as poor polysemy analysis effect of a word in a text and inability to handle different contexts of polysemy, and to sort customer online and offline transaction data sets into keywords that are more in line with the contextual semantics of the text set, and to construct customer labels in a more targeted manner; a k-dimensional tree method is used to obtain similar customers of two label categories, and then recommendations are made based on similar customer purchase histories, thereby achieving a higher recommendation accuracy. The present invention ensures the accuracy of predicted data while ensuring efficient training of data.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and relates to a commodity recommendation method based on AlBert-TextCNN (A Lite BERT-convolutional neural network based on sentence classification), AlBert-BiLSTM-CRF (A Lite BERT-bidirectional long short-term memory network-conditional random field), Encoder-Decoder (encoder-decoder) and k-dimensional tree nearest neighbor search recommendation. Background Art

[0002] The existing customer acquisition methods are basically to acquire customers through various offline promotions and promotional activities, as well as online advertising. However, this method has many pain points, such as a large number of customer leads cannot be effectively followed up in the long term, resulting in a high rate of potential customer churn; online and offline customer acquisition channels are scattered, data is not unified, and cannot be effectively managed, resulting in a large amount of redundant data; the customers obtained are often diverse customer groups, for example, customer groups of different genders, age groups, and occupations, and different customer groups have different needs for products. The current existing customer acquisition methods cannot recommend products to customers in a targeted manner, and cannot meet customer needs well, resulting in poor customer acquisition efficiency and quality.

[0003] Multimodal data refers to data in different forms or formats obtained from different sources and fields. Multimodal data cannot be processed directly. Customer data belongs to multimodal data. Customer data includes the customer's own attribute data, the customer's online transaction data, and the customer's offline transaction data, that is, the customer's gender, age group, occupation, etc.; the customer's offline store transaction records, offline event participation records such as industry exhibitions, etc.; the customer's shopping records on online platforms such as corporate official websites and mini programs, and public account interaction records. ALBERT-TextCNN is one of the multi-label text classification algorithm model frameworks. This model can be used to effectively process the customer's own attribute data. The customer's offline and online transaction data belongs to the time series data type. Using AlBert-BiLSTM-CRF to "memorize" important information in the customer's past transaction records can effectively follow up on a large number of customer leads in the long term during the customer acquisition process.

[0004] Encoder-Decoder is a model architecture. Encoder encodes the input sequence into a fixed-size text feature vector sequence, and Decoder decodes and outputs the expected result. First, align the data of the same customer, and input the three types of data into three different encoders respectively. Then simply add the text feature vector sequences output by the three to complete data fusion and solve the problem of inconsistent data. The result is then input into the decoder, and the target sequence is finally output, and two types of customer labels are constructed.

[0005] According to the two types of customer tags, a binary tree of k-dimensional points can be constructed, and based on the nearest neighbor search of the k-dimensional tree, a set of goods in the purchase records of similar customers can be specifically recommended to customers. Based on this, the present invention integrates multimodal customer data, constructs customer tags, accurately recommends required goods to customers, and improves the efficiency of acquiring customers.

[0006] In view of the current research and development status in this field, designing a method that can accurately recommend the products required by customers has great research significance and application prospects. Summary of the invention

[0007] In order to solve the existing problems of difficulty in accurately acquiring target customer groups and inability to conduct precision marketing, the present invention proposes a product recommendation method based on AlBert-TextCNN, AlBert-BiLSTM-CRF, Encoder-Decoder and k-dimensional tree nearest neighbor search recommendation. This method combines the currently popular deep learning and other technologies to provide a corresponding theoretical basis for intelligent customer acquisition.

[0008] The technical solution implemented by the present invention is as follows:

[0009] A product recommendation method based on multimodal data fusion is mainly divided into two parts.

[0010] The first part is labeling. The customer information is divided into three parts: customer attribute data set, customer online transaction data set and customer offline transaction data set. For customer attribute data, AlBert-TextCNN is used as Encoder to obtain the text feature vector sequence C1. For customer online transaction data and customer offline transaction data, they are both time series data, so two different AlBert-BiLSTM-CRFs are used as Encoders to obtain text feature vector sequences C2 and C3 respectively. C1, C2 and C3 are simply added to obtain the total text feature vector sequence C. The total text feature vector sequence C is sent to a set of LSTM Decoders for decoding, and the target sequence W = {w1, w2, ..., w n} is used to construct two types of customer labels, namely attribute information labels and frequency preference labels.

[0011] The second part is to recommend products. Based on the customer labels obtained in the first step, the similarity between the target customer and other customers is calculated, and a batch of products that the target customer is most likely to buy are recommended.

[0012] The specific steps of the first part are as follows:

[0013] Step 1: Obtain three customer data set texts, namely, customer attribute data set X, customer online transaction data set Y, and customer offline transaction data set Z; and preprocess each data set text by filling missing values ​​and deleting redundant data.

[0014] Step 2: Divide the preprocessed customer data set texts X, Y, and Z into training sets and test sets.

[0015] Step three, the preprocessed test sets of customer data set texts X, Y, and Z are input into the AlBert layer of the Encoder, which mainly uses a multi-layer bidirectional Transformer module, for serialization operations, and the serialized customer attribute data text vector T1, customer online transaction data text vector T2, and customer offline transaction data text vector T3 containing rich semantics are obtained respectively.

[0016] In step 4, the customer attribute data text vector T1 is trained by an encoder (Encoderl) based on TextCNN, and the customer online transaction data text vector T2 and the customer offline transaction data text vector T3 are trained by two different BiLSTM-CRF-based encoders (Encoder2 and Encoder3), respectively, outputting three text feature vector sequences C1, C2 and C3.

[0017] Step 5: The three text feature vector sequences C1, C2 and C3 are simply added to obtain the total text feature vector sequence C. After training the LSTM-based decoder, the target sequence W = {w1, w2, ..., w n}, used to construct two types of customer labels.

[0018] Step 6: Perform the operations in steps 3 to 5 on the training sets of the preprocessed customer data set texts X, Y, and Z to obtain customer labels obtained through the training sets; record the error between the customer labels obtained by testing the model using the test set and the customer labels obtained by the training set, and modify the model parameters according to the error.

[0019] Step 7: Repeat step 6 until the error generated by the model no longer changes.

[0020] Step 8: After the model training process is completed, the model parameters with the smallest error between the customer labels obtained from the training set and the customer labels obtained from the test set are saved as the final model parameters; and the target sequence W = {w1, w2, ..., w n} to construct two label results for the customer.

[0021] In step 1, the text steps of three customer data sets are as follows:

[0022] 1.1 Get customer attribute data set text: X = [x1, x2, ..., x i , ..., x n ], where x i Represents the i-th word in the text of the customer attribute dataset.

[0023] 1.2 Get the text of the customer online transaction dataset: Y = [y1, y2, ..., y j , ..., y n ], where y j Represents the jth word in the text of the customer's online transaction dataset.

[0024] 1.3 Get the text of the customer offline transaction dataset: Z = [z1, z2, ..., z k , ..., z n ], where z k Represents the kth word in the customer's offline transaction dataset text.

[0025] 1.4 Preprocess the texts of the three customer data sets, namely filling missing values ​​and removing redundant data.

[0026] In step 4, the specific process of obtaining three text feature vector sequences is as follows:

[0027] a. For the customer attribute data text vector T1, the customer attribute data text vector T1 is input into the TextCNN-based encoder for training through steps 1)-5). The specific steps are as follows:

[0028] 1) Before inputting the customer attribute data text vector T1 into TextCNN, a zero padding operation is performed first.

[0029] 2) Then input T1 into the input layer of TextCNN to obtain a matrix of dimension n*d representing the sentence; where n represents the number of words in each sentence, and d represents the number of dimensional vectors of each word.

[0030] 3) Next, we enter the convolution layer. The width of the convolution kernel is consistent with the dimension of the word vector. The height of the convolution kernel (i.e., the window value) is selected between 2 and 5 according to the length of the local word sequence, and an appropriate step size is selected.

[0031] 4) In the pooling layer, TextCNN uses max-pooling to filter out the most important features from the feature vector generated in the sliding window, and obtains the input of a fixed-length fully connected layer based on the output of the variable-length convolutional layer. Dropout is added between the pooling layer and the fully connected layer to prevent overfitting.

[0032] 5) Connect the fully connected layer and use the sigmoid activation function to output the text feature vector sequence C1.

[0033] b. For the customer online transaction data text vector T2 and the customer offline transaction data text vector T3, they are respectively input into two different BiLSTM-CRF-based encoders for training; the specific process is as follows:

[0034] After pre-training of the AlBert layer in step three, we obtain dynamic word vectors T2 and T3 that enhance semantic representation capabilities according to the contextual semantics of the sentence. The obtained dynamic word vectors are input into the BiLSTM layer for bidirectional training to extract the feature information of the vectors. The weights of the feature vectors are then allocated through the attention mechanism to maximize the extraction of key features and ignore irrelevant features. Finally, the CRF layer is used to effectively constrain the dependencies between the predicted labels to obtain the global optimal sequence, and the text feature vector sequences C2 and C3 are output respectively.

[0035] In step 5, the steps of constructing a customer tag are as follows:

[0036] 5.1 Simply add the three text feature vector sequences C1, C2 and C3 to obtain a total text feature vector sequence C.

[0037] 5.2 LSTM-based decoder (Decoder) According to the total text feature vector sequence C and the generation information W = {w1, w2, ..., w t-1}Generate the target sequence w corresponding to time t t This method can ensure that the target sequence W = {w1, w2, ..., w n}Has relatively good accuracy and continuity.

[0038] 5.3 By outputting the target sequence W = {w1, w2, ..., w n}, to construct two types of customer labels, namely attribute information labels and frequency preference labels.

[0039] The specific steps of the second part are as follows:

[0040] Step 9: Perform one-hot encoding on the customer attribute information class labels obtained in the first part to obtain a numerical vector sequence L = {L1, L2, ..., L n}, where L i =[l i1 , l i2 ,…,l im ], i∈[1,n],l ij ∈{0, 1}.

[0041] Step 10: For the recommended user, use the k-dimensional tree similarity operation on the vector obtained in step 9, and search for its nearest neighbor vector by constructing a binary tree of k-dimensional points.

[0042] Step 11: Obtain the products purchased by the customer corresponding to the nearest neighbor vector through the customer transaction data, and obtain a set of recommended products for the recommended user.

[0043] Step 12: Perform frequency coding on the frequency preference class labels obtained in the first part to obtain a frequency value vector sequence F = {F1, F2, ..., F n}, where F i =[f i1 , f i2 , ..., f im ], i∈[1,n], f ij ∈N.

[0044] Step 13: For the recommended customers, use the k-dimensional tree similarity operation on the vector obtained in step 12, and search for its nearest neighbor vector by constructing a binary tree of k-dimensional points.

[0045] Step 14: Obtain the products purchased by the customer corresponding to the nearest neighbor vector through the customer transaction data, and obtain another set of recommended products for the recommended user.

[0046] Step 15: Recommend products based on the two recommended product sets.

[0047] In step 10, the steps of searching for the nearest neighbor vector are as follows:

[0048] 10.1 Select any vector in the vector sequence as the root node;

[0049] 10.2 For L = {L1, L2, ..., L n}, L i =[l i1 , l i2 , ..., l im ], select the one that can make {l tj , j∈[1,m]}, the t value with the largest sequence variance, divides the vector sequence into l tk =0 and ltk = two parts of 1;

[0050] 10.3 Repeat steps 10.1-10.2 until all data in the vector sequence are established on the nodes of the k-dimensional tree; the k-dimensional tree is established.

[0051] 10.4 Starting from the root node, move left or right depending on whether the target is smaller or larger than the current node in the segmentation feature;

[0052] 10.5 Once you move to a leaf node, consider that node as the current best node;

[0053] 10.6 Recursively perform steps 10.7-10.8 for each node passed:

[0054] 10.7 If the node is closer to the target node than the current best node, then the node is considered the current best node;

[0055] 10.8 Check if there is a closer point in the other subtree. If there is, search downward from that node.

[0056] 10.9 After the root node search is completed, the best node obtained is the nearest neighbor vector; the nearest neighbor search is completed.

[0057] In step 13, the steps of searching for the nearest neighbor vector are as follows:

[0058] 13.1 Select any vector in the vector sequence as the root node;

[0059] 13.2 For F = {F1, F2, ..., F n}, F i =[f i1 , f i2 , ..., f im ], select the one that makes {f kj , j∈[1,m]} is the k value with the largest variance of the sequence;

[0060] 13.3 Calculation of {f tj , j∈[1,m]} the median of the sequence Divide the vector sequence into and two parts;

[0061] 13.4 Repeat steps 13.1-13.3 until all data are established on the nodes of the k-dimensional tree; the k-dimensional tree is established.

[0062] 13.5 Starting from the root node, move left or right depending on whether the target is smaller or larger than the current node in the segmentation feature;

[0063] 13.6 Once you move to a leaf node, consider that node as the current best node;

[0064] 13.7 Recursively perform steps 13.8-13.9 for each node passed:

[0065] 13.8 If the node is closer to the target node than the current best node, then the node is considered the current best node;

[0066] 13.9 Check if there is a closer point in the other subtree. If there is, search downward from that node.

[0067] 13.10 After the root node search is completed, the best node obtained is the nearest neighbor vector; the nearest neighbor search is completed.

[0068] Beneficial effects of the present invention:

[0069] (1) The AlBert pre-training model can ensure that the representation ability of word vectors is improved to a certain extent. Compared with other time series prediction models, the entire model can better meet actual needs. It divides customer data into three different data sets, and uses different text feature vector extraction methods for different data sets, which can achieve better training results and obtain a more accurate user label system.

[0070] (2) The AlBert-TextCNN model can better improve the representation ability of word vectors extracted from the customer basic attribute data set and retain the semantic information of customers at different levels to the greatest extent, thereby more effectively improving the classification effect of different customer groups and providing the possibility for accurate product recommendations to customers.

[0071] (3) The AlBert-BiLSTM-CRF model can effectively solve problems that traditional language processing models cannot effectively solve, such as poor polysemy analysis of a word in a text and the inability to handle different contexts of polysemous words. The customer's online and offline transaction data sets can be sorted into keywords that are more in line with the contextual semantics of the text set, thereby constructing customer labels in a more targeted and accurate manner.

[0072] (4) By processing attribute information labels through one-hot encoding and frequency preference labels through frequency encoding, the customer information contained in these labels can be well utilized. Using the k-dimensional tree method to obtain similar customers of the two label categories and then making recommendations based on the purchase history of similar customers can achieve a high recommendation accuracy.

[0073] (5) The present invention gives full play to the advantages of frameworks and models such as AlBert-TextCNN, AlBert-BiLSTM-CRF, Encoder-Decoder and k-dimensional tree, and can ensure the accuracy of predicted data while ensuring efficient training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a training flow chart of the present invention based on AlBert-TextCNN, AlBert-BiLSTM-CRF and Encoder-Decoder model.

[0075] Figure 2 It is a training flow chart of the k-dimensional tree nearest neighbor search recommendation model of the present invention.

[0076] Figure 3 This is a basic model architecture diagram of the encoder based on TextCNN in the present invention.

[0077] Figure 4 This is a basic model architecture diagram of the encoder based on BiLSTM-CRF of the present invention.

[0078] Figure 5 This is a basic architecture diagram of the product recommendation model based on AlBert-TextCNN, AlBert-BiLSTM-CRF and Encoder-Decoder and k-dimensional tree nearest neighbor search recommendation in the present invention.

[0079] Figure 6 It is a timing diagram of the present invention from model creation training to application. DETAILED DESCRIPTION

[0080] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the specific implementation modes of the present invention will be described in detail below with reference to the accompanying drawings.

[0081] The three customer data sets, namely the customer attribute data set, the customer online transaction data set and the customer offline transaction data set, are input into the corresponding models respectively. After training, two customer labels are obtained, which are used to predict the types of products and brands that customers may purchase, so as to achieve the purpose of accurately acquiring customers. Figure 1 As shown, it is a training flow chart of the product recommendation method based on AlBert-TextCNN, AlBert-BiLSTM-CRF, Encoder-Decoder and k-dimensional tree nearest neighbor search recommendation provided by the present invention, which specifically includes the following steps:

[0082] Step 1: Obtain three customer data set texts, namely, customer attribute data set X, customer online transaction data set Y, and customer offline transaction data set Z; and preprocess each data set text by filling missing values ​​and deleting redundant data.

[0083] 1) The customer attribute dataset X should include: age, gender, birthday, marital status, fertility status, whether an employee, whether a member, membership level, membership duration, cumulative consumption amount (including online and offline), maximum and minimum consumption amount (including online and offline), cumulative consumption frequency, average customer unit price, average monthly consumption times, etc.

[0084] 2) The customer online transaction data set Y should include: commonly used electronic devices, average length of stay on WiFi login, number of online promotion participation, proportion of participation in online promotions, etc.

[0085] 3) The customer offline transaction data set Z should include: frequently visited stores, frequently visited time periods, provinces, cities, regions and markets, number of offline promotion participations, and percentage of participation in offline promotions.

[0086] Step 2: divide the preprocessed customer data set texts X, Y and Z into training sets and test sets respectively. Among them, 75% of the customer data set is used as the training set, and the remaining 25% is used as the test set (assuming that the customer range and time span of the customer data set are large enough).

[0087] Step three, the preprocessed test sets of customer data set texts X, Y and Z are input into the AlBert layer for serialization operation, and the serialized customer attribute data text vector T1, customer online transaction data text vector T2 and customer offline transaction data text vector T3 are obtained respectively.

[0088] like Figure 3 and 4 These are the basic model architecture diagrams of the TextCNN-based encoder and the BiLSTM-CRF-based encoder respectively.

[0089] Step 4: The customer attribute data text vector T1 is trained by an encoder (Encoder1) based on TextCNN, and the customer online transaction data text vector T2 and the customer offline transaction data text vector T3 are trained by two different BiLSTM-CRF-based encoders (Encoder2 and Encoder3), and three text feature vector sequences C1, C2 and C3 are output respectively. Among them, the d value of the input layer in TextCNN is 5, the convolution kernel height is 2, 3 or 4, the step size is 2, and only one fully connected layer is used, which greatly reduces the number of parameters while maintaining the main features. In BiLSTM-CRF, the hidden vector obtained by the forward LSTM and backward LSTM is converted into a fixed-length text vector, and the fixed-length text vector is weighted and checked in the attention layer to obtain the optimal input for the CRF layer; finally, in the CRF layer, the dependency between the predicted labels is effectively constrained to obtain the global optimal sequence.

[0090] Step 5: The three text feature vector sequences C1, C2 and C3 are simply added to obtain the total text feature vector sequence C. After training the LSTM-based decoder, the target sequence W = {w1, w2, ..., w n}, used to construct two types of customer labels.

[0091] Step 6: Perform the operations in steps 3 to 5 on the training sets of the preprocessed customer data set texts X, Y, and Z to obtain customer labels obtained through the training sets; record the error between the customer labels obtained by testing the model using the test set and the customer labels obtained by the training set, and modify the model parameters according to the error.

[0092] Step 7: Repeat step 6 until the error generated by the model no longer changes.

[0093] Step 8: After the model training process is completed, the model parameters with the smallest error between the customer labels obtained from the training set and the customer labels obtained from the test set are saved as the final model parameters; and the target sequence W = {w1, w2, ..., w n} to construct two label results for the customer.

[0094] 1) Customer information tags: gender (male, female), age group (young, middle-aged, old), membership level (high, medium, low), activity level (high, medium, low, dormant, lost), spending power (strong, medium, low), consumption frequency (more, less), consumption cycle (long, medium, short), promotion sensitivity (high, low), value contribution (important, general, unimportant), etc.

[0095] 2) Customer preference tags: purchase channel preference (online, offline), payment method preference (Alipay, WeChat, cash, bank card), purchase time preference (early morning, morning, morning, noon, afternoon, evening, late night), purchase device preference (mobile phone, tablet, computer), category preference (men's clothing, women's clothing, entertainment, daily necessities, etc.), brand preference, etc.

[0096] Step nine: One-hot encode the customer information labels to obtain a numerical vector of the form [1, 0, 0, 1, ...]. The number of vectors is the number of customers, and the number of features (elements) of the vector is the sum of the number of values ​​of each label.

[0097] Step 10: For the recommended customers, use the k-dimensional tree similarity operation on the vector obtained in step 9, and search for its nearest neighbor vector by constructing a binary tree of k-dimensional points.

[0098] Step 11: Obtain the products purchased by the customer corresponding to the nearest neighbor vector through customer transaction data, and obtain the recommended product set A.

[0099] Step 12: Perform frequency encoding on the customer preference labels obtained in the first part, convert the label values ​​into numerical values, and convert the label vector into a numerical vector representing the frequency.

[0100] Step 13: For the recommended customers, use the k-dimensional tree similarity operation on the vector obtained in step 12, and search for its nearest neighbor vector by constructing a binary tree of k-dimensional points.

[0101] Step 14: Obtain the products purchased by the customer corresponding to the nearest neighbor vector through customer transaction data, and obtain the recommended product set B.

[0102] Step 15: Let the set of products purchased by the recommended customer be C, and recommend products in the order of recommended product set (A∩B)-C, set C, set BAC, and set ABC.

Claims

1. A commodity recommendation method based on multimodal data fusion, characterized in that: The method comprises the following steps: Step 1: Obtain three types of customer data set texts, namely, customer attribute data set X, customer online transaction data set Y, and customer offline transaction data set Z; and preprocess the text of each data set by filling missing values ​​and deleting redundant data; Step 2: divide the preprocessed customer data set texts X, Y and Z into training sets and test sets; Step 3: Input the preprocessed test sets of customer data set texts X, Y, and Z into the AlBert layer of the Encoder that uses a multi-layer bidirectional Transformer module for serialization, and obtain the serialized customer attribute data text vector T1, customer online transaction data text vector T2, and customer offline transaction data text vector T3 containing rich semantics respectively; Step 4: The customer attribute data text vector T1 is trained with an encoder based on TextCNN, and the customer online transaction data text vector T2 and the customer offline transaction data text vector T3 are trained with two different BiLSTM-CRF-based encoders, respectively, to output three text feature vector sequences C1, C2, and C3; Step 5: The three text feature vector sequences C1, C2 and C3 are simply added to obtain the total text feature vector sequence C. After training the LSTM-based decoder, the target sequence W = {w1, w2, ..., w n }, used to construct two types of customer labels: attribute information labels and frequency preference labels; Step 6: Perform the operations in steps 3 to 5 on the training sets of the preprocessed customer data set texts X, Y, and Z to obtain customer labels obtained through the training sets; record the error between the customer labels obtained by testing the model using the test set and the customer labels obtained by the training set, and modify the model parameters according to the error; Step 7: Repeat step 6 until the error generated by the model no longer changes; Step 8: After the model training process is completed, the model parameters with the smallest error between the customer labels obtained from the training set and the customer labels obtained from the test set are saved as the final model parameters; and the target sequence W = {w1, w2, ..., w n } to construct two label results for customers; Step 9: Perform one-hot encoding on the obtained customer attribute information class labels to obtain a numerical vector sequence L = {L1, L2, ..., L n }, where L i =[l i1 ,l i2 ,...,l im ],i∈[1,n],l ij ∈{0,1}; Step 10: For the recommended user, use the k-dimensional tree similarity operation on the vector obtained in step 9, and search for its nearest neighbor vector by constructing a binary tree of k-dimensional points; Step 11: Obtain the products purchased by the customer corresponding to the nearest neighbor vector through the customer transaction data, and obtain a set of recommended products for the recommended user; Step 12: Perform frequency coding on the obtained frequency preference class labels to obtain a frequency value vector sequence F = {F1, F2, ..., F n }, where F i =[f i1 ,f i2 ,...,f im ],i∈[1,n],f ij ∈N; Step 13: For the recommended customers, use the k-dimensional tree similarity operation on the vector obtained in step 12, and search for its nearest neighbor vector by constructing a binary tree of k-dimensional points; Step 14: Obtain the products purchased by the customer corresponding to the nearest neighbor vector through the customer transaction data, and obtain another set of recommended products for the recommended user; Step 15: Recommend products based on the two recommended product sets.

2. The commodity recommendation method based on multimodal data fusion according to claim 1, characterized in that: In step 1, the text steps of three customer data sets are as follows: 1.1 Get customer attribute data set text: X = [x1, x2, ..., x i ,...,x n ], where x i Represents the i-th word in the text of the customer attribute dataset; 1.2 Get the text of the customer online transaction data set: Y = [y1, y2, ..., y j ,...,y n ], where y j Represents the jth word in the text of the customer's online transaction dataset; 1.3 Get the text of the customer offline transaction dataset: Z = [z1, z2, ..., z k ,...,z n ], where z k Represents the kth word in the customer's offline transaction dataset text; 1.4 Preprocess the texts of the three customer data sets, namely filling missing values ​​and removing redundant data.

3. A commodity recommendation method based on multimodal data fusion according to claim 1 or 2, characterized in that: The specific process of step 4 is as follows: a. For the customer attribute data text vector T1, the customer attribute data text vector T1 is input into the TextCNN-based encoder for training through steps 1)-5). The specific steps are as follows: 1) Before inputting the customer attribute data text vector T1 into TextCNN, a zero padding operation is performed first; 2) Then input T1 into the input layer of TextCNN to obtain a matrix representing the sentence with dimension n*d, where n represents the number of words in each sentence and d represents the number of dimensional vectors converted from each word. 3) Next, we enter the convolution layer. The width of the convolution kernel is consistent with the dimension of the word vector. The height of the convolution kernel is selected between 2 and 5 according to the length of the local word sequence, and the appropriate step size is selected. 4) In the pooling layer, TextCNN uses the maximum pooling Max-pool to filter out the most important features from the feature vector generated in the sliding window, and obtain a fixed-length input for the fully connected layer based on the output of the variable-length convolution layer. Dropout is added between the pooling layer and the fully connected layer to prevent overfitting; 5) Connect the fully connected layer and use the sigmoid activation function to output the text feature vector sequence C1; b. For the customer online transaction data text vector T2 and the customer offline transaction data text vector T3, they are respectively input into two different BiLSTM-CRF-based encoders for training; the specific process is as follows: After pre-training of the AlBert layer in step three, we obtain dynamic word vectors T2 and T3 that enhance semantic representation capabilities according to the contextual semantics of the sentence. The obtained dynamic word vectors are input into the BiLSTM layer for bidirectional training to extract the feature information of the vectors. The weights of the feature vectors are then allocated through the attention mechanism to maximize the extraction of key features and ignore irrelevant features. Finally, the CRF layer is used to effectively constrain the dependencies between the predicted labels to obtain the global optimal sequence, and the text feature vector sequences C2 and C3 are output respectively.

4. A commodity recommendation method based on multimodal data fusion according to claim 1 or 2, characterized in that: In step 5, the steps of constructing a customer tag are as follows: 5.1 Simply add the three text feature vector sequences C1, C2 and C3 to obtain a total text feature vector sequence C; 5.2 The LSTM-based decoder is based on the total text feature vector sequence C and the generation information W = {w1,w2,...,w t-1 }Generate the target sequence w corresponding to time t t This method can ensure that the target sequence W = {w1,w2,...,w n }Has relatively good accuracy and continuity; 5.3 By outputting the target sequence W = {w1,w2,...,w n }, to construct two types of customer labels.

5. The commodity recommendation method based on multimodal data fusion according to claim 3, characterized in that: In step 5, the steps of constructing a customer tag are as follows: 5.1 Simply add the three text feature vector sequences C1, C2 and C3 to obtain a total text feature vector sequence C; 5.2 The LSTM-based decoder is based on the total text feature vector sequence C and the generation information W = {w1,w2,...,w t-1 }Generate the target sequence w corresponding to time t t This method can ensure that the target sequence W = {w1,w2,...,w n }Has relatively good accuracy and continuity; 5.3 By outputting the target sequence W = {w1,w2,...,w n }, to construct two types of customer labels.

6. A commodity recommendation method based on multimodal data fusion according to claim 1, 2 or 5, characterized in that: In step 10, the steps of searching for the nearest neighbor vector are as follows: 10.1 Select any vector in the vector sequence as the root node; 10.2 For L = {L1, L2, ..., L n }, L i =[l i1 ,l i2 ,...,l im ], select the one that can make {l tj ,j∈[1,m]}, the t value with the largest sequence variance, divides the vector sequence into l tk =0 and l tk = two parts of 1; 10.3 Repeat steps 10.1-10.2 until all data in the vector sequence are built on the nodes of the k-dimensional tree; The k-dimensional tree is established; 10.4 Starting from the root node, move left or right depending on whether the target is smaller or larger than the current node in the segmentation feature; 10.5 Once you move to a leaf node, consider that node as the current best node; 10.6 Recursively perform steps 10.7-10.8 for each node passed: 10.7 If the node is closer to the target node than the current best node, then the node is considered the current best node; 10.8 Check if there is a closer point in the other subtree, if so, search downward from that node; 10.9 After the root node search is completed, the best node obtained is the nearest neighbor vector; the nearest neighbor search is completed.

7. A commodity recommendation method based on multimodal data fusion according to claim 1, 2 or 5, characterized in that: In step 13, the steps of searching for the nearest neighbor vector are as follows: 13.1 Select any vector in the vector sequence as the root node; 13.2 For F = {F1, F2, ..., F n }, F i =[f i1 ,f i2 ,...,f im ], select the one that makes {f kj ,j∈[1,m]}, the k value with the largest sequence variance; 13.3 Calculation of {f tj ,j∈[1,m]} the median of the sequence Divide the vector sequence into and two parts; 13.4 Repeat steps 13.1-13.3 until all data are built on the nodes of the k-dimensional tree; The k-dimensional tree is established; 13.5 Starting from the root node, move left or right depending on whether the target is smaller or larger than the current node in the segmentation feature; 13.6 Once you move to a leaf node, consider that node as the current best node; 13.7 Recursively perform steps 13.8-13.9 for each node passed: 13.8 If the node is closer to the target node than the current best node, then the node is considered the current best node; 13.9 Check if there is a closer point in the other subtree. If so, search downward from that node; 13.10 After the root node search is completed, the best node obtained is the nearest neighbor vector; the nearest neighbor search is completed.

Images