User classification model training method and device, equipment and medium

Abstract

The invention discloses a user classification model training method, device and system and a medium, and the method comprises the steps: collecting key information related to a user, and converting the key information into a comprehensive feature vector; dialogue text data of a user are obtained, the dialogue text data are input into the improved BigBird model, and the dialogue text data are converted into semantic text vectors through the BigBird model; training the model according to the semantic text vector and the comprehensive feature vector to obtain a user classification model; and obtaining an input text and key information of a target user, performing classification identification on the target user through the user classification model according to the input text and key information of the target user, and outputting a classification result. According to the method, the BigBird model is used for expanding the capacity for the long-sequence text, cross-modal feature fusion is carried out on the structured data and the unstructured data, the user classification model is trained, the long-term dependency relationship is effectively captured, and the user classification accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and is applied to dialogue scenarios such as finance or medicine, and in particular to a user classification model training method, device, equipment and medium. Background Art

[0002] In current conversation scenarios such as finance or medical care, it is necessary to distinguish whether a customer is a potential user or a non-potential user. When determining whether a customer is a potential user, the conversation text needs to be accurately analyzed and identified. However, current models such as ROBERTa and BERT are limited to 512 characters and cannot process thousands of characters, so some semantics will be lost, making the potential customer classification results inaccurate. In addition, although the existing technology collects a large amount of data and a wide range of data, the utilization rate of various types of data is low, and there is a lack of systematic and refined methods to extract the intrinsic information in the data, which makes it difficult to accurately identify the type of customer. Summary of the invention

[0003] The purpose of the embodiments of the present invention is to propose a user classification model training method, device, equipment and medium to solve the problem that the prior art has difficulty in processing long sequence semantic texts, resulting in loss of part of the semantics and inaccurate user classification.

[0004] In order to solve the above technical problems, the embodiment of the present invention provides a user classification model training method, which adopts the following technical solution:

[0005] A user classification model training method comprises the following steps:

[0006] Collect key information related to the user, and convert the key information into a comprehensive feature vector;

[0007] Acquire the user's conversation text data, input the conversation text data into the improved BigBird model, and convert the conversation text data into a semantic text vector by the BigBird model;

[0008] The model is trained according to the semantic text vector and the comprehensive feature vector to obtain a user classification model;

[0009] The input text and key information of the target user are obtained, and the target user is classified and identified through a user classification model according to the input text and key information of the target user, and a classification result is output.

[0010] In one embodiment, collecting key information related to the user and converting the key information into a comprehensive feature vector includes:

[0011] Collect key information of the user and convert the key information into a feature vector;

[0012] constructing a query vector, a key vector and a value vector according to the feature vector;

[0013] Calculate the attention score matrix of the query vector and the key vector through the dot product attention function, and convert the attention score matrix into an attention weight matrix using the soft max function;

[0014] Performing weighted summation on the attention weight matrix and the value vector to obtain a weighted feature representation of each element;

[0015] The weighted feature representation of each element is fused to obtain a comprehensive feature vector.

[0016] In one embodiment, the acquiring of the user's conversation text data, inputting the conversation text data into the improved BigBird model, and converting the conversation text data into a semantic text vector by the BigBird model includes:

[0017] Customize the attention mechanism for the BigBird model to obtain an improved BigBird model;

[0018] Acquire the user's conversation text data, and pre-process the conversation text data;

[0019] Input the preprocessed dialogue text into the improved Big Bird model;

[0020] The improved BigBird model receives the conversation text and processes it to obtain a semantic text vector.

[0021] In one embodiment, the training of the model according to the semantic text vector and the comprehensive feature vector includes:

[0022] Concatenate the comprehensive feature vector with the semantic text vector to obtain a concatenated vector;

[0023] The concatenated vector is input into the classification model for training.

[0024] In one embodiment, the training of the model according to the semantic text vector and the comprehensive feature vector to obtain the user classification model includes:

[0025] Predefine tags for user classification;

[0026] The user classification model generates a prediction result based on the concatenated vector and the classification label;

[0027] The difference between the predicted result and the true label is calculated through the loss function;

[0028] The parameters of the model are updated according to the difference value until the model converges to obtain a user classification model.

[0029] In one embodiment, the user classification model generates a prediction result according to the concatenated vector and the classification label, including:

[0030] Segmenting the spliced ​​vector to generate a plurality of vector segments;

[0031] Encode each vector segment to obtain the feature encoding and hidden state of each vector segment;

[0032] Calculate the relative positions between the elements in each vector segment to obtain the relative position encoding;

[0033] Combining the feature encoding, relative position encoding and hidden state to generate a feature representation;

[0034] Generate prediction results based on the feature representation and the classification label.

[0035] In one embodiment, the target user is classified and identified by a user classification model according to the input text and key information of the target user, and the classification result is output, including:

[0036] Obtaining input text from a target user, and converting the input text into a semantic text vector;

[0037] Acquire key information of target users, and convert the key information into a comprehensive feature vector;

[0038] Concatenate the comprehensive feature vector with the semantic text vector to obtain a concatenated vector, and input the concatenated vector into a user classification model;

[0039] The user classification model receives the concatenated vector to perform label prediction and obtains the probability distribution of each category label;

[0040] The category of the target user is determined according to the probability distribution.

[0041] In order to solve the above technical problems, the embodiment of the present invention also provides a user classification model training device, which adopts the following technical solution:

[0042] A user classification model training device, comprising:

[0043] A comprehensive feature vector module, used to collect key information related to the user and convert the key information into a comprehensive feature vector;

[0044] A semantic text vector module, used for acquiring the user's conversation text data, inputting the conversation text data into the improved BigBird model, and converting the conversation text data into a semantic text vector by the BigBird model;

[0045] A model training module is used to train the model according to the semantic text vector and the comprehensive feature vector to obtain a user classification model;

[0046] The user classification module is used to obtain the input text and key information of the target user, classify and identify the target user through the user classification model according to the input text and key information of the target user, and output the classification result.

[0047] In order to solve the above technical problems, an embodiment of the present invention further provides a computer device, which adopts the following technical solution:

[0048] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the user classification model training method described above when executing the computer-readable instructions.

[0049] In order to solve the above technical problems, an embodiment of the present invention further provides a computer-readable storage medium, which adopts the following technical solution:

[0050] A computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement the steps of the user classification model training method as described above.

[0051] Compared with the prior art, the embodiments of the present invention mainly have the following beneficial effects:

[0052] The user classification model training method described in the embodiment of the present invention collects key information related to the user and converts the key information into a comprehensive feature vector; obtains the user's conversation text data, inputs the conversation text data into an improved BigBird model, and the BigBird model converts the conversation text data into a semantic text vector; trains the model according to the semantic text vector and the comprehensive feature vector to obtain a user classification model; obtains the input text and key information of the target user, and classifies and identifies the target user according to the input text and key information of the target user through the user classification model, and outputs the classification result. The present invention uses the BigBird model to expand the ability of long sequence text, and trains the user classification model by cross-modal feature fusion of structured data and unstructured data, effectively captures long-term dependencies, and improves the accuracy of user classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the scheme in the present invention, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 is an exemplary system architecture diagram in which the present invention can be applied;

[0055] Figure 2 is a flow chart of an embodiment of a user classification model training method according to the present invention;

[0056] Figure 3 It is a schematic diagram of a graph sparsification method of a BigBird model sparse self-attention mechanism of a user classification model training method invented;

[0057] Figure 4 is a structural schematic diagram of an embodiment of a user classification model training device according to the present invention;

[0058] Figure 5 FIG. 1 is a schematic structural diagram of an embodiment of a computer device according to the present invention. DETAILED DESCRIPTION

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present invention; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present invention or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0060] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0061] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0062] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0063] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0064] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) players, laptop computers and desktop computers, etc.

[0065] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0066] It should be noted that the user classification model training method provided in the embodiment of the present invention is generally executed by a server / terminal device, and accordingly, the user classification model training device is generally arranged in the server / terminal device.

[0067] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0068] Continue to refer Figure 2 , shows a flow chart of an embodiment of a user classification model training method according to the present invention. The user classification model training method comprises the following steps:

[0069] S1. Collect key information related to the user and convert the key information into a comprehensive feature vector;

[0070] S2, obtaining the user's conversation text data, inputting the conversation text data into the improved BigBird model, and converting the conversation text data into a semantic text vector by the BigBird model;

[0071] S3, training the model according to the semantic text vector and the comprehensive feature vector to obtain a user classification model;

[0072] S4. Obtain the input text and key information of the target user, classify and identify the target user through a user classification model according to the input text and key information of the target user, and output a classification result.

[0073] In this embodiment, key information related to the user is collected, including basic information of the pet owner and the pet, consumption behavior, and market feedback. These key information are converted into feature vectors and fused into a comprehensive feature vector through the Self-Attention mechanism. The user's conversation text data is obtained and input into the BigBird model. This model specializes in processing long text data and converts the conversation text into semantic text vectors. These semantic text vectors can capture long-distance dependencies in the text. The comprehensive feature vector obtained by feature fusion is spliced ​​with the semantic text vector generated by the BigBird model. The spliced ​​vector combines the features of structured data with the semantic information of unstructured text data. The spliced ​​vector is used as input and input into the Transformer-XL model for training to obtain a user classification model.

[0074] In a specific embodiment, by combining structured data (such as basic customer information, pet information, consumption behavior, market feedback) with unstructured text conversation data, a deep fusion of cross-modal features is achieved. This fusion method can more comprehensively understand the needs and preferences of pet owners and provide a richer and more accurate information basis for personalized recommendations.

[0075] The BigBird model shows significant advantages in processing long sequences of text and can efficiently capture long-distance dependencies in text. Applied to the representation of text conversation data, the model can break through the computational bottleneck encountered by traditional methods when processing large amounts of conversation history, thereby improving the efficiency and accuracy of the model.

[0076] The Transformer-XL model introduces relative position encoding and memory mechanisms, which enables it to process longer contexts without increasing computational complexity. Using this model to further extract features from the text representation output by BigBird can deepen the model's understanding of long-term dependencies and thus improve the accuracy of personalized recommendations.

[0077] The self-attention mechanism enables the model to automatically learn the correlation between parts of the input sequence, which is crucial for achieving deep understanding and generating personalized recommendations. This mechanism enables the model to dynamically focus on the most relevant features, breaking the limitation of processing information only in a fixed order.

[0078] In one embodiment, collecting key information related to the user and converting the key information into a comprehensive feature vector includes:

[0079] S11. Collect key information of the user and convert the key information into a feature vector.

[0080] In this embodiment, the collection of key information covers the basic information of pet owners and pets, consumption behavior and market feedback. Specifically, the collected pet owner information includes basic attributes such as age, gender, occupation and income level. At the same time, the collection of pet information focuses on details such as species, age, health status and past medical history. The record of consumption behavior involves the consumption habits, insurance purchase history and personal preferences of pet owners. The collection of market feedback aims to understand the attitude, demand and feedback of pet owners on insurance products. These collected key information are then converted into feature vectors. This process includes applying One-Hot Encoding or Embedding technology to convert key information data into vectors of fixed length, and using numerical or embedding technology to represent continuous or count data. The generated feature vector will cover aspects such as the basic information of pet owners, pet information, consumption behavior and market feedback. In addition, the vectorization processing of feature vectors uses methods such as bag of words model, TF-IDF, Word2Vec, etc. to effectively convert non-numerical data into numerical feature vectors that the model can process.

[0081] S12, constructing a query vector, a key vector and a value vector according to the feature vector;

[0082] S13, calculating the attention score matrix of the query vector and the key vector by using the dot product attention function, and converting the attention score matrix into an attention weight matrix by using the softmax function;

[0083] S14, performing weighted summation on the attention weight matrix and the value vector to obtain a weighted feature representation of each element;

[0084] S15. Fusing the weighted feature representation of each element to obtain a comprehensive feature vector.

[0085] In this embodiment, the query vector Q (Query), key vector K (Key) and value vector V (Value) are constructed based on the basic information vector of the pet owner, the pet information vector, the consumption behavior vector and the market feedback vector. Query (Q) represents the current feature vector of the pet owner or pet information. Key (K) collects the feature vectors of all pet owners or pet information to form a set. Correspondingly, Value (V) also contains the feature vector set of all pet owners or pet information.

[0086] First, the dot product between Query (Q) and Key (K) is calculated using the dot product attention function to obtain the attention score matrix. Subsequently, the score matrix is ​​normalized by the softmax function and converted into an attention weight matrix. Each element in the weight matrix represents the attention weight of the corresponding element. Next, a matrix multiplication operation is performed to multiply the attention weight matrix with the Value (V) matrix. The Value matrix consists of the Value vectors of all elements, where each row represents the Value vector of an element. After the multiplication operation is completed, the weighted Value matrix is ​​obtained. Finally, the weighted matrix is ​​summed, which is usually performed along the rows or columns of the matrix. The summation operation involves accumulating the weighted Value vectors. The weighted feature representation of each element is obtained.

[0087] In order to capture information in different subspaces, multiple self-attention operations can be performed in parallel, each time applying different linear transformation parameters. The results of these independent operations are then concatenated and further processed through a linear layer to obtain the final weighted feature representation. This process aggregates all weighted feature representations to form a comprehensive feature vector. The advantage of the self-attention mechanism is that it can automatically learn the correlation between different features in the input sequence and highlight the importance of these features through attention weights. The calculation of attention weights allows the model to dynamically focus on the most relevant features, enhancing the model's understanding and processing capabilities of the data.

[0088] In one embodiment, the acquiring of the user's conversation text data, inputting the conversation text data into the improved BigBird model, and converting the conversation text data into a semantic text vector by the BigBird model includes:

[0089] S21. Customize the attention mechanism for the BigBird model to obtain an improved BigBird model.

[0090] In this embodiment, if Figure 3As shown in the figure, the BigBird model incorporates the concept of graph sparsification and replaces the Transformer full attention mechanism in the BERT model with a sparse self-attention mechanism. The computational complexity of this sparse self-attention mechanism is linearly related to the sequence length, which significantly improves the model's ability to process long sequence data.

[0091] BigBird's sparse self-attention mechanism mainly consists of three components:

[0092] 1. The global token set G is a fixed set of tokens in the model that participate in the calculation in all parts of the sequence. This means that no matter where in the sequence, these global tokens will perform self-attention calculations with other tokens.

[0093] 2. Random key vectors For each query vector \(q_i\), the model randomly selects r key vectors from all possible key vectors to participate in the calculation. This random selection can reduce the amount of calculation because not all key vectors are involved in the calculation of each query vector.

[0094] 3. Local sliding window w In order to capture local context information, the model defines a sliding window with a width of w. In the sequence, each query vector only performs self-attention calculations with its neighbors within the window. This can limit the number of key vectors that each query vector needs to consider, thereby reducing the computational complexity.

[0095] The calculation formula for the i-th vector of the self-attention mechanism is as follows:

[0096]

[0097] Where X is the input sequence; x i is the i-th vector in the sequence; Q h , K h and V h are query function, key function and value function respectively; σ is the softmax function; H is the number of heads of multi-head attention; N(i) represents the set of neighbor nodes sent out by the i-th node; x N(i) Corresponding to the neighbor node set {x j :j∈N(i)}, for example, first define an adjacency matrix A∈[0,1] belonging to graph D n×n , where for the (i, j)th element, if the query function q i and key function k j Participates in the self-attention calculation, then a ij=1, otherwise 0. If it is a full attention mechanism, the adjacency matrix A is a full 1 matrix, that is, all words perform self-attention calculations with other words, and the calculation complexity is the square of the sequence length. By considering the attention mechanism as a fully connected graph, we can combine the idea of ​​graph sparsification to reduce the quadratic complexity to linear complexity.

[0098] In a specific embodiment, refer to Figure 3 There are four methods for graph sparseness:

[0099] a. Random graph sparsification method, according to the simplest random graph structure - ER random graph, each edge is randomly selected independently with a fixed probability, that is, r key value vectors are randomly selected for each query vector for self-attention calculation, that is, words and word i are randomly selected for self-attention operation, expressed as formula A(i, ) = 1;

[0100] b. Sliding window graph sparsification method. Many articles show that the correlation between adjacent words and sentences is relatively large, so a word can obtain a lot of information from its neighboring words. Previous studies on the self-attention mechanism in NLP tasks have shown that the attention calculation of adjacent words is very important. Referring to the terminology of graph theory, the clustering coefficient is a measure of the locality of connectivity. When the graph contains many clusters or neighbors (almost completely interconnected subgraphs), the clustering coefficient is very high. In this model, in order to capture the local structure of this context, the author defines a sliding window with a width of w. For the query vector at position i, there will be w key-value vectors to perform attention calculation with it, expressed as the formula A(i,iw / 2:i+w / 2)=1;

[0101] c. Global attention graph sparse method. Select the vocabulary that already exists in the sequence as the global vocabulary, that is, select a subset G, where for each i∈G, A(i,:)=1 and A(:,i)=1;

[0102] d. BigBird graph sparsification method. The BigBird graph sparsification method is a combination of random graph sparsification method, sliding window graph sparsification method and global attention graph sparsification method. It has the following characteristics:

[0103] 1. Random graph sparsification method This method constructs a graph by randomly selecting edges, that is, randomly selecting r key vectors to perform self-attention calculation with each query vector.

[0104] 2. Sliding Window Graph Sparsification Method This method defines a sliding window with a width of w, so that each query vector only performs self-attention calculations with its neighbors within the window.

[0105] 3. Global Attention Graph Sparsification Method This method selects a set of global tokens so that these tokens participate in the calculation in all parts of the sequence.

[0106] 4. BigBird graph sparsification method This is a combination of the above three methods. It allows each query vector to randomly perform attention calculations with r key vectors, and at the same time perform attention calculations with its left and right neighbors w / 2, and include g global tokens.

[0107] Through these methods, the BigBird model can effectively process long sequence data while keeping the computational complexity low. This sparse self-attention mechanism allows the model to focus on the most important parts of the sequence instead of performing comprehensive self-attention calculations on each token.

[0108] S22: Acquire the user's conversation text data, and pre-process the conversation text data.

[0109] In this embodiment, the user's conversation text data is obtained. These text data need to be converted into a format that the model can understand, usually by using a word segmenter and a word embedding pre-trained by the model to represent it. The conversation text data is cleaned, segmented, encoded, and other pre-processing steps are performed to convert it into the format required for model input.

[0110] Preprocessing includes:

[0111] Cleaning: Remove irrelevant characters, punctuation marks, stop words, etc. from the text.

[0112] Tokenization: Splitting text into individual words or tokens.

[0113] Normalize: For example, convert all text to lowercase to eliminate differences in case.

[0114] Encoding: Converting text into a numerical form that the model can understand, such as word indices or word embeddings.

[0115] Serialization: If your model is sequence-based, you may need to convert the text into fixed-length sequences.

[0116] Add special tags: For example, in the BERT model, you may need to add special tags such as [CLS] and [SEP].

[0117] Labeling: Convert the results or categories corresponding to the text into labels, which are used for supervised learning during training.

[0118] S23, inputting the preprocessed dialogue text into the improved Big Bird model;

[0119] S24. The improved BigBird model receives the conversation text and processes it to obtain a semantic text vector.

[0120] In this embodiment, the preprocessed conversation text is fed into the improved BigBird model as input. The BigBird model is a Transformer architecture model designed specifically for processing long texts. The improved BigBird model receives the conversation text and uses its sparse attention mechanism to efficiently process long sequence texts. The model captures the semantic information and long-distance dependencies in the text through self-attention layers and improveable mechanisms (such as Random attention, Window attention, Global attention). After being processed by the model, each input sequence or each element in the sequence will be converted into a semantic text vector. These vectors capture the deep semantic features of the text and can be used for further analysis or as input for downstream tasks. The obtained semantic text vector can be combined with other feature vectors (such as user feature vectors) to train a classification model. The BigBird model can effectively convert unstructured conversation text into structured semantic representations, providing a powerful tool for understanding and processing complex text data.

[0121] In one embodiment, the training of the model according to the semantic text vector and the comprehensive feature vector includes:

[0122] S31, concatenating the comprehensive feature vector with the semantic text vector to obtain a concatenated vector;

[0123] S32, inputting the concatenated vector into a classification model for training.

[0124] In this embodiment, the comprehensive feature vector and the semantic text vector are concatenated to form a longer concatenated vector, which combines the information of structured data and unstructured text data. The concatenated vector is used as input data and provided to the classification model. The classification model is trained using the input concatenated vector and the corresponding label (such as "is a potential customer" or "is not a potential customer"). During the training process, the model learns how to predict the category of the customer based on the input feature vector. During the training process, the model parameters are adjusted by optimizing the loss function, and the model performance is evaluated through the validation set to ensure that the model does not overfit and has good generalization ability.

[0125] In one embodiment, the training of the model according to the semantic text vector and the comprehensive feature vector to obtain the user classification model includes:

[0126] S33, pre-define user classification labels;

[0127] S34, the user classification model generates a prediction result according to the concatenated vector and the classification label;

[0128] S35, calculating the difference between the predicted result and the true label through the loss function;

[0129] S36. Update the parameters of the model according to the difference value until the model converges to obtain a user classification model.

[0130] In this embodiment, before training the classification model, it is necessary to predefine possible user classification labels. The true label is the correct answer provided in the training data for supervised learning. For example, in the pet insurance potential customer classification task, the label may be "a potential customer" or "not a potential customer". The model generates a prediction result based on the input concatenated vector (including a comprehensive feature vector and a semantic text vector), that is, the probability distribution of each category. The loss function measures the difference between the model prediction output and the true label. For classification problems, commonly used loss functions include cross-entropy loss. For each sample, the loss function calculates a loss value, which represents the degree of error between the predicted output and the true label. The goal of model training is to minimize this loss value. Use an optimization algorithm (such as gradient descent or its variants) to adjust the parameters of the model according to the gradient of the loss function. This process is called backpropagation. Through multiple iterative training processes, the model parameters are continuously optimized until the performance of the model on the training set reaches a satisfactory level, and the generalization ability of the model is monitored by the validation set. The model is trained by minimizing the difference between the predicted results and the true labels so that it can accurately classify and predict new and unseen data.

[0131] In one embodiment, the user classification model generates a prediction result according to the concatenated vector and the classification label, including:

[0132] S341, segmenting the spliced ​​vector to generate a plurality of vector segments;

[0133] S342. Encode each vector segment to obtain a feature code and a hidden state of each vector segment.

[0134] In this embodiment, since the sequence length that Transformer-XL can process is limited, it is necessary to divide these long sequences into multiple shorter, non-overlapping vector segments so that the model can process them effectively. For each segmented vector segment, the Transformer-XL model encodes it independently while retaining part of the hidden state of the previous vector segment as memory, so that the information flow and long-term dependencies across segments can be captured. Transformer-XL introduces a memory mechanism that allows the model to retain some hidden states of the previous segment and pass them as "memory" to the processing of the next vector segment. Among them, not all hidden states are passed to the next segment. Usually only a part of the important hidden states is selectively retained. This helps to reduce the amount of calculation and maintain the relevance of information. And these vector segments are cut continuously and do not overlap, ensuring that each vector segment can be processed independently, while retaining the continuity of the sequence through the memory mechanism. In this way, the model can capture long-term dependencies across multiple segments in the sequence. Even in different segments of the sequence, information can flow and maintain.

[0135] S343, calculating the relative positions between the elements in each vector segment, and obtaining a relative position code;

[0136] S344, combining the feature code, relative position code and hidden state to generate a feature representation;

[0137] S345: Generate a prediction result according to the feature representation and classification label.

[0138] In this example, in the standard Transformer model, absolute position encoding is used. This encoding method assigns a unique position identifier to each element in the sequence, which is usually based on the absolute position of the element in the sequence (i.e., an index starting from 1). However, relative position encoding is introduced in Transformer-XL, which is based on the relative distance between elements in the sequence segment, rather than their absolute position. This means that the position encoding reflects the positional relationship of an element relative to other elements.

[0139] Since relative position encoding considers the relative distance between elements, it allows the model to process sequences longer than a single segment. Even in different segments of the sequence, the position encoding of the elements is able to reflect the relative order between them. When the sequence is split into multiple segments, relative position encoding enables the model to capture dependencies across segments. Because the encoding contains the relative position information between elements, the model can understand the order relationship of elements between different segments. Relative position encoding combined with the memory mechanism of Transformer-XL allows the model to utilize the memory state of the previous vector segment when processing a new vector segment. This state not only contains the hidden representation of the sequence, but also implies the relative position information of the sequence. Since relative position encoding does not depend on the absolute position of the elements, it can make the self-attention mechanism more efficient in parallel computing because the position encoding does not need to be recalculated for each vector segment.

[0140] Furthermore, Transformer-XL uses relative position encoding instead of absolute position encoding, which enables the model to handle sequences beyond the length of a single segment. Within each vector segment, the relative positions between elements are calculated. These relative positions are determined based on the relative index difference of the elements in the current segment. Based on the calculated relative positions, relative position encodings are generated. These encodings reflect the relative order of the elements in the current segment. Feature encoding, relative position encoding, and hidden state are combined to generate feature representation.

[0141] Transformer-XL uses the hidden state of the previous vector segment as memory and combines it with the encoding result of the current segment to capture long-term dependencies across segments. The introduction of relative position encoding enables Transformer-XL to effectively process long sequence data. Even when the sequence is split into multiple segments, it can maintain the perception of the relative order of elements in the sequence, thereby capturing and utilizing long-term dependency information.

[0142] In one embodiment, the target user is classified and identified by a user classification model according to the input text and key information of the target user, and the classification result is output, including:

[0143] Obtaining input text from a target user, and converting the input text into a semantic text vector;

[0144] Acquire key information of target users, and convert the key information into a comprehensive feature vector;

[0145] Concatenate the comprehensive feature vector with the semantic text vector to obtain a concatenated vector, and input the concatenated vector into a user classification model;

[0146] The user classification model receives the concatenated vector to perform label prediction and obtain the probability distribution of each category label;

[0147] The category of the target user is determined according to the probability distribution.

[0148] In this embodiment, the input text of the target user is collected, such as the conversation record between the user and the service staff. The input text is processed using the BigBird model and converted into a semantic text vector that can capture the semantic information of the text. The key information of the target user, such as age, gender, occupation, etc., is collected and converted into a comprehensive feature vector. The comprehensive feature vector is spliced ​​with the semantic text vector to form a spliced ​​vector that contains the basic information of the user and the semantic information of the text. The spliced ​​vector is input into the user classification model. The user classification model receives the spliced ​​vector and outputs the probability distribution of each category label. These probabilities represent the model's confidence that the target user belongs to each category. Based on the probability distribution, the category to which the target user is most likely to belong is determined. The category with the highest probability is usually selected as the prediction result.

[0149] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0150] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, large model technology, operation / interaction systems, mechatronics, etc. Artificial intelligence software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0151] In an embodiment of the present invention, key information related to the user is collected and converted into a comprehensive feature vector; the user's conversation text data is obtained and converted into a semantic text vector; the model is trained according to the semantic text vector and the comprehensive feature vector to obtain a user classification model. The present invention uses the BigBird model to expand the ability of long sequence texts, and trains the user classification model by cross-modal feature fusion of structured data and unstructured data, effectively capturing long-term dependencies and improving the accuracy of user classification.

[0152] Further references Figure 4 , as a response to the above Figure 4 The present invention provides an embodiment of a user classification model training device, and the device embodiment is Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0153] like Figure 4 As shown, the user classification model training device described in this embodiment includes: a comprehensive feature vector module 11, a semantic text vector module 12, a model training module 13 and a user classification module 14. Among them:

[0154] A comprehensive feature vector module 11 is used to collect key information related to the user and convert the key information into a comprehensive feature vector;

[0155] A semantic text vector module 12, used for acquiring the user's conversation text data, inputting the conversation text data into the improved BigBird model, and converting the conversation text data into a semantic text vector by the BigBird model;

[0156] A model training module 13 is used to train the model according to the semantic text vector and the comprehensive feature vector to obtain a user classification model;

[0157] The user classification module 14 is used to obtain the input text and key information of the target user, classify and identify the target user through the user classification model according to the input text and key information of the target user, and output the classification result.

[0158] like Figure 4 As shown, the comprehensive feature vector module 11, the semantic text vector module 12, the model training module 13 and the user classification module 14 are connected in sequence. The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, which is more suitable for describing the execution process of user classification model training and conversion than a program. For the specific implementation method of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0159] In one embodiment, the comprehensive feature vector module 11 includes:

[0160] A feature vector unit, used to collect key information of the user and convert the key information into a feature vector;

[0161] A vector construction unit, used to construct a query vector, a key vector and a value vector according to the feature vector;

[0162] An attention weight matrix unit, used to calculate the attention score matrix of the query vector and the key vector through the dot product attention function, and convert the attention score matrix into an attention weight matrix using the soft max function;

[0163] A weighted feature representation unit, used to perform weighted summation on the attention weight matrix and the value vector to obtain a weighted feature representation of each element;

[0164] The comprehensive feature vector unit is used to fuse the weighted feature representation of each element to obtain a comprehensive feature vector.

[0165] In one embodiment, the semantic text vector module 12 includes:

[0166] The model definition unit is used to customize the attention mechanism of the BigBird model to obtain the improved BigBird model;

[0167] A data preprocessing unit, used to obtain the user's conversation text data and preprocess the conversation text data;

[0168] A data input unit, used for inputting the preprocessed dialogue text into the improved Big Bird model;

[0169] The semantic text vector unit is used for the improved BigBird model to receive the conversation text and process it to obtain the semantic text vector.

[0170] In one embodiment, the model training module 13 includes:

[0171] A concatenating vector unit, used for concatenating the comprehensive feature vector with the semantic text vector to obtain a concatenated vector;

[0172] A vector input unit is used to input the concatenated vector into a classification model for training.

[0173] In one embodiment, the model training module 13 includes:

[0174] A label definition unit, used to predefine labels for user classification;

[0175] A prediction result unit, used for a user classification model to generate a prediction result according to the concatenated vector and the classification label;

[0176] The loss unit is used to calculate the difference between the predicted result and the true label through the loss function;

[0177] A parameter updating unit is used to update the parameters of the model according to the difference value until the model converges to obtain a user classification model.

[0178] In one embodiment, the prediction result unit includes:

[0179] Segmenting the spliced ​​vector to generate a plurality of vector segments;

[0180] Encode each vector segment to obtain the feature encoding and hidden state of each vector segment;

[0181] Calculate the relative positions between the elements in each vector segment to obtain the relative position encoding;

[0182] Combining the feature encoding, relative position encoding and hidden state to generate a feature representation;

[0183] Generate prediction results based on the feature representation and the classification label.

[0184] In one embodiment, the user classification module 14 includes:

[0185] A target semantic text unit, used to obtain an input text of a target user and convert the input text into a semantic text vector;

[0186] A target comprehensive feature vector unit, used to obtain key information of the target user and convert the key information into a comprehensive feature vector;

[0187] A target concatenated vector unit, used for concatenating the comprehensive feature vector with the semantic text vector to obtain a concatenated vector, and inputting the concatenated vector into a user classification model;

[0188] A target label prediction unit, used for the user classification model to receive the concatenated vector for label prediction and obtain the probability distribution of each category label;

[0189] The target classification unit is used to determine the category of the target user according to the probability distribution.

[0190] In the above embodiment, the present invention discloses a user classification model training device, which belongs to the field of data processing technology. The present invention collects key information related to the user, converts the key information into a comprehensive feature vector; obtains the user's conversation text data, converts the conversation text data into a semantic text vector; trains the model according to the semantic text vector and the comprehensive feature vector to obtain a user classification model. The present invention uses the BigBird model to expand the ability of long sequence texts, and trains the user classification model by cross-modal feature fusion of structured data and unstructured data, effectively capturing long-term dependencies and improving the accuracy of user classification.

[0191] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0192] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0193] To solve the above technical problems, the embodiment of the present invention also provides a computer device. Figure 5 , Figure 5 This is a basic structural block diagram of the computer device in this embodiment.

[0194] The computer device 8 includes a memory 8a, a processor 8b, and a network interface 8c that are interconnected through a system bus. It should be noted that the figure only shows a computer device 8 with components 8a-8c, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0195] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0196] The memory 8a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 8a can be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 8a can also be an external storage device of the computer device 8, such as a plug-in hard disk equipped on the computer device 8, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 8a can also include both the internal storage unit of the computer device 8 and its external storage device. In this embodiment, the memory 8a is generally used to store the operating system and various application software installed on the computer device 8, such as a computer-readable instruction of a user classification model training method, etc. In addition, the memory 8a can also be used to temporarily store various types of data that have been output or are to be output.

[0197] The processor 8b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other user classification model training chip in some embodiments. The processor 8b is generally used to control the overall operation of the computer device 8. In this embodiment, the processor 8b is used to run the computer-readable instructions or process data stored in the memory 8a, such as computer-readable instructions for running the user classification model training method.

[0198] The network interface 8c may include a wireless network interface or a wired network interface. The network interface 8c is generally used to establish a communication connection between the computer device 8 and other electronic devices.

[0199] The computer device proposed in this embodiment belongs to the field of data processing technology and is applied to dialogue scenarios such as finance or medical care. The present invention collects key information related to the user, converts the key information into a comprehensive feature vector; obtains the user's dialogue text data, converts the dialogue text data into a semantic text vector; trains the model according to the semantic text vector and the comprehensive feature vector to obtain a user classification model. The present invention uses the BigBird model to expand the ability of long sequence texts, and trains the user classification model by cross-modal feature fusion of structured data and unstructured data, effectively capturing long-term dependencies and improving the accuracy of user classification.

[0200] The present invention also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the user classification model training method as described above.

[0201] The computer-readable storage medium proposed in this embodiment belongs to the field of data processing technology and is applied to conversation scenarios such as finance or medical care. The present invention collects key information related to the user, converts the key information into a comprehensive feature vector; obtains the user's conversation text data, converts the conversation text data into a semantic text vector; trains the model according to the semantic text vector and the comprehensive feature vector to obtain a user classification model. The present invention uses the BigBird model to expand the ability of long sequence texts, and trains the user classification model by cross-modal feature fusion of structured data and unstructured data, effectively capturing long-term dependencies and improving the accuracy of user classification.

[0202] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0203] Obviously, the embodiments described above are only some embodiments of the present invention, rather than all embodiments. The preferred embodiments of the present invention are given in the accompanying drawings, but they do not limit the patent scope of the present invention. The present invention can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive. Although the present invention has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific embodiments, or to perform equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of the present invention, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present invention.

Claims

1. A user classification model training method, characterized in that: include: Collect key information related to the user, and convert the key information into a comprehensive feature vector; Acquire the user's conversation text data, input the conversation text data into the improved BigBird model, and convert the conversation text data into a semantic text vector by the BigBird model; The model is trained according to the semantic text vector and the comprehensive feature vector to obtain a user classification model; The input text and key information of the target user are obtained, and the target user is classified and identified through a user classification model according to the input text and key information of the target user, and a classification result is output.

2. The user classification model training method according to claim 1, characterized in that: The collecting of key information related to the user and converting the key information into a comprehensive feature vector includes: Collect key information of the user and convert the key information into a feature vector; constructing a query vector, a key vector and a value vector according to the feature vector; Calculate the attention score matrix of the query vector and the key vector through the dot product attention function, and convert the attention score matrix into an attention weight matrix using the soft max function; Performing weighted summation on the attention weight matrix and the value vector to obtain a weighted feature representation of each element; The weighted feature representation of each element is fused to obtain a comprehensive feature vector.

3. The user classification model training method according to claim 1, characterized in that: The step of obtaining the user's conversation text data, inputting the conversation text data into the improved BigBird model, and converting the conversation text data into a semantic text vector by the BigBird model includes: Customize the attention mechanism for the BigBird model to obtain an improved BigBird model; Acquire the user's conversation text data, and pre-process the conversation text data; Input the preprocessed dialogue text into the improved Big Bird model; The improved BigBird model receives the conversation text and processes it to obtain a semantic text vector.

4. The user classification model training method according to claim 1, characterized in that: The training of the model according to the semantic text vector and the comprehensive feature vector comprises: Concatenate the comprehensive feature vector with the semantic text vector to obtain a concatenated vector; The concatenated vector is input into the classification model for training.

5. The user classification model training method according to claim 4, characterized in that: The model is trained according to the semantic text vector and the comprehensive feature vector to obtain a user classification model, including: Predefine tags for user classification; The user classification model generates a prediction result based on the concatenated vector and the classification label; The difference between the predicted result and the true label is calculated through the loss function; The parameters of the model are updated according to the difference value until the model converges to obtain a user classification model.

6. The user classification model training method according to claim 5, characterized in that: The user classification model generates a prediction result according to the concatenated vector and the classification label, including: Segmenting the spliced ​​vector to generate a plurality of vector segments; Encode each vector segment to obtain the feature encoding and hidden state of each vector segment; Calculate the relative positions between the elements in each vector segment to obtain the relative position encoding; Combining the feature encoding, relative position encoding and hidden state to generate a feature representation; Generate prediction results based on the feature representation and the classification label.

7. The user classification model training method according to claim 1, characterized in that: The method of classifying and identifying the target user through a user classification model according to the input text and key information of the target user and outputting the classification result includes: Obtaining input text from a target user, and converting the input text into a semantic text vector; Acquire key information of target users, and convert the key information into a comprehensive feature vector; Concatenate the comprehensive feature vector with the semantic text vector to obtain a concatenated vector, and input the concatenated vector into a user classification model; The user classification model receives the concatenated vector to perform label prediction and obtains the probability distribution of each category label; The category of the target user is determined according to the probability distribution.

8. A user classification model training device, characterized in that: include: A comprehensive feature vector module, used to collect key information related to the user and convert the key information into a comprehensive feature vector; A semantic text vector module, used for acquiring the user's conversation text data, inputting the conversation text data into the improved BigBird model, and converting the conversation text data into a semantic text vector by the BigBird model; A model training module is used to train the model according to the semantic text vector and the comprehensive feature vector to obtain a user classification model; The user classification module is used to obtain the input text and key information of the target user, classify and identify the target user through the user classification model according to the input text and key information of the target user, and output the classification result.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the user classification model training method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the user classification model training method according to any one of claims 1 to 7.

Images