Customer group determination method and device, storage medium and electronic equipment

Abstract

The invention discloses a customer group determination method and device, a storage medium and electronic equipment, and the method comprises the steps: obtaining multi-modal data of a plurality of customers, the multi-modal data comprising voice data, image data and structured data; processing the multi-modal data of the plurality of clients through a target model to obtain a target feature vector set; the target feature vector set is classified through a target classifier, a plurality of customer groups are obtained, the target classifier is a classifier trained in advance based on a plurality of category labels, each customer group corresponds to one category label, and the customer groups are used for determining potential customer groups of the target recommendation activity. According to the method and the device, the technical problem of low customer grouping accuracy caused by incomplete customer attribute selection due to the fact that only structured data of customers are analyzed during customer grouping in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to a method, device, storage medium and electronic device for determining a customer group. Background Art

[0002] In order to improve customer experience and conversion rates of recommendation activities and better manage customer relationships, financial institutions usually need to conduct potential customer analysis on customer groups. Currently, when dividing customer groups, the analysis is often based on the structured data of customers stored in the database. The selection of customer attributes is not comprehensive, resulting in low accuracy in customer segmentation.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, storage medium and electronic device for determining a customer group, so as to at least solve the technical problem in the prior art that when performing customer grouping, only the structured data of the customer is analyzed, the selection of customer attributes is incomplete, and the accuracy of customer grouping is low.

[0005] According to one aspect of an embodiment of the present invention, a method for determining a customer group is provided, comprising: acquiring multimodal data of a plurality of customers, wherein the multimodal data comprises voice data, image data and structured data; processing the multimodal data of the plurality of customers through a target model to obtain a target feature vector set, wherein the target model comprises at least a target convolutional recurrent neural network model and a target classifier, the target convolutional recurrent neural network model is obtained by training the convolutional recurrent neural network with sample data, the target feature vector set comprises a plurality of multimodal fusion feature vectors, the multimodal fusion feature vectors are used to characterize emotional features corresponding to the voice data, image features corresponding to the image data and object features corresponding to the structured data, the object features at least comprising behavioral features; classifying the target feature vector set through a target classifier to obtain a plurality of customer groups, wherein the target classifier is a classifier pre-trained based on a plurality of category labels, each customer group corresponds to a category label, and the plurality of customer groups are used to determine potential customer groups for a target recommendation activity.

[0006] Furthermore, the target model also includes a fully connected layer, wherein the multimodal data of multiple customers are processed by the target model to obtain a target feature vector set, including: extracting features of voice data and image data respectively by a target convolutional recurrent neural network model to obtain emotional features and image features; extracting features of structured data by a fully connected layer to obtain object features, and performing feature fusion of emotional features, image features and object features in the fully connected layer to obtain multiple multimodal fusion feature vectors; generating a target feature vector set based on the multiple multimodal fusion feature vectors.

[0007] Furthermore, a target model is generated through the following steps: sample data is obtained, and the sample data is divided into a training set and a test set, wherein the sample data includes multimodal data and true labels of multiple sample customers, and the true labels are used to identify the group categories corresponding to the sample customers; an initial model is constructed based on a convolutional recurrent neural network, and the initial model is trained according to the training set to obtain a trained model; the trained model is tested and optimized according to a test set and preset evaluation indicators to obtain a target convolutional recurrent neural network model, and a target model is generated based on the target convolutional recurrent neural network model and a target classifier.

[0008] Furthermore, the initial model is trained based on the training set to obtain a trained model, including: extracting features from the speech data in the training set through the initial model to obtain sample emotion features; extracting features from the image data in the training set through the initial model to obtain sample image features; and training the initial model based on the sample emotion features and the sample image features to obtain a trained model.

[0009] Furthermore, the initial model is used to extract features from the speech data in the training set to obtain sample emotion features, including: performing speech recognition processing on the speech data in the training set to obtain speech text data corresponding to the speech data in the training set, and performing word embedding processing on the speech text data to obtain processed speech text data; performing feature extraction on the processed speech text data through the convolutional neural network in the initial model to obtain semantic emotion features corresponding to the speech data in the training set; fusing the semantic emotion features into the fully connected layer of the initial model through the recurrent neural network in the initial model to obtain multiple emotion fusion feature vectors; and classifying the multiple emotion fusion feature vectors through the first classifier in the initial model to obtain sample emotion features.

[0010] Furthermore, feature extraction is performed on the image data in the training set through the initial model to obtain sample image features, including: feature extraction is performed on the image data in the training set through the convolution layer of the convolutional neural network to obtain initial image features corresponding to the image data in the training set; down-sampling the initial image features through the pooling layer of the convolutional neural network to obtain processed image features; and the processed image features are processed through the recurrent neural network to obtain sample image features.

[0011] Furthermore, the initial model is trained based on the sample emotion features and the sample image features to obtain a trained model, including: extracting features of the structured data in the training set through the fully connected layer of the initial model to obtain sample object features; concatenating the sample object features, the sample emotion features and the sample image features according to preset dimensions to obtain a sample feature vector matrix; and training the initial model based on the sample feature vector matrix to obtain a trained model.

[0012] According to another aspect of an embodiment of the present invention, a device for determining a customer group is also provided, comprising: a first acquisition module, used to acquire multimodal data of multiple customers, wherein the multimodal data includes voice data, image data and structured data; a first processing module, used to process the multimodal data of multiple customers through a target model to obtain a target feature vector set, wherein the target model at least includes a target convolutional recurrent neural network model and a target classifier, the target convolutional recurrent neural network model is obtained by training the convolutional recurrent neural network with sample data, and the target feature vector set includes multiple multimodal fusion feature vectors, the multimodal fusion feature vectors are used to characterize emotional features corresponding to voice data, image features corresponding to image data and object features corresponding to structured data, and the object features include at least behavioral features; a second processing module, used to classify the target feature vector set through a target classifier to obtain multiple customer groups, wherein the target classifier is a classifier pre-trained based on multiple category labels, each customer group corresponds to a category label, and multiple customer groups are used to determine potential customer groups for target recommendation activities.

[0013] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method for determining a customer group when running.

[0014] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, comprising one or more processors; a memory for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the above-mentioned method for determining a customer group when running.

[0015] In an embodiment of the present invention, the emotional features of customers are added to the customer grouping process, and the customers are grouped in combination with multimodal data. First, the multimodal data of multiple customers are obtained, and then the multimodal data of multiple customers are processed by the target model to obtain a target feature vector set, and then the target feature vector set is classified by the target classifier to obtain multiple customer groups. Among them, the multimodal data includes voice data, image data and structured data, the target model includes at least a target convolutional recurrent neural network model and a target classifier, the target convolutional recurrent neural network model is obtained by training the convolutional recurrent neural network with sample data, the target feature vector set includes multiple multimodal fusion feature vectors, the multimodal fusion feature vectors are used to characterize the emotional features corresponding to the voice data, the image features corresponding to the image data, and the object features corresponding to the structured data, the object features at least include behavioral features, the target classifier is a classifier pre-trained based on multiple category labels, each customer group corresponds to a category label, and multiple customer groups are used to determine the potential customer groups for the target recommendation activities.

[0016] In the above process, by acquiring multimodal data of multiple customers, a data basis is provided for data processing of the target model; by processing the multimodal data of multiple customers through the target model, a target feature vector set including multiple multimodal fusion feature vectors can be obtained, which realizes the comprehensive consideration of emotional features, image features and object features, enriches the selection of customer attributes, and thus can effectively improve the accuracy of customer segmentation; by classifying the target feature vector set through the target classifier, multiple customer groups can be accurately determined, so as to better perform customer relationship management, improve customer experience and customer stickiness, and then improve the conversion rate of recommendation activities.

[0017] It can be seen that through the technical solution of the present invention, the purpose of comprehensive analysis and accurate grouping of customers based on multimodal customer data is achieved, thereby realizing the technical effect of improving the accuracy of customer grouping, and further solving the technical problem that in the prior art, when performing customer grouping, only the structured data of the customer is analyzed, the selection of customer attributes is incomplete, and the accuracy of customer grouping is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0019] Figure 1 is a flow chart of an optional method for determining a customer group according to an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of an optional emotion analysis and discrimination framework based on speech recognition according to an embodiment of the present invention;

[0021] Figure 3 is a schematic diagram of an optional customer group classification model framework based on multimodal sentiment analysis according to an embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of an optional device for determining a customer group according to an embodiment of the present invention;

[0023] Figure 5 is a schematic diagram of an optional electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set between the system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0027] Example 1

[0028] According to an embodiment of the present invention, an embodiment of a method for determining a customer group is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] Figure 1 is a flow chart of an optional method for determining a customer group according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0030] Step S101, obtaining multimodal data of multiple customers, wherein the multimodal data includes voice data, image data and structured data.

[0031] In the above steps, multimodal data of multiple customers can be obtained through application systems, processors, electronic devices and other devices. Optionally, multimodal data of multiple customers can be obtained through a customer group determination system. Multimodal data refers to data from different sources in different forms, including unstructured data such as text, images, and voice, as well as structured data such as historical transaction data, behavioral data, and preference data of customers stored in a database. For example, unstructured data and structured data of customers in the process of handling business are collected. For example, voice data can be data collected during customer telephone consultation or complaints.

[0032] Step S102, processing the multimodal data of multiple customers through the target model to obtain a target feature vector set, wherein the target model at least includes a target convolutional recurrent neural network model and a target classifier, the target convolutional recurrent neural network model is obtained by training the convolutional recurrent neural network with sample data, and the target feature vector set includes multiple multimodal fusion feature vectors, and the multimodal fusion feature vectors are used to characterize the emotional features corresponding to the voice data, the image features corresponding to the image data, and the object features corresponding to the structured data, and the object features include at least behavioral features.

[0033] Optionally, the target model may be a customer clustering model, the target convolutional recurrent neural network model may be a convolutional recurrent neural network (CRNN) model, and the target classifier may be a softmax classifier. For example, by processing the multimodal data of multiple customers through the customer clustering model, a target feature vector set including multiple multimodal fusion feature vectors may be obtained. Specifically, by extracting features from speech data and image data respectively through the CRNN model, emotional features and image features may be obtained, and by fusing emotional features, image features and object features, a multimodal fusion feature vector may be obtained.

[0034] Step S103, classifying the target feature vector set by a target classifier to obtain multiple customer groups, wherein the target classifier is a classifier pre-trained based on multiple category labels, each customer group corresponds to a category label, and the multiple customer groups are used to determine the potential customer groups of the target recommendation activity.

[0035] Optionally, after obtaining a target feature vector set including multiple multimodal fusion feature vectors, the target feature vector set can be classified by a trained softmax classifier to obtain multiple customer groups to promote better customer relationship management and product promotion. For example, after obtaining multiple customer groups, more personalized and intelligent financial products and services can be launched based on the category characteristics corresponding to the multiple customer groups, thereby improving customer satisfaction and customer stickiness.

[0036] Based on the scheme defined by the above steps S101 to S103, it can be known that in an embodiment of the present invention, the emotional features of customers are added to the customer grouping process, and the customers are grouped in combination with multimodal data. First, the multimodal data of multiple customers are obtained, and then the multimodal data of multiple customers are processed by the target model to obtain a target feature vector set, and then the target feature vector set is classified by the target classifier to obtain multiple customer groups. Among them, the multimodal data includes voice data, image data and structured data, the target model includes at least a target convolutional recurrent neural network model and a target classifier, the target convolutional recurrent neural network model is obtained by training the convolutional recurrent neural network with sample data, the target feature vector set includes multiple multimodal fusion feature vectors, the multimodal fusion feature vector is used to characterize the emotional features corresponding to the voice data, the image features corresponding to the image data, and the object features corresponding to the structured data, the object features at least include behavioral features, the target classifier is a classifier pre-trained based on multiple category labels, each customer group corresponds to a category label, and multiple customer groups are used to determine the potential customer groups for the target recommendation activity.

[0037] It is easy to notice that in the above process, by obtaining multimodal data of multiple customers, a data basis is provided for data processing of the target model; by processing the multimodal data of multiple customers through the target model, a target feature vector set including multiple multimodal fusion feature vectors can be obtained, which realizes the comprehensive consideration of emotional features, image features and object features, enriches the selection of customer attributes, and thus can effectively improve the accuracy of customer segmentation; by classifying the target feature vector set through the target classifier, multiple customer groups can be accurately determined, so as to better perform customer relationship management, improve customer experience and customer stickiness, and then improve the conversion rate of recommendation activities.

[0038] It can be seen that through the technical solution of the present invention, the purpose of comprehensive analysis and accurate grouping of customers based on multimodal customer data is achieved, thereby realizing the technical effect of improving the accuracy of customer grouping, and further solving the technical problem that in the prior art, when performing customer grouping, only the structured data of the customer is analyzed, the selection of customer attributes is incomplete, and the accuracy of customer grouping is low.

[0039] In an optional embodiment, the target model also includes a fully connected layer, wherein the multimodal data of multiple customers are processed by the target model to obtain a target feature vector set, including: extracting features of voice data and image data respectively by a target convolutional recurrent neural network model to obtain emotion features and image features; extracting features of structured data by a fully connected layer to obtain object features, and performing feature fusion of emotion features, image features and object features in the fully connected layer to obtain multiple multimodal fusion feature vectors; generating a target feature vector set based on the multiple multimodal fusion feature vectors.

[0040] Optionally, in the process of processing the multimodal data of multiple customers through the target model to obtain a target feature vector set, the customer's emotional features and image features are extracted through the target convolutional recurrent neural network model, the customer's object features are extracted through the fully connected layer of the target model, and the emotional features, image features and object features are fused in the fully connected layer to obtain multiple multimodal fusion feature vectors to generate a target feature vector set.

[0041] For example, the CRNN model is used to extract features from speech data and image data respectively to obtain emotional features and image features, and the fully connected layer is used to extract features from structured data to obtain object features. The emotional features, image features, and object features are fused in the fully connected layer to obtain multiple multimodal fusion feature vectors, thereby generating a target feature vector set.

[0042] It should be noted that by processing the multimodal data of multiple customers, it is possible to discover potential patterns and relationships hidden in large amounts of data. For example, by analyzing the emotional state of customers, we can obtain the emotional characteristics of customers, thereby determining the emotional category label of customers (such as irritability, calmness, optimism, etc.); by analyzing the structured data of customers, we can obtain the behavioral characteristics, consumption characteristics and other object characteristics of customers, discover the consumption habits, consumption levels, income status, purchasing preferences and needs of customers, and achieve a more comprehensive and accurate grasp of customer needs and behaviors, so that when customers are grouped according to these characteristics, the accuracy of customer grouping can be improved, and multiple customer groups can be accurately determined.

[0043] In an optional embodiment, a target model is generated by the following steps: obtaining sample data, and dividing the sample data into a training set and a test set, wherein the sample data includes multimodal data of multiple sample customers and true labels, and the true labels are used to identify the group categories corresponding to the sample customers; constructing an initial model based on a convolutional recurrent neural network, and training the initial model according to the training set to obtain a trained model; testing and optimizing the trained model according to a test set and preset evaluation indicators to obtain a target convolutional recurrent neural network model, and generating a target model based on the target convolutional recurrent neural network model and a target classifier.

[0044] Optionally, in this embodiment, the sample data may be historical data obtained from different data sources, for example, structured data such as customer transaction records may be obtained from a database, customer consultation or complaint voice data may be obtained from customer service voice records, and customer image data may be obtained from relevant certification materials and certificates uploaded by customers. Optionally, after obtaining the sample data, the data may be pre-processed such as cleaning, deduplication, and format conversion to ensure the accuracy and consistency of the data so as to facilitate the next step of analysis and modeling.

[0045] Optionally, a widely used method is used to divide the sample data into a training set and a test set, and an initial CRNN model is constructed and trained, that is, the model is trained with the training set, and the accuracy and generalization ability of the model are evaluated with the test set, so as to obtain a target convolutional recurrent neural network model. For example, the constructed initial CRNN model is trained using a prepared multimodal sample data set, appropriate hyperparameters are set, and the training error and test error of the model are monitored to avoid overfitting, and the model is evaluated using a test data set and preset evaluation indicators (such as accuracy, recall, etc.) to ensure the performance and generalization ability of the model.

[0046] Optionally, after the initial CRNN model is trained, evaluated and optimized, a trained CRNN model is obtained, and a customer clustering model is generated based on the trained CRNN model and the trained softmax classifier.

[0047] It should be noted that in this embodiment, data analysis is performed on the historical customer information, pictures, and voices of the financial institution, and a customer group classification model based on multimodal sentiment analysis, namely, a customer clustering model, is established. Thus, through the customer clustering model and multimodal customer data, customers can be comprehensively analyzed and accurately clustered, thereby solving the problem of incomplete customer behavior information.

[0048] In an optional embodiment, the initial model is trained based on the training set to obtain a trained model, including: extracting features from the speech data in the training set through the initial model to obtain sample emotion features; extracting features from the image data in the training set through the initial model to obtain sample image features; and training the initial model based on the sample emotion features and the sample image features to obtain a trained model.

[0049] Optionally, in the process of training the initial model based on the training set to obtain the trained model, for the voice data in the historical customer service call records, the customer's emotional state characteristics are extracted through the initial CRNN model to obtain sample emotional characteristics, and for the customer's image data, the customer's image characteristics are extracted through the initial CRNN model to obtain sample image characteristics, and then based on the sample emotional characteristics and sample image characteristics, a widely used method is used to train the initial CRNN model to obtain the trained model.

[0050] It should be noted that in the above process, various forms of data are used for analysis, and by extracting features from the data, the customer characteristics are portrayed in multiple aspects, so that we can have a more comprehensive understanding of customer needs and behavioral habits, and then provide customers with more accurate services and improve customer satisfaction.

[0051] In an optional embodiment, feature extraction is performed on speech data in a training set through an initial model to obtain sample emotion features, including: performing speech recognition processing on the speech data in the training set to obtain speech text data corresponding to the speech data in the training set, and performing word embedding processing on the speech text data to obtain processed speech text data; feature extraction is performed on the processed speech text data through a convolutional neural network in the initial model to obtain semantic emotion features corresponding to the speech data in the training set; the semantic emotion features are fused into the fully connected layer of the initial model through a recurrent neural network in the initial model to obtain multiple emotion fusion feature vectors; and multiple emotion fusion feature vectors are classified through a first classifier in the initial model to obtain sample emotion features.

[0052] Optionally, speech recognition processing is performed on the speech data in the training set by using multimodal speech recognition technology to obtain speech text data corresponding to the speech data in the training set, and word embedding processing is performed on the speech text data to obtain processed speech text data.

[0053] Specifically, the CRNN model includes a CNN feature extraction layer and an RNN (e.g., LSTM) sequence feature extraction layer. The CNN (i.e., convolutional neural network) in the initial CRNN model is used to extract features from the processed speech text data, so that the semantic emotional features corresponding to the speech data in the training set can be obtained. The RNN (i.e., recurrent neural network) in the CRNN model is used to fuse the semantic emotional features into the fully connected layer of the initial model, so that multiple emotional fusion feature vectors can be obtained, thereby realizing the conversion of raw data into feature vectors that can be understood by the deep learning model.

[0054] Figure 2 is a schematic diagram of an optional emotion analysis and discrimination framework based on speech recognition according to an embodiment of the present invention, such as Figure 2 As shown in the figure, the local time features of the audio are extracted through the convolution layer. The input of the convolution layer is the time domain waveform of the speech signal, and the output is the convolution features of the time domain waveform in different time windows. The convolution features are then downsampled through the pooling layer to reduce the dimension of the features. The features are then processed by the LSTM in the recurrent neural network RNN ​​with maximum pooling, minimum pooling, and average pooling, and then fused into the fully connected layer (FC).

[0055] Furthermore, by classifying multiple emotion fusion feature vectors through the first classifier (e.g., softmax classifier) ​​for emotion analysis in the initial CRNN model, sample emotion features can be obtained, thereby realizing emotion discrimination of customers. Figure 2 As shown in FIG. 1 , multiple emotion fusion feature vectors are classified by softmax to obtain the customer emotion class.

[0056] It should be noted that when extracting speech features in multimodality, the customer's emotional state can be analyzed through multimodal speech recognition technology. Since the customer's emotional state can reveal the customer's real needs, preferences and intentions, using emotional characteristics as a characteristic indicator for customer segmentation can better understand the customer's behavior and decision-making, thereby effectively improving the accuracy of customer segmentation.

[0057] In an optional embodiment, feature extraction is performed on image data in a training set through an initial model to obtain sample image features, including: feature extraction is performed on image data in a training set through a convolutional layer of a convolutional neural network to obtain initial image features corresponding to the image data in the training set; downsampling the initial image features through a pooling layer of a convolutional neural network to obtain processed image features; and processing the processed image features through a recurrent neural network to obtain sample image features.

[0058] Optionally, for image data, convolutional layers and pooling layers are used for feature extraction. Specifically, by extracting features from the image data in the training set through the convolutional layer of CNN, the initial image features corresponding to the image data in the training set can be obtained. The input of the convolutional layer is the pixel value matrix of the image, and the output is the convolution features at different positions (i.e., the initial image features). The initial image features are downsampled through the pooling layer to reduce the dimension of the features and obtain the processed image features. Furthermore, the processed image features are processed through the RNN to obtain the sample image features.

[0059] It should be noted that by extracting sample image features, a data basis is provided for feature fusion, which can better understand customer behaviors and decisions, thereby effectively improving the accuracy of customer segmentation.

[0060] In an optional embodiment, the initial model is trained based on sample emotion features and sample image features to obtain a trained model, including: extracting features of structured data in a training set through a fully connected layer of the initial model to obtain sample object features; concatenating sample object features, sample emotion features, and sample image features based on preset dimensions to obtain a sample feature vector matrix; and training the initial model based on the sample feature vector matrix to obtain a trained model.

[0061] Optionally, for structured data, after preprocessing operations such as standardization and normalization, feature extraction is performed through the fully connected layer of the model to obtain sample object features. The input of the fully connected layer is the feature vector of the structured data, and the output is a one-dimensional feature vector.

[0062] Furthermore, after the above-mentioned feature extraction, the sample object features, sample emotion features and sample image features are spliced ​​according to a preset dimension to obtain a sample feature vector matrix. Optionally, when fusing multimodal features, the features of different layers are spliced ​​in a concatenated manner to construct a complete feature vector matrix to obtain a sample feature vector matrix, wherein concatenate refers to concatenating two or more vectors into a longer vector according to a certain dimension. For example, in the process of fusing multimodal features, the features of different layers are usually spliced ​​according to the dimension of the channel to obtain a multi-channel sample feature vector matrix containing all features.

[0063] Furthermore, based on the sample feature vector matrix, a widely used method is used to train the initial model to obtain a trained model.

[0064] Figure 3 is a schematic diagram of an optional customer group classification model framework based on multimodal sentiment analysis according to an embodiment of the present invention, such as Figure 3 As shown in the figure, the customer classification model consists of a CRNN model, a fully connected layer and a classifier. The input of the customer classification model is structured data, image data and voice data. Specifically, the CRNN model is used to extract features of voice data and image data to obtain affective characteristics and image features. The structured data is extracted through a fully connected layer to obtain structural features. The affective features, image features and structural features (i.e., object features) are fused in the fully connected layer, and then the softmax classifier is used to divide the customer groups to obtain customer classification.

[0065] It should be noted that in this embodiment, when classifying customer groups, the concept of multimodality is introduced to continuously enrich the customer's attribute characteristics, and to explore the customer's emotional state in the voice characteristics. By using a deep learning algorithm and combining multimodal customer data to group customers, it is possible to grasp customer characteristics more comprehensively and achieve more accurate customer grouping.

[0066] It can be seen that through the technical solution of the present invention, the purpose of comprehensive analysis and accurate grouping of customers based on multimodal customer data is achieved, thereby realizing the technical effect of improving the accuracy of customer grouping, and further solving the technical problem that in the prior art, when performing customer grouping, only the structured data of the customer is analyzed, the selection of customer attributes is incomplete, and the accuracy of customer grouping is low.

[0067] Example 2

[0068] According to an embodiment of the present invention, an embodiment of a device for determining a customer group is provided, wherein: Figure 4 is a schematic diagram of an optional device for determining a customer group according to an embodiment of the present invention, such as Figure 4 As shown, the device includes: a first acquisition module 401, which is used to acquire multimodal data of multiple customers, wherein the multimodal data includes voice data, image data and structured data; a first processing module 402, which is used to process the multimodal data of multiple customers through a target model to obtain a target feature vector set, wherein the target model at least includes a target convolutional recurrent neural network model and a target classifier, the target convolutional recurrent neural network model is obtained by training the convolutional recurrent neural network with sample data, and the target feature vector set includes multiple multimodal fusion feature vectors, and the multimodal fusion feature vectors are used to characterize the emotional features corresponding to the voice data, the image features corresponding to the image data, and the object features corresponding to the structured data, and the object features at least include behavioral features; a second processing module 403, which is used to classify the target feature vector set through a target classifier to obtain multiple customer groups, wherein the target classifier is a classifier pre-trained based on multiple category labels, each customer group corresponds to a category label, and multiple customer groups are used to determine the potential customer groups for the target recommendation activity.

[0069] It should be noted that the above-mentioned first acquisition module 401, first processing module 402 and second processing module 403 correspond to steps S101 to S103 in the above-mentioned embodiment, and the examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiment 1.

[0070] Optionally, the target model also includes a fully connected layer, and the first processing module includes: a third processing module, used to perform feature extraction on speech data and image data respectively through a target convolutional recurrent neural network model to obtain emotion features and image features; a fourth processing module, used to perform feature extraction on structured data through a fully connected layer to obtain object features, and perform feature fusion on emotion features, image features and object features in the fully connected layer to obtain multiple multimodal fusion feature vectors; a first generation module, used to generate a target feature vector set based on multiple multimodal fusion feature vectors.

[0071] Optionally, the device for determining the customer group also includes the following modules, which are used to generate a target model through the following steps: a second acquisition module, used to acquire sample data, and divide the sample data into a training set and a test set, wherein the sample data includes multimodal data and true labels of multiple sample customers, and the true labels are used to identify the group categories corresponding to the sample customers; a construction module, used to construct an initial model based on a convolutional recurrent neural network, and train the initial model according to the training set to obtain a trained model; a second generation module, used to test and optimize the trained model according to a test set and preset evaluation indicators to obtain a target convolutional recurrent neural network model, and generate a target model based on the target convolutional recurrent neural network model and a target classifier.

[0072] Optionally, the construction module includes: a fifth processing module, used to extract features of speech data in a training set through an initial model to obtain sample emotion features; a sixth processing module, used to extract features of image data in a training set through an initial model to obtain sample image features; and a training module, used to train the initial model based on sample emotion features and sample image features to obtain a trained model.

[0073] Optionally, the fifth processing module includes: a first extraction module, which is used to perform speech recognition processing on the speech data in the training set to obtain speech text data corresponding to the speech data in the training set, and perform word embedding processing on the speech text data to obtain processed speech text data; a second extraction module, which is used to perform feature extraction on the processed speech text data through the convolutional neural network in the initial model to obtain semantic emotional features corresponding to the speech data in the training set; a first fusion module, which is used to fuse the semantic emotional features into the fully connected layer of the initial model through the recurrent neural network in the initial model to obtain multiple emotion fusion feature vectors; a first classification module, which is used to classify multiple emotion fusion feature vectors through the first classifier in the initial model to obtain sample emotion features.

[0074] Optionally, the sixth processing module includes: a third extraction module, used to extract features of image data in the training set through the convolution layer of the convolutional neural network, and obtain initial image features corresponding to the image data in the training set; a fourth extraction module, used to downsample the initial image features through the pooling layer of the convolutional neural network to obtain processed image features; and a seventh processing module, used to process the processed image features through a recurrent neural network to obtain sample image features.

[0075] Optionally, the training module includes: a fifth extraction module, used to extract features of the structured data in the training set through the fully connected layer of the initial model to obtain sample object features; a splicing module, used to splice sample object features, sample emotion features and sample image features according to preset dimensions to obtain a sample feature vector matrix; a first training module, used to train the initial model based on the sample feature vector matrix to obtain a trained model.

[0076] Example 3

[0077] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method for determining a customer group when running.

[0078] Example 4

[0079] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, wherein: Figure 5 is a schematic diagram of an optional electronic device according to an embodiment of the present invention, such as Figure 5 As shown, the electronic device includes one or more processors; a memory for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors are used to run the programs, wherein the programs are set to execute the above-mentioned method for determining the customer group when running. When the processor executes the program, the following steps are implemented: obtaining multimodal data of multiple customers, wherein the multimodal data includes voice data, image data and structured data; processing the multimodal data of multiple customers through a target model to obtain a target feature vector set, wherein the target model at least includes a target convolutional recurrent neural network model and a target classifier, the target convolutional recurrent neural network model is obtained by training the convolutional recurrent neural network with sample data, the target feature vector set includes multiple multimodal fusion feature vectors, the multimodal fusion feature vectors are used to characterize the emotional features corresponding to the voice data, the image features corresponding to the image data and the object features corresponding to the structured data, and the object features at least include behavioral features; classifying the target feature vector set through a target classifier to obtain multiple customer groups, wherein the target classifier is a classifier pre-trained based on multiple category labels, each customer group corresponds to a category label, and multiple customer groups are used to determine the potential customer groups of the target recommendation activity.

[0080] Optionally, the target model also includes a fully connected layer, and the processor also implements the following steps when executing the program: processing the multimodal data of multiple customers through the target model to obtain a target feature vector set, including: extracting features from the voice data and image data respectively through the target convolutional recurrent neural network model to obtain emotional features and image features; extracting features from the structured data through the fully connected layer to obtain object features, and performing feature fusion on the emotional features, image features and object features in the fully connected layer to obtain multiple multimodal fusion feature vectors; generating a target feature vector set based on the multiple multimodal fusion feature vectors.

[0081] Optionally, when executing the program, the processor also implements the following steps: generating a target model by the following steps: obtaining sample data, and dividing the sample data into a training set and a test set, wherein the sample data includes multimodal data and true labels of multiple sample customers, and the true labels are used to identify the group categories corresponding to the sample customers; constructing an initial model based on a convolutional recurrent neural network, and training the initial model according to the training set to obtain a trained model; testing and optimizing the trained model according to the test set and preset evaluation indicators to obtain a target convolutional recurrent neural network model, and generating a target model based on the target convolutional recurrent neural network model and a target classifier.

[0082] Optionally, when the processor executes the program, the following steps are also implemented: training the initial model based on the training set to obtain a trained model, including: extracting features from the speech data in the training set through the initial model to obtain sample emotion features; extracting features from the image data in the training set through the initial model to obtain sample image features; training the initial model based on the sample emotion features and the sample image features to obtain a trained model.

[0083] Optionally, when the processor executes the program, the following steps are also implemented: feature extraction of speech data in a training set is performed through an initial model to obtain sample emotion features, including: speech recognition processing is performed on the speech data in the training set to obtain speech text data corresponding to the speech data in the training set, and word embedding processing is performed on the speech text data to obtain processed speech text data; feature extraction of the processed speech text data is performed through a convolutional neural network in the initial model to obtain semantic emotion features corresponding to the speech data in the training set; the semantic emotion features are fused into the fully connected layer of the initial model through a recurrent neural network in the initial model to obtain multiple emotion fusion feature vectors; and multiple emotion fusion feature vectors are classified through a first classifier in the initial model to obtain sample emotion features.

[0084] Optionally, when the processor executes the program, the following steps are also implemented: extracting features from the image data in the training set through the initial model to obtain sample image features, including: extracting features from the image data in the training set through the convolution layer of the convolutional neural network to obtain initial image features corresponding to the image data in the training set; downsampling the initial image features through the pooling layer of the convolutional neural network to obtain processed image features; processing the processed image features through a recurrent neural network to obtain sample image features.

[0085] Optionally, when the processor executes the program, the following steps are also implemented: based on the sample emotion features and the sample image features, the initial model is trained to obtain a trained model, including: extracting features of the structured data in the training set through the fully connected layer of the initial model to obtain sample object features; based on preset dimensions, the sample object features, the sample emotion features and the sample image features are concatenated to obtain a sample feature vector matrix; based on the sample feature vector matrix, the initial model is trained to obtain a trained model.

[0086] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0087] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0088] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0089] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0090] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0091] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0092] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0093] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for determining a customer group, characterized in that: include: Acquire multimodal data of multiple customers, wherein the multimodal data includes voice data, image data, and structured data; The multimodal data of the multiple customers are processed by a target model to obtain a target feature vector set, wherein the target model at least includes a target convolutional recurrent neural network model and a target classifier, the target convolutional recurrent neural network model is obtained by training the convolutional recurrent neural network with sample data, and the target feature vector set includes a plurality of multimodal fusion feature vectors, the multimodal fusion feature vectors are used to characterize the emotion features corresponding to the voice data, the image features corresponding to the image data, and the object features corresponding to the structured data, and the object features at least include behavioral features; The target feature vector set is classified by the target classifier to obtain multiple customer groups, wherein the target classifier is a classifier pre-trained based on multiple category labels, each customer group corresponds to one category label, and the multiple customer groups are used to determine potential customer groups for target recommendation activities.

2. The method according to claim 1, characterized in that: The target model also includes a fully connected layer, wherein The multimodal data of the multiple customers are processed by the target model to obtain a target feature vector set, including: Performing feature extraction on the speech data and the image data respectively through the target convolutional recurrent neural network model to obtain the emotion feature and the image feature; Extracting features from the structured data through the fully connected layer to obtain the object features, and fusing the emotion features, the image features, and the object features through the fully connected layer to obtain the multiple multimodal fusion feature vectors; The target feature vector set is generated according to the multiple multimodal fusion feature vectors.

3. The method according to claim 1, characterized in that The target model is generated by following the steps below: Acquire the sample data, and divide the sample data into a training set and a test set, wherein the sample data includes multimodal data of a plurality of sample customers and true labels, and the true labels are used to identify the group category corresponding to the sample customers; Constructing an initial model based on the convolutional recurrent neural network, and training the initial model according to the training set to obtain a trained model; The trained model is tested and optimized according to the test set and preset evaluation indicators to obtain the target convolutional recurrent neural network model, and the target model is generated according to the target convolutional recurrent neural network model and the target classifier.

4. The method according to claim 3, characterized in that The initial model is trained according to the training set to obtain a trained model, including: Extracting features from the speech data in the training set using the initial model to obtain sample emotion features; Extracting features from the image data in the training set using the initial model to obtain sample image features; The initial model is trained according to the sample emotion features and the sample image features to obtain the trained model.

5. The method according to claim 4, characterized in that The initial model is used to extract features from the speech data in the training set to obtain sample emotion features, including: Performing speech recognition processing on the speech data in the training set to obtain speech text data corresponding to the speech data in the training set, and performing word embedding processing on the speech text data to obtain processed speech text data; Performing feature extraction on the processed speech text data through the convolutional neural network in the initial model to obtain semantic emotion features corresponding to the speech data in the training set; The semantic emotion feature is fused into the fully connected layer of the initial model through the recurrent neural network in the initial model to obtain a plurality of emotion fusion feature vectors; The plurality of emotion fusion feature vectors are classified by the first classifier in the initial model to obtain the sample emotion feature.

6. The method according to claim 5, characterized in that Extracting features from the image data in the training set using the initial model to obtain sample image features includes: Performing feature extraction on the image data in the training set through the convolution layer of the convolutional neural network to obtain initial image features corresponding to the image data in the training set; Downsampling the initial image features through the pooling layer of the convolutional neural network to obtain processed image features; The processed image features are processed by the recurrent neural network to obtain the sample image features.

7. The method according to claim 4, characterized in that The initial model is trained according to the sample emotion feature and the sample image feature to obtain the trained model, including: Extracting features from the structured data in the training set through the fully connected layer of the initial model to obtain sample object features; According to a preset dimension, the sample object feature, the sample emotion feature and the sample image feature are spliced ​​to obtain a sample feature vector matrix; The initial model is trained according to the sample feature vector matrix to obtain the trained model.

8. A device for determining a customer group, characterized in that: include: A first acquisition module is used to acquire multimodal data of multiple customers, wherein the multimodal data includes voice data, image data and structured data; A first processing module is used to process the multimodal data of the multiple customers through a target model to obtain a target feature vector set, wherein the target model at least includes a target convolutional recurrent neural network model and a target classifier, the target convolutional recurrent neural network model is obtained by training the convolutional recurrent neural network with sample data, and the target feature vector set includes a plurality of multimodal fusion feature vectors, the multimodal fusion feature vectors are used to characterize the emotion features corresponding to the voice data, the image features corresponding to the image data, and the object features corresponding to the structured data, and the object features at least include behavioral features; The second processing module is used to classify the target feature vector set through the target classifier to obtain multiple customer groups, wherein the target classifier is a classifier pre-trained based on multiple category labels, each customer group corresponds to one category label, and the multiple customer groups are used to determine potential customer groups for target recommendation activities.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method for determining a customer group as claimed in any one of claims 1 to 7 when running.

10. An electronic device, characterized in that: The electronic device includes one or more processors; A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to run the programs, wherein the program is configured to execute the method for determining the customer group described in any one of claims 1 to 7 when running.

Images