Chat prompt word-based financial digital marketing opportunity analysis method and system

Abstract

The application discloses a financial digital marketing opportunity analysis method based on chat prompt words, and relates to the field of marketing analysis.The method comprises the following steps: collecting dialogue information of multiple target users; extracting multi-modal dialogue features of visual information and non-visual information of the target users; completing multi-modal fusion of the multi-modal dialogue features; constructing a user feature supergraph; completing community division of the user feature supergraph to obtain multiple user feature communities; completing feature matching between each user feature community and pre-acquired financial product features, and outputting a financial marketing strategy according to the feature matching result.The application can effectively improve the extraction accuracy of user features.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of marketing analysis, and particularly relate to a financial digital marketing opportunity analysis method and system based on chat prompt words. BACKGROUND

[0002] With the deep integration of Internet technology and financial business, precise marketing through data driving has become the core demand of industry development. Among them, the core step to realize digital marketing is to accurately match suitable financial products for target users. However, the existing method of accurately matching products for target users mainly extracts financial keywords of target user chat content, and completes the matching of users and products according to the feature similarity between the financial keywords of target users and financial products. This method is difficult to accurately reflect the real user portrait due to sparse data and single user feature dimension. SUMMARY

[0003] Embodiments of the present application provide a financial digital marketing opportunity analysis based on chat prompt words, which is used to solve the problem that the prior art is difficult to accurately extract user features.

[0004] To achieve the above purpose, the embodiments of the present application adopt the following technical solutions:

[0005] In a first aspect, a financial digital marketing opportunity analysis method based on chat prompt words is provided, which comprises:

[0006] Collecting conversation information of multiple target users, the conversation information including visual information and non-visual information;

[0007] For any target user, extracting multi-modal conversation features of the target user's visual information and non-visual information;

[0008] Completing multi-modal fusion of multi-modal conversation features through a tensor fusion network to obtain multi-modal fusion features of the target user;

[0009] Using a clustering algorithm to complete feature clustering of the multi-modal fusion features of all target users to obtain multiple feature clusters, taking all feature clusters as hyper-edges and taking target users as hyper-nodes to construct a user feature supergraph;

[0010] Completing community division of the user feature supergraph through a random walk algorithm to obtain multiple user feature communities;

[0011] Completing feature matching between each user feature community and pre-acquired financial product features.

[0012] Optionally, the visual information includes text dialogue information and non-text dialogue information, and the multi-modal dialogue feature of the target user extracted from the visual information and the non-visual information includes the following steps:

[0013] Metadata in the non-text dialogue information is extracted, and the metadata is mapped into dialogue sentiment features by using a pre-trained language model;

[0014] Acoustic signal features of the non-visual information are extracted based on a mel-frequency scale, and sentiment classification of the acoustic signal features is completed by a classification algorithm to obtain acoustic sentiment features;

[0015] The non-text dialogue information is converted into acoustic text information according to the acoustic signal features and by using a text conversion model, and the text conversion model is constructed based on a deep learning model;

[0016] Semantic extraction of the text dialogue information and the acoustic text information is respectively completed based on a topic model to obtain text semantic features and acoustic semantic features of the target user;

[0017] The dialogue sentiment features, the acoustic sentiment features, the text semantic features, and the acoustic semantic features are integrated to obtain the multi-modal dialogue feature.

[0018] Optionally, the acoustic signal features of the non-visual information extracted based on the mel-frequency scale include the following steps:

[0019] A sampling and quantization coding step of the non-visual information is completed to obtain dialogue digital signals;

[0020] The dialogue digital signals are pre-emphasized, and the dialogue digital signals after pre-emphasis are windowed and framed to obtain a plurality of stationary dialogue signals;

[0021] For any stationary dialogue signal, the stationary dialogue signal is decomposed into a plurality of dialogue signal components, and fast Fourier transform of each dialogue signal component is completed to obtain a stationary dialogue spectrum;

[0022] The stationary dialogue spectrum is input into a mel-frequency spectrum filter, and logarithmic compression of the stationary dialogue spectrum is completed by the mel-frequency spectrum filter to obtain a spectrum logarithmic sequence;

[0023] After discrete cosine transform of all the spectrum logarithmic sequences is completed, the acoustic signal features of the non-visual information are output.

[0024] Optionally, the text conversion model includes an acoustic module and a language module, the acoustic module is constructed based on a Conformer encoder, and the language module includes a first language unit and a second language unit, the first language unit is constructed based on a CTC decoder, and the second language unit is constructed based on an RNN-T decoder.

[0025] Optionally, converting the non-text dialogue information into acoustic text information according to the acoustic signal feature and using the text conversion model comprises the following steps:

[0026] The acoustic fundamental frequency feature of the non-visual information is extracted by a fundamental frequency extraction algorithm, and the feature fusion of the acoustic fundamental frequency feature and the acoustic signal feature is completed to obtain an acoustic fusion feature;

[0027] The acoustic fusion feature is input into an acoustic module, and the acoustic fusion feature is converted into a high-dimensional acoustic feature based on the acoustic module and by capturing the context dependence of the acoustic fusion feature;

[0028] The high-dimensional acoustic feature is mapped to a preset character label space by a first language unit, and a plurality of initial text sequences are generated according to the mapping result and by a search algorithm;

[0029] The rescoring of all initial text sequences is completed by a second language unit, and an optimal text sequence is selected according to the rescoring result;

[0030] The optimal text sequence is output as acoustic text information by the text conversion model.

[0031] Optionally, the multi-modal fusion of the multi-modal dialogue feature is completed by a tensor fusion network to obtain the multi-modal fusion feature of the target user, comprising the following steps:

[0032] The multi-modal dialogue feature is vectorized to obtain a multi-modal feature vector;

[0033] The multi-modal feature vector is normalized;

[0034] Each sub-feature vector in the normalized multi-modal feature vector is expanded into a dialogue sub-tensor, and each dialogue sub-tensor is spliced into an initial dialogue tensor by tensor outer product operation;

[0035] The tensor dimension reduction and tensor compression of the initial dialogue tensor are completed based on the tensor decomposition technology to obtain the multi-modal fusion feature of the target user.

[0036] Optionally, the community division of the user feature supergraph is completed by a random walk algorithm to obtain a plurality of user feature communities, comprising the following steps:

[0037] The intra-cluster density of all target users in all feature clusters is calculated, and a plurality of seed user sets are selected according to the intra-cluster density;

[0038] The user feature supergraph is converted into a supergraph expansion graph by a supergraph expansion method;

[0039] For any sub-user set, starting from the seed user in the seed user set, a random walk is performed in the supergraph expansion graph to obtain a walk probability vector of all target users;

[0040] extracting a user reference set from the user feature expansion graph in combination with the walk probability vector and the node degree of each node in the user feature expansion graph;

[0041] expanding the user feature supergraph based on the user reference set to obtain the user feature expansion graph;

[0042] calculating a user overlap degree between the user reference set and the whole seed user set, and defining a walk constraint function based on the user overlap degree;

[0043] performing community division of the user feature expansion graph by taking the walk constraint function as an objective function and using a random walk algorithm to obtain a plurality of user feature communities.

[0044] Optionally, the extracting of the user reference set from the user feature expansion graph in combination with the walk probability vector and the node degree of each node in the user feature expansion graph comprises the following steps:

[0045] calculating a comprehensive score of all target users in combination with the walk probability vector and the node degree of each node in the user feature expansion graph and using a user score calculation formula, the user score calculation formula being as follows:

[0046]

[0047] wherein, is a preset weight parameter, is the maximum node degree in the supergraph expansion graph, is the node degree of node , and is the walk probability vector of node .

[0048] integrating the target users with a comprehensive score greater than a preset score threshold into the user reference set.

[0049] In a second aspect, the present application provides a machine readable storage medium, characterized in that the machine readable storage medium has instructions stored thereon, the instructions being used to cause a machine to execute the method for financial digital marketing opportunity analysis based on chat prompt words according to any one of the first aspect.

[0050] In a third aspect, the present application provides a system for financial digital marketing opportunity analysis based on chat prompt words, characterized in that the system comprises:

[0051] a memory configured to store instructions; and

[0052] a processor configured to call the instructions from the memory and capable of implementing the method for financial digital marketing opportunity analysis based on chat prompt words according to any one of the first aspect when executing the instructions.

[0053] By the technical solution, the dialogue information of each target user is first collected, and the information of each target user is divided into visual information and non-visual information, and dialogue emotion features, acoustic signal features, text semantic features and acoustic semantic features of the visual information and the non-visual information are extracted based on a topic model and a mel frequency scale respectively. The features of the visual information and the non-visual information are extracted by different methods, so that the extracted features are more accurate, which is conducive to improving the matching accuracy between the target user and the financial product. In addition, in order to mine deep features of the target user, the corresponding features of the visual information and the non-visual information are fused into multi-modal fusion features by a tensor fusion network. The user feature supergraph between all target users is constructed by using the multi-modal fusion features, so as to realize feature aggregation of all target users, thereby improving the extraction accuracy of the user features and the extraction efficiency of the user features.

[0054] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific embodiments part. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 A flowchart of a financial digital marketing opportunity analysis method based on chat prompt words provided by the embodiments of the present application;

[0056] Figure 2 A structure diagram of a text conversion model provided by the embodiments of the present application;

[0057] Figure 3 An example diagram of supergraph conversion provided by the embodiments of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the specific embodiments described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0059] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement condition, etc. between components in a certain specific posture (as shown in the drawings), and if the specific posture changes, the directional indications also change accordingly.

[0060] In addition, if the description of "first", "second" and the like is involved in the embodiments of the present application, the description of "first", "second" and the like is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can be explicitly or implicitly included at least one of the features. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the realization of ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor in the protection scope claimed by the present application.

[0061] Figure 1 The flowchart of the method for analyzing financial digital marketing opportunities based on chat prompt words according to the embodiments of the present application is schematically shown. As shown in Figure 1 , the embodiments of the present application provide a method for analyzing financial digital marketing opportunities based on chat prompt words, which can include the following steps:

[0062] S101. Collecting the conversation information of multiple target users, the conversation information including visual information and non-visual information.

[0063] In the present embodiment, through the third-party interface, such as the SDK tool of the applet or the cooperation platform, the conversation data of the target user is captured in real time under the premise of authorization of the target user. The visual information includes the text sent by the target user to the conversation object in the conversation process, and the image information such as the emoticon package sent by the target user to the conversation object in the conversation process. The non-visual information refers to the voice information sent by the target user to the conversation object. In addition, in the process of collecting the conversation information, the dynamic data filtering mechanism is established in advance, and the bank card number, password and other private data are excluded in real time in the collection process through the technical means such as keyword shielding and sensitive information desensitization (such as replacing the ID card number with "##"), to ensure the information security of the target user. For the collected conversation information, standardization processing is required, such as unified text encoding, standard time format, data source channel labeling, etc., to lay a foundation for subsequent data analysis and model training. In addition, the integration of the collection process will be recorded through the log system throughout the process, including collection time, data volume, etc., in order to trace and audit, and ensure the monitorability of the collection process and the traceability of the data.

[0064] S102. For any target user, the multi-modal conversation features of the visual information and the non-visual information of the target user are extracted.

[0065] ​In this embodiment, the visualization information includes text conversation information and non-text conversation information, the text conversation information is text sent by the target user to the conversation object during the conversation process, the non-text data refers to image information such as emoticon packs sent by the target user to the conversation object during the conversation process, and the metadata refers to label information stored when the emoticon pack is created, such as "happy", "sad", "angry", etc. The metadata of the emoticon pack can be read through a corresponding target user chat platform open programming interface (API). Then the obtained metadata is input into a pre-trained language model, and the conversation emotion features of the metadata are extracted through the pre-trained language model. Common pre-trained language models include BERT (Bidirectional Encoder), RoBERTa (Optimized Bidirectional Encoder), etc.

[0066] Then the acoustic signal features of the non-visualization information are extracted based on the mel frequency scale, and the acoustic emotion features are classified according to the acoustic emotion features. The mel frequency scale is a kind of nonlinear frequency scale based on the sensory judgment of the human ear to the equal distance of pitch change. Since the perception of frequency by the human ear is not linear, there are more filters in the low frequency area and fewer in the high frequency area, so the use of mel scale can better simulate the hearing characteristics of the human ear. Then the text semantic features and acoustic semantic features of the text conversation information and acoustic text information are extracted by using the topic model respectively. Common topic models include LDA (Latent Dirichlet Allocation), BERTopic, etc. The conversation emotion features, acoustic emotion features, text semantic features and acoustic semantic features are integrated to obtain multi-modal conversation features.

[0067] S103. The multi-modal fusion of the multi-modal conversation features is completed through the tensor fusion network to obtain the multi-modal fusion features of the target user.

[0068] In the embodiment, the multi-modal dialogue features of different sources are first processed into multi-modal feature vectors, and then the multi-modal feature vectors are normalized. The Z-score standardization method or the Min-Max normalization method can be used to compress the features into the interval of 0 to 1. The normalization eliminates the dimensional differences of different modal features and avoids the large numerical sub-feature vectors from dominating the fusion process. After normalization, a feature dimension is added to each sub-feature vector in the multi-modal dialogue features to obtain dialogue sub-tensors. Then, the outer product operation is performed between each dialogue sub-tensor, and the outer product operation result is used as the initial dialogue tensor. Since the dimension of the initial dialogue tensor can be large, the initial dialogue tensor needs to be reduced in dimension, i.e., tensor dimension reduction and tensor compression. Common tensor decomposition techniques include C decomposition and Tucker decomposition. Taking Tucker decomposition as an example, the Tucker decomposition decomposes the initial dialogue tensor into a core tensor and multiple factor matrices. The core tensor captures the high-order interaction between modalities, and the factor matrices represent the low-dimensional representation of each modality, which can be extended by singular value decomposition (SVD). Then, the alternating least squares (ALS) is used to alternately optimize each factor matrix and the core tensor to gradually approach the optimal solution until the maximum iteration number is reached, and the final core tensor and multiple factor matrices are obtained, thereby realizing tensor dimension reduction and tensor compression. The final core tensor and multiple factor matrices are output as the multi-modal fusion features of the target user. By this method, the dimension is reduced, and the interaction information between modalities is preserved.

[0069] S104. The feature clustering of the multi-modal fusion features of all target users is completed by using the clustering algorithm to obtain multiple feature clusters. All feature clusters are used as hyper-edges, and the target user is used as a hyper-node to construct a user feature hypergraph.

[0070] In this embodiment, commonly used clustering algorithms include density peak clustering (DPC), HDBSCAN, K-means, and spectral clustering algorithms. Taking HDBSCAN as an example, DBSCAN (Density-based Spatial Clustering of Applications with Noise) is a density-based clustering algorithm based on expansion. The principle is to find core objects and continuously expand to include samples in the neighborhood of core objects (determine whether it is a core object, and then continue to expand), thereby forming a cluster. Samples that are not in the neighborhood of any core object and cannot form core objects are considered outliers and are not assigned to any cluster. Specifically, the minimum number of samples in the core point neighborhood and the minimum cluster size are set. The minimum number of samples affects the sensitivity of the density threshold, and the minimum cluster size determines the minimum size of the cluster. Then calculate the core distance of each feature point in the multi-modal fusion feature, that is, the distance from the feature point to the adjacent point, and calculate the mutual reachable distance between each feature point using the Euclidean distance or cosine similarity formula. Then, the feature points are taken as nodes, and the reachable distance between the feature points is taken as the weight to construct a minimum spanning tree. Starting from the leaf nodes of the minimum spanning tree, the sub-clusters are merged upwards to generate parent clusters, and the reachable distance between each node and other nodes during each merge is recorded. Then, the reachable distance when each cluster is first formed (split from the parent cluster) and the reachable distance when merged into a larger cluster (parent cluster) are calculated, and the stability of each cluster is calculated at this time. Then, starting from the root node, recursively check each sub-cluster downward. If the sub-cluster is greater than the minimum cluster, and the stability of the sub-cluster is higher than that of its parent cluster, then it is cut out as a feature cluster. Otherwise, if the sub-cluster is less than or equal to the minimum cluster, or the stability of the sub-cluster is less than or equal to that of its parent cluster, then it is not cut and the entire parent cluster is taken as a feature cluster. The stability calculation formula is as follows:

[0071]

[0072] wherein, is the target parent cluster, is the node in the target parent cluster, is the node is the reachable distance when the cluster is first formed (split from the parent cluster), is the node is the reachable distance when merged into a larger cluster (parent cluster).

[0073] Each target user is taken as a hyper node, and each feature cluster is taken as a hyper edge containing all target users belonging to the feature cluster. The weight of the hyper edge can be assigned according to the number of target users in the feature cluster. The more the number of target users, the greater the weight assigned to the corresponding hyper edge. After the above steps, the user feature supergraph containing the multi-modal fusion features of all target users is obtained. The user feature supergraph is used for subsequent batch matching of financial products with high matching degrees for all target users.

[0074] S105. The community division of the user feature supergraph is completed by using a random walk algorithm, and a plurality of user feature communities are obtained.

[0075] In this embodiment, the intra-cluster density of all target users in all feature clusters is first calculated. All feature clusters are arranged in descending order according to the intra-cluster density. K feature clusters with the top K intra-cluster densities (for example, K = 5) are selected as seed feature clusters. The m hyper nodes with the highest node degrees in the seed feature clusters are selected as seed users. The seed users in each seed feature cluster are integrated into a seed user set. Then, the user feature supergraph is expanded into a supergraph expansion graph by using a supergraph expansion method. Common supergraph expansion methods include clique expansion, star expansion, and line graph method. Then, for any seed user set, the seed users in the seed user set are taken as the starting point for random walk in the supergraph expansion graph, and the walk probability vector of all target users is obtained. The comprehensive score of each target user is calculated by combining the walk probability vector and the node degree of each node in the user feature expansion graph. The plurality of target users with the highest comprehensive scores are integrated into a user reference set. Then, the user feature supergraph is expanded into a user feature expansion graph with the user reference set as the core. The user overlap degree between the user reference set and all seed users, i.e., the seed user set, is calculated. Then, the random walk penalty is defined, and the walk constraint function is defined by combining the user overlap degree and the random walk penalty. The minimum walk penalty coefficient is solved based on the walk constraint function, and the user feature community is obtained.

[0076] S106. The feature matching between each user feature community and the pre-acquired financial product features is completed.

[0077] In this embodiment, the cut size and community volume of each user feature community, and the community volume of the non-user feature community are first calculated. The non-user feature community refers to the set of hyper nodes in the user feature supergraph after removing the user feature community. The cut size refers to the sum of the weights of all hyper edges connecting the user feature community and the non-user feature community. The community volume refers to the sum of the node degrees of all hyper nodes in the user feature community. The node degree refers to the number of hyper edges connected to the hyper node. Then, the community conductance of the user feature community is calculated by using the conductance formula. The conductance formula is as follows:

[0078]

[0079] wherein, denotes the size of a user feature community denotes the size of a user feature community denotes the size of a user feature community denotes the size of a user feature community denotes the size of a user feature community denotes the set of other supernodes in the user feature hypergraph after removing the user feature community from the user feature hypergraph.

[0080] Then, the contribution of each hyperedge in each user feature community to the community conductance is calculated according to the community conductance and the hyperedge weight, for example, the weight proportion of the weight of a certain hyperedge in the total weight can be calculated first, and the product of the weight proportion and the community conductance is calculated to obtain the contribution of the hyperedge. Then, the community centrality of each user feature community is calculated, the node degree of each supernode is calculated respectively, and the node degrees of all supernodes are integrated to obtain the community centrality of the user feature community. Then, the core hyperedges with a contribution greater than a preset contribution threshold and the core supernodes with a node degree greater than a preset degree threshold are screened out, and then the multi-modal fusion features corresponding to the core hyperedges and the core supernodes are taken as the user core features.

[0081] The sources of the pre-acquired financial product features include structured documents (such as product brochures, filing documents), unstructured texts (such as market analysis reports, regulatory announcements), etc. For structured documents, the fields such as risk level, investment period, expected return, etc. are automatically extracted through table recognition technology (OCR), and the extracted fields are cleaned using regular expressions, such as converting “12 months” to the numerical format of “1 year”, thereby obtaining the financial product features. For unstructured texts, features can be extracted through natural language processing techniques, such as using a named entity recognition (NER) model to identify keywords from product promotional texts, such as “national debt” and “Shanghai and Shenzhen 300 index”, and then extracting key attributes of the keywords through syntactic analysis, such as “10 million yuan of initial investment” and “T+3 redemption T+3”, thereby obtaining the financial product features.

[0082] Then the user core features and the pre-acquired financial product features are mapped into feature vectors to obtain user feature vectors and product feature vectors. The feature mapping method includes TF-IDF, Word2Vec or BERT embedding method, or entity embedding, which maps the user core features and the financial product features into low-dimensional vectors, such as mapping a certain feature into [0.12, -0.45, …, 0.28]. Finally, the similarity formula (such as cosine similarity) can be used to calculate the feature similarity between each user feature vector and other all product feature vectors. According to the feature similarity, the target user and the financial product are matched, that is, the product feature vectors with a feature similarity greater than a preset similarity threshold are screened out, the financial products corresponding to the screened product feature vectors are matched with the target user corresponding to the user feature vector, and then a financial product recommendation list is generated for each target user according to the matching result.

[0083] The application also provides a machine-readable storage medium, characterized in that the machine-readable storage medium stores instructions for causing a machine to execute the method of chat prompt word-based financial digital marketing business analysis according to any one of the above.

[0084] In one embodiment, the visualization information includes text conversation information and non-text conversation information, and the multi-modal conversation features of the target user are extracted by the following steps:

[0085] Metadata in the non-text conversation information is extracted, and the metadata is mapped into conversation sentiment features by using a pre-trained language model;

[0086] Acoustic signal features of the non-visual information are extracted based on the Mel frequency scale, and the acoustic signal features are classified by a classification algorithm to obtain acoustic sentiment features;

[0087] The non-text conversation information is converted into acoustic text information based on the acoustic signal features and by using a text conversion model, and the text conversion model is constructed based on deep learning;

[0088] The semantic extraction of the text conversation information and the acoustic text information is completed based on a topic model to obtain text semantic features and acoustic semantic features of the target user;

[0089] The conversation sentiment features, the acoustic sentiment features, the text semantic features and the acoustic semantic features are integrated to obtain multi-modal conversation features.

[0090] In the embodiment, the visualization information includes text conversation information and non-text conversation information, the text conversation information is text sent by the target user to the conversation object during the conversation process, the non-text data refers to image information such as sticker sent by the target user to the conversation object during the conversation process, and the metadata refers to label information stored when the sticker is created, such as "happy", "sad", "angry", etc. The metadata of the sticker can be read through a corresponding programming interface (API) opened by the chat platform of the target user. Then the obtained metadata is input into a pre-trained language model, and the conversation emotion features of the metadata are extracted through the pre-trained language model. Common pre-trained language models include BERT (Bidirectional Encoder), RoBERTa (Optimized Bidirectional Encoder), etc. Taking BERT as an example, the metadata is preprocessed first, and the preprocessing steps include data cleaning of the metadata to remove noise information, and splitting the description words into basic units through word segmentation technology to obtain metadata word segmentation. In addition, for composite emotion words, semantic disassembly or emotion weight labeling methods can be used for processing, for example, "angry and laugh" can be disassembled into "angry" and "joy" and assigned with corresponding proportions. Finally, word embedding technology is used, such as mapping metadata word segmentation to a vector through Word2Vec or the word segmenter of the pre-trained model. The metadata word vector ensures that its input format meets the input requirements of the BERT model. The metadata word vector is input into the BERT model, and the semantic association between words is captured through the multi-layer self-attention mechanism of the BERT model, such as the semantic relationship between "tears" and "sadness" and "moved". Then the hidden layer representation containing emotion information is generated. Then the full connection layer and the activation function of the BERT model are used for classification, and the conversation emotion features are output.

[0091] Then, the acoustic signal features of the non-visual information are extracted based on a mel-frequency scale. The mel-frequency scale is a non-linear frequency scale based on the perceptual judgment of the human ear to the equal distance of the pitch change. Since the human ear is not linear in frequency perception, there are more filters in the low frequency region and fewer in the high frequency region, so the use of mel scale can better simulate the auditory characteristics of the human ear. Specifically, the analog-to-digital conversion of the non-visual information is completed through a sampling quantization encoding step, converting the analog signal of the non-visual information into a digital signal to obtain a dialogue digital signal. Then, the dialogue digital signal is preprocessed, and the preprocessing step includes pre-emphasis processing and windowing and framing processing, which makes the dialogue digital signal smoother to obtain a stable dialogue digital signal, i.e., a stable dialogue signal. Then, any stable dialogue signal is decomposed into multiple dialogue signal components using an empirical mode decomposition or ensemble empirical mode decomposition method. Then, the fast Fourier transform is performed on each dialogue signal component to convert the time domain signal to the frequency domain to obtain the frequency spectrum information of each dialogue signal component. The frequency spectrum information of all dialogue signal components is added to obtain the stable dialogue spectrum. Then, the stable dialogue spectrum is input into a Mel filter to complete the weighted average of the stable dialogue spectrum to obtain a filter output value, and the filter output value is logarithmically operated to obtain a frequency spectrum logarithmic sequence. Finally, the frequency spectrum logarithmic sequence is converted from the frequency domain to the inverse frequency domain by discrete cosine transform to obtain the mel-frequency cepstral coefficient (MFCC), i.e., the acoustic signal feature.

[0092] Common classification algorithms include support vector machines, random forests, etc. Taking the support vector machine as an example, the core of SVM is to find the optimal hyperplane that maximizes the sample interval. First, select a suitable kernel function, such as linear kernel, polynomial kernel and Gaussian kernel, etc. And use grid search and K-fold cross-validation to optimize the parameters of SVM, then use the SVM with optimized parameters to perform emotion classification on the acoustic signal features, and finally output the classification result according to the decision function value and the confidence threshold to obtain the acoustic emotion features.

[0093] Then the topic model is used to extract the text semantic features and acoustic semantic features of the text dialogue information and acoustic text information respectively. Commonly used topic models include LDA (Latent Dirichlet Allocation), BERTopic, etc. Taking the BERTopic model as an example, first, the text dialogue information and acoustic text information are respectively cleaned of noise information (such as links and invalid symbols, etc.) and standardized in text format. Then, the text is segmented into basic lexical units, and stop words such as "of", "of", "in" and other words without substantive semantics are removed to reduce irrelevant information interference. Then, using the segmentation tool such as jieba, the text dialogue information and acoustic text information after data cleaning are respectively segmented to obtain the text dialogue word set and acoustic text word set. After the above steps, the semantic features of the two are extracted using the BERTopic model. BERTopic combines BERT embedding and traditional topic modeling methods, generates high-dimensional semantic embedding through pre-trained language models, and thus better handles context information and captures the complex relationships between words. It is a topic modeling technology based on BERT pre-trained model, and its core process includes text embedding generation, dimensionality reduction clustering and topic representation. Specifically, the pre-trained BERT model (such as bert-base-uncased or Chinese bert-base-chinese) is used to vectorize the text dialogue word set and acoustic text word set respectively, and each word in it is mapped to a high-dimensional semantic vector. Then, text-level embedding is generated through average pooling or CLS token, etc. These embeddings capture the deep semantic associations of the text, for example, the embedding vectors of "movie plot wonderful" and "film plot attractive" are close in semantic space. After generating text-level embedding, UMAP (Uniform Manifold Approximation and Projection) algorithm is used to reduce the dimension of high-dimensional text-level embedding. UMAP can map data to low-dimensional space while preserving semantic similarity, facilitating subsequent clustering. After dimensionality reduction, the data is clustered by HDBSCAN (Hierarchical Density-Based Clustering) algorithm. HDBSCAN can automatically identify clustering boundaries based on data density, adapt to different sizes of topics, and avoid the limitations of manually setting the number of clusters, for example, texts discussing "stocks" and "funds" are clustered into different categories, and a small amount of irrelevant text is identified as noise. After clustering, BERTopic generates the distribution probability of the topic keywords by calculating the probability distribution of each cluster and the embedding similarity of the words. The distribution probability of the topic keywords is used as the text semantic feature and acoustic semantic feature of the target user respectively. The dialogue sentiment features, acoustic sentiment features, text semantic features and acoustic semantic features are integrated to obtain multi-modal dialogue features.

[0094] In one embodiment, the acoustic signal features of the non-visual information based on the Mel frequency scale include the following steps:

[0095] The sampling and quantization coding step of the non-visualized information is completed to obtain a dialogue digital signal;

[0096] The dialogue digital signal is pre-emphasized, and the dialogue digital signal after pre-emphasis is windowed and framed to obtain a plurality of stationary dialogue signals;

[0097] For any stationary dialogue signal, the stationary dialogue signal is decomposed into a plurality of dialogue signal components, and the fast Fourier transform of each dialogue signal component is completed to obtain a stationary dialogue spectrum;

[0098] The stationary dialogue spectrum is input into a mel-frequency spectrum filter, and the logarithmic compression of the stationary dialogue spectrum is completed by the mel-frequency spectrum filter to obtain a spectrum logarithmic sequence;

[0099] The discrete cosine transform of all spectrum logarithmic sequences is completed to output the acoustic signal characteristics of the non-visualized information.

[0100] In the embodiment, the purpose of the sampling and quantization coding step is to convert the analog signal of the non-visualized information into a digital signal. Through a preset sampling frequency, such as 8000 Hz or 16000 Hz, the continuous analog signal (non-visualized information) is sampled to obtain discrete sampling points, and the amplitude of each sampling point is quantized and mapped to a limited number of quantization levels, such as 8-bit or 16-bit quantization. After quantization, the non-visualized information is encoded, which means that the non-visualized information after sampling and quantization is converted into a binary value, i.e., a dialogue digital signal is obtained.

[0101] Pre-emphasis is a method of performing first-order high-pass filtering on the dialogue digital signal, which reduces the influence of pulses on the dialogue digital signal and makes the dialogue digital signal smoother. The principle is that the high-frequency part of the dialogue digital signal usually has weak energy, so the convolution operation of the dialogue digital signal is completed through a filter (such as a first-order filter) to enhance the high-frequency part. After pre-emphasis, the dialogue digital signal needs to be windowed and framed. The dialogue digital signal is first divided into a plurality of signal segments, each signal segment becomes a frame, and a frame is usually 20 to 30 milliseconds, and the frame shift can be about 10 milliseconds. In addition, in order to reduce the discontinuity of the frame boundary of each signal segment, a window function needs to be applied to each signal segment, i.e., the product of each frame signal and the window function is calculated. Common window functions include Hamming window, Hanning window, etc. The function of the window function is to make the signal at both ends of the frame smoothly transition to avoid spectral leakage, thereby obtaining a stationary dialogue signal, i.e., a stationary dialogue signal.

[0102] Then any stationary dialogue signal is decomposed into multiple dialogue signal components using methods such as empirical mode decomposition or ensemble empirical mode decomposition. Taking empirical mode decomposition as an example, empirical mode decomposition (EMD) is an adaptive signal decomposition method that decomposes a signal into a series of intrinsic mode functions (IMF) by continuously filtering and decomposing. Each IMF is a single component signal that reflects the change pattern of the signal in a few time periods. The decomposition steps include: extracting the local extreme points of each stationary dialogue signal, including local maximum points and local minimum points, connecting all local maximum points and local minimum points with cubic spline lines to form upper and lower envelope lines, calculating the mean of the local extreme points corresponding to the upper and lower envelope lines, and subtracting the mean from the stationary dialogue signal to obtain a signal without low frequency trend. Repeat the above steps of extracting extreme points, constructing upper and lower envelope lines, calculating mean and removing until the stopping condition is met (for example, the standard deviation of IMF is less than the standard deviation threshold). Repeat the above steps until no more IMF can be decomposed, and obtain multiple IMFs, i.e. dialogue signal components. Then perform fast Fourier transform on each dialogue signal component to convert the time domain signal to the frequency domain and obtain the frequency spectrum information of each dialogue signal component. Add the frequency spectrum information of all dialogue signal components to obtain the stationary dialogue spectrum.

[0103] Then map the stationary dialogue spectrum to the Mel frequency scale, which is a nonlinear frequency scale based on the sensory judgment of the human ear to the equal interval of pitch change. Since the perception of frequency by the human ear is not linear, but has more filters in the low frequency region and fewer in the high frequency region, using the Mel scale can better simulate the auditory characteristics of the human ear. Specifically, the stationary dialogue spectrum is passed through a set of Mel filters, i.e. Mel frequency spectrum filters, the weighted average of the stationary dialogue spectrum is completed through the Mel filter, the filter output value is obtained, and the logarithmic operation is performed on the filter output value to obtain the spectrum logarithmic sequence. The formula of this process is as follows:

[0104]

[0105] wherein, is the stationary dialogue spectrum is the spectrum logarithmic sequence obtained after completing the Mel frequency scale mapping.

[0106] Finally, the log-spectrum sequence is converted from the frequency domain to the cepstrum domain by a discrete cosine transform, and the resulting coefficients are called mel-frequency cepstral coefficients (MFCCs). The acoustic signal feature is the mel-frequency cepstral coefficient. The discrete cosine transform is a transform related to the Fourier transform, which is similar to the discrete Fourier transform, but only uses real numbers. The discrete cosine transform is equivalent to a discrete Fourier transform of approximately twice its length on a real even function (because the Fourier transform of a real even function is still a real even function). The core is to represent the log-spectrum sequence as a linear combination of a series of cosine functions.

[0107] Through the above steps, the acoustic signal features in the non-visual information that conform to the human auditory perception law can be extracted, so that they can more accurately reflect the essential features of the user's voice in the subsequent application process, thereby making the matching result between the subsequent user and the financial product more accurate.

[0108] In one of the embodiments, the text conversion model includes an acoustic module and a language module, the acoustic module is constructed based on a Conformer encoder, and the language module includes a first language unit and a second language unit, the first language unit is constructed based on a CTC decoder, and the second language unit is constructed based on an RNN-T decoder.

[0109] In the present embodiment, the Conformer encoder is stacked by multiple Conformer units, each of which includes a feed-forward subunit (FFN), a multi-head self-attention subunit (MHSA) and a convolution subunit. The feed-forward subunit (FFN) is mainly composed of a normalization layer, a linear layer, an activation layer and a regularization layer. The feed-forward subunit (FFN) can enhance the expression ability of acoustic fusion features through two layers of linear transformation (expansion coefficient 4) and Swish activation function. The purpose of the multi-head self-attention subunit (MHSA) is to find the correlation between vectors and vectors, so as to capture the context dependence of acoustic fusion features, which is the key to understanding the context information of non-text information. It mainly projects the input acoustic fusion features into three feature dimensions, which are called Q, K, V, representing Query, Key, Value respectively. In the process of calculating the attention weight, Q and K are first mapped to the same high-dimensional space to calculate the similarity between them. Then the above calculation result is normalized by using the softmax function to obtain the weight distribution of attention. Finally, the Value value is weighted and summed to obtain the final self-attention weight. The convolution subunit includes depth separable convolution and gated linear unit (GLU), which is used to extract local features of acoustic fusion features. The CTC decoder can automatically align the frame sequence of speech in time and the corresponding transcription text sequence in the model training process, without labeling the start and end time period of each character or phoneme, so as to realize direct classification on time sequence, and also avoid the relatively complex alignment operation of traditional algorithm. The RNN-T decoder is a decoder for sequence-to-sequence tasks, which includes a prediction network and a fusion network, mainly based on multi-layer long short-term memory network (LSTM) and gated recurrent unit (GRU) construction.

[0110] The training steps of the text conversion model include: first, a large number of labeled speech-text pairs are collected, wherein the speech data needs to be pre-processed into digital signals through sampling, framing, windowing, etc., and then acoustic features are extracted through Mel frequency cepstral coefficient (MFCC) or filter bank energy, and fundamental frequency features are extracted through YIN or SWIPE algorithm and fused with acoustic features to form acoustic fusion features; the text data needs to be processed through character standardization and encoding to form a label sequence recognizable by the model. The model training process can adopt a multi-stage optimization strategy, that is, the acoustic module is first pre-trained, and the parameters of the Conformer encoder are optimized using the CTC loss function to accurately convert the acoustic fusion features into high-dimensional acoustic features. This stage focuses on enabling the model to learn the preliminary mapping relationship between acoustic features and character labels, and generates an initial text sequence through greedy search or beam search and calculates the CTC loss, that is, maximizes the alignment probability of acoustic features and text labels. Then, the RNN-T decoder is introduced for end-to-end joint training, and the input of the model is the acoustic fusion features and the output is the text sequence, and the training target is to optimize the CTC loss and the RNN-T loss. The RNN-T loss calculates the joint probability of acoustic features, text sequences and alignment paths through dynamic programming, which not only considers acoustic matching degree, but also models language context dependence through a prediction network to make the generated text more consistent with language rules. During training, the weights of the two losses need to be balanced, and the super parameter is usually adjusted, for example, a higher weight is given to the CTC loss to ensure accurate mapping of acoustic features, and the weight of the RNN-T loss is gradually increased to strengthen the language modeling capability. In addition, in order to improve the robustness and generalization ability of the text conversion model, data enhancement techniques can be used during training, such as adding background noise to the speech data, performing speed perturbation, or replacing synonyms in the text data, to simulate speech changes in different scenarios. The optimizer usually selects adaptive algorithms such as Adam, combined with learning rate scheduling strategies such as cosine annealing or exponential decay, to avoid the model falling into local optimum and accelerate convergence. At the same time, during the training process, the model performance of the text conversion model is verified through a pre-constructed validation set, and the word error rate or sentence error rate evaluation indicators can be used, and when the word error rate or sentence error rate decreases to a preset error rate threshold, the training process is stopped, and the trained text conversion model is obtained.

[0111] In one embodiment, converting non-text dialogue information into acoustic text information according to acoustic signal features and using a text conversion model includes the following steps:

[0112] extracting acoustic fundamental frequency features of non-visual information through a fundamental frequency extraction algorithm, and completing feature fusion of acoustic fundamental frequency features and acoustic signal features to obtain acoustic fusion features;

[0113] input the acoustic fusion feature into an acoustic module, and convert the acoustic fusion feature into a high-dimensional acoustic feature based on the acoustic module and by capturing context dependency of the acoustic fusion feature;

[0114] map the high-dimensional acoustic feature to a preset character label space through a first language unit, and generate a plurality of initial text sequences according to a mapping result and by searching algorithm;

[0115] complete re-scoring of all the initial text sequences through a second language unit, and screen out an optimal text sequence according to a re-scoring result;

[0116] output the optimal text sequence as acoustic text information through a text conversion model.

[0117] The base frequency extraction algorithm commonly used in the present embodiment includes YIN algorithm, SWIPE algorithm and the like. Taking the YIN algorithm as an example, the YIN algorithm is a time-domain base frequency extraction algorithm widely used in audio signal processing, mainly used for pitch detection and base frequency estimation, and has the characteristics of fast operation speed and high accuracy. Its core idea is to analyze the periodicity of the signal through the time-domain autocorrelation function. The specific steps of extracting the acoustic base frequency feature of the non-visual information include: pre-processing the dialogue digital signal. The pre-processing step mainly divides the dialogue digital signal into multiple fixed-length signal frames through windowing and framing processing, such as dividing it into frames with a time length of about 20 to 40 milliseconds. This is done to allow each signal frame to be analyzed independently, while adding a Hanning window to each signal frame to reduce spectral leakage and make subsequent analysis more accurate. Then, normalize each signal frame to eliminate the influence of amplitude differences between different signal frames. Then, for each signal frame, define its delay parameter, which is usually 1 to 1 / 2 of the signal frame length, and calculate the original difference function of each signal frame according to the delay parameter. Then, in order to eliminate the influence of signal amplitude and noise on the original difference function, the original difference function is normalized to obtain the normalized difference function. Then, search for the first delay point that makes the normalized difference function less than the preset empirical threshold (such as 0.15). If the delay point is within the preset delay range, such as between 20Hz and 500Hz, then the delay point is taken as a candidate base frequency period point. If the delay point is not within the preset delay range and no other delay point that meets the conditions can be found, then the signal frame is determined to be aperiodic, such as a clear tone or silence, and the base frequency at this time is set to 0 or the signal frame is skipped. Finally, in order to further improve the base frequency feature extraction accuracy, perform parabolic interpolation around the candidate base frequency period point, that is, take the candidate base frequency period point and its two adjacent delay points, fit a parabola with the normalized difference function values of the three delay points, and then calculate the minimum point of the parabola to obtain the base frequency period point. Then, in the sampling quantization encoding step, the sampling frequency of the non-visual information is divided by the period value of the base frequency period point to obtain the base frequency in hertz, which is the acoustic base frequency feature. After extracting the acoustic base frequency feature, the acoustic base frequency feature and the acoustic signal feature are fused to obtain the acoustic fusion feature. Common feature fusion methods include feature splicing, attention mechanism fusion, and fusion based on a gating mechanism.

[0118] Reference Figure 2The acoustic fusion feature is input into an acoustic module in a text conversion model, long-distance context dependence of the acoustic fusion feature is captured by a Conformer encoder in the acoustic module, the input acoustic fusion feature is first subjected to one convolutional secondary sampling, then a plurality of Conformer units are stacked to process the acoustic fusion feature subjected to the secondary sampling, the Conformer unit comprises a feedforward subunit (FFN), a multi-head self-attention subunit (MHSA) and a convolutional subunit. The feedforward subunit (FFN) mainly comprises a normalization layer, a linear layer, an activation layer and a regularization layer, the feedforward subunit (FFN) can enhance the expression capability of the acoustic fusion feature through two layers of linear transformation (expansion coefficient 4) and a Swish activation function. The multi-head self-attention subunit (MHSA) aims to find the correlation between vectors and vectors, so as to capture the context dependence of the acoustic fusion feature, and is the key to understanding the context information of non-text information. The input acoustic fusion feature is mainly projected onto three feature dimensions, which are referred to as Q, K and V, representing Query, Key and Value respectively. In the process of calculating the attention weight, Q and K are first mapped to the same high-dimensional space to obtain the similarity of the two. Then, the above calculation result is normalized by using a softmax function to obtain the weight distribution of the attention. Finally, the Value value is weighted and summed to obtain the final self-attention weight. The convolutional subunit comprises a depth separable convolution and a gated linear unit (GLU), which is used to extract local features of the acoustic fusion feature. In addition, batch normalization (BN) is added in the convolutional module to normalize the value of a single neuron to stabilize the data distribution and accelerate the training of the model. After the acoustic fusion feature passes through the Conformer encoder, high-level representations containing context information are learned through multi-layer self-attention and convolution operations, and high-dimensional acoustic features are output. This process is similar to the process of human auditory system extracting semantic information from sound, and the acoustic module gradually abstracts phonemes, words and other high-dimensional acoustic features from low-level acoustic fusion features, laying a foundation for subsequent language modules.

[0119] The language module comprises a first language unit and a second language unit, the first language unit is constructed based on a CTC decoder, and the second language unit is constructed based on an RNN-T decoder. The character label space is a discrete character set comprising all basic units of text to be recognized by the text conversion model. The CTC decoder automatically aligns the speech frames and the character sequence by introducing a blank label “_”, without explicit frame labeling. The processing procedure thereof comprises: firstly, converting high-dimensional acoustic features into a character probability distribution at each time step through a full connection layer; secondly, using a search algorithm to find the most possible character sequence in the character probability distribution, and removing continuous repetitions and blank blocks to generate an initial text sequence. Common search algorithms include greedy method, beam search method and prefix beam search method, etc. Taking the beam search method as an example, the K characters with the highest probability are selected from the probability distribution as K initial candidate sequences. For each initial candidate sequence, all possible next characters are expanded to obtain a new sequence, and then the cumulative probability of the new sequence is calculated, which is equivalent to the product of the original distribution probability. Each initial candidate sequence can be expanded into V new sequences (V is the size of the vocabulary), and K initial candidate sequences can generate KxV new sequences. Finally, the K sequences with the highest probability are selected from the KxV new sequences as the next candidates, until all sequences meet the termination condition (such as the appearance of a termination symbol or the maximum length is reached), and K initial text sequences are obtained.

[0120] Then, the second language unit is used to complete the rescoring of all initial text sequences, and the optimal text sequence is selected according to the rescoring result. The second language unit is based on an RNN-T decoder, and the RNN-T decoder comprises multiple layers of long short-term memory networks (LSTM) and gated recurrent units (GRU). The LSTM is used as a predictor unit, which generates a current context state based on the previous characters for each character position in any initial text sequence, and passes the context state of each character in the time step to capture the time-dependent sequence of the initial text sequence, thereby obtaining the semantic features of each character position. Then, the semantic features of the character position and its corresponding high-dimensional acoustic features are transformed into joint representations through a nonlinear transformation, and then a character probability distribution of all joint representations is generated through a softmax function. Then, the joint probability of each initial text sequence is calculated through a dynamic programming algorithm (such as a forward algorithm). Then, the initial text sequence with the highest joint probability is selected as the optimal text sequence, and the optimal text sequence is taken as the final output result, i.e., the acoustic text information.

[0121] The CTC encoder can quickly align speech and text when processing high-dimensional acoustic features, ensuring the accuracy of character-level matching. The re-scoring using the RNN-T encoder introduces the ability of the language model, which can judge whether the initial text sequence conforms to the logic and habits of the language as a whole, such as avoiding grammatical errors and incoherent situations. By fusing the CTC decoder and the RNN-T decoder, the optimal text sequence can be accurately screened from two different perspectives of the local and the global. While accurately identifying each character, the fluency and rationality of the entire sentence are also ensured, greatly improving the quality of the final text.

[0122] In one embodiment, the multi-modal fusion of the multi-modal dialogue features is completed by a tensor fusion network to obtain the multi-modal fusion features of the target user, including the following steps:

[0123] Vectorizing the multi-modal dialogue features to obtain multi-modal feature vectors;

[0124] Normalizing the multi-modal feature vectors;

[0125] Each sub-feature vector in the multi-modal feature vector after normalization is expanded into a dialogue sub-tensor, and each dialogue sub-tensor is spliced into an initial dialogue tensor through tensor outer product operation;

[0126] Tensor dimension reduction and tensor compression of the initial dialogue tensor are completed based on tensor decomposition technology to obtain the multi-modal fusion features of the target user.

[0127] In the embodiment, the multi-modal dialogue features of different sources are first processed into multi-modal feature vectors, and then the multi-modal feature vectors are normalized. The Z-score standardization method can be used, that is, the mean and standard deviation of each sub-feature vector in each multi-dimensional feature are calculated, and the feature values of each sub-feature vector are converted into a distribution with a mean of 0 and a standard deviation of 1, or the Min-Max normalization is used to compress the features into the interval of 0 to 1. The dimensional difference of different modal features is eliminated through normalization, and the fusion process is avoided to be dominated by the sub-feature vector with large value. The sub-feature vector is obtained by vectorizing the dialogue emotion feature, acoustic emotion feature, text semantic feature and acoustic semantic feature. After normalization, a feature dimension is added to each sub-feature vector in the multi-modal dialogue feature, and a dialogue sub-tensor is obtained. For example, the text semantic feature is generally in the form of word order, and then the BERT model is used to generate word embedding vectors, that is, sub-feature vectors, each word order corresponds to a 768-dimensional word embedding vector. At this time, the original format of the multi-modal dialogue feature is a two-dimensional matrix, for example, sentence length x word vector dimension. By adding a unit dimension to the two-dimensional matrix, a dialogue sub-tensor is obtained, for example, sentence length x word vector dimension x 1. Then, the outer product operation is performed between each dialogue sub-tensor, and the outer product operation result is taken as the initial dialogue tensor. The outer product operation refers to calculating the product of all elements between two dialogue sub-tensors. Then, since the dimension of the initial dialogue tensor can be very large, it is necessary to perform dimension reduction processing on the initial dialogue tensor, that is, to perform tensor dimension reduction and tensor compression. Common tensor decomposition techniques include CP decomposition, Tucker decomposition, etc. Taking Tucker decomposition as an example, Tucker decomposition decomposes the initial dialogue tensor into a core tensor and multiple factor matrices. The core tensor captures the high-order interaction between modalities, and the factor matrix represents the low-dimensional representation of each modality, which can be expanded by singular value decomposition (SVD). Then, the alternating least squares (ALS) is used to alternately optimize each factor matrix and the core tensor, and the optimal solution is gradually approached until the maximum iteration number is reached, and the final core tensor and multiple factor matrices are obtained, thereby realizing tensor dimension reduction and tensor compression. The final core tensor and multiple factor matrices are taken as the multi-modal fusion features of the target user. Through this method, the dimension is reduced, and the interaction information between modalities is preserved.

[0128] In one embodiment, the community division of the user feature supergraph is completed by a random walk algorithm, and a plurality of user feature communities are obtained, including the following steps:

[0129] The intra-cluster density of all target users in all feature clusters is calculated, and a plurality of seed user sets are selected according to the intra-cluster density;

[0130] The user feature supergraph is converted into a supergraph expansion graph by using a supergraph expansion method;

[0131] For any one of the seed user sets, a random walk is performed in the hypergraph expansion graph starting from a seed user in the seed user set to obtain a walk probability vector of all target users;

[0132] The node degrees of each node in the user feature expansion graph are combined with the walk probability vector, and a comprehensive score of all target users is calculated using a user score calculation formula as follows:

[0133]

[0134] wherein, is a preset weight parameter, is the maximum node degree in the hypergraph expansion graph, is the node degree of node , and is the walk probability vector of node .

[0135] Target users with a comprehensive score greater than a preset score threshold are integrated into a user reference set.

[0136] The user feature hypergraph is expanded based on the user reference set to obtain a user feature expansion graph.

[0137] The user overlap degree between the user reference set and all seed user sets is calculated, and a walk constraint function is defined based on the user overlap degree.

[0138] The community division of the user feature expansion graph is completed using a random walk algorithm with the minimum walk constraint function as an objective function to obtain a plurality of user feature communities.

[0139] In this embodiment, the intra-cluster density of all target users in all feature clusters is first calculated. The intra-cluster density refers to the ratio of the sum of the weights of all hyperedges in each feature cluster to the sum of the node degrees of all hypernodes in each feature cluster. All feature clusters are arranged in descending order of intra-cluster density, and the top K feature clusters (e.g., K = 5) in terms of intra-cluster density are selected as seed feature clusters. The top m hypernodes in terms of node degree in the seed feature clusters are selected as seed users, and the seed users in each seed feature cluster are integrated into a seed user set. For example, if a feature cluster has a high intra-cluster density Q = 0.7 and the target user A1 in the feature cluster has the highest degree d = 50, the target user A1 is selected as a seed user and is included in the seed user set. This step ensures that the seed users come from high-correlation feature clusters and provides a high-quality starting point for subsequent random walks.

[0140] Next, the user feature hypergraph is expanded into an expanded hypergraph using hypergraph expansion methods. Commonly used hypergraph expansion methods include clique expansion, star expansion, and line graph expansion. Taking star expansion as an example, each hyperedge e of the user feature hypergraph is mapped to a vertex in the ordinary graph. If two hyperedges ei and ej share at least one common vertex in the user feature hypergraph, an edge is added between the vertices corresponding to hyperedges ei and ej in the ordinary graph, thus obtaining the expanded hypergraph. (Refer to...) Figure 3 The vertices of the extended hypergraph represent the hyperedges of the user feature hypergraph, and the edges represent intersections of hyperedges. Next, for any seed user set, a random walk is performed in the extended hypergraph starting with the seed users in the seed user set, obtaining the walk probability vector for all target users. The random walk steps include: when transitioning from a supernode to a superedge, if the user is located at supernode v, the probability of selecting the superedge e connected to it is proportional to the weight of the superedge. For example, if the set of superedges connected to supernode v is E(v), and the weight of each superedge e is w(e), then the probability of selecting superedge e is the ratio of w(e) to the sum of the weights of the superedge set E(v). When transitioning from a superedge to a supernode, if the user is located at the superedge node... When a node (a node in the extended hypergraph) is selected, the next hypernode needs to be chosen from all the hypernodes connected by the hyperedge e. Typically, any hypernode in hyperedge e is selected with equal probability; that is, if hyperedge e connects k hypernodes, the probability of each hypernode being selected is 1 / k. After each random walk, the number of times each hyperedge node and hypernode in the extended hypergraph is recorded. The ratio of the number of times each node is visited to the total number of random walks is taken as the original walk probability of that node. For example, in 1000 random walks, if node U is visited 200 times, its original walk probability is initially recorded as 200 / 1000 = 0.2. Then, the relative value between the original walk probability and the degree of each node is calculated to obtain the walk probability vector.

[0141] The comprehensive score for each target user is calculated by combining the wander probability vector and the node degree of each node in the user feature expansion graph. Target users with the highest comprehensive scores are then grouped into a user reference set; that is, target users with comprehensive scores greater than a preset score threshold are grouped into the user reference set. The formula for calculating the comprehensive score is as follows:

[0142]

[0143] in, The preset weighting parameter is, for example, 0.6. The maximum node degree in the extended graph of the hypergraph. For nodes The degree of the node. For nodes The walking probability vector.

[0144] Then, taking the user reference set as the core, the user feature supergraph is expanded into a user feature expanded graph. Specifically, the superedges associated with the user reference set in the user feature supergraph are retained, each superedge containing at least one node in the user reference set, and if two users are associated in the user feature supergraph through multiple superedges, a new superedge with a weight being the sum of the weights of the superedges is added, to obtain the user feature expanded graph.

[0145] The user overlap degree between the user reference set and all seed users, i.e., the seed user set, is calculated, and the user overlap degree calculation formula is as follows:

[0146]

[0147] wherein, is the intersection of the user reference set and the seed user set , is the sum of the node degrees of all nodes in the intersection of the user reference set and the seed user set , is the random walk penalty of all nodes belonging to the user reference set but not belonging to the seed user set . is a preset non-negative parameter, usually greater than 0 and less than 1, used as an adjustment factor. The random walk penalty is as follows:

[0148]

[0149] wherein, is a penalty coefficient, such as 0.5, is the walk probability vector of node , is the sum of the node degrees of all nodes in the user reference set .

[0150] The higher the random walk penalty is, the closer the target user corresponding to the node is associated with the seed user set, and needs to be retained when dividing the community.

[0151] The walk constraint function is defined in combination with the user overlap degree and the random walk penalty, and the walk constraint function is as follows:

[0152]

[0153] wherein, is the walk penalty coefficient of the node set , is the sum of the weights of all superedges divided into the node set .

[0154] The user feature community can be obtained by finding the minimum walk penalty coefficient. The solution steps include: initializing the user feature community. Assuming user characteristic community For user reference set The current wander penalty coefficient is calculated as the initial wander penalty coefficient. Next, a reference hypergraph is constructed containing two types of virtual nodes: source nodes and sink nodes. Source nodes represent the "core region of the seed user set," guiding the community partitioning process towards nodes strongly associated with the seed user set. Sink nodes represent the "non-seed user set region," penalizing nodes weakly associated with the seed user set. The reference hypergraph contains all hypernodes, source nodes, and sink nodes from the user feature hypergraph. Specifically, for each node u in the user reference set, a directed edge is added from the source node to u, with a weight of... ,in, Preset adjustment parameters, Let be the degree of node u. The penalty for random walks at node u. For each node w in the non-user reference set, add a directed edge from node w to the sink node with weight . , It is a preset non-negative parameter. Let w be the degree of node w.

[0155] After constructing the reference hypergraph, a node is found that divides the set of nodes in the reference hypergraph into two complementary and intersecting subsets. Then, the wander penalty coefficient of the partitioned subset is calculated. If the wander penalty coefficient of the subset is less than the initial wander penalty coefficient, the reference hypergraph is repeatedly constructed based on the subset until the wander penalty coefficient of the partitioned subset no longer decreases. The iteration stops, and the final subset is taken as the user feature community.

[0156] This application also provides a machine-readable storage medium, characterized in that the machine-readable storage medium stores instructions for causing a machine to perform a method for analyzing financial digital marketing opportunities based on chat prompt words according to any one of the above.

[0157] This application also provides a financial digital marketing opportunity analysis system based on chat prompts, characterized in that it includes:

[0158] The memory is configured to store instructions; and

[0159] The processor is configured to retrieve instructions from memory and, when executing the instructions, to implement a method for analyzing financial digital marketing opportunities based on chat prompts, according to any of the above.

[0160] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), programmable logic devices (PLD), or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, and the present application is not limited in this regard.

[0161] The memory can be an internal storage unit of the computer device, for example, a hard disk or a memory of the computer device, or an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (SMC), a secure digital card (SD), or a flash memory card (FC) provided on the computer device, etc. The memory can also be a combination of the internal storage unit and the external storage device of the computer device. The memory is used to store computer programs and other programs and data required by the computer device. The memory can also be used to temporarily store data that has been output or will be output, and the present application is not limited in this regard.

[0162] The embodiments of the present application also provide a machine readable storage medium having instructions stored thereon, the instructions being used to cause a machine to execute the method of the financial digital marketing merchant opportunity analysis based on a chat prompt word of the first aspect.

[0163] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can be in the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks

[0165] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0167] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0168] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the computer stores information about an operating system, application software, and / or the like. Memory is an example of computer readable media.

[0169] Computer readable media includes permanent and non-permanent, moveable and non- moveable media that can be implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that is accessible to a computing device. According to the definition provided herein, a computer readable medium excludes transitory media, such as modulated data signals and carrier waves.

[0170] It should also be noted that the terms "comprising", "comprises" or other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0171] The above embodiments are only used to illustrate the present application, but not to limit it. Instead of the above, various modifications and changes can be made to the application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall fall into the scope of the claims of the application.

Claims

1. A chat prompt word-based financial digital marketing lead analysis method, characterized in that, The method comprises the following steps: Collecting dialogue information of multiple target users, the dialogue information comprising visual information and non-visual information; For any target user, extracting multi-modal dialogue features of the target user from the visual information and the non-visual information; Vectorizing the multi-modal dialogue features to obtain multi-modal feature vectors; Normalizing the multi-modal feature vectors; Expanding each sub-feature vector in the normalized multi-modal feature vectors into a dialogue sub-tensor, and splicing each dialogue sub-tensor into an initial dialogue tensor through tensor outer product operation; Completing tensor dimension reduction and tensor compression of the initial dialogue tensor based on tensor decomposition technology to obtain multi-modal fusion features of the target user; Completing feature clustering of the multi-modal fusion features of all target users by using a clustering algorithm to obtain multiple feature clusters, taking all feature clusters as hyper-edges and taking target users as hyper-nodes to construct a user feature hypergraph; Calculating the intra-cluster density of all target users in all feature clusters, and screening multiple seed user sets according to the intra-cluster density; Converting the user feature hypergraph into a hypergraph expansion graph by using a hypergraph expansion method; For any sub-user set, starting from a seed user in the seed user set and randomly walking in the hypergraph expansion graph to obtain a walk probability vector of all target users; Extracting a user reference set from the user feature expansion graph in combination with the walk probability vector and the node degree of each node in the user feature expansion graph; Expanding the user feature hypergraph based on the user reference set to obtain a user feature expansion graph; Calculating the user overlap degree between the user reference set and all seed user sets, and defining a walk constraint function based on the user overlap degree; Completing community division of the user feature expansion graph by taking the minimization of the walk constraint function as an objective function and using a random walk algorithm to obtain multiple user feature communities; Completing feature matching between each user feature community and pre-acquired financial product features.

2. The method of claim 1, wherein, The visual information comprises text dialogue information and non-text dialogue information, and the multi-modal dialogue features of the target user extracted from the visual information and the non-visual information comprise the following steps: Extracting metadata in the non-text dialogue information, and mapping the metadata into dialogue sentiment features by using a pre-trained language model; Extracting acoustic signal features of the non-visual information based on a mel frequency scale, and completing sentiment classification of the acoustic signal features by using a classification algorithm to obtain acoustic sentiment features; Converting the non-text dialogue information into acoustic text information according to the acoustic signal features and by using a text conversion model, the text conversion model being constructed based on a deep learning model; Respectively completing semantic extraction of the text dialogue information and the acoustic text information based on a topic model to obtain text semantic features and acoustic semantic features of the target user; Integrating the dialogue sentiment features, the acoustic sentiment features, the text semantic features and the acoustic semantic features to obtain the multi-modal dialogue features.

3. The method of claim 2, wherein, The method for extracting acoustic signal features of the non-visual information based on a mel frequency scale comprises the following steps: Completing a sampling and quantization encoding step of the non-visual information to obtain dialogue digital signals; Pre-emphasizing the dialogue digital signals, and performing windowing and framing processing on the dialogue digital signals after pre-emphasis to obtain multiple stationary dialogue signals; For any stationary dialogue signal, the stationary dialogue signal is decomposed into multiple dialogue signal components, and a fast Fourier transform of each dialogue signal component is completed to obtain a stationary dialogue spectrum; The stationary dialogue spectrum is input into a mel-frequency spectrum filter, and the stationary dialogue spectrum is logarithmically compressed through the mel-frequency spectrum filter to obtain a spectrum logarithm sequence; After completing the discrete cosine transform of all the spectrum logarithm sequences, the acoustic signal features of the non-visual information are output.

4. The method of claim 2, wherein, The text conversion model includes an acoustic module and a language module, the acoustic module is constructed based on a Conformer encoder, the language module includes a first language unit and a second language unit, the first language unit is constructed based on a CTC decoder, and the second language unit is constructed based on an RNN-T decoder.

5. The method of claim 4, wherein, The conversion of the non-text dialogue information into acoustic text information according to the acoustic signal features and by using the text conversion model includes the following steps: An acoustic fundamental frequency feature of the non-visual information is extracted through a fundamental frequency extraction algorithm, and feature fusion of the acoustic fundamental frequency feature and the acoustic signal feature is completed to obtain an acoustic fusion feature; The acoustic fusion feature is input into the acoustic module, and the acoustic fusion feature is converted into a high-dimensional acoustic feature based on the acoustic module and by capturing context dependence of the acoustic fusion feature; The high-dimensional acoustic feature is mapped to a preset character label space through the first language unit, and a plurality of initial text sequences are generated according to a mapping result and by using a search algorithm; Rescoring of all the initial text sequences is completed through the second language unit, and an optimal text sequence is selected according to a rescoring result. The optimal text sequence is output as the acoustic text information through the text conversion model.

6. The method of claim 1, wherein, The user reference set is extracted from the user feature expansion graph by combining the walk probability vector and the node degree of each node in the user feature expansion graph, and the comprehensive score of all target users is calculated by using a user score calculation formula, and the user score calculation formula is as follows: The target users with a comprehensive score greater than a preset score threshold are integrated into the user reference set. ， wherein, is a preset weight parameter, is the maximum node degree in the hypergraph expansion graph, is a node of the node degree, is a walk probability vector of a node ; The machine-readable storage medium stores instructions for causing a machine to execute the method for financial digital marketing opportunity analysis based on a chat prompt word according to any one of claims 1 to 6.

7. A machine-readable storage medium, characterized in that, Comprise:

8. A chat prompt word-based financial digital marketing lead analysis system, characterized in that, a memory configured to store instructions; and a processor configured to call the instructions from the memory and capable of implementing the method for financial digital marketing opportunity analysis based on a chat prompt word according to any one of claims 1 to 6 when executing the instructions. ​

Images