Financial digital marketing business opportunity analysis method and system based on chat prompt words

Abstract

The invention discloses a financial digital marketing business opportunity analysis method based on chat cues, and relates to the field of marketing analysis, and the method comprises the steps: collecting the dialogue information of a plurality of target users; extracting multi-modal dialogue features of the visual information and the non-visual information of the target user; multi-modal fusion of the multi-modal dialogue features is completed; constructing a user feature hypergraph; community division of the user feature hypergraph is completed, and a plurality of user feature communities are obtained; and completing feature matching between each user feature community and pre-acquired financial product features, and outputting a financial marketing strategy according to a feature matching result. The method can effectively improve the extraction precision of the user features.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of marketing analysis, and in particular to a method and system for analyzing financial digital marketing opportunities based on chat prompts. Background Art

[0002] With the deep integration of internet technology and financial services, data-driven precision marketing has become a core requirement for industry development. A key step in achieving digital marketing is accurately matching target users with appropriate financial products. However, existing methods for accurately matching products to target users primarily extract financial keywords from target users' chats and match users with products based on the similarity between the target users' financial keywords and the characteristics of financial products. Due to sparse data and a single dimension of user characteristics, this method's extracted user features fail to accurately reflect the user's true profile. Summary of the Invention

[0003] The embodiment of the present application provides a financial digital marketing opportunity analysis based on chat prompt words, which is used to solve the problem that the existing technology is difficult to accurately extract user features.

[0004] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions: In a first aspect, a method for analyzing financial digital marketing business opportunities based on chat prompts is provided, the method comprising: Collect conversation information from multiple target users, including visual and non-visual information; For any target user, extract the multimodal conversation features of the target user's visual information and non-visual information; The multimodal fusion of multimodal conversation features is completed through the tensor fusion network to obtain the multimodal fusion features of the target user; The clustering algorithm is used to complete the feature clustering of the multimodal fusion features of all target users, and multiple feature clusters are obtained. All feature clusters are used as hyperedges, and the target users are used as supernodes to construct a user feature hypergraph; The user feature hypergraph is divided into communities using a random walk algorithm to obtain multiple user feature communities. Complete feature matching between each user feature community and pre-acquired financial product features.

[0005] Optionally, the visual information includes text conversation information and non-text conversation information, and extracting multimodal conversation features of the visual information and non-visual information of the target user includes the following steps: Extract metadata from non-text conversation information and use a pre-trained language model to map the metadata into conversation sentiment features; The acoustic signal features of non-visual information are extracted based on the Mel frequency scale, and the sentiment classification of the acoustic signal features is completed through the classification algorithm to obtain the acoustic sentiment features; Based on the acoustic signal characteristics, the non-text conversation information is converted into acoustic text information using a text conversion model built on a deep learning model. Based on the topic model, semantic extraction of text conversation information and acoustic text information is completed to obtain the text semantic features and acoustic semantic features of the target user; The multimodal dialogue features are obtained by integrating dialogue emotion features, acoustic emotion features, text semantic features and acoustic semantic features.

[0006] Optionally, extracting acoustic signal features of non-visual information based on the Mel frequency scale includes the following steps: Complete the sampling, quantization and encoding steps of non-visual information to obtain the dialogue digital signal; Pre-emphasize the conversation digital signal, and perform windowing and framing processing on the pre-emphasized conversation digital signal to obtain multiple smooth conversation signals; For any stationary conversation signal, the stationary conversation signal is decomposed into multiple conversation signal components, and the fast Fourier transform of each conversation signal component is performed to obtain the stationary conversation spectrum; The spectrum of the steady conversation is input into the Mel frequency spectrum filter, and the Mel frequency spectrum filter is used to perform logarithmic compression of the spectrum of the steady conversation to obtain a spectrum logarithmic sequence; After completing the discrete cosine transform of the entire spectral logarithmic sequence, the acoustic signal features of non-visual information are output.

[0007] Optionally, the text conversion model includes an acoustic module and a language module, the acoustic module is constructed based on a Conformer encoder, the language module includes a first language unit and a second language unit, the first language unit is constructed based on a CTC decoder, and the second language unit is constructed based on an RNN-T decoder.

[0008] Optionally, converting the non-text conversation information into acoustic text information based on the acoustic signal features and using a text conversion model includes the following steps: The acoustic fundamental frequency features of non-visual information are extracted through the fundamental frequency extraction algorithm, and the acoustic fundamental frequency features and acoustic signal features are fused to obtain the acoustic fusion features. Input the acoustic fusion features into the acoustic module, and based on the acoustic module and by capturing the context dependency of the acoustic fusion features, convert the acoustic fusion features into high-dimensional acoustic features; Mapping high-dimensional acoustic features to a preset character label space through the first language unit, and generating multiple initial text sequences based on the mapping results through a search algorithm; Complete the rescoring of all initial text sequences using the second language unit, and select the optimal text sequence based on the rescoring results; The optimal text sequence is output as acoustic text information through the text conversion model.

[0009] Optionally, performing multimodal fusion of multimodal conversation features through a tensor fusion network to obtain multimodal fusion features of the target user includes the following steps: Vectorize the multimodal conversation features to obtain a multimodal feature vector; Normalize multimodal feature vectors; Expand each sub-feature vector in the normalized multimodal feature vector into a dialogue sub-tensor, and concatenate each dialogue sub-tensor into the initial dialogue tensor through a tensor outer product operation; Based on tensor decomposition technology, tensor dimensionality reduction and tensor compression of the initial conversation tensor are completed to obtain the multimodal fusion features of the target user.

[0010] Optionally, the user feature hypergraph is divided into communities using a random walk algorithm to obtain multiple user feature communities, including the following steps: Calculate the intra-cluster density of all target users in all feature clusters, and select multiple seed user sets based on the intra-cluster density; Using the hypergraph expansion method to transform the user feature hypergraph into a hypergraph extension graph; For any seed user set, randomly walk in the hypergraph expansion graph starting from the seed user in the seed user set to obtain the walk probability vector of all target users; The user reference set is extracted from the user feature expansion graph by combining the walk probability vector and the node degree of each node in the user feature expansion graph; Expand the user feature hypergraph based on the user reference set to obtain a user feature expansion graph; Calculate the user overlap between the user reference set and the entire seed user set, and define the walk constraint function based on the user overlap; Taking minimizing the walk constraint function as the objective function and using the random walk algorithm to complete the community division of the user feature expansion graph, multiple user feature communities are obtained.

[0011] Optionally, extracting a user reference set from the user feature expansion graph by combining the walk probability vector and the node degree of each node in the user feature expansion graph includes the following steps: The comprehensive scores of all target users are calculated by combining the walk probability vector and the node degree of each node in the user feature expansion graph and using the user score calculation formula. The user score calculation formula is as follows:

[0012] in, is the preset weight parameter, is the maximum node degree in the hypergraph expansion graph, For nodes The node degree of For nodes The walk probability vector of .

[0013] Target users whose comprehensive scores are greater than the preset score threshold are integrated into the user reference set.

[0014] In a second aspect, the present application provides a machine-readable storage medium, characterized in that instructions are stored on the machine-readable storage medium, and the instructions are used to enable a machine to execute any one of the methods for analyzing financial digital marketing opportunities based on chat prompts in the first aspect.

[0015] In a third aspect, the present application provides a financial digital marketing business opportunity analysis system based on chat prompt words, characterized by including: a memory configured to store instructions; and The processor is configured to call instructions from a memory and, when executing the instructions, implement the method for analyzing financial digital marketing business opportunities based on chat prompt words according to any one of the first aspects.

[0016] Through the above technical solution, conversation information of each target user is first collected and divided into visual and non-visual information. Based on the topic model and Mel-frequency scale, conversation sentiment features, acoustic signal features, text semantic features, and acoustic semantic features are extracted from these two types of information, respectively. By extracting features of visual and non-visual information through different methods, the extracted features are more accurate, which helps improve the accuracy of subsequent matching between target users and financial products. Furthermore, to explore the deeper characteristics of target users, the corresponding features of visual and non-visual information are fused into multimodal fusion features through a tensor fusion network. Multimodal fusion features are then used to construct a user feature hypergraph between all target users, achieving feature aggregation for all target users. This improves both the accuracy and efficiency of user feature extraction.

[0017] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a method for analyzing financial digital marketing opportunities based on chat prompts provided in an embodiment of the present application; Figure 2 A structural diagram of a text conversion model provided in an embodiment of the present application; Figure 3 This is an example diagram of the hypergraph conversion provided in the embodiments of the present application. DETAILED DESCRIPTION

[0019] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0020] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), such directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0021] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0022] Figure 1 The following schematically shows a flow chart of a method for analyzing financial digital marketing business opportunities based on chat prompts according to an embodiment of the present application. Figure 1 As shown, the embodiment of the present application provides a method for analyzing financial digital marketing business opportunities based on chat prompt words, which may include the following steps: S101. Collect conversation information of multiple target users, where the conversation information includes visual information and non-visual information.

[0023] In this embodiment, through a third-party interface, such as a mini-program or SDK tool of a cooperative platform, the conversation data of the target user is captured in real time under the premise of authorization. Visual information includes text sent by the target user to the conversation partner during the conversation, as well as image information such as emoticons sent by the target user to the conversation partner during the conversation. Non-visual information refers to the voice information sent by the target user to the conversation partner. In addition, during the process of collecting conversation information, a dynamic data filtering mechanism will be pre-established to block keywords, desensitize sensitive information (such as ID card numbers with " "" replacement) and other technical means are used to remove private data such as bank card numbers and passwords in real time during the collection process to ensure the information security of target users. The collected conversation information needs to be standardized, such as unified text encoding, standardized time format, and annotation of data source channels, laying the foundation for subsequent data analysis and model training. In addition, the integrated collection process will be fully recorded in the log system, including collection time and data volume, for traceability and auditing, ensuring the monitorability of the collection process and traceability of the data.

[0024] S102. For any target user, extract multimodal conversation features of the target user's visual information and non-visual information.

[0025] In this embodiment, the visualized information includes both textual and non-textual conversation information. Textual conversation information refers to the text sent by the target user to the conversation partner during the conversation. Non-textual data refers to image information, such as emoticons, sent by the target user to the conversation partner during the conversation. Metadata refers to the label information stored when the emoticon is created, such as "happy," "sad," or "angry." This helps users quickly search for desired emoticons. The emoticon metadata can be read through the API provided by the target user's chat platform. The metadata is then fed into a pre-trained language model to extract emotional features from the metadata. Commonly used pre-trained language models include BERT (Bidirectional Encoder) and RoBERTa (Optimized Bidirectional Encoder).

[0026] Next, acoustic signal features of non-visual information are extracted based on the Mel frequency scale, and acoustic emotion features are classified based on acoustic emotion features. The Mel frequency scale is a nonlinear frequency scale based on the human ear's sensory perception of equally spaced pitch changes. Because the human ear's frequency perception is not linear, with more filters in the low-frequency region and fewer in the high-frequency region, the Mel frequency scale better simulates the human ear's auditory characteristics. Next, topic models are used to extract textual semantic features and acoustic semantic features of the textual conversation information and acoustic text information, respectively. Common topic models include LDA (Latent Dirichlet Allocation) and BERTopic. By integrating conversation emotion features, acoustic emotion features, textual semantic features, and acoustic semantic features, multimodal conversation features are obtained.

[0027] S103. Complete multimodal fusion of multimodal conversation features through a tensor fusion network to obtain multimodal fusion features of the target user.

[0028] In this embodiment, multimodal conversation features from different sources are first quantized to obtain a multimodal feature vector. The multimodal feature vector is then normalized, either using the Z-score method or Min-Max normalization to compress the features to the range of 0 to 1. Normalization eliminates dimensional differences between modal features and prevents large sub-feature vectors from dominating the fusion process. After normalization, each sub-feature vector in the multimodal conversation feature is added with a feature dimension to obtain a conversation sub-tensor. The outer product operation is then performed on each conversation sub-tensor, and the result is used as the initial conversation tensor. Because the initial conversation tensor may have a large dimensionality, it is necessary to perform dimensionality reduction, namely, tensor dimensionality reduction and compression. Common tensor decomposition techniques include C decomposition and Tucker decomposition. Taking Tucker decomposition as an example, it decomposes the initial conversation tensor into a core tensor and multiple factor matrices. The core tensor captures the high-order interactions between the modalities, while the factor matrices represent the low-dimensional representation of each modality, which can generally be expanded through singular value decomposition (SVD). The alternating least squares (ALS) method is then used to alternately optimize each factor matrix and the core tensor, gradually approaching the optimal solution until the maximum number of iterations is reached. The final core tensor and multiple factor matrices are obtained, thus achieving tensor dimensionality reduction and compression. The final core tensor and multiple factor matrices are then output as the multimodal fusion features of the target user. This method not only reduces the dimensionality but also preserves the interaction information between the modalities.

[0029] S104. Use a clustering algorithm to complete feature clustering of the multimodal fusion features of all target users to obtain multiple feature clusters. Use all feature clusters as hyperedges and target users as supernodes to construct a user feature hypergraph.

[0030] In this embodiment, commonly used clustering algorithms include density peak clustering (DPC), HDBSCAN, K-means, and spectral clustering. Taking HDBSCAN as an example, DBSCAN (Density-based Spatial Clustering of Applications with Noise) is a clustering algorithm based on density expansion. Its principle is to form a cluster by finding a core object and continuously expanding the samples added to the neighborhood of the core object (determining whether it is a core object and then continuing to expand). Samples that are not in the neighborhood of any core object and cannot form a core object are regarded as outliers and are not assigned to any cluster. Specifically, the minimum number of samples and the minimum cluster size in the neighborhood of the core point are first set. The minimum number of samples affects the sensitivity of the density threshold, and the minimum cluster size determines the minimum size of the cluster. Then, the core distance of each feature point in the multimodal fusion feature is calculated, that is, the distance from the feature point to the neighboring point, and the Euclidean distance or cosine similarity formula is used to calculate the mutual reachable distance between each feature point. Then, the feature points are used as nodes, and the reachable distance between the feature points is used as the weight to construct a minimum spanning tree. Starting from the leaf node of the minimum spanning tree, the subclusters are merged upward to generate the parent cluster, and the reachable distance between each merged node and other nodes is recorded. Then, the reachable distance of each cluster when it is first formed (split from the parent cluster) and the reachable distance when it is merged into the larger cluster (parent cluster) are calculated. This time is also marked, and the stability of each cluster is calculated by combining the two. Then, starting from the root node, each subcluster is recursively checked downward. If the subcluster is larger than the minimum cluster and the stability of the subcluster is higher than that of its parent cluster, it is cut out as a feature cluster. Conversely, if the subcluster is less than or equal to the minimum cluster, or the stability of the subcluster is less than or equal to the stability of its parent cluster, it is not cut, and the entire parent cluster is used as the feature cluster. The stability calculation formula is as follows:

[0031] in, is the target parent cluster, is the node in the target parent cluster, For nodes The reachability distance when the cluster is first formed (split from the parent cluster), For nodes The reachable distance when merging into a larger cluster (parent cluster).

[0032] Each target user is considered a supernode, and each feature cluster is considered a hyperedge. This includes all target users belonging to that feature cluster. Weights are assigned to hyperedges based on the number of target users in the feature cluster. The greater the number of target users, the greater the weight assigned to the corresponding hyperedge. After completing the above steps, a user feature hypergraph containing the multimodal fusion features of all target users is obtained. This is then used to batch match all target users with highly compatible financial products.

[0033] S105. Complete community division of the user feature hypergraph through a random walk algorithm to obtain multiple user feature communities.

[0034] In this embodiment, the intra-cluster density of all target users within all feature clusters is first calculated. All feature clusters are then sorted in descending order based on intra-cluster density. The top K feature clusters (e.g., K = 5) with the highest intra-cluster density are selected as seed feature clusters. The top m supernodes in the seed feature clusters by node degree are selected as seed users. The seed users within each seed feature cluster are then integrated into a seed user set. The user feature hypergraph is then expanded into an extended hypergraph using a hypergraph expansion method. Common hypergraph expansion methods include clique expansion, star expansion, and line graph. Next, for any seed user set, a random walk is performed within the extended hypergraph, starting from a seed user in the seed user set, to obtain a walk probability vector for all target users. A comprehensive score for each target user is calculated by combining the walk probability vector and the node degree of each node in the user feature extended graph. The target users with the highest comprehensive scores are then integrated into a user reference set. The user feature hypergraph is then expanded into an extended user feature graph, with the user reference set as the core. The user overlap between the user reference set and all seed users, i.e., the seed user set, is calculated. Then, a random walk penalty is defined. The walk constraint function is defined by combining the user overlap and the random walk penalty. The minimum walk penalty coefficient is solved based on the walk constraint function to obtain the user feature community.

[0035] S106. Complete feature matching between each user feature community and pre-acquired financial product features.

[0036] In this embodiment, the cut size and community volume of each user-feature community, as well as the community volume of the non-user-feature community, are first calculated. The non-user-feature community refers to the set of other supernodes in the user-feature hypergraph after removing the user-feature community. The cut size refers to the sum of all hyperedge weights connecting the user-feature community and the non-user-feature community. The community volume refers to the sum of the node degrees of all supernodes in the user-feature community. The node degree refers to the number of hyperedges connected to the supernode. Then, the community conductivity of the user-feature community is calculated using the conductivity formula. The conductivity formula is as follows:

[0037] in, Refers to user characteristic communities The cutting size, Refers to user characteristic communities The community volume, refers to the community volume of non-user characteristic communities, Refers to the user feature hypergraph Remove user characteristics from the community The set of other super nodes after .

[0038] Next, based on the community conductivity and hyperedge weights, the contribution of each hyperedge in each user-characteristic community to the community conductivity is calculated. For example, the weight of a particular hyperedge in the total weight can be calculated first, and the product of the weight percentage and the community conductivity can be calculated to obtain the hyperedge's contribution. The community centrality of each user-characteristic community is then calculated, and the node degree of each supernode is calculated separately. The node degrees of all supernodes are then integrated to obtain the community centrality of the user-characteristic community. Core hyperedges with contributions greater than a preset contribution threshold and core supernodes with node degrees greater than a preset degree threshold are then screened out. The multimodal fusion features corresponding to the core hyperedges and core supernodes are then used as the user's core features.

[0039] The sources of pre-acquired financial product features include structured documents (such as product manuals and filing documents) and unstructured text (such as market analysis reports and regulatory announcements). For structured documents, fields such as risk level, investment period, and expected return are automatically extracted through optical character recognition (OCR) technology, and the extracted fields are cleaned using regular expressions, such as converting "12 months" to the numerical format of "1 year" to obtain financial product features. For unstructured text, features can be extracted through natural language processing technology, such as using named entity recognition (NER) models to identify keywords from product promotional copy, such as "government bonds" and "Shanghai and Shenzhen 300 Index". Then, through syntactic analysis, the key attributes of the keywords are extracted, such as "minimum investment amount of 100,000 yuan" and "redemption T+3 arrival", thereby obtaining financial product features.

[0040] Next, the user's core features and pre-acquired financial product features are mapped into feature vectors, resulting in user and product feature vectors. Feature mapping methods include embedding methods such as TF-IDF, Word2Vec, or BERT. Alternatively, entity embedding can be used to map the user's core features and financial product features into low-dimensional vectors, such as mapping a feature vector to [0.12, -0.45, ..., 0.28]. Finally, a similarity formula (such as cosine similarity) is used to calculate the feature similarity between each user feature vector and all other product feature vectors. Based on this feature similarity, the target user is matched with the financial products. Specifically, product feature vectors whose feature similarity with the user feature vector exceeds a preset similarity threshold are selected. The financial products corresponding to these selected product feature vectors are then matched with the target user corresponding to the user feature vector. Based on the matching results, a list of recommended financial products is generated for each target user.

[0041] The present application also provides a machine-readable storage medium, characterized in that the machine-readable storage medium stores instructions, which are used to enable a machine to execute the method of financial digital marketing business opportunity analysis based on chat prompt words according to any of the above items.

[0042] In one embodiment, the visual information includes text conversation information and non-text conversation information, and extracting multimodal conversation features of the visual information and non-visual information of the target user includes the following steps: Extract metadata from non-text conversation information and use a pre-trained language model to map the metadata into conversation sentiment features; The acoustic signal features of non-visual information are extracted based on the Mel frequency scale, and the sentiment classification of the acoustic signal features is completed through the classification algorithm to obtain the acoustic sentiment features; Based on the acoustic signal characteristics, the non-text conversation information is converted into acoustic text information using a text conversion model built on deep learning. Based on the topic model, semantic extraction of text conversation information and acoustic text information is completed to obtain the text semantic features and acoustic semantic features of the target user; The multimodal dialogue features are obtained by integrating dialogue emotion features, acoustic emotion features, text semantic features and acoustic semantic features.

[0043] In this embodiment, the visualized information includes both textual and non-textual conversation information. Textual conversation information refers to the text sent by the target user to the conversation partner during the conversation. Non-textual data refers to image information, such as emoticons, sent by the target user to the conversation partner during the conversation. Metadata refers to the label information stored when the emoticon is created, such as "happy," "sad," or "angry." This helps users quickly search for desired emoticons. The emoticon metadata can be accessed through the API (Application Programming Interface) provided by the target user's chat platform. The obtained metadata is then input into a pre-trained language model to extract the sentiment characteristics of the metadata. Common pre-trained language models include BERT (Bidirectional Encoder) and RoBERTa (Optimized Bidirectional Encoder). Taking BERT as an example, the metadata is first pre-processed. This pre-processing step includes data cleaning to remove noise and using word segmentation to break down descriptive words into basic units, resulting in metadata segmentation. Furthermore, for compound sentiment words, semantic decomposition or sentiment weighting can be employed. For example, the phrase "angry and laughing" can be decomposed into "angry" and "joyful," with corresponding proportions assigned. Finally, word embedding techniques, such as Word2Vec or the pre-trained model's built-in tokenizer, are used to map metadata tokens into vectors. The metadata word vectors are generated, ensuring that their input format meets the BERT model's input requirements. The metadata word vectors are then fed into the BERT model. The model's multi-layer self-attention mechanism captures semantic connections between words, such as the semantic link between "tears" and "sadness" and "touched," generating a hidden layer representation containing sentiment information. Classification is then performed using the BERT model's fully connected layers and activation functions, outputting sentiment features for the conversation.

[0044] Next, the acoustic signal features of the non-visual information are extracted using the Mel frequency scale, a nonlinear frequency scale based on the human ear's sensory perception of equally spaced pitch changes. Since the human ear's frequency perception is not linear, with more filters in the low-frequency region and fewer in the high-frequency region, the Mel scale better simulates the human ear's auditory characteristics. Specifically, the analog-to-digital conversion of the non-visual information is completed through sampling, quantization, and encoding, converting the analog signal into a digital signal to obtain a conversational digital signal. The conversational digital signal is then preprocessed, including pre-emphasis and windowing and framing, to smooth the conversational digital signal and obtain a stationary conversational digital signal, also known as a stationary conversational signal. The stationary conversational signal is then decomposed into multiple conversational signal components using methods such as empirical mode decomposition (EMD) or ensemble empirical mode decomposition (EMD). Each conversational signal component is then subjected to a fast Fourier transform (FFT) to convert the time domain signal to the frequency domain, obtaining the spectrum of each conversational signal component. The spectrum of all conversational signal components is summed to obtain the stationary conversational spectrum. The spectrum of the stationary conversation is then fed into a Mel filter, which performs a weighted average of the spectrum to obtain the filter output. This filter output is then logarithmically calculated to produce a logarithmic spectrum sequence. Finally, a discrete cosine transform is used to convert the logarithmic spectrum sequence from the frequency domain to the cepstral domain. The resulting coefficients are called Mel-Frequency Cepstral Coefficients (MFCCs), which are the acoustic signal features.

[0045] Common classification algorithms include support vector machines and random forests. Taking support vector machines as an example, the core of SVMs is to find the optimal hyperplane that maximizes the sample margin. First, an appropriate kernel function is selected, such as a linear kernel, a polynomial kernel, or a Gaussian kernel. Grid search and K-fold cross-validation are then used to optimize the SVM parameters. The optimized SVM is then used to perform sentiment classification based on acoustic signal features. Finally, the classification result is output based on the decision function value and confidence threshold, resulting in the acoustic sentiment signature.

[0046] Next, the topic model is used to extract the text semantic features and acoustic semantic features of the text dialogue information and acoustic text information respectively. Commonly used topic models include LDA (Latent Dirichlet Allocation), BERTopic, etc. Taking the BERTopic model as an example, first, data cleaning is performed on the text dialogue information and acoustic text information respectively to remove noise information (such as links and invalid symbols, etc.) and standardize the text format. Then, word segmentation is carried out, splitting the text into basic lexical units, and at the same time removing stop words such as "的", "了", "在" and other words without substantial semantics to reduce interference from irrelevant information. Then, using word segmentation tools such as jieba, word segmentation is performed on the text dialogue information and acoustic text information that have completed data cleaning respectively, obtaining the text dialogue word set and the acoustic text word set. After completing the above steps, the BERTopic model is used to extract the semantic features of the two. BERTopic combines BERT embeddings and traditional topic model methods, generating high-dimensional semantic embeddings through pre-trained language models, so as to better process context information and capture complex relationships between words. It is a topic model technology based on the BERT pre-trained model, and its core processes include text embedding generation, dimensionality reduction clustering, and topic representation. Specifically, first, the pre-trained BERT model (such as bert-base-uncased or Chinese bert-base-chinese) is used to vectorize the text dialogue word set and the acoustic text word set respectively, mapping each word in them to a high-dimensional semantic vector, and then text-level embeddings are generated through methods such as average pooling or CLS token. These embeddings capture the deep semantic associations of the text. For example, the embedding vectors of "电影剧情精彩" and "影片情节吸引人" are close in the semantic space. After generating the text-level embeddings, the UMAP (Uniform Manifold Approximation and Projection) algorithm is used to reduce the dimensionality of the high-dimensional text-level embeddings. UMAP can map the data to a low-dimensional space while preserving semantic similarity, facilitating subsequent clustering. After dimensionality reduction, the data is clustered by the HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) algorithm. HDBSCAN can automatically identify clustering boundaries according to data density, adapt to different scales of topics, and avoid the limitations of manually setting the number of clusters. For example, texts discussing "股票" and "基金" are clustered into different categories respectively, and at the same time, a small amount of irrelevant texts are identified as noise. After clustering, BERTopic generates the distribution probability of topic keywords by calculating the probability distribution and embedding similarity of words in each cluster, and takes the distribution probability of topic keywords as the text semantic features and acoustic semantic features of the target user respectively. Integrating the dialogue emotion features, acoustic emotion features, text semantic features, and acoustic semantic features, multi-modal dialogue features are obtained.

[0047] In one embodiment, the steps for extracting the acoustic signal features of non-visual information based on the Mel frequency scale are as follows: Complete the sampling, quantization and encoding steps of non-visual information to obtain the dialogue digital signal; Pre-emphasize the conversation digital signal, and perform windowing and framing processing on the pre-emphasized conversation digital signal to obtain multiple smooth conversation signals; For any stationary conversation signal, the stationary conversation signal is decomposed into multiple conversation signal components, and the fast Fourier transform of each conversation signal component is performed to obtain the stationary conversation spectrum; The spectrum of the steady conversation is input into the Mel frequency spectrum filter, and the Mel frequency spectrum filter is used to perform logarithmic compression of the spectrum of the steady conversation to obtain a spectrum logarithmic sequence; After completing the discrete cosine transform of the entire spectral logarithmic sequence, the acoustic signal features of non-visual information are output.

[0048] In this embodiment, the purpose of the sampling, quantization and encoding step is to convert the non-visual information, an analog signal, into a digital signal. The continuous analog signal (non-visual information) is sampled at a preset sampling frequency, such as 8000 Hz or 16000 Hz, to obtain discrete sampling points. The amplitude of each sampling point is then quantized and mapped to a finite number of quantization levels, such as 8-bit or 16-bit quantization. After quantization, the non-visual information is encoded. Encoding refers to converting the sampled and quantized non-visual information into binary values ​​to obtain a conversational digital signal.

[0049] Pre-emphasis involves applying a first-order high-pass filter to a conversational digital signal. This reduces the impact of pulses on the signal and makes it smoother. The principle is that the high-frequency portion of a conversational digital signal is typically weak. Therefore, convolution of the conversational digital signal with a filter (such as a first-order filter) enhances the high-frequency portion. After pre-emphasis, the conversational digital signal needs to be windowed and framed. First, the conversational digital signal is divided into multiple segments, each of which is called a frame. A frame is typically 20 to 30 milliseconds, and the frame shift can be around 10 milliseconds. Furthermore, to reduce discontinuities at the frame boundaries, a window function is applied to each frame. This involves multiplying the frame signal by a window function. Common window functions include Hamming and Hanning windows. The window function smoothes the transition between the two ends of the frame, preventing spectral leakage and resulting in a smooth conversational digital signal.

[0050] Next, any stationary conversation signal is decomposed into multiple conversation signal components using methods such as empirical mode decomposition (EMD) or ensemble empirical mode decomposition (EMD). Taking EMD as an example, EMD is an adaptive signal decomposition method that, through continuous screening and decomposition, decomposes the signal into a series of intrinsic mode functions (IMFs). Each IMF is a single-component signal that reflects the signal's variation pattern over several time periods. The decomposition steps include extracting local extreme points (including local maxima and local minima) from each stationary conversation signal. Using cubic splines, all local maxima and minima are connected to form upper and lower envelopes. The mean of the local extreme points corresponding to the upper and lower envelopes is calculated and subtracted from the stationary conversation signal to obtain a signal with low-frequency detrending. The above steps of extreme point extraction, constructing upper and lower envelopes, calculating the mean, and removing these extreme points are repeated for the detrended signal until a stopping condition is met (for example, the standard deviation of the IMF is less than a standard deviation threshold). Repeat the above steps until no more IMFs can be decomposed, resulting in multiple IMFs, or conversation signal components. Each conversation signal component is then subjected to a Fast Fourier Transform (FFT), converting the time domain signal to the frequency domain to obtain its spectrum. The spectrum of all conversation signal components is then summed to obtain the stationary conversation spectrum.

[0051] The spectrum of the smooth conversation is then mapped to the Mel frequency scale, which is a nonlinear frequency scale based on the human ear's sensory judgment of equally spaced pitch changes. Since the human ear's perception of frequency is not linear, but has more filters in the low-frequency area and fewer in the high-frequency area, the Mel scale can better simulate the auditory characteristics of the human ear. Specifically, the spectrum of the smooth conversation is passed through a set of Mel filters, namely the Mel frequency spectrum filter. The weighted average of the smooth conversation spectrum is completed by the Mel filter to obtain the filter output value, and the filter output value is logarithmically operated to obtain the spectrum logarithmic sequence. The formula for this process is as follows:

[0052] in, For a smooth conversation spectrum The logarithmic sequence of spectra obtained after completing the Mel frequency scale mapping.

[0053] Finally, the discrete cosine transform (DCT) is used to convert the logarithmic spectral sequence from the frequency domain to the inverse frequency domain. The resulting coefficients are called Mel-Frequency Cepstral Coefficients (MFCCs), meaning that the acoustic signal is characterized by Mel-Frequency Cepstral Coefficients. The DCT is a transform related to the Fourier transform, similar to the DFT but using only real numbers. The DCT is equivalent to a DFT of approximately twice the length, performed on a real even function (because the Fourier transform of a real even function is still a real even function). Its core principle is to represent the logarithmic spectral sequence as a linear combination of cosine functions.

[0054] Through the above steps, acoustic signal features that conform to the laws of human auditory perception can be extracted from non-visual information, so that they can more accurately reflect the essential characteristics of user voices in the application process of subsequent steps, thereby making the subsequent matching results between users and financial products more accurate.

[0055] In one embodiment, the text conversion model includes an acoustic module and a language module, the acoustic module is constructed based on a Conformer encoder, the language module includes a first language unit and a second language unit, the first language unit is constructed based on a CTC decoder, and the second language unit is constructed based on an RNN-T decoder.

[0056] In this embodiment, the Conformer encoder is composed of multiple stacked Conformer units. Each Conformer unit includes a feedforward subunit (FFN), a multi-head self-attention subunit (MHSA), and a convolution subunit. The feedforward subunit (FFN) primarily consists of a normalization layer, a linear layer, an activation layer, and a regularization layer. Through two layers of linear transformation (with an expansion factor of 4) and a Swish activation function, the FFN enhances the expressiveness of acoustic fusion features. The purpose of the multi-head self-attention subunit (MHSA) is to find the correlation between vectors, thereby capturing the contextual dependencies of acoustic fusion features and is key to understanding the contextual information of non-text information. It primarily projects the input acoustic fusion features onto three feature dimensions, designated Q, K, and V, representing query, key, and value, respectively. When calculating attention weights, Q and K are first mapped into the same high-dimensional space and their similarity is calculated. The results are then normalized using the softmax function to obtain the attention weight distribution. Finally, the Value values ​​are weighted and summed to obtain the final self-attention weight. The convolution subunit includes depthwise separable convolution and gated linear units (GLUs), which are used to extract local features of the acoustic fusion feature. The CTC decoder can automatically align the temporal frame sequence of speech and the corresponding transcribed text sequence during model training, eliminating the need to mark the start and end time periods of each character or phoneme. This allows for direct classification on the time series and eliminates the more complex alignment operations of traditional algorithms. The RNN-T decoder is a decoder for sequence-to-sequence tasks. It includes a prediction network and a fusion network, primarily based on a multi-layer long short-term memory network (LSTM) and a gated recurrent unit (GRU).

[0057] The training steps for a text-to-text model include: first, collecting a large number of annotated speech-to-text pairs. The speech data is converted into digital signals through preprocessing such as sampling, framing, and windowing. Acoustic features are then extracted using methods such as Mel-Frequency Cepstral Coefficients (MFCCs) or filter bank energy. Fundamental frequency features are extracted using the YIN or SWIPE algorithm and fused with the acoustic features to form acoustic fusion features. The text data undergoes character normalization and encoding to convert it into a label sequence recognizable by the model. The model training process can employ a multi-stage optimization strategy. First, the acoustic module is pre-trained. The Conformer encoder parameters are optimized using the CTC loss function to accurately convert the acoustic fusion features into high-dimensional acoustic features. This stage focuses on learning the initial mapping between acoustic features and character labels. An initial text sequence is generated through greedy search or beam search, and the CTC loss is calculated to maximize the probability of alignment between the acoustic features and the text labels. Next, an RNN-T decoder is introduced for end-to-end joint training. The model input is the acoustic fusion features, and the output is a text sequence. The training objective is to simultaneously optimize the CTC loss and the RNN-T loss. The RNN-T loss uses dynamic programming to calculate the joint probability of acoustic features, text sequences, and alignment paths. It not only considers acoustic matching but also models language context dependencies through prediction networks, making the generated text more linguistically consistent. During training, the weights of the two losses must be balanced. This is typically achieved through hyperparameter tuning, such as giving the CTC loss a higher weight to ensure accurate mapping of acoustic features while gradually increasing the weight of the RNN-T loss to enhance language modeling capabilities. Furthermore, to improve the robustness and generalization of the text conversion model, data augmentation techniques can be employed during training, such as adding background noise to speech data, perturbing the tempo, or performing synonym substitution on text data to simulate speech variations in different scenarios. The optimizer typically uses an adaptive algorithm such as Adam, combined with a learning rate scheduling strategy such as cosine annealing or exponential decay, to avoid model regression and accelerate convergence. During training, the performance of the text conversion model is verified using a pre-constructed validation set. Word error rate or sentence error rate metrics can be used for evaluation. Training is terminated when the word error rate or sentence error rate drops below a preset threshold, resulting in a fully trained text conversion model.

[0058] In one embodiment, converting non-text conversation information into acoustic text information based on acoustic signal features and using a text conversion model includes the following steps: The acoustic fundamental frequency features of non-visual information are extracted through the fundamental frequency extraction algorithm, and the acoustic fundamental frequency features and acoustic signal features are fused to obtain the acoustic fusion features. Input the acoustic fusion features into the acoustic module, and based on the acoustic module and by capturing the context dependency of the acoustic fusion features, convert the acoustic fusion features into high-dimensional acoustic features; Mapping high-dimensional acoustic features to a preset character label space through the first language unit, and generating multiple initial text sequences based on the mapping results through a search algorithm; Complete the rescoring of all initial text sequences using the second language unit, and select the optimal text sequence based on the rescoring results; The optimal text sequence is output as acoustic text information through the text conversion model.

[0059] Commonly used fundamental frequency extraction algorithms in this embodiment include the YIN algorithm and the SWIPE algorithm. Taking the YIN algorithm as an example, it is a time-domain fundamental frequency extraction algorithm widely used in audio signal processing. It is primarily used for pitch detection and fundamental frequency estimation, and features fast computational speed and high accuracy. Its core concept is to analyze signal periodicity through the time-domain autocorrelation function. The specific steps for extracting the acoustic fundamental frequency characteristics of non-visual information include: preprocessing the conversational digital signal. This preprocessing step primarily involves dividing the conversational digital signal into multiple frames of fixed duration, such as frames of approximately 20 to 40 milliseconds, through windowing and framing. This ensures that each frame can be analyzed independently. A Hanning window is applied to each frame to reduce spectral leakage and enhance subsequent analysis accuracy. Each frame is then normalized to eliminate the effects of amplitude differences between frames. Then, for each signal frame, its delay parameter is defined, which usually ranges from 1 to 1 / 2 of the signal frame length. The original differential function of each signal frame is calculated based on the delay parameter. Then, in order to eliminate the influence of signal amplitude and noise on the original differential function, the original differential function is normalized to obtain a normalized differential function. Then, a preset empirical threshold (such as 0.15) is used to search for the first delay point that makes the normalized differential function less than the empirical threshold. If the delay point is within the preset delay range, such as between 20Hz and 500Hz, then the delay point is used as a candidate fundamental frequency period point. If the delay point is not within the preset delay range and no other delay points that meet the conditions can be found, then the signal frame is determined to be a non-periodic signal, such as clear or silent, and the fundamental frequency at this time is set to 0 or the signal frame is skipped. Finally, to further improve the accuracy of fundamental frequency feature extraction, parabolic interpolation is performed around the candidate fundamental frequency period point. That is, the candidate fundamental frequency period point and its two adjacent delay points are taken, and the normalized difference function values ​​of these three delay points are fitted into a parabola. The minimum point of the parabola is then calculated to obtain the fundamental frequency period point. Next, the sampling frequency of the non-visual information in the sampling quantization encoding step is divided by the period value of the fundamental frequency period point to obtain the fundamental frequency Hertz number, which is the acoustic fundamental frequency feature. After extracting the acoustic fundamental frequency feature, the acoustic fundamental frequency feature is fused with the acoustic signal feature to obtain the acoustic fusion feature. Common feature fusion methods include feature splicing, attention mechanism fusion, and gating mechanism-based fusion.

[0060] Reference Figure 2 The acoustic fusion features are input into the acoustic module of the text conversion model. The Conformer encoder in the acoustic module captures the long-range contextual dependencies of the acoustic fusion features. The input acoustic fusion features are first subsampled by a convolution. Multiple Conformer units are then stacked to produce the subsampled acoustic fusion features. The Conformer unit consists of a feedforward subunit (FFN), a multi-head self-attention subunit (MHSA), and a convolution subunit. The feedforward subunit (FFN) primarily consists of a normalization layer, a linear layer, an activation layer, and a regularization layer. The FFN uses two layers of linear transformation (with a dilation factor of 4) and a Swish activation function to enhance the expressive power of the acoustic fusion features. The purpose of the multi-head self-attention subunit (MHSA) is to find the correlation between vectors, thereby capturing the contextual dependencies of the acoustic fusion features. This is key to understanding the contextual information of non-text information. It primarily projects the input acoustic fusion features onto three feature dimensions: Q, K, and V, representing query, key, and value, respectively. When calculating attention weights, Q and K are first mapped into the same high-dimensional space to calculate their similarity. The results are then normalized using the softmax function to obtain the attention weight distribution. Finally, the Value values ​​are weighted and summed to obtain the final self-attention weight. The convolutional subunits include depthwise separable convolutions and gated linear units (GLUs) to extract local features from the acoustic fusion features. Furthermore, batch normalization (BN) is incorporated into the convolutional module to normalize the values ​​of individual neurons, thereby stabilizing the data distribution and accelerating model training. After passing the acoustic fusion features through the Conformer encoder, they undergo multiple layers of self-attention and convolution operations to learn high-level representations that incorporate contextual information, outputting high-dimensional acoustic features. This process is similar to how the human auditory system extracts semantic information from sound. The acoustic module gradually abstracts high-dimensional acoustic features such as phonemes and words from low-level acoustic fusion features, laying the foundation for the subsequent language module.

[0061] The language module consists of a first language unit and a second language unit. The first language unit is built based on the CTC decoder, and the second language unit is built based on the RNN-T decoder. The character label space is a discrete set of characters that contains the basic units of all text that the text conversion model needs to recognize. The CTC decoder automatically aligns speech frames with character sequences by introducing a blank label "_", eliminating the need for explicit frame annotation. Its processing pipeline involves first converting high-dimensional acoustic features into a character probability distribution for each time step through a fully connected layer. Then, a search algorithm is used to find the most likely character sequence within the character probability distribution, removing consecutive repetitions and blank blocks to generate the initial text sequence. Common search algorithms include greedy search, beam search, and prefix beam search. Taking beam search as an example, the K characters with the highest probability are selected from a probability distribution as K initial candidate sequences. For each initial candidate sequence, all possible next characters are extended to obtain a new extended sequence. The cumulative probability of the new sequence is then calculated, which is equivalent to the product of the original distribution probabilities. Each initial candidate sequence can be expanded into V new sequences (V is the vocabulary size). K original initial candidate sequences can generate K×V new sequences. Finally, the K sequences with the highest probability are selected from these K×V new sequences as candidates for the next step. This process continues until all sequences meet the termination condition (such as the appearance of a terminator or reaching the maximum length), resulting in K initial text sequences.

[0062] Next, the second language unit re-scored the entire initial text sequence and selected the optimal text sequence based on the re-scoring results. The second language unit is based on an RNN-T decoder, which consists of multiple layers of long short-term memory (LSTM) networks and gated recurrent units (GRUs). The LSTM serves as a predictor. For each character position in the initial text sequence, the predictor generates a current context state based on the previous characters and propagates this context state across time steps to capture the temporal dependencies of the initial text sequence, resulting in semantic features for each character position. The semantic features of each character position are then combined with their corresponding high-dimensional acoustic features through a nonlinear transformation to form a joint representation. A softmax function is then applied to generate a character probability distribution over all the joint representations. A dynamic programming algorithm (such as the forward algorithm) is then used to calculate the joint probability for each initial text sequence. The initial text sequence with the highest joint probability is then selected as the optimal text sequence and used as the final output, representing the acoustic text information.

[0063] The CTC encoder, when processing high-dimensional acoustic features, can quickly align speech and text, ensuring character-level matching accuracy. The RNN-T encoder, while rescoring, incorporates the power of a language model to comprehensively assess whether the initial text sequence conforms to the logic and conventions of the language, for example, avoiding grammatical errors and semantic incoherence. By integrating the CTC decoder with the RNN-T decoder, the optimal text sequence can be precisely selected from both local and global perspectives. This allows for accurate character recognition while ensuring the fluency and rationality of the entire sentence, significantly improving the quality of the final text.

[0064] In one embodiment, performing multimodal fusion of multimodal conversation features through a tensor fusion network to obtain multimodal fusion features of a target user includes the following steps: Vectorize the multimodal conversation features to obtain a multimodal feature vector; Normalize multimodal feature vectors; Expand each sub-feature vector in the normalized multimodal feature vector into a dialogue sub-tensor, and concatenate each dialogue sub-tensor into the initial dialogue tensor through a tensor outer product operation; Based on tensor decomposition technology, tensor dimensionality reduction and tensor compression of the initial conversation tensor are completed to obtain the multimodal fusion features of the target user.

[0065] In this embodiment, multimodal conversation features from different sources are first quantized to obtain a multimodal feature vector. The multimodal feature vector is then normalized. Z-score normalization can be used, which calculates the mean and standard deviation of each sub-feature vector within each multidimensional feature and transforms the eigenvalues ​​of each sub-feature vector into a distribution with a mean of 0 and a standard deviation of 1. Alternatively, Min-Max normalization can be used to compress the features to a range between 0 and 1. Normalization eliminates dimensional differences between modal features and prevents sub-feature vectors with large values ​​from dominating the fusion process. Sub-feature vectors are obtained by quantizing conversation sentiment features, acoustic sentiment features, textual semantic features, and acoustic semantic features. After normalization, a feature dimension is added to each sub-feature vector within the multimodal conversation feature to obtain a conversation sub-tensor. For example, textual semantic features are generally in the form of word sequences. These sub-feature vectors are then processed through the BERT model to generate word embedding vectors, or sub-feature vectors, where each word sequence corresponds to a 768-dimensional word embedding vector. In this case, the original format of the multimodal conversation feature is a two-dimensional matrix, such as sentence length × word vector dimension. By increasing the two-dimensional matrix by a unit dimension, we obtain a dialog tensor tensor, for example, sentence length × word vector dimension × 1. We then perform an outer product operation on each dialog tensor tensor, and use the result as the initial dialog tensor. The outer product operation multiplies all elements in the two dialog tensors. Since the initial dialog tensor may have a large dimension, we need to perform dimensionality reduction on it, i.e., tensor dimensionality reduction and compression. Common tensor decomposition techniques include CP decomposition and Tucker decomposition. Taking Tucker decomposition as an example, it decomposes the initial conversation tensor into a core tensor and multiple factor matrices. The core tensor captures the high-order interactions between the modalities, while the factor matrices represent the low-dimensional representation of each modality, which can generally be expanded through singular value decomposition (SVD). The alternating least squares (ALS) method is then used to alternately optimize each factor matrix and the core tensor, gradually approaching the optimal solution until the maximum number of iterations is reached. The final core tensor and multiple factor matrices are obtained, thus achieving tensor dimensionality reduction and compression. The final core tensor and multiple factor matrices are then output as the multimodal fusion features of the target user. This method not only reduces the dimensionality but also preserves the interaction information between the modalities.

[0066] In one embodiment, the random walk algorithm is used to divide the user feature hypergraph into communities, and obtaining multiple user feature communities includes the following steps: Calculate the intra-cluster density of all target users in all feature clusters, and select multiple seed user sets based on the intra-cluster density; Using the hypergraph expansion method to transform the user feature hypergraph into a hypergraph extension graph; For any seed user set, randomly walk in the hypergraph expansion graph starting from the seed user in the seed user set to obtain the walk probability vector of all target users; The comprehensive scores of all target users are calculated by combining the walk probability vector and the node degree of each node in the user feature expansion graph and using the user score calculation formula. The user score calculation formula is as follows:

[0067] in, is the preset weight parameter, is the maximum node degree in the hypergraph expansion graph, For nodes The node degree of For nodes The walk probability vector of Integrate target users whose comprehensive scores are greater than a preset score threshold into a user reference set; Expand the user feature hypergraph based on the user reference set to obtain a user feature expansion graph; Calculate the user overlap between the user reference set and the entire seed user set, and define the walk constraint function based on the user overlap; Taking minimizing the walk constraint function as the objective function and using the random walk algorithm to complete the community division of the user feature expansion graph, multiple user feature communities are obtained.

[0068] In this embodiment, the intra-cluster density of all target users in all feature clusters is first calculated. The intra-cluster density refers to the ratio of the sum of the weights of all hyperedges in each feature cluster to the sum of the node degrees of all supernodes in each feature cluster. All feature clusters are sorted in descending order based on intra-cluster density, and the top K feature clusters (e.g., K = 5) with the highest intra-cluster density are selected as seed feature clusters. The top m supernodes in the seed feature clusters with the highest node degrees are selected as seed users, and the seed users in each seed feature cluster are integrated into a seed user set. For example, if a feature cluster ranks high with an intra-cluster density of Q = 0.7, and the target user A1 in this feature cluster has the highest degree d = 50, then the target user A1 is selected as a seed user and included in the seed user set. This step ensures that the seed users come from highly correlated feature clusters, providing a high-quality starting point for subsequent random walks.

[0069] Then, the user feature hypergraph is expanded into a hypergraph extension graph using the hypergraph expansion method. Common hypergraph expansion methods include clique expansion method, star expansion method and line graph method. Taking the star expansion method as an example, each hyperedge e of the user feature hypergraph is mapped to a vertex in the ordinary graph. If there are two hyperedges ei and ej in the user feature hypergraph that share at least one common vertex, then an edge is added between the vertices corresponding to the hyperedges ei and ej in the ordinary graph to obtain the hypergraph extension graph. Figure 3. The vertices of the hypergraph expansion graph represent the hyperedges of the user feature hypergraph, and the edges represent the intersection of the hyperedges. Then, for any seed user set, randomly walk in the hypergraph expansion graph starting from the seed user in the seed user set to obtain the walk probability vector of all target users. The steps of random walk include: when transferring from a supernode to a hyperedge, if it is located at a supernode v, the probability of selecting the hyperedge e connected to it is proportional to the weight of the hyperedge. For example, the set of hyperedges connected to the supernode v is E(v), and the weight of each hyperedge e is w(e), then the probability of selecting the hyperedge e is the ratio of w(e) to the sum of the weights of the hyperedge set E(v). When transferring from a hyperedge to a supernode, if it is located at a hyperedge node When (a node in the hypergraph expansion graph), the next supernode needs to be selected from all supernodes connected by hyperedge e. At this time, any supernode in hyperedge e is usually selected with equal probability. That is, if hyperedge e connects k supernodes, the probability of each supernode being selected is 1 / k. After each random walk, the number of visits to each hyperedge node and supernode in the hypergraph expansion graph is recorded, and the ratio of the number of visits to each node to the total number of random walks is used as the original walk probability of the node. For example, if node U is visited 200 times in 1000 random walks, its original walk probability is initially recorded as 200 / 1000=0.2. Then, the relative value of the original walk probability and the node degree of each node is calculated to obtain the walk probability vector.

[0070] The comprehensive score of each target user is calculated by combining the walk probability vector and the node degree of each node in the user feature expansion graph. The target users with the highest comprehensive scores are integrated into the user reference set, that is, the target users with comprehensive scores greater than the preset score threshold are integrated into the user reference set. The comprehensive score calculation formula is as follows:

[0071] in, is the preset weight parameter, such as 0.6, is the maximum node degree in the hypergraph expansion graph, For nodes The node degree of For nodes The walk probability vector of .

[0072] Next, with the user reference set as the core, the user feature hypergraph is expanded into a user feature extension graph. Specifically, the hyperedges associated with the user reference set in the user feature hypergraph are retained. Each hyperedge contains at least one node in the user reference set. If two users are connected by multiple hyperedges in the user feature hypergraph, a new hyperedge with a weight equal to the sum of the weights of these hyperedges is added to obtain the user feature extension graph.

[0073] Calculate the user overlap between the user reference set and all seed users, that is, the seed user set. The formula for calculating user overlap is as follows:

[0074] in, Refers to the user reference set and seed user set The sum of the node degrees of all nodes in the intersection of Refers to the seed user set But not in the user reference set The sum of the node degrees of all nodes of Refers to the user reference set But not in the seed user set The random walk penalty of . Is a preset non-negative parameter, usually greater than 0 and less than 1, used as a regulation factor. The random walk penalty is as follows:

[0075] in, is the penalty coefficient, such as 0.5, For nodes The walk probability vector of Reference set for users The sum of the node degrees of all nodes in .

[0076] The higher the random walk penalty, the closer the association between the target user corresponding to the node and the seed user set is, and this user needs to be retained when dividing the community.

[0077] The walk constraint function is defined by combining user overlap and random walk penalty. The walk constraint function is as follows:

[0078] in, For node sets The walk penalty coefficient, To be divided into node sets The sum of the weights of all hyperedges.

[0079] By solving the minimum walk penalty coefficient, the user feature community can be obtained. The solution steps include: initializing the user feature community , assuming user feature community Reference set for users , calculate the current walk penalty coefficient as the initial walk penalty coefficient. Then construct a reference hypergraph containing two virtual nodes, source nodes and sink nodes. The source node represents the "core area of ​​the seed user set" and is used to guide the process of dividing the community to lean towards nodes that are strongly associated with the user seed set. The sink node represents the "non-seed user set area" and is used to punish nodes that are weakly associated with the seed user set. The reference hypergraph contains all supernodes, source nodes, and sink nodes in the user feature hypergraph. Specifically, for each node u in the user reference set, add a directed edge from the source node to u with a weight of ,in, Preset adjustment parameters, is the node degree of node u, is the random walk penalty of node u. For each node w that is not in the user reference set, add a directed edge from node w to the sink node with weight , is a preset non-negative parameter, is the node degree of node w.

[0080] After constructing the reference hypergraph, find a node to divide the node set in the reference hypergraph into two complementary and intersecting subsets, and then calculate the walk penalty coefficient of the divided subset. If the walk penalty coefficient of the subset is less than the initial walk penalty coefficient, repeatedly construct the reference hypergraph based on the subset until the walk penalty coefficient of the divided subset no longer decreases. Stop the iteration and use the final subset as the household characteristic community.

[0081] The present application also provides a machine-readable storage medium, characterized in that the machine-readable storage medium stores instructions, which are used to enable a machine to execute the method of financial digital marketing business opportunity analysis based on chat prompt words according to any of the above items.

[0082] The present application also provides a financial digital marketing business opportunity analysis system based on chat prompt words, which is characterized by including: a memory configured to store instructions; and The processor is configured to call instructions from a memory and implement the method for analyzing financial digital marketing business opportunities based on chat prompts according to any of the above items when executing the instructions.

[0083] Among them, the processor can adopt a central processing unit (CPU). Of course, according to actual usage, other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be adopted. The general-purpose processor can adopt a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0084] Among them, the memory can be an internal storage unit of a computer device, such as a hard disk or memory of a computer device, or an external storage device of a computer device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD) or flash memory card (FC) equipped on the computer device. In addition, the memory can also be a combination of an internal storage unit and an external storage device of a computer device. The memory is used to store computer programs and other programs and data required by the computer device. The memory can also be used to temporarily store data that has been output or is to be output. This application does not impose any restrictions on this.

[0085] An embodiment of the present application also provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute one of the above-mentioned methods for analyzing financial digital marketing business opportunities based on chat prompt words.

[0086] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0088] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0089] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0090] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0091] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0092] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0093] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0094] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included within the scope of the claims of the present application.

Claims

1. A financial digital marketing business opportunity analysis method based on chat prompt words, characterized in that: The method comprises the following steps: Collect conversation information from multiple target users, including visual and non-visual information; For any target user, extract the multimodal conversation features of the target user's visual information and non-visual information; The multimodal fusion of multimodal conversation features is completed through the tensor fusion network to obtain the multimodal fusion features of the target user; Use clustering algorithms to complete feature clustering of multimodal fusion features of all target users, obtain multiple feature clusters, use all feature clusters as hyperedges, and construct a user feature hypergraph with the target users as supernodes; The user feature hypergraph is divided into communities using a random walk algorithm to obtain multiple user feature communities. Complete feature matching between each user feature community and pre-acquired financial product features.

2. The method according to claim 1, characterized in that The visual information includes text conversation information and non-text conversation information. Extracting multimodal conversation features of the target user's visual information and non-visual information includes the following steps: Extract metadata from non-text conversation information and use a pre-trained language model to map the metadata into conversation sentiment features; The acoustic signal features of non-visual information are extracted based on the Mel frequency scale, and the sentiment classification of the acoustic signal features is completed through the classification algorithm to obtain the acoustic sentiment features; Based on the acoustic signal characteristics, the non-text conversation information is converted into acoustic text information using a text conversion model built on a deep learning model. Based on the topic model, semantic extraction of text conversation information and acoustic text information is completed respectively to obtain the text semantic features and acoustic semantic features of the target user; The multimodal dialogue features are obtained by integrating dialogue emotion features, acoustic emotion features, text semantic features and acoustic semantic features.

3. The method according to claim 2, characterized in that The method of extracting acoustic signal features of non-visual information based on the Mel frequency scale comprises the following steps: Complete the sampling, quantization and encoding steps of non-visual information to obtain the dialogue digital signal; Pre-emphasize the conversation digital signal, and perform windowing and framing processing on the pre-emphasized conversation digital signal to obtain multiple smooth conversation signals; For any stationary conversation signal, the stationary conversation signal is decomposed into multiple conversation signal components, and the fast Fourier transform of each conversation signal component is performed to obtain the stationary conversation spectrum; The spectrum of the steady conversation is input into the Mel frequency spectrum filter, and the Mel frequency spectrum filter is used to perform logarithmic compression of the spectrum of the steady conversation to obtain a spectrum logarithmic sequence; After completing the discrete cosine transform of the entire spectral logarithmic sequence, the acoustic signal features of non-visual information are output.

4. The method according to claim 2, characterized in that The text conversion model includes an acoustic module and a language module. The acoustic module is constructed based on a Conformer encoder. The language module includes a first language unit and a second language unit. The first language unit is constructed based on a CTC decoder, and the second language unit is constructed based on an RNN-T decoder.

5. The method according to claim 4, characterized in that The method of converting non-text conversation information into acoustic text information based on acoustic signal features and using a text conversion model comprises the following steps: The acoustic fundamental frequency features of non-visual information are extracted through the fundamental frequency extraction algorithm, and the acoustic fundamental frequency features and acoustic signal features are fused to obtain the acoustic fusion features. Input the acoustic fusion features into the acoustic module, and based on the acoustic module and by capturing the context dependency of the acoustic fusion features, convert the acoustic fusion features into high-dimensional acoustic features; Mapping high-dimensional acoustic features to a preset character label space through the first language unit, and generating multiple initial text sequences based on the mapping results through a search algorithm; Complete the rescoring of all initial text sequences using the second language unit, and select the optimal text sequence based on the rescoring results; The optimal text sequence is output as acoustic text information through the text conversion model.

6. The method according to claim 1, characterized in that The multimodal fusion of multimodal conversation features using a tensor fusion network to obtain the multimodal fusion features of the target user includes the following steps: Vectorize the multimodal conversation features to obtain a multimodal feature vector; Normalize multimodal feature vectors; Expand each sub-feature vector in the normalized multimodal feature vector into a dialogue sub-tensor, and concatenate each dialogue sub-tensor into the initial dialogue tensor through a tensor outer product operation; Based on tensor decomposition technology, tensor dimensionality reduction and tensor compression of the initial conversation tensor are completed to obtain the multimodal fusion features of the target user.

7. The method according to claim 1, characterized in that The method of completing the community division of the user feature hypergraph by the random walk algorithm to obtain multiple user feature communities includes the following steps: Calculate the intra-cluster density of all target users in all feature clusters, and select multiple seed user sets based on the intra-cluster density; Using the hypergraph expansion method to transform the user feature hypergraph into a hypergraph extension graph; For any seed user set, randomly walk in the hypergraph expansion graph starting from the seed user in the seed user set to obtain the walk probability vector of all target users; The user reference set is extracted from the user feature expansion graph by combining the walk probability vector and the node degree of each node in the user feature expansion graph; Expand the user feature hypergraph based on the user reference set to obtain a user feature expansion graph; Calculate the user overlap between the user reference set and the entire seed user set, and define the walk constraint function based on the user overlap; Taking minimizing the walk constraint function as the objective function and using the random walk algorithm to complete the community division of the user feature expansion graph, multiple user feature communities are obtained.

8. The method according to claim 7, characterized in that The method of extracting a user reference set from the user feature expansion graph by combining the walk probability vector and the node degree of each node in the user feature expansion graph comprises the following steps: The comprehensive scores of all target users are calculated by combining the walk probability vector and the node degree of each node in the user feature expansion graph and using the user score calculation formula. The user score calculation formula is as follows: ； in, is the preset weight parameter, is the maximum node degree in the hypergraph expansion graph, For nodes The node degree of For nodes The walk probability vector of Target users whose comprehensive scores are greater than the preset score threshold are integrated into the user reference set.

9. A machine-readable storage medium, characterized in that The machine-readable storage medium stores instructions for enabling a machine to execute the method for analyzing financial digital marketing business opportunities based on chat prompts according to any one of claims 1 to 8.

10. A financial digital marketing business opportunity analysis system based on chat prompt words, characterized in that: include: a memory configured to store instructions; as well as The processor is configured to call instructions from a memory and implement the method for analyzing financial digital marketing business opportunities based on chat prompts according to any one of claims 1 to 8 when executing the instructions.

Images