Multi-modal data clustering method and system based on attention between integrated heterogeneous graphs

Abstract

The invention discloses a multi-modal data clustering method and system based on attention between integrated heterogeneous graphs, and belongs to the technical field of data processing, and the method comprises the steps: obtaining multi-modal data, carrying out the feature extraction of the multi-modal data, and obtaining a D-dimensional node feature representation; projecting the D-dimensional node feature representation to a joint learning space, generating a similarity graph, and generating an availability graph; constructing an integrated heterogeneous graph corresponding to each mode; performing intra-graph attention calculation on the integrated heterogeneous graph, performing inter-graph attention calculation based on a neighborhood aggregation result of nodes in each modal graph, and combining an intra-graph attention result and an inter-graph attention result to obtain multi-modal joint node representation; calculating original distribution and target distribution to obtain clustering loss; and establishing a basic clustering model, reconstructing loss and clustering loss, and outputting a clustering result of the user portrait and the user intention through a self-supervised learning optimization model. According to the invention, the semantic comprehension capability and service accuracy of the intelligent customer service system are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a multi-modal data clustering method and system based on integrated heterogeneous graph inter-graph attention. BACKGROUND

[0002] The multi-modal intelligent customer service system is a key technical direction in the current digital transformation of power marketing. Relying on large models and multi-modal learning capabilities, it can simultaneously process heterogeneous data such as text, images, and voice generated in user interactions, enabling automatic identification and response to user portraits and intentions, and significantly improving service efficiency, professionalism, and stability. It has been piloted in several provincial power companies and is gradually being promoted.

[0003] However, existing technologies face serious challenges in actual deployment: user interaction data often contains only a single modality, such as text or images only, and the combination of modalities varies greatly between different samples. Traditional multi-modal clustering methods rely on fixed modal structures and are difficult to adapt to such dynamic missing scenarios. Existing graph clustering methods usually construct fully connected graphs directly based on feature similarity, without distinguishing between "modality missing" and "semantic irrelevance", leading to false edges interfering with clustering results. In real business, there is a lack of labeled labels, and existing methods are difficult to drive clustering structure convergence through self-supervised signals under high missing conditions, resulting in blurred user portraits and intentions and low accuracy. The above problems seriously hinder the semantic understanding ability and service accuracy of intelligent customer service in complex interaction scenarios. SUMMARY

[0004] In view of the above problems, the present application is proposed.

[0005] Therefore, the technical problem solved by the present application is: how to integrate heterogeneous graph construction and inter-graph attention mechanism to realize structured modeling, cross-modal semantic alignment, and robust clustering of multi-modal data in high missing rate and complex modality existence scenarios, accurately distinguish user portraits and user intentions, and avoid false associations and clustering biases caused by modality missing.

[0006] To solve the above technical problems, the present application provides the following technical solutions: a multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention, comprising: obtaining multi-modal data generated in the interaction process between a user and an intelligent customer service system, performing feature extraction on the multi-modal data to obtain D-dimensional node feature representations corresponding to each modality; projecting the D-dimensional node feature representations into a joint learning space, generating a similarity graph based on the D-dimensional node feature representations in the joint learning space, and generating an availability graph according to the existence of each modality in each sample; based on the similarity graph and the availability graph, constructing an integrated heterogeneous graph corresponding to each modality; Intra-graph attention calculation is performed on the integrated heterogeneous graph, inter-graph attention calculation is performed based on neighborhood aggregation results of nodes in each modal graph, intra-graph attention results and inter-graph attention results are combined, and multi-modal joint node representation is obtained through multi-head attention; Based on the multi-modal joint node representation and the preset clustering center, the Student-t distribution is used to calculate the original distribution and the target distribution, and the clustering loss is obtained by minimizing the KL divergence between the two. A basic clustering model is established using artificial preprocessed data and low missing rate simulated data, high missing rate simulated data is introduced to adjust the reconstruction loss and clustering loss, and a large amount of real data is used to optimize the model through self-supervised learning, and the clustering results of user portraits and user intentions are output.

[0007] As a preferred scheme of the multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention, the feature extraction includes: Multi-modal data generated in the interaction process of the user and the intelligent customer service system is obtained, and the multi-modal data includes one or more of text, image and voice; For each modality, a corresponding encoder is configured, and the multi-modal data is input into the corresponding encoder for processing; The feature extraction is performed on the modal data through the encoder to obtain the original feature input of each sample under each modality. The original feature input is mapped to a D-dimensional vector as the D-dimensional node feature representation of the sample under the modality.

[0008] As a preferred scheme of the multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention, the generation of the availability graph includes: The D-dimensional node feature representation of each sample under each modality is input into the joint learning space to obtain the D-dimensional node feature representation in the unified semantic space; Based on the D-dimensional node feature representation in the joint learning space, K nearest neighbor samples of each sample are selected, the similarity between the sample and each neighbor sample is calculated, and a similarity graph is generated. According to the actual existence of each modality in each sample, it is judged whether each pair of samples exists simultaneously under each modality, and an availability graph is generated.

[0009] As a preferred scheme of the multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention, the construction of the integrated heterogeneous graph corresponding to each modality includes: The similarity graph generated by the D-dimensional node feature representation in the joint learning space is obtained, and the similarity graph is an N*N matrix, where N is the total number of single batch samples; obtain an availability graph generated by each modality existing in each sample, the availability graph is independently generated according to the modality, and each modality corresponds to an N*N binary matrix; For each modality, the similarity graph is element-wise fused with the availability graph corresponding to the selected modality to obtain an integrated heterogeneous graph corresponding to the modality.

[0010] As a preferred scheme of the multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention provided by the application, wherein: For each modality corresponding integrated heterogeneous graph, the D-dimensional node feature representation in the joint learning space is linearly transformed to obtain a feature vector suitable for graph attention calculation; Based on the adjacency relationship of the integrated heterogeneous graph, the intra-graph attention coefficient of the node pair in each modality is calculated, and the connection state in the integrated heterogeneous graph is used for missing mask processing; Based on the intra-graph attention coefficient, the neighborhood features of the node in each modality are weighted and aggregated to obtain the neighborhood aggregation result of the node in the selected modality; Based on the neighborhood aggregation result of the same node in each modality, the inter-graph attention value corresponding to each modality is calculated to measure the contribution degree of different modality graphs to the global feature representation; The intra-graph attention coefficient and the inter-graph attention value are fused to obtain the final fusion attention weight, and the node features of each modality are integrated through the multi-head attention mechanism to generate the multi-modal joint node representation.

[0011] As a preferred scheme of the multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention provided by the application, wherein: Obtain a preset clustering center, the clustering center corresponds to a user portrait category and a user intent category, and is in the same feature space as the multi-modal joint node representation; Based on the Euclidean distance between the multi-modal joint node representation and the clustering center, the Student-t distribution is used to calculate the original distribution to obtain the attribution probability of each sample to each clustering center; Based on the original distribution, the probability of each sample on each clustering is normalized again to generate a target distribution, so that a high confidence allocation obtains a higher weight; Calculate the KL divergence between the original distribution and the target distribution as the clustering loss, which is used to drive the model parameter update.

[0012] As a preferred scheme of the multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention provided by the application, wherein: The artificial preprocessed data and the low missing rate simulation data are used, preset clustering centers are combined, a basic clustering model is established through reconstruction loss and clustering loss; High missing rate simulation data is introduced, the established clustering center and the standardized feature representation are used, the proportion of the reconstruction loss and the clustering loss is adjusted, and the adaptability of the model to the complex missing mode scene is optimized; A large amount of unprocessed real user interaction data is input, the clustering ability of the trained model is used, the feature representation and the clustering result are continuously optimized through self-supervised learning; Based on the finally optimized multi-modal joint node representation, the belonging probability of each sample to the clustering center is calculated, and the clustering category with the highest probability is output as the user portrait and user intention clustering result of the selected sample.

[0013] The application provides a multi-modal data clustering system based on integrated inter-graph attention of heterogeneous graphs.

[0014] To solve the above technical problems, the application further provides the following technical scheme: a multi-modal data clustering system based on integrated inter-graph attention of heterogeneous graphs, comprising: a multi-modal feature extraction module, configured to obtain multi-modal data generated in the interaction process of a user and an intelligent customer service system, perform feature extraction on the multi-modal data, and obtain D-dimensional node feature representations corresponding to each modality; A joint space projection module is configured to project the D-dimensional node feature representations into a joint learning space, generate a similarity graph based on the D-dimensional node feature representations in the joint learning space, and generate an availability graph according to the existence of each modality in each sample; An integrated graph construction module is configured to construct an integrated heterogeneous graph corresponding to each modality based on the similarity graph and the availability graph; An inter-graph attention fusion module is configured to perform intra-graph attention calculation on the integrated heterogeneous graph, perform inter-graph attention calculation based on the neighborhood aggregation result of nodes in each modality graph, combine the intra-graph attention result and the inter-graph attention result, and obtain multi-modal joint node representations through multi-head attention; A multi-modal clustering module is configured to calculate original distribution and target distribution using Student-t distribution based on the multi-modal joint node representations and preset clustering centers, and obtain clustering loss by minimizing the KL divergence between the original distribution and the target distribution; A self-supervised optimization module is configured to establish a basic clustering model using artificial preprocessed data and low missing rate simulation data, introduce high missing rate simulation data to adjust reconstruction loss and clustering loss, and optimize the model based on a large amount of real data through self-supervised learning, and output clustering results of user portraits and user intentions.

[0015] The application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements steps of the multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention when executing the computer program.

[0016] The application provides a computer readable storage medium, which stores a computer program, wherein the computer program implements steps of the multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention when executed by a processor.

[0017] The application has the following beneficial effects: the application effectively solves the problem of invalidation of traditional clustering methods caused by high missing rate of multi-modal data and complex modality existence in the power marketing intelligent customer service scene by constructing an integrated heterogeneous graph and introducing an inter-graph attention mechanism. Specifically, by projecting each modality feature into a joint learning space and generating a similarity graph and an availability graph respectively, semantic similarity and modality existence are decoupled and modeled; by fusing the two to construct an integrated heterogeneous graph, invalid connections are explicitly shielded, and the graph structure is ensured to truly reflect the effective association between samples; by double calculation of intra-graph attention and inter-graph attention, local neighborhood importance and global modality credibility are simultaneously captured, and discriminative multi-modal joint node representation is generated; by constructing the original distribution and the target distribution through Student-t distribution and minimizing the KL divergence, the self-optimization of the clustering structure under unsupervised conditions is realized; and by the three-stage progressive training strategy of establishing a basic model from low missing simulation data, strengthening robustness from high missing data, and self-supervised optimization from real data, the model gradually adapts to the extreme missing scene in actual business. The above steps work together to finally realize accurate, stable and interpretable clustering of user portraits and user intents under the conditions of no label, high missing and multi-modal heterogeneity, and significantly improve the semantic understanding ability and service accuracy of the intelligent customer service system. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0019] Figure 1 The application provides a computer readable storage medium, which stores a computer program, wherein the computer program implements steps of the multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention when executed by a processor. DETAILED DESCRIPTION

[0020] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present application.

[0021] Embodiment 1, reference Figure 1 For the first embodiment of the present application, the embodiment provides a multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention, comprising: S1: obtaining multi-modal data generated in the interaction process of a user and an intelligent customer service system, performing feature extraction on the multi-modal data to obtain D-dimensional node feature representation corresponding to each modality; S2: projecting the D-dimensional node feature representation to a joint learning space, generating a similarity graph based on the D-dimensional node feature representation in the joint learning space, and generating an availability graph according to the existence of each modality in each sample; S3: based on the similarity graph and the availability graph, constructing an integrated heterogeneous graph corresponding to each modality; S4: performing intra-graph attention calculation on the integrated heterogeneous graph, and performing inter-graph attention calculation based on the neighborhood aggregation result of the nodes in each modality graph, combining the intra-graph attention result and the inter-graph attention result, and obtaining a multi-modal joint node representation through multi-head attention; S5: based on the multi-modal joint node representation and the preset clustering center, using Student-t distribution to calculate the original distribution and the target distribution, and obtaining the clustering loss by minimizing the KL divergence between the two; S6: using artificial preprocessed data and low missing rate simulated data to establish a basic clustering model, introducing high missing rate simulated data to adjust the reconstruction loss and clustering loss, and based on a large amount of real data, optimizing the model through self-supervised learning, and outputting the clustering results of user portrait and user intent.

[0022] It should be noted that in the power marketing intelligent customer service system, the multi-modal data generated in the interaction process of the user and the intelligent customer service generally has the problems of high modality missing rate and complex modality existence, and the modality combination difference between different samples is significant, which makes it difficult for traditional multi-modal clustering methods to effectively model the semantic association between samples, and cannot accurately distinguish user portrait and user intent. At the same time, due to the interference of missing modalities on feature consistency, the clustering process is easily affected by noise, and the clustering center is difficult to converge stably, and in the real scene, there is a lack of label supervision, and the model is difficult to optimize through conventional supervision signals; therefore, there is an urgent need for a technical solution that can fuse modality existence information, adaptively model sample relationships, and realize robust clustering under unsupervised conditions.

[0023] Therefore, in order to solve the problems of inaccurate clustering and weak semantic distinction in the multi-modal high-missing scenario, the steps S1-S6 are used to construct an integrated heterogeneous graph to explicitly encode the modal existence and sample similarity, introduce an intra-graph and inter-graph attention mechanism to jointly model the node level and modal level importance, and generate discriminative multi-modal joint node representation; a clustering probability model is constructed based on Student-t distribution, and the clustering structure is self-optimized by KL divergence; and the reconstruction loss and clustering loss are combined, and in three training stages, low-missing simulation data, high-missing simulation data and a large amount of real data are used in turn to gradually improve the adaptability of the model to complex missing patterns, and finally realize accurate clustering of user portraits and user intents.

[0024] Embodiment 2, with reference to 1, is a second embodiment of the application, which provides a multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention.

[0025] S1: Obtain multi-modal data generated in the interaction process of a user and an intelligent customer service system, and perform feature extraction on the multi-modal data to obtain D-dimensional node feature representation corresponding to each modality; S11: Obtain multi-modal data generated in the interaction process of a user and an intelligent customer service system, wherein the multi-modal data includes one or more of text, image, and voice; In an optional embodiment, the multi-modal data is derived from the consultation content submitted by a user on an intelligent customer service platform for power marketing, wherein the text data is a problem description input by the user, the image data is a meter photo or fault screenshot uploaded by the user, and the voice data is a voice message recorded by the user through a mobile terminal.

[0026] In an optional embodiment, the multi-modal data is stored in a structured form in a system log, and each interaction record contains a user identifier, a timestamp, and a storage path or embedded content of each modality data, ensuring that each modality can be aligned by sample in subsequent processing.

[0027] S12: For each modality, configure a corresponding encoder respectively, and input the multi-modal data into the corresponding encoder for processing; In an optional embodiment, the text encoder uses a pre-trained BERT model, the image encoder uses a ResNet-50 network, and the voice encoder uses a 1D convolutional neural network, all of which are independently deployed and do not share parameters.

[0028] S13: Perform feature extraction on the multi-modal data through the encoder to obtain the original feature input of each sample under each modality; In an optional embodiment, the text encoder outputs a sequence of 768-dimensional context vectors after tokenization and embedding of the user question text, and inputs the corresponding vector marked by [CLS] as the original text feature of the sample; the image encoder extracts a 2048-dimensional global feature from the 224x224 pixel meter image; and the speech encoder outputs a 512-dimensional time-frequency statistical feature from the 1-second speech segment sampled at 16 kHz.

[0029] S14: Map the original feature input into a D-dimensional vector as the D-dimensional node feature representation of the sample under the modality; In an optional embodiment, D is 128, and the original feature input of each modality is mapped to a 128-dimensional space through a learnable linear projection layer, and the projection layer parameters are optimized jointly with other model parameters during the training process.

[0030] In an optional embodiment, for the case where the image modality is missing for a sample, the original feature input of the sample under the image modality is empty, the system sets its corresponding D-dimensional node feature representation as a 128-dimensional zero vector, and records the non-existence of the modality in the subsequent availability graph to maintain the alignment of the feature structure.

[0031] S2: Project the D-dimensional node feature representation into a joint learning space, generate a similarity graph based on the D-dimensional node feature representation in the joint learning space, and generate an availability graph according to the existence of each modality in each sample; S21: Input the D-dimensional node feature representation of each sample under each modality into the joint learning space to obtain the D-dimensional node feature representation in the unified semantic space; In an optional embodiment, the joint learning space is a shared embedding space, and the D-dimensional node feature representation of each modality is directly used as the node representation in the space without additional transformation for subsequent graph construction.

[0032] In an optional embodiment, the joint learning space is optimized by minimizing the reconstruction loss, which is calculated based on the difference between the original modality input and the modality data reconstructed from the D-dimensional node feature representation, to ensure that the representation after projection retains the original semantic information.

[0033] S22: Based on the D-dimensional node feature representation in the joint learning space, select K nearest neighbors for each sample, calculate the similarity between the sample and each neighbor, and generate a similarity graph; In an optional embodiment, K is 10, and for each sample i, the Euclidean distance between it and all other samples in the joint learning space is calculated, and the 10 samples with the smallest distance are selected as neighbors to form an adjacency set.

[0034] In an optional embodiment, the similarity is calculated using a Student-t distribution kernel function, with the formula wherein, is the similarity graph element value of sample i and sample j in modality v, representing the normalized similarity of the two in the joint learning space; is the D-dimensional node feature representation of sample i in the joint learning space; is the D-dimensional node feature representation of sample j in the joint learning space; is the degree of freedom parameter of the Student-t distribution, which is a preset positive integer, used to control the similarity decay rate; is the K nearest neighbor sample set of sample i, and K is a preset positive integer; S23: According to the actual existence of each modality in each sample, it is judged whether each pair of samples exists in each modality at the same time, and if so, it is marked as 1, otherwise it is marked as 0, and an availability graph is generated; In an optional embodiment, for modality v, the availability graph element of sample i and sample j is defined as follows: wherein, is the element value of sample i and sample j in modality v in the availability graph, which is 0 or 1; is an indicator variable, which is 1 when sample i contains modality v, otherwise it is 0; is an indicator variable, which is 1 when sample j contains modality v, otherwise it is 0.

[0035] In an optional embodiment, the availability graph is generated independently by modality, and each modality corresponds to an N x N binary matrix, which is 1 only when both samples are not missing in the modality, and 0 otherwise, which is used for subsequent element-wise fusion with the similarity graph.

[0036] S3: Based on the similarity graph and the availability graph, an integrated heterogeneous graph corresponding to each modality is constructed; S31: Obtain a similarity graph generated by the D-dimensional node feature representation in the joint learning space, and the similarity graph is an N x N matrix, wherein N is the total number of single batch samples; In an optional embodiment, the similarity graph is calculated by normalizing the feature representation distance between each sample and its several nearest neighbor samples in the joint learning space, and the normalization process uses a probability distribution function with a long tail feature to enhance the robustness to abnormal samples.

[0037] S32: Obtain the availability graph generated by the modality existence of each sample, the availability graph is independently generated by modality, and each modality corresponds to an N*N binary matrix; In an optional embodiment, the connection state of any two samples in the availability graph in a certain modality is determined by whether the two samples contain the modality at the same time. Only when the two samples contain the modality, the corresponding matrix element is 1, otherwise it is 0.

[0038] In an optional embodiment, the availability graph is stored in a sparse form, only the element position with value 1 is reserved, so as to reduce the storage overhead and improve the subsequent graph calculation efficiency.

[0039] S33: For each modality v, element-wise fusion is performed on the similarity graph and the availability graph corresponding to the modality, to obtain the integrated heterogeneous graph corresponding to the modality v; In an optional embodiment, the connection strength of any two samples in the integrated heterogeneous graph in modality v is equal to the product of the corresponding element values in the similarity graph and the availability graph. If the availability graph indicates that the sample pair is not connectable in modality v, the corresponding edge in the integrated heterogeneous graph is forced to be zero.

[0040] In an optional embodiment, the integrated heterogeneous graph corresponding to each modality is independently used for subsequent graph attention calculation, so that the structural information of different modalities is propagated in the graph neural network respectively, and invalid information interference in the neighborhood aggregation process is avoided due to the lack of modalities.

[0041] S4: Intra-graph attention calculation is performed on the integrated heterogeneous graph, and inter-graph attention calculation is performed based on the neighborhood aggregation results of the nodes in each modality graph. The multi-modal joint node representation is obtained by combining the intra-graph attention result and the inter-graph attention result through multi-head attention; S41: For each modality corresponding integrated heterogeneous graph, linear transformation is performed on the D-dimensional node feature representation in the joint learning space to obtain a feature vector suitable for graph attention calculation; In an optional embodiment, the linear transformation is realized by a learnable matrix, which maps the D-dimensional node feature representation of each sample in the joint learning space to another D-dimensional vector, which is used for subsequent attention coefficient calculation, to ensure that the features in different modalities are in a comparable semantic space.

[0042] S42: Based on the adjacency relationship of the integrated heterogeneous graph, the intra-graph attention coefficient is calculated for each node pair in each modality, and the missing mask processing is performed by using the connection state in the integrated heterogeneous graph; In an optional embodiment, the intra-graph attention coefficient is calculated by inputting the concatenated feature vectors of the two nodes into a shared attention vector. If the corresponding position in the integrated heterogeneous graph is zero, the attention coefficient of the node pair is forced to be negative infinity, and after softmax, it is zero, which realizes the shielding of invalid connections in the missing mode.

[0043] In an optional embodiment, the shared attention vector is a learnable parameter, which is optimized together with the linear transformation matrix during the training process, and is used to measure the semantic correlation of any two adjacent nodes in the current modal graph.

[0044] S43: Based on the intra-graph attention coefficient, the neighborhood features of the nodes in each mode are weighted and aggregated to obtain the neighborhood aggregation result of the node in the mode; In an optional embodiment, the neighborhood aggregation result is the weighted sum of the transformed feature vectors of all neighbors of the node according to the intra-graph attention coefficient, which reflects the local semantic representation of the node under the current modal graph structure.

[0045] In an optional embodiment, the weighted aggregation process only acts on the neighbor nodes of the non-zero connection in the integrated heterogeneous graph, and the neighbors corresponding to the missing mode do not participate in the aggregation, avoiding the interference of invalid information.

[0046] S44: Based on the neighborhood aggregation result of the same node in each mode, the inter-graph attention value corresponding to each mode is calculated to measure the contribution degree of different modal graphs to the global feature representation. In an optional embodiment, the inter-graph attention value is calculated by inputting the neighborhood aggregation result of each mode into a learnable global attention vector, and the relative importance score of each mode is formed after softmax normalization.

[0047] In an optional embodiment, the global attention vector is independent of the intra-graph attention parameter and is specially used to evaluate the overall credibility of different modal graphs on the current sample, thereby guiding the direction of multi-modal information fusion.

[0048] S45: The intra-graph attention coefficient and the inter-graph attention value are fused to obtain the final fusion attention weight, and the node features of each mode are integrated through a multi-head attention mechanism to generate a multi-modal joint node representation. In an optional embodiment, the fusion attention weight is the product of the intra-graph attention coefficient and the inter-graph attention value of the corresponding mode, and the multi-head attention mechanism calculates multiple independent fusion attention heads in parallel, concatenates the outputs of each head, and linearly maps them to the final multi-modal joint node representation.

[0049] In an optional embodiment, each head in the multi-head attention mechanism uses independent linear transformation parameters, and the high-dimensional representation after splicing is compressed back to the original dimension through an output mapping layer, ensuring that the output representation is consistent with the input dimension, facilitating subsequent clustering processing.

[0050] In an optional embodiment, the output of the multi-head attention is processed through a nonlinear activation function, enhancing the model's ability to express complex semantic relationships while preserving the discriminative nature of the original features.

[0051] S5: Based on the multi-modal joint node representation and the preset clustering center, the Student-t distribution is used to calculate the original distribution and the target distribution, and the clustering loss is obtained by minimizing the KL divergence between the two. S51: Obtain a preset clustering center, which corresponds to user portrait categories and user intent categories and is in the same feature space as the multi-modal joint node representation. In an optional embodiment, the clustering center is initialized by artificial preprocessing data and low missing rate simulation data in the initial training stage, ensuring its semantic interpretability and covering the main user portrait and intent types.

[0052] In an optional embodiment, the clustering center is updated with the model optimization during the training process, and its update method is to take the mean of the multi-modal joint node representations of all samples belonging to the same cluster in the current batch.

[0053] S52: Based on the Euclidean distance between the multi-modal joint node representation and the clustering center, the Student-t distribution is used to calculate the original distribution to obtain the belonging probability of each sample to each clustering center. In an optional embodiment, the belonging probability of a sample to a clustering center in the original distribution is negatively correlated with the square of the Euclidean distance between their feature representations, and the degree of freedom parameter of the Student-t distribution is a preset constant, used to control the probability decay rate.

[0054] S53: Based on the original distribution, the probability of each sample on each cluster is normalized again to generate a target distribution, so that high confidence allocation obtains higher weight. In an optional embodiment, the calculation of the target distribution includes: first square the probability of each cluster in the original distribution and divide it by the total probability of that cluster, and then normalize the result, so that the model is more inclined to strengthen high-probability clusters and suppress low-probability noise allocation.

[0055] S54: Calculate the KL divergence between the original distribution and the target distribution as the clustering loss, which is used to drive model parameter update. In an optional embodiment, the KL divergence is calculated element by element for each sample and cluster center, and the smaller the value is, the more consistent the current multimodal joint node representation is with the cluster structure, and the stronger the model clustering ability is.

[0056] In an optional embodiment, the cluster loss and the reconstruction loss jointly constitute a total loss function, and the total loss is minimized by backpropagation during the training process to achieve joint optimization of feature representation and cluster structure.

[0057] S6: Use artificial preprocessed data and low missing rate simulated data to establish a basic clustering model, introduce high missing rate simulated data to adjust the reconstruction loss and the cluster loss, and based on a large amount of real data, optimize the model through self-supervised learning to output the clustering results of user portraits and user intents.

[0058] S61: Use artificial preprocessed data and low missing rate simulated data, combined with preset cluster centers, to establish a basic clustering model through reconstruction loss and cluster loss; In an optional embodiment, the artificial preprocessed data is a user interaction sample with standardized annotation, containing complete text, image or voice modalities, which is used to initialize the cluster center and guide the model to learn the basic semantic structure of user portraits and user intents.

[0059] In an optional embodiment, the low missing rate simulated data is generated by randomly masking a small number of modalities in complete multimodal samples, so that the model learns feature representation and cluster association under the condition of relatively complete structural information.

[0060] S62: Introduce high missing rate simulated data, use the established cluster center and standardized feature representation, and adjust the proportion of reconstruction loss and cluster loss to optimize the model's adaptability to complex modal missing scenarios; In an optional embodiment, the high missing rate simulated data is generated by randomly removing multiple modalities in samples, so that most samples only retain a single modality, forcing the model to rely on integrated heterogeneous graphs and inter-graph attention mechanisms to mine cross-sample and cross-modality implicit associations.

[0061] In an optional embodiment, the proportion of reconstruction loss and cluster loss is adjusted by manually set parameters to balance the feature fidelity and cluster structure consistency, and improve the discrimination ability of user portraits and user intents.

[0062] S63: Input a large amount of unprocessed real user interaction data, based on the clustering ability of the trained model, continuously optimize the feature representation and clustering results through self-supervised learning; In an optional embodiment, the real data is derived from the historical logs of the power marketing intelligent customer service system, contains a high proportion of missing modalities and complex modality combinations, and the model does not rely on manual annotation in this stage, but is driven by reconstruction loss and clustering loss for self-iteration.

[0063] In an optional embodiment, in the self-supervised learning process, the clustering center is updated with the clustering result of each batch of data, and the updating method is to take the mean of the multi-modal joint node representations of all samples belonging to the same cluster, so that the clustering structure gradually approximates the real user distribution.

[0064] S64: Based on the final optimized multi-modal joint node representation, the attribution probability of each sample to the clustering center is calculated, and the cluster category with the highest probability is output as the user portrait and user intent clustering result of the sample.

[0065] In an optional embodiment, the attribution probability is calculated by Student-t distribution, and the output result contains user portrait categories (such as resident users and enterprise users) and user intent categories (such as electricity fee consultation and fault repair), which are used for subsequent intelligent customer service response generation.

[0066] In an optional embodiment, the clustering result is output in the form of structured labels, which can be directly connected to the downstream task system to realize automatic routing of user intent and personalized service recommendation.

[0067] Embodiment 3 is a third embodiment of the application, which provides a multi-modal data clustering system based on integrated heterogeneous graph inter-graph attention, comprising: A multi-modal feature extraction module is used to obtain multi-modal data generated in the interaction process of users and an intelligent customer service system, extract features of the multi-modal data, and obtain D-dimensional node feature representations corresponding to each modality. A joint space projection module is used to project the D-dimensional node feature representations into a joint learning space, generate a similarity graph based on the D-dimensional node feature representations in the joint learning space, and generate an availability graph according to the existence of each modality in each sample. An integrated graph construction module is used to construct an integrated heterogeneous graph corresponding to each modality based on the similarity graph and the availability graph. A graph attention fusion module is used to perform intra-graph attention calculation on the integrated heterogeneous graph, perform inter-graph attention calculation based on the neighborhood aggregation results of nodes in each modality graph, combine the intra-graph attention results and the inter-graph attention results, and obtain multi-modal joint node representations through multi-head attention. A multi-modal clustering module is used to calculate the original distribution and the target distribution based on the multi-modal joint node representations and the preset clustering center by using Student-t distribution, and obtain a clustering loss by minimizing the KL divergence between the two. The self-supervised optimization module is used for establishing a basic clustering model by using artificial preprocessed data and low-missing-rate simulated data, introducing high-missing-rate simulated data to adjust reconstruction loss and clustering loss, and optimizing the model based on a large amount of real data through self-supervised learning to output clustering results of user portraits and user intents.

[0068] Embodiment 4, which is different from the first three embodiments, is that the function, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0069] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered a list of executable instructions for implementing logical functions, and can be specifically embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch the instructions from an instruction execution system, apparatus or device and execute the instructions. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in connection with an instruction execution system, apparatus or device, or in conjunction with these instruction execution systems, apparatus or devices.

[0070] More specific examples (non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CD ROM). In addition, the computer-readable medium can even be paper or other suitable medium on which the program can be printed, because the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by editing, interpretation or processing, if necessary, in other suitable ways, and then stored in a computer memory.

[0071] It should be understood that portions of the present application can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, implementation can be with any or a combination of the following technologies, which are all well known in the art: a discrete logic circuit having logic gates for implementing logic functions upon an application of data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0072] It should be noted that the above examples are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, and all should be covered in the scope of the claims of the present application.

Claims

1. A multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention, characterized in that: The method comprises the following steps: Obtaining multi-modal data generated in the interaction process of a user and an intelligent customer service system, and performing feature extraction on the multi-modal data to obtain D-dimensional node feature representations corresponding to each modality; Projecting the D-dimensional node feature representations into a joint learning space, generating a similarity graph based on the D-dimensional node feature representations in the joint learning space, and generating an availability graph according to the existence of each modality in each sample; Based on the similarity graph and the availability graph, an integrated heterogeneous graph corresponding to each modality is constructed; Intra-graph attention calculation is performed on the integrated heterogeneous graph, inter-graph attention calculation is performed based on the neighborhood aggregation results of the nodes in each modality graph, and multi-modal joint node representations are obtained by combining the intra-graph attention results and the inter-graph attention results through multi-head attention; Based on the multi-modal joint node representations and the preset clustering center, the Student-t distribution is used to calculate the original distribution and the target distribution, and the clustering loss is obtained by minimizing the KL divergence between the two. The method comprises the following steps:

2. The multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention according to claim 1, wherein: Obtaining multi-modal data generated in the interaction process of a user and an intelligent customer service system, and performing feature extraction on the multi-modal data to obtain D-dimensional node feature representations corresponding to each modality; For each modality, a corresponding encoder is configured, and the multi-modal data is input into the corresponding encoder for processing; Feature extraction is performed on each modality data through the encoder to obtain the original feature input of each sample under each modality; The original feature input is mapped to a D-dimensional vector as the D-dimensional node feature representation of the sample under the modality. The method comprises the following steps:

3. The multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention of claim 2, wherein: Inputting the D-dimensional node feature representations of each sample under each modality into a joint learning space to obtain D-dimensional node feature representations in a unified semantic space; Based on the D-dimensional node feature representations in the joint learning space, selecting K nearest neighbor samples for each sample, calculating the similarity between the sample and each nearest neighbor sample, and generating a similarity graph; According to the actual existence of each modality in each sample, it is judged whether each pair of samples exists simultaneously under each modality to generate an availability graph. The method comprises the following steps:

4. The multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention of claim 3, wherein: Obtaining the similarity graph generated by the D-dimensional node feature representations in the joint learning space, the similarity graph being an N×N matrix, where N is the total number of single batch samples; Obtaining the availability graph generated by the existence of each modality in each sample, the availability graph being independently generated according to the modality, and each modality corresponding to an N×N binary matrix; For each modality, the similarity graph and the availability graph corresponding to the selected modality are fused element by element to obtain an integrated heterogeneous graph corresponding to the modality. The method comprises the following steps:

5. The multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention of claim 4, wherein: For each modality corresponding to the integrated heterogeneous graph, performing linear transformation on the D-dimensional node feature representations in the joint learning space to obtain a feature vector suitable for graph attention calculation; ​ Based on the adjacency relationship of the integrated heterogeneous graph, the intra-graph attention coefficients are calculated for each node pair in each modality, and missing mask processing is performed using the connection state in the integrated heterogeneous graph; Based on the intra-graph attention coefficients, the neighborhood features of the nodes in each modality are weighted and aggregated to obtain the neighborhood aggregation results of the nodes in the selected modality; Based on the neighborhood aggregation results of the same node in each modality, the inter-graph attention values corresponding to each modality are calculated to measure the contribution of different modality graphs to the global feature representation; The intra-graph attention coefficients and the inter-graph attention values are fused to obtain the final fusion attention weight, and the node features of each modality are integrated through the multi-head attention mechanism to generate a multi-modal joint node representation.

6. The multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention of claim 5, wherein: The clustering loss is obtained as follows: Obtain a preset cluster center, which corresponds to a user portrait category and a user intent category and is in the same feature space as the multi-modal joint node representation; Based on the Euclidean distance between the multi-modal joint node representation and the cluster center, the original distribution is calculated using the Student-t distribution to obtain the membership probability of each sample to each cluster center; Based on the original distribution, the probability of each sample on each cluster is normalized again to generate a target distribution, so that a high confidence allocation obtains a higher weight; The KL divergence between the original distribution and the target distribution is calculated as the clustering loss to drive model parameter updating.

7. The multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention of claim 6, wherein: The output of the clustering results of the user portrait and the user intent includes: Use artificial preprocessed data and low missing rate simulated data to establish a basic clustering model by combining a preset cluster center, reconstruction loss and clustering loss; Introduce high missing rate simulated data, use the established cluster center and standardized feature representation, and adjust the proportion of reconstruction loss and clustering loss to optimize the adaptability of the model to complex modality missing scenarios; Input a large amount of unprocessed real user interaction data, and based on the clustering ability of the trained model, continuously optimize the feature representation and clustering results through self-supervised learning; Based on the final optimized multi-modal joint node representation, the membership probability of each sample to the cluster center is calculated, and the cluster category with the highest probability is output as the user portrait and user intent clustering result of the selected sample.

8. A multi-modal data clustering system based on integrated heterogeneous graph inter-graph attention, applying a multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention according to any one of claims 1-7, characterized in that, It includes: A multi-modal feature extraction module is configured to obtain multi-modal data generated during the interaction between a user and an intelligent customer service system, extract features from the multi-modal data, and obtain D-dimensional node feature representations corresponding to each modality. A joint space projection module is configured to project the D-dimensional node feature representations into a joint learning space, generate a similarity graph based on the D-dimensional node feature representations in the joint learning space, and generate an availability graph according to the existence of each modality in each sample. An integrated graph construction module is configured to construct an integrated heterogeneous graph corresponding to each modality based on the similarity graph and the availability graph. An inter-graph attention fusion module is configured to calculate intra-graph attention for the integrated heterogeneous graph, calculate inter-graph attention based on the neighborhood aggregation results of the nodes in each modality graph, and obtain a multi-modal joint node representation through multi-head attention by combining the intra-graph attention results and the inter-graph attention results. The multi-modal clustering module is configured to calculate an original distribution and a target distribution based on the multi-modal joint node representation and preset clustering centers, calculate a clustering loss by minimizing a KL divergence between the original distribution and the target distribution, and output a clustering result of a user portrait and a user intent. The self-supervised optimization module is configured to establish a basic clustering model by using artificial preprocessed data and low-missing-rate simulated data, introduce high-missing-rate simulated data to adjust a reconstruction loss and the clustering loss, and optimize the model by self-supervised learning based on a large amount of real data, and output the clustering result of the user portrait and the user intent. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the multi-modal data clustering method based on integrated heterogeneous graph inter-graph attention according to any one of claims 1 to 7.

Images