Smart museum virtual interaction method and system fusing multi-modal data

Abstract

The invention relates to the technical field of data fusion, in particular to a smart museum virtual interaction method and system fusing multi-modal data. According to the method, self-attention mechanism calculation is carried out on a feature set in a single mode, and feature vectors are clustered to obtain feature clusters according to the correlation degree between the attention attention degrees of the feature vectors and local feature vectors; matching the feature clustering clusters among different modes by using a cross attention mechanism to obtain the most matched clustering cluster of each feature clustering cluster in other modes; and carrying out multi-modal data fusion by utilizing a matching relationship among different modals. According to the method, a self-attention mechanism is combined with clustering segmentation, so that the accurate recognition capability of local features is improved, clusters of different modals are corrected and matched by using a cross-modal attention mechanism, the condition of excessive crossing is reduced, the data fusion quality is improved, and richer and more real virtual interaction experience is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of data fusion, and in particular to a smart museum virtual interaction method and system for fusing multimodal data. Background Art

[0002] As the core node of smart city culture display and dissemination, smart museums are facing new challenges and requirements. With the development of technology, smart museums are no longer just an extension of digital museums, but are based on multimodal perception of "data" to establish a comprehensive and in-depth interconnection system, eliminate information islands, and realize systematic collaborative working methods between people, people and objects, and objects. The development model of smart museums has changed from technology-led to demand-driven and business-led. By reconstructing the relationship between the various elements of the museum, a joint force is formed to strengthen the coordination of museum services, protection and management. In addition, smart museums also provide a two-way multi-information interaction channel between "objects, people, and data", realizing intelligent adaptive control and optimization of museum services, protection and management.

[0003] There are two main technical points in the multimodal data fusion of smart museums: alignment and fusion. The existing technology usually obtains the multimodal data that needs to be displayed in the operation of smart museums, specifically image data, text data and voice data, and uses machine learning and deep learning methods to obtain the global and local semantic features of image data, the global and local semantic features of text data, and the global and local semantic features of voice data. Deep learning is used to achieve multimodal alignment and automatically discover and learn the corresponding relationship between different modalities.

[0004] However, the current cross-modal attention-based multimodal data fusion is to perform cross-modal attention judgment on the overall data. When applied to smart museums, the virtual display and interaction of the museum are usually for exhibits, and the exhibits are demonstrated from multiple angles. Therefore, the use of the existing cross-modal attention mechanism may lead to excessive cross-modal attention, resulting in a certain degree of fusion and alignment deviation, affecting the final data fusion quality. Summary of the invention

[0005] In order to solve the technical problem that excessive crossover occurs when performing the cross-modal attention mechanism in the prior art, resulting in a certain degree of fusion and alignment deviation, which affects the final data fusion quality, the purpose of the present invention is to provide a smart museum virtual interaction method and system for fusing multimodal data. The technical solutions adopted are as follows: According to one aspect of an embodiment of the present invention, a method for virtual interaction of a smart museum integrating multimodal data is provided, the method comprising: Get the feature vectors in each modality data and form a feature set, the modalities include image, text and audio; Based on the self-attention mechanism, the correlation between the feature vector in the feature set of each modality and other local feature vectors is analyzed to obtain the local correlation index of each feature vector; the feature vectors in each modality are clustered according to the local correlation index to obtain feature clustering clusters; Based on the cross-attention mechanism, the matching degree of feature vector correlation between each feature cluster in each modality and each feature cluster in other modalities is analyzed to determine the best matching cluster for each feature cluster. Multimodal fusion alignment is performed based on the most matching clustering clusters of feature clusters between different modalities.

[0006] In the above scheme, considering the global alignment difference of the cross-attention mechanism when processing multimodal data, firstly, the self-attention mechanism is combined with clustering to calculate the feature set in the single mode, and the changes in the content features of the identified items are reflected through the attention degree association of the local feature vector, and clustering is performed to better capture local details and improve the accurate fusion of local features of different modalities. At the same time, there will be excessive cross-influence of redundant information between multimodal data. The cross-attention mechanism is used to further correct and match the feature clusters between different modalities, and the most matching cluster clusters are obtained to optimize the information alignment between modalities, improve the accuracy of fine-grained information matching, and use the matching relationship between different modalities to perform multimodal data fusion and reduce inconsistency. The present invention combines clustering segmentation with a self-attention mechanism to improve the accurate identification of local features, and uses a cross-modal attention mechanism to correct and match clusters of different modalities, reduce excessive cross-influence, and improve the quality of data fusion to provide a richer and more realistic virtual interactive experience.

[0007] Furthermore, the method for obtaining the local correlation index includes: In the feature set of each modality, the self-attention score of each feature vector and each other feature vector is obtained through the self-attention mechanism, as well as the self-attention weight of each feature vector in the feature set. For any feature vector in each modal feature set, the feature vector in a preset local range in the feature set is used as the local feature vector of the feature vector; according to the relative degree between the feature vector and the local feature vector in the distribution of the self-attention score and the self-attention weight, the local correlation of the feature vector is obtained; The local correlation index of the feature vector is obtained by combining the local correlation and self-attention weight of the feature vector.

[0008] In the above scheme, the local correlation index of the feature vector is obtained by combining the self-attention score that represents the relationship with other feature vectors and the features that represent the overall contribution, and the local range is limited. The self-attention weight is adjusted to obtain the local correlation index, which more finely quantifies the importance and mutual relationship of the feature vector in its local context, can capture more detailed correlation relationships, reduce the computational redundancy of global analysis from local relationships, improve the robustness of feature vectors in correlation analysis, and make subsequent feature clustering division more accurate.

[0009] Furthermore, the method for obtaining the local correlation includes: Take each local feature vector as the target vector in turn, calculate the projection of the self-attention weight of the target vector on the self-attention weight of the feature vector, and obtain the relative attention of the target vector; The product of the self-attention score and the relative attention of the feature vector and the target vector is taken as the correlation between the feature vector and the target vector; The average value of the correlation between the feature vector and all local feature vectors is taken as the local correlation of the feature vector.

[0010] Furthermore, the method for obtaining the feature clustering clusters includes: For any mode, the set of local correlation degrees corresponding to the eigenvectors in the mode is taken as the correlation set; the number of clusters is determined by the elbow method for the correlation set of the mode; The feature vectors are clustered according to the local correlation degree of the feature vectors of the mode, and the number of clusters is obtained as the feature clustering clusters of the mode.

[0011] In the above scheme, the fact that changes in the main body will also cause large changes in the correlation of local data is taken into consideration. The number of clustering segmentations is determined by the change in local correlation, and the feature vectors are clustered so that each cluster corresponds to the local detail features more accurately, providing a data basis for subsequent local fine matching.

[0012] Furthermore, the method for obtaining the best matching cluster includes: The cross-attention mechanism is used to obtain the cross-attention score between each feature vector in each modality and each feature vector in other modalities, as well as the cross-attention weight of each feature vector in the feature set of other modalities; for any modality, each other modality is taken as the cross-target modality in turn; For any feature vector in the modality, the matching degree between the feature vector and each feature cluster in the cross-target modality is obtained according to the cross-attention score and cross-attention weight between the feature vector and the feature vector in each feature cluster in the cross-target modality, as well as the cross-attention weight of the feature vector in each feature cluster in the cross-target modality in the modality; Combined with the matching degree of all feature vectors in the feature clustering cluster where the feature vector is located and each feature clustering cluster in the cross-target modality, the cluster matching index of the feature clustering cluster where the feature vector is located and each feature clustering cluster in the cross-target modality is obtained; based on the cluster matching index of each feature clustering cluster in the modality and the feature clustering cluster in the cross-target modality, the most matching clustering cluster of each feature clustering cluster in the modality is determined.

[0013] In the above scheme, by combining the cross-attention mechanism with the matching of feature clusters, the cross-attention scores and cross-attention weights between the feature vectors in each modality and the feature clusters in other modalities are analyzed to analyze the correlation between the feature vectors and the feature clusters. From the correlation between the single feature vector and the feature cluster in each modality, the feature vectors are integrated to obtain the matching relationship between the clusters, improve the accuracy and robustness of the cross-modal matching analysis, and thus determine the most matching cluster of each feature cluster in other modalities. With the most matching cluster of feature clusters, when it comes to information of multiple modalities, the mismatch of clusters of different modalities can be avoided, so that the cluster of each modality can find the most suitable match in the multimodal space, thereby achieving a more refined clustering relationship result.

[0014] Furthermore, the method for obtaining the matching degree includes: For any feature cluster in the cross-target modality, calculate the dot product of the cross-attention weight of the feature vector in the cross-target modality and the cross-attention weight of each feature vector in the feature cluster in the modality as the similarity weight between the feature vector and each feature vector in the feature cluster; Calculate the product of the cross attention score and the similarity weight between the feature vector and each feature vector in the feature cluster to obtain the cross similarity between the feature vector and each feature vector in the feature cluster; The cross similarity between the feature vector and all feature vectors in the feature cluster is combined to obtain the matching degree between the feature vector and the feature cluster.

[0015] Furthermore, the method for obtaining the cluster matching index includes: For any feature cluster in the cross-target mode, the feature cluster where the feature vector is located is taken as the analysis cluster; The mean of the matching degree between all feature vectors in the analysis clustering cluster and the feature clustering cluster is calculated as the cluster matching index between the analysis clustering cluster and the feature clustering cluster.

[0016] Furthermore, the determining the most matching cluster of each feature cluster in the modality according to the cluster matching index between each feature cluster in the modality and the feature cluster in the cross target modality includes: Each feature cluster in the modality is taken as the target cluster in turn; When the cluster matching index between the target cluster and the feature cluster in the cross-target modality is the largest, the feature cluster in the corresponding cross-target modality is used as the best matching cluster of the target cluster in the cross-target modality.

[0017] Furthermore, performing multimodal fusion alignment according to the most matching clustering clusters of feature clustering clusters between different modalities includes: For any modality, data cross-matching is performed between each feature cluster in the modality and the most matching cluster in each other modality to obtain the cross-matching result between the modality and other modalities; data alignment and fusion are performed based on the cross-matching result.

[0018] In the above scheme, by finding the most matching cluster cluster for cross-modal cross-matching, the accuracy range of cross-matching is limited, the interference of redundant information or noise is reduced, the impact of excessive cross-matching caused by mismatching is reduced, and the accuracy of data alignment and fusion is improved.

[0019] According to another aspect of an embodiment of the present invention, a smart museum virtual interaction system integrating multimodal data is provided to implement the above-mentioned smart museum virtual interaction method integrating multimodal data, and the system includes: The data acquisition module is used to obtain the feature vectors in each modality data and form a feature set. The modalities include images, texts and audios. The single-modal segmentation module is used to analyze the correlation between the feature vectors in the feature set of each modality and other local feature vectors based on the self-attention mechanism, and obtain the local correlation index of each feature vector; cluster the feature vectors in each modality according to the local correlation index to obtain feature clustering clusters; The multimodal matching module is used to analyze the degree of correlation matching between the feature vectors of each feature cluster in each modality and each feature cluster in other modalities based on the cross-attention mechanism, and determine the best matching cluster of each feature cluster in each modality in other modalities; The multimodal fusion module is used to perform multimodal fusion alignment based on the most matching clustering clusters of feature clustering clusters between different modalities.

[0020] According to yet another aspect of an embodiment of the present invention, a computer device is provided.

[0021] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of a virtual interaction method for a smart museum integrating multimodal data are implemented as described above.

[0022] According to yet another aspect of an embodiment of the present invention, a computer-readable storage medium is provided.

[0023] A computer-readable storage medium stores a computer program, which can be executed by at least one processor. When the computer program code runs on the processor of a computer, the computer executes the steps of a virtual interaction method for a smart museum that integrates multimodal data as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A flow chart of a smart museum virtual interaction method integrating multimodal data provided by an embodiment of the present invention; Figure 2 A flow chart of a method for obtaining the best matching cluster provided by an embodiment of the present invention; Figure 3 A structural diagram of a smart museum virtual interaction system integrating multimodal data provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of a virtual interactive method and system for a smart museum integrating multimodal data proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0027] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0028] The specific scheme of a smart museum virtual interaction method and system integrating multimodal data provided by the present invention is described in detail below with reference to the accompanying drawings.

[0029] Feature extraction of multimodal data refers to extracting feature vectors that can represent information from data of different modalities. These feature vectors can then be used for model training and prediction. The method of feature extraction depends on the specific modality and task, for example: Visual modality: Use convolutional neural network (CNN) to extract image features. Text modality: Use word embeddings or pre-trained language models such as BERT to extract text features. Audio modality: Extract acoustic features such as Mel frequency cepstral coefficients (MFCC).

[0030] In multimodal learning, the key to feature extraction is how to effectively represent and fuse information from different modalities so that the model can capture the complex relationships and interactions between different modalities. Through self-attention and cross-modal attention mechanisms, the model can better understand and utilize these features and improve the performance of multimodal tasks.

[0031] Multimodal data fusion alignment refers to the integration of data from different modalities, such as vision, text, and audio, at the feature level in multimodal learning so that the model can understand and utilize information from these different sources at the same time. The purpose of alignment is to solve the temporal and semantic correspondence problem of different modal data, which is the basis of multimodal fusion.

[0032] Methods for multimodal data fusion and alignment include: Early Fusion: merging data of different modalities at the feature extraction stage and then inputting them into the model for processing; Late Fusion: extracting and learning features of data of different modalities separately and then fusing them at the decision layer or higher layers.

[0033] The Cross-Attention Mechanism is a very important mechanism in multimodal learning. It focuses on exploring the relationship between different modalities and is usually used in scenarios where it is necessary to model associations between two different sequences, such as multimodal learning (images and text), machine translation (encoders and decoders), generation tasks, etc.

[0034] The self-attention mechanism, also known as the internal attention mechanism, is an attention mechanism that associates different positions of a single sequence to calculate the representation of the same sequence. The core idea of ​​the self-attention mechanism is to allow the model to consider the information of all positions in the sequence when processing each element in the sequence, thereby capturing the long-distance dependencies within the sequence. This mechanism does not rely on the absolute position of the elements in the sequence, but is achieved by calculating the relative relationship between the elements.

[0035] The main steps are to generate queries (Q), keys (K) and values ​​(V) for each element in a sequence. The model calculates the similarity or matching degree of each query with all keys to obtain an attention score matrix. The attention scores are normalized by the softmax function to obtain the attention weight of each element.

[0036] Embodiment 1: The embodiment of the present invention provides a virtual interaction method for a smart museum integrating multimodal data. Figure 1 , which shows a flow chart of a smart museum virtual interaction method integrating multimodal data provided by an embodiment of the present invention, the method comprising the following steps: S1: Obtain feature vectors from each modality data and form a feature set. The modalities include image, text, and audio.

[0037] In the virtual interactive system of the smart museum, the collection of multimodal data is the basis for building an immersive experience. The embodiments of the present invention mainly involve data in three modalities: image, text, and audio, and aim to achieve a richer and more realistic virtual interactive experience by integrating these data sources.

[0038] Multimodal data feature extraction is a prerequisite and key step to achieve multimodal alignment and enhancement. Machine learning and deep learning methods can be used to extract features from the three modal data of image, text and audio respectively, and form a feature set to represent global and local semantic features for subsequent analysis.

[0039] In the embodiment of the present invention, image feature extraction mainly includes: for image data, a convolutional neural network (CNN) is used to perform preliminary feature extraction, by capturing local features of the image, such as edges, textures, etc. In order to further enhance the object detection and recognition capabilities, a region proposal network (RPN) is introduced to generate candidate regions, and a feature pyramid network (FPN) is combined to extract multi-scale features. Finally, the local features in the image are converted into feature vectors to form a feature set of the image modality.

[0040] Text feature extraction mainly includes: For text data, first use word embedding technology, such as Word2Vec or BERT, to convert words into vector representations to capture the semantic relationship between words. Then, use a sentence-level encoder based on Transformer to further extract the semantic features of key sentences and phrases in the text and generate a feature vector set of the text.

[0041] Audio feature extraction mainly includes: for audio data, first convert the audio signal into frequency domain representation through Mel Frequency Cepstral Coefficient (MFCC) to capture the frequency characteristics of the audio. Then, further extract local features in the audio, such as the speech pattern of a specific word or phrase, and represent it as a feature vector to form a feature set of the audio modality.

[0042] Features are extracted from image, text and audio data respectively, and the features of each modality are represented as a vector set, providing a basis for subsequent multimodal analysis. It should be noted that the techniques used for local feature extraction are well known to those skilled in the art and will not be described in detail here.

[0043] S2: Based on the self-attention mechanism, the correlation between the feature vector in the feature set of each modality and other local feature vectors is analyzed to obtain the local correlation index of each feature vector; the feature vectors in each modality are clustered according to the local correlation index to obtain feature clustering clusters.

[0044] In a smart museum, the content displayed is usually the data of the exhibits in the museum and the humanistic information such as history and characters related to the exhibits. The multimodal data of different exhibits may have similar components, such as appearance and background. When fusing multimodal data in a smart museum, it is necessary to align the multimodal data to provide a better visual and interactive experience.

[0045] At the same time, there will be a large information gap between the multi-angle information of the exhibits, which will lead to a certain information gap. As a result, when using the cross-attention mechanism to fuse the multimodal data of museum exhibits, a certain attention cross-bias will be generated, affecting the fusion and alignment effects of multimodal data.

[0046] Therefore, the multimodal data of the smart museum can be clustered and segmented. By segmenting the entire multimodal data through clustering and self-attention mechanisms, the cross-modal attention calculation of the multimodal data can be limited to a certain range, thereby improving the accuracy of the subsequent cross-modal attention between multimodal data, thereby improving the final fusion and alignment accuracy of the multimodal data.

[0047] The self-attention mechanism can obtain relatively accurate and good results in a smaller data set, but when processing long data, the attention score on the long time axis is distorted because it ignores the position relationship in the data. At the same time, when the smart museum is displaying, when the displayed information changes greatly, the attention score of the feature data in the single-modal data is usually low. Therefore, clustering or data segmentation within the single modality can be completed by clustering within the single-modal data and making the attention score within a single cluster more concentrated.

[0048] The self-attention mechanism can capture the relationship between different feature vectors, effectively evaluate the relationship between feature vectors and other features, identify local potential associations through local analysis, and then optimize the clustering results of features. Therefore, we first analyze the degree of association between feature vectors and other local feature vectors in the feature set of each modality based on the self-attention mechanism, obtain the local association index of each feature vector, and evaluate each feature vector through the attention association of local features.

[0049] Preferably, in an embodiment of the present invention, the method for obtaining the local correlation index includes: The self-attention mechanism can obtain the score of each element in the set, that is, the attention weight, which reflects the degree of attention the model pays to other elements in the set when processing the current element. In the virtual interactive system of the smart museum, there are usually one or more main display objects, which appear as significant elements in the overall or local features of the multimodal data. Since the self-attention mechanism can automatically assign weights, these main display objects usually appear as higher self-attention scores in the relevant feature data of each modality. Therefore, first calculating the self-attention mechanism on the unimodal data helps to identify and highlight these main display objects.

[0050] First, in the feature set of each modality, the self-attention score of each feature vector and each other feature vector, as well as the self-attention weight of each feature vector in the feature set are obtained through the self-attention mechanism. The self-attention score is calculated by the self-attention mechanism to characterize the degree of relationship between each feature vector and other vectors, and the self-attention weight is further obtained to characterize the contribution of each feature vector to the overall feature set. It should be noted that the method of obtaining the self-attention score and self-attention weight by the self-attention mechanism is a technical means well known to those skilled in the art, and will not be elaborated here.

[0051] In the self-attention mechanism, the self-attention weight of each element is a vector, because each element will calculate the similarity with all other elements. The self-attention weight contains the global attention distribution of an element to the entire set. For example, assuming that the length of the set is n, then for an input element, the self-attention weight after Softmax processing is an n-dimensional vector, and each dimension represents the attention distribution of the input element to the n elements in the set.

[0052] Further, for any feature vector in each modal feature set, the feature vector within a preset local range of the feature vector in the feature set is used as the local feature vector of the feature vector to determine the local analysis range. In an embodiment of the present invention, the preset local range is set to be a range with the feature vector as the center and a radius of 20 data lengths to the left and right of the set. When the feature vector can reach the range limit on both sides, the number of local feature vectors of the feature vector is 40. The implementer of the specific range value setting can adjust it according to the specific implementation scenario, and no limitation is made here.

[0053] Further, according to the relative degree between the distribution of the feature vector and the local feature vector in the self-attention score and the self-attention weight, the local correlation of the feature vector is obtained, and the accuracy of the local feature representation is improved through the relative attention and correlation. In an embodiment of the present invention, the method for obtaining the local correlation includes: First, take each local eigenvector as the target vector in turn, calculate the projection of the self-attention weight of the target vector on the self-attention weight of the eigenvector, and obtain the relative attention of the target vector. Projection is to map the self-attention weight distribution of the target vector to the current eigenvector, and calculate the attention of the target vector relative to the current eigenvector.

[0054] Furthermore, the product of the self-attention score and the relative attention of the feature vector and the target vector is used as the correlation between the feature vector and the target vector. The correlation is reflected by the self-attention score and weighted by the relative attention. This can more accurately measure the relationship between features and avoid the precision of simple self-attention score measurement.

[0055] Finally, the average of the correlations between the feature vector and all local feature vectors is taken as the local correlation of the feature vector, and the correlation between the feature vector and the local feature vectors is comprehensively considered. As an example, the expression of local correlation is: ; In the formula, Expressed as The local correlation of the eigenvectors, Expressed as The total number of local eigenvectors of eigenvectors, Expressed as The eigenvector and The self-attention score between local feature vectors, Expressed as The relative attention between local feature vectors, Expressed as The eigenvector and The correlation of local eigenvectors.

[0056] Finally, the local correlation index of the feature vector is obtained by combining the local correlation and self-attention weight of the feature vector. The local correlation characteristics of the feature vector are comprehensively characterized by combining the correlation of the local analysis with the attention of the feature vector in the global analysis. In an embodiment of the present invention, the product of the local correlation of the feature vector and the self-attention weight is used as the local correlation index of the feature vector.

[0057] The feature set in a single modality can be further clustered based on local relevance. When the main content changes, the local relevance will change significantly. Therefore, different local feature parts are clustered for subsequent analysis and identification. Preferably, in an embodiment of the present invention, the method for obtaining a feature clustering cluster includes: For any mode, the set of local correlations corresponding to the feature vectors in the mode is taken as the association set. By analyzing the distribution changes of the local correlations, the changes between the local features can be reflected. The number of clusters is determined by the elbow method for the association set of the mode. It should be noted that the elbow method is a commonly used method for determining the number of clusters, which is a technical means well known to those skilled in the art and will not be described in detail here.

[0058] Then, the feature vectors are clustered according to the local correlation of the feature vectors of the mode, and the number of clusters is obtained as the feature cluster clusters of the mode. In the embodiment of the present invention, the K-means clustering algorithm is used for clustering. During clustering, the seed points are evenly distributed in the data, and the local correlation is used as the distance metric to obtain the cluster clusters. The clustering algorithm is a technical means well known to those skilled in the art and is not limited here.

[0059] At this point, each single mode is clustered and the scope of subsequent analysis is limited by clustering clusters.

[0060] S3: Based on the cross-attention mechanism, the feature vector correlation matching degree between each feature cluster in each modality and each feature cluster in other modalities is analyzed to determine the best matching cluster of each feature cluster in each modality in other modalities.

[0061] The cross-attention mechanism can accurately align similar features and semantic relationships in different modalities. The feature clusters of each modality can be matched based on their correlation with clusters in other modalities, thereby ensuring the semantic consistency of each modality in the cluster. By determining the most relevant matching clusters of the feature clusters of each modality with other modalities, the system can avoid over-fusion of irrelevant information, thereby reducing the risk of overfitting. Therefore, by participating in the matching of clusters through cross-attention, multimodal information fusion is more efficient and compact.

[0062] Preferably, in the embodiment of the present invention, the method for obtaining the most matching cluster can be found in Figure 2 , which shows a flow chart of a method for obtaining the most matching cluster provided by an embodiment of the present invention, the method comprising the following steps: S301: Obtain the cross-attention score between each feature vector in each modality and each feature vector in other modalities, as well as the cross-attention weight of each feature vector in the feature set of other modalities through the cross-attention mechanism; for any modality, take each other modality as the cross-target modality in turn.

[0063] The cross-attention mechanism is an extension of the self-attention mechanism, which allows the model to directly calculate the degree of attention of each feature vector in one modality to all feature vectors in another modality when processing multimodal data. In this way, the model can understand the mutual correlation between different modalities, rather than just the internal information of a single modality.

[0064] The cross-attention score reflects the correlation between feature vectors, and the cross-attention weight represents the relative importance of a feature vector in another modality in the cross-modal feature space. Each other modality of the modality is analyzed in turn to match the feature clusters between the modalities.

[0065] S302: For any feature vector in the modality, the matching degree between the feature vector and each feature cluster in the cross-target modality is obtained according to the cross-attention score and cross-attention weight between the feature vector and the feature vector in each feature cluster in the cross-target modality, as well as the cross-attention weight of the feature vector in each feature cluster in the cross-target modality in the modality.

[0066] Combined with the attention between the feature vector and the feature clustering clusters whose matching degree is analyzed, as well as the global attention in the mutual feature set, the matching degree between the feature vector and the feature clustering cluster is evaluated to characterize the overall relationship between the feature vector and the feature clustering cluster.

[0067] Preferably, in an embodiment of the present invention, the method for obtaining the matching degree includes: First, for any feature cluster in the cross-target modality, the dot product of the cross-attention weight of the feature vector in the cross-target modality and the cross-attention weight of each feature vector in the feature cluster in the modality is calculated as the similarity weight between the feature vector and each feature vector in the feature cluster. The dot product result of the cross-attention weight reflects the similarity between the two feature vectors at the modality level, characterizing the potential attention similarity between these feature vectors.

[0068] Furthermore, the product of the cross-attention score and the similarity weight of the feature vector and each feature vector in the feature cluster is calculated to obtain the cross-similarity between the feature vector and each feature vector in the feature cluster. The cross-attention score reflects the overall similarity between the feature vector and all vectors of the cross-target modality. The similarity between the feature vectors is adjusted in combination with the similarity weight to reflect the similarity relationship in a finer granularity.

[0069] Finally, the cross-similarity between the feature vector and all feature vectors in the feature clustering cluster is combined to obtain the matching degree between the feature vector and the feature clustering cluster. In an embodiment of the present invention, the mean of the cross-similarity between the feature vector and all feature vectors in the feature clustering cluster is used as the matching degree between the feature vector and the feature clustering cluster, which comprehensively reflects the overall correspondence between the feature vector and the feature clustering cluster.

[0070] S303: Combining the matching degree of all feature vectors in the feature clustering cluster where the feature vector is located with each feature clustering cluster in the cross-target modality, obtain the cluster matching index of the feature clustering cluster where the feature vector is located and each feature clustering cluster in the cross-target modality; according to the cluster matching index of each feature clustering cluster in the modality and the feature clustering cluster in the cross-target modality, determine the most matching clustering cluster of each feature clustering cluster in the modality.

[0071] By combining the matching degree relationship between a single feature vector in a cluster and other feature clusters, the matching relationship between clusters is obtained. Preferably, in an embodiment of the present invention, the method for obtaining the cluster matching index includes: For any feature clustering cluster in the cross-target modality, the feature clustering cluster where the feature vector is located is used as the analysis clustering cluster. The mean of the matching degree between all feature vectors in the analysis clustering cluster and the feature clustering cluster is calculated as the cluster matching index between the analysis clustering cluster and the feature clustering cluster. By analyzing the matching between all feature vectors in the clustering cluster and the feature clustering cluster, the matching between clustering clusters is obtained.

[0072] Then, the best matching feature cluster of each feature cluster in other modalities can be screened out through the matching between clusters. According to the cluster matching index of each feature cluster in the modality and the feature cluster in the cross target modality, the best matching cluster of each feature cluster in the modality is determined, including: Each feature cluster in the modality is taken as the target cluster in turn. When the cluster matching index between the target cluster and the feature cluster in the cross-target modality is the largest, the feature cluster in the corresponding cross-target modality is taken as the best matching cluster of the target cluster in the cross-target modality. The feature cluster in other modalities with the best match is determined by the maximum cluster matching index of the target cluster.

[0073] At this point, the relevant matching at a local fine-grained level is completed, and the matching relationship between the clusters of different modalities is obtained.

[0074] S4: Multimodal fusion alignment is performed based on the most matching clustering clusters of feature clusters between different modalities.

[0075] By cross-matching the feature clusters in one modality with the best matching clusters in other modalities, the similarities and potential associations between different modalities can be captured more accurately. The alignment of each cluster in different modalities is not only based on local features, but also optimized through global matching, thereby improving the accuracy of cross-modal matching.

[0076] In an embodiment of the present invention, for any modality, data cross-matching is performed between the most matching clusters of each feature cluster in the modality in each other modality to obtain a cross-matching result between the modality and the other modalities, and data alignment and fusion are performed based on the cross-matching result.

[0077] Improve the expressiveness of multimodal information through more precise alignment and fusion, and enhance the interactive experience between smart museums and visitors in different exhibits. For example, the color and shape features in the image modality can be more accurately combined with the historical background and cultural connotation in the text modality, making the expression of exhibit information more complete and vivid. Or when a visitor says the name of an artist or historical period, the system can more accurately identify the relevant image and text information based on voice commands, quickly switch exhibits or provide detailed background explanations.

[0078] In summary, the present invention considers the global alignment difference of the cross-attention mechanism when processing multimodal data. First, the self-attention mechanism is combined with clustering to calculate the feature set in the single mode. The changes in the content features of the identified items are reflected through the association of the attention degree of the local feature vector, and clustering is performed to better capture local details and improve the accurate fusion of local features of different modalities. At the same time, there will be excessive cross-influence of redundant information between multimodal data. The cross-attention mechanism is used to further correct and match the feature clusters between different modalities, and the most matching cluster clusters are obtained to optimize the information alignment between modalities, improve the accuracy of fine-grained information matching, and use the matching relationship between different modalities to perform multimodal data fusion and reduce inconsistency. The present invention combines clustering segmentation with a self-attention mechanism to improve the accurate identification of local features, and uses a cross-modal attention mechanism to correct and match clusters of different modalities, reduce excessive cross-influence, and improve the quality of data fusion to provide a richer and more realistic virtual interactive experience.

[0079] Embodiment 2: The embodiment of the present invention provides a smart museum virtual interaction system integrating multimodal data, please refer to Figure 3, which shows a structural diagram of a smart museum virtual interaction system that integrates multimodal data provided by an embodiment of the present invention. The system 400 includes: a data acquisition module 401, a single modality division module 402, a multimodal matching module 403 and a multimodal fusion module 404.

[0080] A data acquisition module 401 is used to acquire feature vectors in each modality data and form a feature set, and the modalities include images, texts and audios; The single-modality segmentation module 402 is used to analyze the correlation between the feature vector in the feature set of each modality and other local feature vectors based on the self-attention mechanism to obtain the local correlation index of each feature vector; cluster the feature vectors in each modality according to the local correlation index to obtain feature clustering clusters; A multimodal matching module 403 is used to analyze the degree of correlation matching between feature vectors of each feature cluster in each modality and each feature cluster in other modalities based on a cross-attention mechanism, and determine the best matching cluster of each feature cluster in each modality in other modalities; The multimodal fusion module 404 is used to perform multimodal fusion alignment according to the most matching clustering clusters of feature clustering clusters between different modalities.

[0081] It should be noted that the system provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functional distribution can be completed by different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. Since the specific implementation process of a smart museum virtual interaction system integrating multimodal data in this embodiment is the same as the specific implementation process of a smart museum virtual interaction method integrating multimodal data in the above embodiment, it will not be repeated here.

[0082] Embodiment 3: The present invention also provides a computer device, see Figure 4 , which shows a schematic diagram of the structure of a computer device provided by an embodiment of the present invention, the computer device 500 at least includes, but is not limited to, a memory 501, a processor 502 and a communication interface 503 that can be interconnected and communicated via a bus 510.

[0083] The memory 501 may include a large-capacity memory for data or instructions, including at least one type of computer-readable storage medium, the readable storage medium including a flash memory, a hard disk, a read-only memory, a magnetic disk or an optical disk, etc. The memory 501 may be inside or outside the computer device. In an embodiment of the present invention, the memory 501 may be a read-only memory for storing computer program instructions.

[0084] The processor 502 may include one or more integrated circuits for controlling the overall operation of the computer device 500. In the embodiment of the present invention, the processor 502 reads and executes computer program instructions stored in the memory 501 to implement Figure 1 The steps of a smart museum virtual interaction method integrating multimodal data in Example 1 are shown.

[0085] The communication interface 503 is used to establish a communication connection between the computer device 500 and other electronic devices. In an embodiment of the present invention, the communication interface 503 is used to connect the computer device 500 to an external terminal through a network, establish a data transmission channel and a communication connection between the computer device 500 and the external terminal, etc.

[0086] Embodiment 4: The embodiment of the present invention also provides a computer-readable storage medium, such as a flash memory, a hard disk, an optical disk or a server, etc., on which computer program instructions are stored, and the program instructions are executed by a processor to implement corresponding functions. The computer-readable storage medium in the embodiment of the present invention is used to store computer program instructions for a virtual interactive method for a smart museum that integrates multimodal data, and the processor executes the steps of the virtual interactive method for a smart museum that integrates multimodal data.

[0087] It should be noted that the sequence of the above embodiments of the present invention is for description only and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0088] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A virtual interaction method for a smart museum integrating multimodal data, characterized in that: The method comprises: Get the feature vectors in each modality data and form a feature set, the modalities include image, text and audio; Based on the self-attention mechanism, the correlation between the feature vector in the feature set of each modality and other local feature vectors is analyzed to obtain the local correlation index of each feature vector; the feature vectors in each modality are clustered according to the local correlation index to obtain feature clustering clusters; Based on the cross-attention mechanism, the matching degree of feature vector correlation between each feature cluster in each modality and each feature cluster in other modalities is analyzed to determine the best matching cluster of each feature cluster in each modality in other modalities. Multimodal fusion alignment is performed based on the most matching clustering clusters of feature clusters between different modalities.

2. According to claim 1, a smart museum virtual interaction method integrating multimodal data is characterized in that: The method for obtaining the local correlation index includes: In the feature set of each modality, the self-attention score of each feature vector and each other feature vector is obtained through the self-attention mechanism, as well as the self-attention weight of each feature vector in the feature set. For any feature vector in each modal feature set, the feature vector in a preset local range in the feature set is used as the local feature vector of the feature vector; according to the relative degree between the feature vector and the local feature vector in the distribution of the self-attention score and the self-attention weight, the local correlation of the feature vector is obtained; The local correlation index of the feature vector is obtained by combining the local correlation and self-attention weight of the feature vector.

3. According to claim 2, a smart museum virtual interaction method integrating multimodal data is characterized in that: The method for obtaining the local correlation comprises: Take each local feature vector as the target vector in turn, calculate the projection of the self-attention weight of the target vector on the self-attention weight of the feature vector, and obtain the relative attention of the target vector; The product of the self-attention score and the relative attention of the feature vector and the target vector is taken as the correlation between the feature vector and the target vector; The average value of the correlation between the feature vector and all local feature vectors is taken as the local correlation of the feature vector.

4. According to claim 1, a method for virtual interaction of a smart museum integrating multimodal data, characterized in that: The method for obtaining the feature clustering clusters includes: For any mode, the set of local correlation degrees corresponding to the eigenvectors in the mode is taken as the correlation set; the number of clusters is determined by the elbow method for the correlation set of the mode; The feature vectors are clustered according to the local correlation degree of the feature vectors of the mode, and the number of clusters is obtained as the feature clustering clusters of the mode.

5. According to claim 1, a method for virtual interaction of a smart museum integrating multimodal data, characterized in that: The method for obtaining the best matching cluster includes: The cross-attention mechanism is used to obtain the cross-attention score between each feature vector in each modality and each feature vector in other modalities, as well as the cross-attention weight of each feature vector in the feature set of other modalities; for any modality, each other modality is taken as the cross-target modality in turn; For any feature vector in the modality, the matching degree between the feature vector and each feature cluster in the cross-target modality is obtained according to the cross-attention score and cross-attention weight between the feature vector and the feature vector in each feature cluster in the cross-target modality, as well as the cross-attention weight of the feature vector in each feature cluster in the cross-target modality in the modality; Combined with the matching degree of all feature vectors in the feature clustering cluster where the feature vector is located and each feature clustering cluster in the cross-target modality, the cluster matching index of the feature clustering cluster where the feature vector is located and each feature clustering cluster in the cross-target modality is obtained; based on the cluster matching index of each feature clustering cluster in the modality and the feature clustering cluster in the cross-target modality, the most matching clustering cluster of each feature clustering cluster in the modality is determined.

6. According to claim 5, a method for virtual interaction of a smart museum integrating multimodal data is characterized in that: The method for obtaining the matching degree includes: For any feature cluster in the cross-target modality, calculate the dot product of the cross-attention weight of the feature vector in the cross-target modality and the cross-attention weight of each feature vector in the feature cluster in the modality as the similarity weight between the feature vector and each feature vector in the feature cluster; Calculate the product of the cross attention score and the similarity weight between the feature vector and each feature vector in the feature cluster to obtain the cross similarity between the feature vector and each feature vector in the feature cluster; The cross similarity between the feature vector and all feature vectors in the feature cluster is combined to obtain the matching degree between the feature vector and the feature cluster.

7. The method for virtual interaction of a smart museum integrating multimodal data according to claim 5, characterized in that: The method for obtaining the cluster matching index includes: For any feature cluster in the cross-target mode, the feature cluster where the feature vector is located is taken as the analysis cluster; The mean of the matching degree between all feature vectors in the analysis clustering cluster and the feature clustering cluster is calculated as the cluster matching index between the analysis clustering cluster and the feature clustering cluster.

8. The method for virtual interaction of a smart museum integrating multimodal data according to claim 5, characterized in that: The step of determining the most matching cluster of each feature cluster in the modality according to the cluster matching index between each feature cluster in the modality and the feature cluster in the cross target modality includes: Each feature cluster in the modality is taken as the target cluster in turn; When the cluster matching index between the target cluster and the feature cluster in the cross-target modality is the largest, the feature cluster in the corresponding cross-target modality is used as the best matching cluster of the target cluster in the cross-target modality.

9. The method for virtual interaction of a smart museum integrating multimodal data according to claim 1, characterized in that: The multimodal fusion alignment is performed according to the most matching clustering clusters of feature clustering clusters between different modalities, including: For any modality, data cross-matching is performed between each feature cluster in the modality and the most matching cluster in each other modality to obtain the cross-matching result between the modality and other modalities; data alignment and fusion are performed based on the cross-matching result.

10. A smart museum virtual interaction system integrating multimodal data, used to implement a smart museum virtual interaction method integrating multimodal data as described in any one of claims 1 to 9, characterized in that: The system comprises: The data acquisition module is used to obtain the feature vectors in each modality data and form a feature set. The modalities include images, texts and audios. The single-modal segmentation module is used to analyze the correlation between the feature vectors in the feature set of each modality and other local feature vectors based on the self-attention mechanism, and obtain the local correlation index of each feature vector; cluster the feature vectors in each modality according to the local correlation index to obtain feature clustering clusters; The multimodal matching module is used to analyze the degree of correlation matching between the feature vectors of each feature cluster in each modality and each feature cluster in other modalities based on the cross-attention mechanism, and determine the best matching cluster of each feature cluster in each modality in other modalities; The multimodal fusion module is used to perform multimodal fusion alignment based on the most matching clustering clusters of feature clustering clusters between different modalities.

Images