Multi-modal feature fusion method and device, electronic equipment and storage medium

Abstract

The invention provides a multi-modal feature fusion method and device, electronic equipment and a storage medium, and relates to the technical field of electric power intelligent services, and the method comprises the steps: obtaining multi-modal data, carrying out the coding of each modal data in the multi-modal data, and obtaining an embedded vector corresponding to each modal data; based on each embedded vector and historical data, determining a modal weight corresponding to each embedded vector; and based on the modal weight and each embedded vector, a cross-modal attention mechanism is adopted to obtain weighted features. And matching and determining corresponding entity features in a pre-constructed knowledge graph vector library according to the weighted features, and splicing the weighted features and the entity features to obtain target fusion features. An industrial semantic space is constructed through a knowledge graph and a retrieval enhancement mechanism, so that the system can understand technical terms, business logic and context semantics, and deep fusion of semantic recognition and knowledge understanding is realized.

Description

Technical Field

[0001] This application relates to the field of intelligent power service technology, and in particular to a multimodal feature fusion method, fusion device, electronic device and storage medium. Background Technology

[0002] Against the backdrop of digital transformation in electricity marketing, power companies have widely deployed intelligent service systems based on digital humans for scenarios such as customer consultation, business processing, explanation of electricity pricing policies, and energy consumption guidance. However, existing digital human systems rely on single-modal information input (such as voice or text) during user interaction, lacking the comprehensive understanding of multimodal information such as tone of voice, semantic context, facial expressions, and user emotions. This makes it difficult for the system to accurately identify the user's true intentions in complex contexts. For example, when a user's tone carries doubt or dissatisfaction, traditional systems often fail to perceive these emotional changes, resulting in responses that do not match user expectations, thus affecting service experience and corporate image. Summary of the Invention

[0003] In view of this, the purpose of this application is to propose a modal feature fusion method, fusion device, electronic device and storage medium to solve the problem of inaccurate identification of the user's true intent.

[0004] To achieve the above objectives, the first aspect of this application provides a multimodal feature fusion method, comprising: Acquire multimodal data; Each modality in the multimodal data is encoded to obtain the embedding vector corresponding to each modality. Based on each embedding vector and historical data, determine the modal weights corresponding to each embedding vector; Based on the modal weights and each embedding vector, a cross-modal attention mechanism is used to obtain weighted features; Based on the weighted features, the corresponding entity features are determined by matching in a pre-constructed knowledge graph vector library; The weighted features and the entity features are concatenated to obtain the target fusion features.

[0005] Based on the same inventive concept, a second aspect of this application also provides a multimodal feature fusion apparatus, comprising: The acquisition module is configured to acquire multimodal data; The encoding module is configured to encode each modality data in the multimodal data to obtain the embedding vector corresponding to each modality data. The weighting module is configured to determine the modal weights corresponding to each embedding vector based on each embedding vector and historical data. The weighting module is configured to obtain weighted features based on the modal weights and each embedding vector using a cross-modal attention mechanism; The matching module is configured to match and determine the corresponding entity features in a pre-built knowledge graph vector library based on the weighted features; The fusion module is configured to concatenate the weighted features and the entity features to obtain the target fused features.

[0006] Based on the same inventive concept, a third aspect of this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method described above when executing the computer program.

[0007] Based on the same inventive concept, a fourth aspect of this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method described above.

[0008] As can be seen from the above, the multimodal feature fusion method, fusion device, electronic device, and storage medium provided in this application include: acquiring multimodal data; encoding each modality data in the multimodal data to obtain an embedding vector corresponding to each modality data; determining the modal weights corresponding to each embedding vector based on each embedding vector and historical data; obtaining weighted features using a cross-modal attention mechanism based on the modal weights and each embedding vector, achieving adaptive weighting in different scenarios through dynamic weighting, significantly improving the robustness and dynamic response capability of multimodal feature fusion; matching and determining corresponding entity features in a pre-constructed knowledge graph vector library according to the weighted features; concatenating the weighted features and the entity features to obtain the target fusion feature; and constructing an industry-specific semantic space through knowledge graph and retrieval enhancement mechanisms, enabling the system to understand professional terms, business logic, and contextual semantics, achieving deep fusion of semantic recognition and knowledge understanding. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating the multimodal feature fusion method according to an embodiment of this application; Figure 2 This is a diagram of the multimodal feature fusion architecture of an embodiment of this application; Figure 3 This is a schematic diagram of the structure of the multimodal feature fusion device according to an embodiment of this application; Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0012] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0013] Intelligent electricity billing has always been a key research area in the power sector. Numerous scholars and practitioners have conducted in-depth analyses of intelligent electricity billing from different perspectives, and the research has gone through the following three stages of development: (1) User intent recognition methods based on unimodal features: User intent recognition methods mainly rely on unimodal text features, that is, through natural language processing (NLP) techniques, extracting keywords or semantic embeddings from sentences for classification. Typical approaches are to use the Bag-of-Words model, TF-IDF (Term Frequency–Inverse Document Frequency), or later deep semantic embedding models (such as Word2Vec, BERT) to convert text into vector representations. Then input it into the classifier for intent prediction: , in, Indicates weight, This indicates bias. The advantage of this stage is its simplicity and low computational cost, but it has the significant drawback of text modality failing to reflect information such as tone, speech rate, and emotion. This is due to the lack of speech information. With visual information The system suffers from low intent recognition accuracy, making it prone to misjudgments or omissions, and failing to meet the demands of complex user interactions in electricity marketing scenarios. Especially in electricity marketing, when users' semantics are ambiguous (e.g., "I don't quite understand why electricity prices have increased") or their tone is emotional, the system often misjudges it as a regular inquiry rather than a complaint or challenge, leading to inaccurate service responses.

[0014] (2) Speech-text intent recognition method based on dual-modal fusion With the development of speech recognition and acoustic feature analysis technologies, researchers have begun to combine speech features with text semantics for intent recognition. This method typically extracts low-level features of the speech signal, such as Mel-Frequency Cipstal Coefficients (MFCCs), speech rate, and energy intensity, and encodes them into acoustic vectors. and text features The splicing creates a composite feature: , The data is then fed into a classification network for intent prediction. While this method significantly improves recognition accuracy compared to a single modality, it still has three major limitations: first, the concatenation operation only achieves static fusion, lacking semantic-level interactive modeling; second, speech and text are asynchronous on a temporal scale, making feature alignment difficult; and third, in multi-turn dialogues, the model's perception of contextual emotional changes is insufficient. Therefore, when users speak with sarcasm or intonation fluctuations, the system still struggles to accurately identify their true intent.

[0015] (3) Multimodal fusion-based speech-text-visual collaborative recognition method With the development of deep learning and the Transformer stage, multimodal learning has become the mainstream approach for user intent recognition. The technical approach at this stage is to simultaneously incorporate speech, text, and visual modal features, capturing the semantic relationships between modalities through joint modeling. For example, speech features... Text features and visual features Embedded in a unified semantic space respectively: , Then, the correlation between modes is calculated using a cross-modal attention mechanism: , in, The elements of the attention weight matrix represent the correlation between mode i and mode j. The query vector representing modality i. The key vector representing mode j. Represents the dimension of the vector, and represents the normalization function.

[0016] It significantly improves the ability to understand complex contexts, simultaneously capturing the relationships between semantics, emotion, and expression. However, existing research still has the following shortcomings: First, multimodal feature fusion lacks a dynamic weight adjustment mechanism, and the contributions of different modalities in different scenarios are not adaptively adjusted; second, the multimodal model lacks sufficient understanding of knowledge in the electricity marketing domain, and the semantic space is not optimized for industry tasks; third, interpretability is insufficient: it is difficult to quantify the contribution of each modality to the final intent judgment, reducing the credibility of the digital human's response, especially in electricity marketing complaint and billing dispute scenarios, which poses regulatory and user trust risks.

[0017] In view of this, this application proposes a multimodal feature fusion method, aiming to construct a unified semantic understanding model by integrating multimodal feature information such as speech, text, and vision. This enables digital humans to accurately identify the true intentions of users in power marketing scenarios, improving recognition accuracy in multi-turn dialogues and emotional change scenarios. Furthermore, the method is optimized for the specific semantics and user energy consumption behavior of the power marketing industry, achieving high accuracy and interpretability of the model in complex business scenarios. This significantly improves the digital human system's comprehensive perception of user intentions and emotions, ensuring the naturalness and intelligence of the interaction.

[0018] The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0019] This application proposes a multimodal feature fusion method, referencing... Figure 1 This includes the following steps: Step 101: Obtain multimodal data.

[0020] Specifically, multimodal data refers to data originating from different sensory channels, including text data, voice data, and visual data. Text data can be text directly input by the user or text converted from voice data. Text data directly reveals the specific content the user wants to inquire about. Voice data is collected through sound sensors, allowing us to understand the user's tone and emotions. Visual data is collected through image sensors, also providing insights into the user's emotions.

[0021] Step 102: Encode each modal data in the multimodal data to obtain the embedding vector corresponding to each modal data.

[0022] Specifically, when encoding multimodal data, each modality can be encoded using its corresponding encoder to obtain the corresponding embedding vector. Encoding one modality per encoder helps improve the accuracy of that modality's encoding.

[0023] Step 103: Based on each embedding vector and historical data, determine the modal weights corresponding to each embedding vector.

[0024] Specifically, since different embedding vectors have different information reliability and contribution in different interaction scenarios, this step can dynamically determine the modal weights corresponding to each embedding vector based on each embedding vector and historical data, thereby achieving adaptive adjustment of the modal weights of each modal data.

[0025] Step 104: Based on the modal weights and each embedding vector, a cross-modal attention mechanism is used to obtain weighted features.

[0026] Specifically, a cross-modal attention mechanism is employed, which weights and fuses each embedding vector according to the modal weights, enabling real-time optimization of the contribution of information from different modalities. Through this mechanism, the modal fusion ratio is automatically adjusted in response to scene changes and semantic shifts, thereby achieving more stable and intelligent multimodal interaction and intent recognition in the digital human system.

[0027] Step 105: Match and determine the corresponding entity features in the pre-constructed knowledge graph vector library based on the weighted features.

[0028] Specifically, the knowledge graph vector library is pre-built based on domain-specific knowledge. Domain data is collected, entities and attributes are extracted from the domain data, and triples are constructed based on the entities and attributes. A knowledge graph is then built based on these triples. For example, in the electricity domain, entities include "residential users" and "tiered electricity pricing," and relations include "applies to." The constructed triple is ("residential users," "applies to," "tiered electricity pricing").

[0029] Through graph embedding functions The knowledge graph is converted into a vectorized representation, resulting in a knowledge graph vector library. During this conversion, the optimization objective is expressed by the following formula: , in, Represents a knowledge triple. This represents the true embedding vector of the entity. By minimizing the above optimization objective, the model continuously adjusts the vector representation of each entity and relation, ultimately achieving a vectorized representation ( As close as possible .

[0030] Weighted features are matched for similarity in a knowledge graph vector database to determine the corresponding entity features. These entity features serve as domain contextual knowledge in the subsequent intent recognition process, enabling the model to not only understand literal meanings but also connect them to business rules within the domain, achieving a leap from perception to cognition. Furthermore, entity features help trace the model's decision-making process, thereby enhancing the model's interpretability.

[0031] Step 106: Concatenate the weighted features and the entity features to obtain the target fusion features.

[0032] Specifically, after determining the weighted features and entity features, the two are concatenated to obtain the target fusion feature. The target fusion feature can be used for subsequent user intent recognition, significantly improving the accuracy of intent recognition.

[0033] Target fusion features It can be determined by the following formula:

[0034] in, Indicates weighted features, Represent entity characteristics, This represents a multi-layer nonlinear mapping function. Then, the target fusion features are input into a two-layer fully connected neural network and a softmax classification layer, outputting the corresponding user intent category (such as inquiry, complaint, confirmation, denial, etc.).

[0035] Based on steps 101 to 106 above, the multimodal feature fusion method provided in this embodiment includes: acquiring multimodal data; encoding each modality of the multimodal data to obtain an embedding vector corresponding to each modality; determining the modal weights corresponding to each embedding vector based on each embedding vector and historical data; obtaining weighted features using a cross-modal attention mechanism based on the modal weights and each embedding vector, achieving adaptive weighting in different scenarios through dynamic weighting, significantly improving the robustness and dynamic response capability of multimodal feature fusion; matching and determining corresponding entity features in a pre-constructed knowledge graph vector library according to the weighted features; concatenating the weighted features and the entity features to obtain the target fusion feature; and constructing an industry-specific semantic space through knowledge graph and retrieval enhancement mechanisms, enabling the system to understand professional terms, business logic, and contextual semantics, achieving deep integration of semantic recognition and knowledge understanding.

[0036] In some embodiments, determining the modal weights corresponding to each embedding vector based on each embedding vector and historical data includes: The effectiveness of each embedding vector is evaluated based on a preset evaluation rule to determine the confidence score corresponding to each embedding vector; the modal weights corresponding to each embedding vector are determined based on the confidence scores and historical data.

[0037] Specifically, historical data includes users' historical dialogue intentions, emotional states, and environmental factors. Pre-defined evaluation rules can be set in advance according to specific technical fields to evaluate the effectiveness of each embedding vector. These pre-defined evaluation rules can be implemented using a Modality Reliability Estimator (MRE). The MRE performs real-time evaluation of the effectiveness of each embedding vector, obtaining a confidence score for each embedding vector. This can reflect the reliability of the information of the embedding vector in the current interaction state.

[0038] Furthermore, based on the confidence scores and historical data, the modal weights corresponding to each embedding vector are determined, including: Determine the context semantic vector based on the historical data; Based on the context semantic vector, a context gating function is used to determine the modal weights corresponding to each embedding vector.

[0039] Specifically, historical data is converted into contextual semantic vectors, based on the confidence scores. Construct a context gating function using context semantic vectors. The modal weights corresponding to each embedding vector are determined by the following formula. :

[0040] in, Represents an embedding vector. This represents a vector concatenation operation. This represents the Sigmoid activation function. and These represent the learnable parameter matrix and the bias term, respectively. Let m = {s, v, t}, where s represents the speech modality, v represents the visual modality, and t represents the text modality. The method in this embodiment allows for the adjustment of the weights of different embedding vectors in intent determination, thus enabling correct identification of user intent even when speech, text, or visual signals are inconsistent. This not only improves the system's robustness in multimodal conflict scenarios but also enhances the digital human's sensitivity to the user's true emotions and intent, making interaction more natural and intelligent.

[0041] In some embodiments, the multimodal data includes text data, speech data, and visual data, and the embedding vectors corresponding to each modal data include text embedding vectors, speech embedding vectors, and visual embedding vectors. The step of encoding each modality data in the multimodal data to obtain the embedding vector corresponding to each modality data includes: The text data is encoded using a pre-trained text encoder to obtain a text embedding vector; The speech data is encoded using a pre-trained speech encoder to obtain a speech embedding vector; The visual data is encoded by a pre-trained visual encoder to obtain a visual embedding vector.

[0042] Specifically, different encoders are used when encoding data for each modality. Text data is encoded using a text encoder to obtain text features. For example, the text encoder can be a pre-trained language model BERT. Using Mel-frequency coefficients, pitch, energy curves, and speech rate features, a speech encoder encodes the speech data to obtain speech features. For example, a speech encoder can be a bidirectional LSTM (Long Short-Term Memory) network to encode speech data and obtain temporal features. Visual features are obtained by encoding visual data using a visual encoder. For example, a visual encoder can be a residual neural network (ResNet) and a facial action coding system (FACS) to encode visual data and extract visual representations of the user's facial expression and posture features. .

[0043] Each modal feature is mapped to a unified semantic space through an independent encoder. , in, This represents the embedding vector corresponding to each modality feature. The embedding function represents the data of each modality. Representing the modal characteristics, Represents a text embedding vector. Represents the speech embedding vector. This represents the visual embedding vector.

[0044] This embodiment utilizes temporal modeling methods such as bidirectional LSTM to model the changes in speech, visual, and text features over time, enabling the perception of the evolution of user intent over time. It can identify changes in intent and emotional fluctuations in multi-turn dialogues, allowing the digital human to more naturally understand the user's continuous intent and provide timely responses.

[0045] In some embodiments, the training process of the text encoder includes: Build the initial text encoder; Collect domain-specific text data to construct a domain-specific training dataset; Based on the aforementioned domain training dataset, the initial text encoder is trained by minimizing the domain mask language modeling loss function to obtain the text encoder; Among them, the domain mask language modeling loss function It is expressed as follows: , Where N represents the total number of masked words in a batch. This represents the i-th word that is masked in the sentence. It refers to all other words in the sentence besides the ones mentioned above. Represents the parameters of the domain model. This indicates that, given the context and domain model parameters, the model predicts the vocabulary. The probability of.

[0046] Specifically, the initial text encoder can be a model pre-trained on general massive text datasets, such as BERT. Domain text data can be from the power marketing domain, specifically including documents such as the "Power Marketing Business Rules" and "Power Supply Business Rules," historical customer service work orders, dialogue records, power policy documents, power equipment manuals, industry analysis reports, etc. A domain training dataset is constructed based on this data. The initial text encoder is trained on this dataset, randomly masking some words from the input sentence, and then predicting the masked words using the initial text encoder. This is achieved through multiple rounds of training on the domain training dataset, minimizing the domain masking language modeling loss function. Domain model parameters It is continuously adjusted so that the text encoder obtained after training can have the ability to deeply understand the semantic expression of the domain, thus solving the problems of insufficient understanding of the semantics of power business and incomplete coverage of domain knowledge of existing digital humans.

[0047] In some embodiments, the training process of the speech encoder includes: Construct the initial speech encoder; Acquire domain-specific speech data; Construct positive speech sample pairs and negative speech sample pairs based on the text embedding samples and the domain speech data, respectively; The initial speech encoder is trained by minimizing the speech contrast learning loss function using the positive speech sample pairs and the negative speech sample pairs to obtain the speech encoder. Among them, the speech contrast learning loss function It is expressed as follows:

[0048] in, Represents the cosine similarity function. Indicates temperature parameter, This represents the embedding vector of positive speech sample pairs. This represents the embedding vector of the negative speech sample pair. Let K represent the text embedding sample, and K represent the total number of speech negative sample pairs.

[0049] Specifically, in this embodiment, using text embedding samples as anchors, the speech encoder is trained so that the speech embedding vector output by the speech encoder is close to the text embedding sample in the vector space, facilitating semantic alignment with the text embedding sample. In practice, an initial speech encoder is first constructed. Domain speech data is acquired, including historical speech data input by the user. Positive and negative speech sample pairs are constructed based on the domain speech data and the text embedding samples. For example, if the text embedding sample is "electricity bill is too high" and the first historical user input speech data is also "electricity bill is too high," then the text embedding sample and the first historical user input speech constitute a positive speech sample pair. If the second historical user input speech data is "apply for capacity expansion," then the text embedding sample and the second historical user input speech constitute a negative speech sample pair. The speech encoder is trained using the positive and negative speech sample pairs, minimizing the speech contrastive learning loss function during training. This aims to increase the sample similarity in positive speech sample pairs and decrease the sample similarity in negative speech sample pairs. This is achieved by minimizing the speech contrast learning loss function. The parameters in the speech encoder are continuously updated. Ultimately, the trained speech encoder outputs speech embedding vectors that are identical to the text embedding vectors in the vector space, forming a unified domain semantic space. Finally, regardless of the modality of the data, as long as the expressed domain semantics are the same or similar, the embedding vectors output by the encoder will cluster together in the space, which is beneficial for improving the efficiency and effectiveness of the weighted fusion of various embedding vectors in subsequent cross-modal attention mechanisms.

[0050] In some embodiments, the training process of the visual encoder includes: Construct the initial visual encoder; Acquire domain visual data; Visual positive samples and visual negative samples are constructed based on the text embedding samples and the domain visual data, respectively. The initial visual encoder is trained by minimizing the visual contrast learning loss function using the visual positive samples and the visual negative samples to obtain the visual encoder. Among them, the visual contrast learning loss function It is expressed as follows: , in, Represents the cosine similarity function. Indicates temperature parameter, Represents the visual positive sample embedding vector. This represents the visual negative sample embedding vector. L represents the total number of visual negative samples.

[0051] Specifically, in this embodiment, using text embedding samples as anchors, a visual encoder is trained so that the visual embedding vector output by the visual encoder is close to the text embedding sample in the vector space, facilitating semantic alignment with the text embedding sample. In practice, an initial visual encoder is first constructed. Domain visual data is acquired, including video images of users conducting business. Positive and negative visual sample pairs are constructed based on the domain visual data and the text embedding samples. For example, if the text embedding sample is "electricity bill is too high" and the first historical user input visual data is a "frowning expression," then the text embedding sample and the first historical user input visual data constitute a positive visual sample pair. If the second historical user input visual data is a "calm expression," then the text embedding sample and the second historical user input visual data constitute a negative visual sample pair. The visual encoder is trained using the positive and negative visual sample pairs, minimizing the visual contrastive learning loss function during training. This aims to increase the sample similarity in visually positive sample pairs and decrease the sample similarity in visually negative sample pairs. This is achieved by minimizing the visual contrast learning loss function. The parameters in the visual encoder are continuously updated. Ultimately, the trained visual encoder outputs visual embedding vectors that are identical to the text embedding vectors in the vector space, forming a unified domain semantic space. Finally, regardless of the modality of the data, as long as the domain semantics expressed are the same or similar, the embedding vectors output by the encoder will cluster together in the space, which is beneficial for improving the efficiency and effectiveness of the weighted fusion of various embedding vectors in subsequent cross-modal attention mechanisms.

[0052] It should be noted that the embodiments of this application can also be further described in the following ways: Figure 2 A diagram of the multimodal feature fusion architecture is shown. (For example...) Figure 2 As shown, text data, speech data, and visual data are encoded using their respective encoders to obtain embedding vectors. A context gating function is used to determine the modal weights corresponding to each embedding vector. Based on these modal weights, a cross-modal attention mechanism is employed to fuse the embedding vectors, resulting in weighted features. Simultaneously, a domain-based model is introduced during the fusion process, and a graph embedding function is used. The converted knowledge graph vector library is used to retrieve entity features. Finally, the weighted features and entity features are concatenated to obtain the target fusion features.

[0053] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0054] It should be noted that some embodiments of this application have been described above. In some cases, the actions or steps described in the above embodiments can be performed in a different order than that shown in the above embodiments and the desired result can still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0055] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a multimodal feature fusion device.

[0056] refer to Figure 3 The multimodal feature fusion device includes: The acquisition module 301 is configured to acquire multimodal data; The encoding module 302 is configured to encode each modal data in the multimodal data to obtain the embedding vector corresponding to each modal data. The weight module 303 is configured to determine the modal weights corresponding to each embedding vector based on each embedding vector and historical data. The weighting module 304 is configured to obtain weighted features based on the modal weights and each embedding vector using a cross-modal attention mechanism; Matching module 305 is configured to match and determine the corresponding entity features in a pre-built knowledge graph vector library based on the weighted features; The fusion module 306 is configured to concatenate the weighted features and the entity features to obtain the target fused features.

[0057] In some embodiments, the weighting module 303 is configured to evaluate the effectiveness of each embedding vector based on a preset evaluation rule, determine the confidence score corresponding to each embedding vector, and determine the modal weight corresponding to each embedding vector based on the confidence score and historical data.

[0058] In some embodiments, the weighting module 303 is configured to determine a context semantic vector based on the historical data; and based on the context semantic vector, to determine the modal weights corresponding to each embedding vector using a context gating function.

[0059] In some embodiments, the multimodal data includes text data, speech data, and visual data, and the embedding vectors corresponding to each modality data include text embedding vectors, speech embedding vectors, and visual embedding vectors; the encoding module 302 is configured to encode the text data using a pre-trained text encoder to obtain a text embedding vector; to encode the speech data using a pre-trained speech encoder to obtain a speech embedding vector; and to encode the visual data using a pre-trained visual encoder to obtain a visual embedding vector.

[0060] In some embodiments, a training module is also included, configured to build an initial text encoder; Collect domain-specific text data to construct a domain-specific training dataset; Based on the aforementioned domain training dataset, the initial text encoder is trained by minimizing the domain mask language modeling loss function to obtain the text encoder; Among them, the domain mask language modeling loss function It is expressed as follows: , Where N represents the total number of masked words in a batch. This represents the i-th word that is masked in the sentence. It refers to all other words in the sentence besides the ones mentioned above. Represents the parameters of the domain model. This indicates that, given the context and domain model parameters, the model predicts the vocabulary. The probability of.

[0061] In some embodiments, the training module is configured to build an initial speech encoder; Acquire domain-specific speech data; Construct positive speech sample pairs and negative speech sample pairs based on the text embedding samples and the domain speech data, respectively; The initial speech encoder is trained by minimizing the speech contrast learning loss function using the positive speech sample pairs and the negative speech sample pairs to obtain the speech encoder. Among them, the speech contrast learning loss function It is expressed as follows:

[0062] in, Represents the cosine similarity function. Indicates temperature parameter, This represents the embedding vector of positive speech sample pairs. This represents the embedding vector of the negative speech sample pair. Let K represent the text embedding sample, and K represent the total number of speech negative sample pairs.

[0063] In some embodiments, the training module is configured to build an initial visual encoder; Acquire domain visual data; Visual positive samples and visual negative samples are constructed based on the text embedding samples and the domain visual data, respectively. The initial visual encoder is trained by minimizing the visual contrast learning loss function using the visual positive samples and the visual negative samples to obtain the visual encoder. Among them, the visual contrast learning loss function It is expressed as follows: , in, Represents the cosine similarity function. Indicates temperature parameter, Represents the visual positive sample embedding vector. This represents the visual negative sample embedding vector. L represents the total number of visual negative samples.

[0064] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0065] The apparatus described above is used to implement the corresponding multimodal feature fusion method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0066] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multimodal feature fusion method described in any of the above embodiments.

[0067] Figure 4 This embodiment illustrates a more specific hardware structure of an electronic device. The device may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0068] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0069] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0070] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0071] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0072] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0073] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0074] The electronic devices described above are used to implement the corresponding multimodal feature fusion methods in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0075] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the multimodal feature fusion method as described in any of the above embodiments.

[0076] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0077] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the multimodal feature fusion method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0078] Based on the same concept, corresponding to any of the above embodiments, this application also provides a computer program product, including computer program instructions, which, when run on a computer, cause the computer to perform the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0079] It is understood that before using the technical solutions of the various embodiments in this disclosure, users will be informed of the type, scope of use, and usage scenarios of the personal information involved in an appropriate manner, and user authorization will be obtained.

[0080] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to choose, based on the prompt message, whether to provide personal information to the software or hardware such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution.

[0081] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" regarding the provision of personal information by the electronic device.

[0082] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0083] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application is limited to these examples; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0084] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0085] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0086] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A multimodal feature fusion method, characterized in that, include: Acquire multimodal data; Each modality in the multimodal data is encoded to obtain the embedding vector corresponding to each modality. Based on each embedding vector and historical data, determine the modal weights corresponding to each embedding vector; Based on the modal weights and each embedding vector, a cross-modal attention mechanism is used to obtain weighted features; Based on the weighted features, the corresponding entity features are determined by matching in a pre-constructed knowledge graph vector library; The weighted features and the entity features are concatenated to obtain the target fusion features.

2. The method according to claim 1, characterized in that, The process of determining the modal weights corresponding to each embedding vector based on each embedding vector and historical data includes: The validity of each embedding vector is evaluated based on the preset evaluation rules, and the confidence score corresponding to each embedding vector is determined. Based on the confidence scores and historical data, determine the modal weights corresponding to each embedding vector.

3. The method according to claim 2, characterized in that, The step of determining the modality weights corresponding to each embedding vector based on the confidence scores and historical data includes: Determine the context semantic vector based on the historical data; Based on the context semantic vector, a context gating function is used to determine the modal weights corresponding to each embedding vector.

4. The method according to claim 1, characterized in that, The multimodal data includes text data, speech data, and visual data, and the embedding vectors corresponding to each modal data include text embedding vectors, speech embedding vectors, and visual embedding vectors. The step of encoding each modality data in the multimodal data to obtain the embedding vector corresponding to each modality data includes: The text data is encoded using a pre-trained text encoder to obtain a text embedding vector; The speech data is encoded using a pre-trained speech encoder to obtain a speech embedding vector; The visual data is encoded by a pre-trained visual encoder to obtain a visual embedding vector.

5. The method according to claim 4, wherein the training process of the text encoder includes: Build the initial text encoder; Collect domain-specific text data to construct a domain-specific training dataset; Based on the aforementioned domain training dataset, the initial text encoder is trained by minimizing the domain mask language modeling loss function to obtain the text encoder; Among them, the domain mask language modeling loss function It is expressed as follows: Where N represents the total number of masked words in a batch. This represents the i-th word that is masked in the sentence. It refers to all other words in the sentence besides the ones mentioned above. Represents the parameters of the domain model. This indicates that, given the context and domain model parameters, the model predicts the vocabulary. The probability of.

6. The method according to claim 4, wherein the training process of the speech encoder includes: Construct the initial speech encoder; Acquire domain-specific speech data; Construct positive speech sample pairs and negative speech sample pairs based on the text embedding samples and the domain speech data, respectively; The initial speech encoder is trained by minimizing the speech contrast learning loss function using the positive speech sample pairs and the negative speech sample pairs to obtain the speech encoder. Among them, the speech contrast learning loss function It is expressed as follows: in, Represents the cosine similarity function. Indicates temperature parameter, This represents the embedding vector of positive speech sample pairs. This represents the embedding vector of the negative speech sample pair. Let K represent the text embedding sample, and K represent the total number of speech negative sample pairs.

7. The method according to claim 4, wherein the training process of the visual encoder includes: Construct the initial visual encoder; Acquire domain visual data; Visual positive samples and visual negative samples are constructed based on the text embedding samples and the domain visual data, respectively. The initial visual encoder is trained by minimizing the visual contrast learning loss function using the visual positive samples and the visual negative samples to obtain the visual encoder. Among them, the visual contrast learning loss function It is expressed as follows: ， in, Represents the cosine similarity function. Indicates temperature parameter, Represents the visual positive sample embedding vector. This represents the visual negative sample embedding vector. L represents the total number of visual negative samples.

8. A multimodal feature fusion device, characterized in that, include: The acquisition module is configured to acquire multimodal data; The encoding module is configured to encode each modality data in the multimodal data to obtain the embedding vector corresponding to each modality data. The weighting module is configured to determine the modal weights corresponding to each embedding vector based on each embedding vector and historical data. The weighting module is configured to obtain weighted features based on the modal weights and each embedding vector using a cross-modal attention mechanism; The matching module is configured to match and determine the corresponding entity features in a pre-built knowledge graph vector library based on the weighted features; The fusion module is configured to concatenate the weighted features and the entity features to obtain the target fused features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 8.

Images