Multi-modal entity recognition model training method and device, equipment and storage medium

Abstract

The invention relates to the field of artificial intelligence, and discloses a multi-modal entity recognition model training method, which comprises the following steps of: obtaining voice data and text data, and respectively carrying out feature processing on the voice data and the text data; performing framing processing and Fourier transform on the voice data to obtain voice features, and encoding the voice features through an encoder to generate voice feature vectors; text embedding features are generated by text data through word embedding and position embedding, and then text feature vectors are generated through processing of a decoder. The voice feature vector is processed by a CTC layer to generate a voice entity vector, and the text feature vector is processed by a CRF layer to generate a text entity vector. And training the text entity vector, the voice entity vector and a comparative learning method, and optimizing the initial model to obtain a target multi-modal entity recognition model. The voice data and the text data are processed at the same time, the multi-mode entity recognition model can capture complementary information between different modes, and the entity recognition capacity of the model is improved.

Description

Technical Field

[0001] The present invention relates to the field of multimodal learning technology in the field of artificial intelligence, and in particular to a multimodal entity recognition model training method, device, equipment and storage medium. Background Art

[0002] Multimodal Entity Recognition is an important direction in the field of artificial intelligence, which aims to identify and understand entities in data by combining different types of data sources (such as speech and text). This method can be applied to various scenarios, such as virtual assistants, automatic speech recognition systems, and multimodal information retrieval.

[0003] With the rapid development of deep learning, researchers have gradually realized that relying solely on data from a single modality often cannot fully understand complex real-world information. Therefore, multimodal learning has gradually emerged, aiming to enhance the perception ability of the model by integrating data from different modalities. Voice and text, as the two most common modalities, are often used in combination in many application scenarios, such as understanding voice commands and natural language processing (NLP) tasks.

[0004] The main challenge of multimodal learning is how to efficiently fuse the features of different modalities and retain their unique information during the fusion process. Future development directions may include more intelligent modality fusion technology, more efficient sequence labeling algorithms, and training on larger datasets to improve the generalization ability of the model. Summary of the invention

[0005] The main purpose of the present invention is to provide a multimodal entity recognition model training method, device, equipment and storage medium, aiming to solve the technical problem that the existing technology is difficult to effectively integrate voice and text features, resulting in insufficient accuracy of multimodal entity recognition.

[0006] To achieve the above object, the present invention provides a multimodal entity recognition model training method, the multimodal entity recognition model training method comprising the following steps:

[0007] Acquire speech data and text data, obtain speech features by performing frame processing and Fourier transform on the speech data, and encode the speech features by a multi-layer Transformer encoder to obtain a speech feature vector;

[0008] Processing the text data by word embedding and position embedding to obtain text embedding features, and processing the text embedding features by a Transformer decoder to obtain a text feature vector;

[0009] The speech feature vector is processed by the CTC layer to obtain a speech entity vector, and the text feature vector is processed by the CRF layer to obtain a text entity vector. The text entity vector and the speech entity vector are used as input, and an initial model is trained by a comparative learning method to obtain a target multimodal entity recognition model.

[0010] Optionally, an initial model is trained by using text entity vectors, speech entity vectors, and a contrastive learning method to obtain a target multimodal entity recognition model, including:

[0011] The text entity vector and the speech entity vector obey Gaussian distribution, and the difference between two tokens belonging to the same entity label in the text entity vector and the speech entity vector is calculated;

[0012] Calculate the difference between the feature vectors from the first token to the second token and from the second token to the first token respectively, and take the average of the difference values ​​in the two directions to measure the symmetric difference;

[0013] In the contrastive learning process, the difference between token pairs with the same label is minimized, the difference between token pairs with different labels is maximized, and the total loss function is obtained by combining CRF loss and CTC loss.

[0014] The parameters of the initial model are optimized through the back-propagation algorithm until the total loss function converges, completing the training of the initial model and obtaining the target multimodal entity recognition model.

[0015] Optionally, before calculating the difference between two tokens belonging to the same entity tag in the text entity vector and the speech entity vector, the method further includes:

[0016] The text entity vector and the speech entity vector are standardized, and the dimensions of the processed text entity vector and the speech entity vector are reduced to a target dimension range through a principal component analysis algorithm to obtain low-dimensional text entity vectors and speech entity vectors, which are used to calculate the symmetric difference between two tokens of the same entity label.

[0017] Optionally, the parameters of the initial model are optimized by a back-propagation algorithm, including:

[0018] The gradient of the total loss function relative to each model parameter is calculated through the back-propagation algorithm. The learning rate of each model parameter is dynamically adjusted by applying the adaptive learning rate algorithm according to the gradient of each model parameter. Each model parameter is updated and optimized using the learning rate adjusted by the adaptive learning rate.

[0019] Optionally, after obtaining the target multimodal entity recognition model, the method further includes:

[0020] The data to be predicted is obtained, the data to be predicted is input into the target multimodal entity recognition model, and the target entity prediction result is output through the CRF layer of the target multimodal entity recognition model.

[0021] Optionally, encoding the speech feature through a multi-layer Transformer encoder to obtain a speech feature vector includes:

[0022] The multi-layer Transformer encoder is a 6-layer Transformer encoder, each layer of the Transformer encoder includes a multi-head attention mechanism, a first residual connection and normalization, a feedforward neural network, a second residual connection and normalization, and the speech features are encoded by the 6-layer Transformer encoder to obtain the speech feature vector.

[0023] Optionally, the text data is processed by word embedding and position embedding to obtain text embedding features, including:

[0024] Convert each word in the text data into a vector representation of a fixed dimension, i.e., a word embedding vector;

[0025] The position code of each word is generated by sine and cosine functions, and the position code is added to each word embedding vector to obtain the text embedding feature.

[0026] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a multimodal entity recognition model training device, the multimodal entity recognition model training device comprising:

[0027] A speech processing and encoding module is used to obtain speech data and text data, obtain speech features by performing frame processing and Fourier transform on the speech data, and encode the speech features by a multi-layer Transformer encoder to obtain a speech feature vector;

[0028] A text processing and decoding module, used to process the text data through word embedding and position embedding to obtain text embedding features, and process the text embedding features through a Transformer decoder to obtain a text feature vector;

[0029] The feature fusion and model training module is used to process the speech feature vector through the CTC layer to obtain a speech entity vector, process the text feature vector through the CRF layer to obtain a text entity vector, take the text entity vector and the speech entity vector as input, train the initial model through a comparative learning method, and obtain a target multimodal entity recognition model.

[0030] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a multimodal entity recognition model training device, which includes a memory, a processor, and a multimodal entity recognition model training program stored in the memory and executable on the processor, and the multimodal entity recognition model training program, when executed by the processor, implements the steps of the multimodal entity recognition model training method as described above.

[0031] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer storage medium, on which a multimodal entity recognition model training program is stored, and when the multimodal entity recognition model training program is executed by a processor, the steps of the multimodal entity recognition model training method as described above are implemented.

[0032] The present invention relates to the field of artificial intelligence, and discloses a multimodal entity recognition model training method, which obtains voice data and text data, and performs feature processing on the voice data and text data respectively. The voice data is subjected to frame processing and Fourier transform to obtain voice features, and is encoded by an encoder to generate a voice feature vector; the text data is subjected to word embedding and position embedding to generate text embedding features, and then processed by a decoder to generate a text feature vector. The voice feature vector is processed by a CTC layer to generate a voice entity vector, and the text feature vector is processed by a CRF layer to generate a text entity vector. By training the text entity vector, the voice entity vector and the contrast learning method, the initial model is optimized to obtain a target multimodal entity recognition model. The present invention processes the voice data and the text data simultaneously, and the multimodal entity recognition model can capture the complementary information between different modes, and improve the model's recognition ability of entities. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of the structure of the device hardware operating environment involved in the embodiment of the multimodal entity recognition model training device of the present invention;

[0034] Figure 2 A schematic diagram of a flow chart of a first embodiment of a multimodal entity recognition model training method of the present invention;

[0035] Figure 3 Schematic diagram of functional modules of a preferred embodiment of the multimodal entity recognition model training device of the present invention.

[0036] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0037] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0038] It should be noted that multimodal entity recognition is an important direction in the field of artificial intelligence, which aims to identify and understand entities in data by combining different types of data sources (such as voice and text). This method can be applied to various scenarios, such as virtual assistants, automatic speech recognition systems, and multimodal information retrieval.

[0039] With the rapid development of deep learning, researchers have gradually realized that relying solely on data from a single modality often cannot fully understand complex real-world information. Therefore, multimodal learning has gradually emerged, aiming to enhance the perception ability of the model by integrating data from different modalities. Voice and text, as the two most common modalities, are often used in combination in many application scenarios, such as understanding voice commands and natural language processing (NLP) tasks.

[0040] The main challenge of multimodal learning is how to efficiently fuse the features of different modalities and retain their unique information during the fusion process. Future development directions may include more intelligent modality fusion technology, more efficient sequence labeling algorithms, and training on larger datasets to improve the generalization ability of the model.

[0041] Based on the above-mentioned defects, the present invention provides a multimodal entity recognition model training device, referring to Figure 1 , Figure 1 It is a structural schematic diagram of the device hardware operating environment involved in the embodiment scheme of the multimodal entity recognition model training device of the present invention.

[0042] like Figure 1 As shown, the multimodal entity recognition model training device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0043] Those skilled in the art will understand that Figure 1The hardware structure of the multimodal entity recognition model training device shown in does not constitute a limitation of the multimodal entity recognition model training device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0044] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a multimodal entity recognition model training program. The operating system is a program for managing and controlling the multimodal entity recognition model training device and software resources, and supports the operation of the network communication module, the user interface module, the multimodal entity recognition model training program, and other programs or software; the network communication module is used to manage and control the network interface 1004; the user interface module is used to manage and control the user interface 1003.

[0045] exist Figure 1 In the hardware structure of the multimodal entity recognition model training device shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; the processor 1001 can call the multimodal entity recognition model training program stored in the memory 1005 and perform the following operations:

[0046] Acquire speech data and text data, obtain speech features by performing frame processing and Fourier transform on the speech data, and encode the speech features by a multi-layer Transformer encoder to obtain a speech feature vector;

[0047] Processing the text data by word embedding and position embedding to obtain text embedding features, and processing the text embedding features by a Transformer decoder to obtain a text feature vector;

[0048] The speech feature vector is processed by the CTC layer to obtain a speech entity vector, and the text feature vector is processed by the CRF layer to obtain a text entity vector. The text entity vector and the speech entity vector are used as input, and an initial model is trained by a comparative learning method to obtain a target multimodal entity recognition model.

[0049] Furthermore, the initial model is trained by using text entity vectors, speech entity vectors and contrastive learning methods to obtain the target multimodal entity recognition model, including:

[0050] The text entity vector and the speech entity vector obey Gaussian distribution, and the difference between two tokens belonging to the same entity label in the text entity vector and the speech entity vector is calculated;

[0051] Calculate the difference between the feature vectors from the first token to the second token and from the second token to the first token respectively, and take the average of the difference values ​​in the two directions to measure the symmetric difference;

[0052] In the contrastive learning process, the difference between token pairs with the same label is minimized, the difference between token pairs with different labels is maximized, and the total loss function is obtained by combining CRF loss and CTC loss.

[0053] The parameters of the initial model are optimized through the back-propagation algorithm until the total loss function converges, completing the training of the initial model and obtaining the target multimodal entity recognition model.

[0054] Furthermore, before calculating the difference between two tokens belonging to the same entity tag in the text entity vector and the speech entity vector, the method further includes:

[0055] The text entity vector and the speech entity vector are standardized, and the dimensions of the processed text entity vector and the speech entity vector are reduced to a target dimension range through a principal component analysis algorithm to obtain low-dimensional text entity vectors and speech entity vectors, which are used to calculate the symmetric difference between two tokens of the same entity label.

[0056] Furthermore, the parameters of the initial model are optimized by back propagation algorithm, including:

[0057] The gradient of the total loss function relative to each model parameter is calculated through the back-propagation algorithm. The learning rate of each model parameter is dynamically adjusted by applying the adaptive learning rate algorithm according to the gradient of each model parameter. Each model parameter is updated and optimized using the learning rate adjusted by the adaptive learning rate.

[0058] Furthermore, after obtaining the target multimodal entity recognition model, it also includes:

[0059] The data to be predicted is obtained, the data to be predicted is input into the target multimodal entity recognition model, and the target entity prediction result is output through the CRF layer of the target multimodal entity recognition model.

[0060] Furthermore, the speech feature is encoded by a multi-layer Transformer encoder to obtain a speech feature vector, including:

[0061] The multi-layer Transformer encoder is a 6-layer Transformer encoder, each layer of the Transformer encoder includes a multi-head attention mechanism, a first residual connection and normalization, a feedforward neural network, a second residual connection and normalization, and the speech features are encoded by the 6-layer Transformer encoder to obtain the speech feature vector.

[0062] Furthermore, the text data is processed by word embedding and position embedding to obtain text embedding features, including:

[0063] Convert each word in the text data into a vector representation of a fixed dimension, i.e., a word embedding vector;

[0064] The position code of each word is generated by sine and cosine functions, and the position code is added to each word embedding vector to obtain the text embedding feature.

[0065] The specific implementation of the multimodal entity recognition model training device of the present invention is basically the same as the various embodiments of the multimodal entity recognition model training method described below, and will not be repeated here.

[0066] The present invention also provides a multimodal entity recognition model training method based on the multimodal entity recognition model training device.

[0067] Reference Figure 2 , Figure 2 Schematic diagram of the process of the first embodiment of the multimodal entity recognition model training method of the present invention.

[0068] An embodiment of the present invention provides an embodiment of a multimodal entity recognition model training method. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown here.

[0069] In various embodiments of the multimodal entity recognition model training method, the execution entity is a multimodal entity recognition model training device.

[0070] The multimodal entity recognition model training method is applied to a multimodal entity recognition model training device, and the multimodal entity recognition model training method includes:

[0071] Step S10, acquiring voice data and text data, performing frame processing and Fourier transform on the voice data to obtain voice features, encoding the voice features through a multi-layer Transformer encoder to obtain a voice feature vector;

[0072] Multimodal Entity Recognition is an important direction in the field of artificial intelligence, which aims to identify and understand entities in data by combining different types of data sources (such as speech and text). This method can be applied to various scenarios, such as virtual assistants, automatic speech recognition systems, and multimodal information retrieval.

[0073] With the rapid development of deep learning, researchers have gradually realized that relying solely on data from a single modality often cannot fully understand complex real-world information. Therefore, multimodal learning has gradually emerged, aiming to enhance the perception ability of the model by integrating data from different modalities. Voice and text, as the two most common modalities, are often used in combination in many application scenarios, such as understanding voice commands and natural language processing (NLP) tasks.

[0074] The main challenge of multimodal learning is how to efficiently fuse the features of different modalities and retain their unique information during the fusion process. Future development directions may include more intelligent modality fusion technology, more efficient sequence labeling algorithms, and training on larger datasets to improve the generalization ability of the model.

[0075] In this embodiment, the voice data can be obtained through a microphone or other acquisition devices, or through file transfer or API call. The text data can be obtained through file reading, database query or real-time input.

[0076] Before processing, speech data is first divided into multiple short time frames (usually 20 milliseconds), each of which represents a short continuous speech signal. Subsequently, Fourier transform (usually short-time Fourier transform STFT) is applied to each frame to convert the time domain signal into a frequency domain signal to extract the spectral features of the speech signal. The spectral features contain the energy distribution of the speech signal at different frequencies and are an important basis for subsequent speech feature encoding.

[0077] The processed speech features are fed into a multi-layer Transformer encoder. The encoder processes these features through a self-attention mechanism, capturing the temporal dependencies between different frames and generating a high-dimensional speech feature vector. The multi-layer structure enables the encoder to gradually extract complex patterns in the speech data, thereby forming a more advanced feature representation.

[0078] Specific implementations may be:

[0079] Voice data acquisition: Real-time voice data can be collected through the microphone, or through file transfer or API calls. Voice data collected by the microphone will be transmitted to the system in real time, which is suitable for real-time voice processing scenarios. Voice data can be received in real time through network transmission, or obtained from third-party services through APIs, which is suitable for applications that integrate voice recognition, speech synthesis and other functions.

[0080] Text data acquisition: Text data can be extracted from local or remote servers by reading files, or obtained from databases or third-party services through APIs. Real-time text data can be captured directly from user input or other applications. For example, extracting stored text files from a database, or obtaining text information entered by users from a real-time chat system.

[0081] Speech data preprocessing: The acquired speech data is divided into several 20 millisecond frames. These frames can preserve the short-term characteristics of the speech signal.

[0082] Apply Short Time Fourier Transform (STFT) to each frame to convert it from time domain signal to frequency domain signal and extract the spectral features of speech. The spectral features generated by the STFT operation will serve as the input of the subsequent Transformer encoder.

[0083] Transformer encoder processing: The spectral features are input into a network consisting of 6 layers of Transformer encoders. Each layer of the encoder processes the input features through a multi-head self-attention mechanism to capture the relationships and dependencies between different time frames.

[0084] The feedforward neural network in each layer of the encoder further performs nonlinear transformation on the processed features, making the feature representation more complex and effective. After multiple layers of processing, the generated speech feature vector can capture the high-level information in the speech data.

[0085] Example description:

[0086] Assume that the system receives a 10-second speech data with a sampling rate of 16kHz. The speech is first divided into 500 frames, each of which is 20 milliseconds. For each frame, a short-time Fourier transform is applied to generate frequency domain features. The obtained spectral features contain the energy distribution of the speech signal at multiple frequencies.

[0087] These spectral features are input into a 6-layer Transformer encoder. Each layer of the Transformer encoder uses an 8-head self-attention mechanism and a feedforward neural network to process and extract features. After being processed by the encoder, a sequence of speech feature vectors with a length of 500 is generated, and the dimension of each vector is 512. These feature vectors are used together with the text feature vectors for subsequent multimodal entity recognition model training and reasoning.

[0088] Through the above steps, the temporal dependency and spectral characteristics of speech signals can be fully captured, and high-dimensional feature vectors can be generated to provide richer and more accurate speech representations for subsequent recognition tasks, significantly improving the accuracy of multimodal entity recognition and the generalization ability of the model.

[0089] Step S20, the text data is processed by word embedding and position embedding to obtain text embedding features, and the text embedding features are processed by a Transformer decoder to obtain a text feature vector;

[0090] In this embodiment, the text data is converted into a vector form that the model can understand. Word embedding converts each word in the text into a vector of fixed dimension, which captures the semantic relationship between words. Positional encoding can be added to the word embedding vector by the position encoding generated by sine and cosine functions to retain the order information of words in the sentence, ensuring that the model can understand the order of sequence data.

[0091] The processed text embedding features are input into the Transformer decoder, which further processes these features through a multi-head self-attention mechanism and a feedforward neural network. The decoder can combine contextual information to generate more complex text feature representations.

[0092] Specific implementations may be:

[0093] Text data acquisition: Text data can be acquired from files, databases or real-time input. Assume that the text data contains several sentences, each sentence consists of several words.

[0094] Word embedding and position embedding:

[0095] Word embedding: Convert each word into a fixed-dimensional word embedding vector that captures the semantic relationship between words. For example, if the word embedding dimension is 512, each word will be represented as a 512-dimensional vector.

[0096] Position embedding: Use sine and cosine functions to generate position information with the same dimension as the word embedding vector, and add these position codes to the corresponding word embedding vector. The text embedding features processed in this way not only contain the semantic information of the words, but also retain their position information in the sentence.

[0097] Transformer decoder processing: The generated text embedding features are input into the Transformer decoder. The multi-head self-attention mechanism of the decoder first processes these features to capture the dependencies and contextual information between words. Then, after further processing by the feedforward neural network, text feature vectors are generated. These vectors can better represent the semantics of the text and are used for subsequent recognition or generation tasks.

[0098] In addition, word embedding can also be:

[0099] Static word embeddings:

[0100] Word2Vec: Word embeddings trained from large-scale text data through unsupervised learning. The vector representation of each word is fixed in the entire corpus.

[0101] GloVe: Similar to Word2Vec, but generates word embeddings through matrix decomposition and statistical information (such as co-occurrence matrix), which can capture global semantic information.

[0102] Dynamic Word Embeddings:

[0103] ELMo (Embeddings from Language Models): Generates word embeddings from the context, taking into account the polysemy of words in different contexts.

[0104] BERT (Bidirectional Encoder Representations from Transformers): Utilizes the bidirectional Transformer architecture to generate context-aware word embeddings that can capture the meaning of words in different contexts.

[0105] Position embedding can also be:

[0106] Learnable position embedding: Position encoding can be done without fixed sine and cosine functions. Instead, position information is considered as part of model training and learned through the training process. Each position corresponds to a learnable vector, which is used in conjunction with word embedding.

[0107] Relative position encoding: Unlike absolute position encoding, relative position encoding considers the relative distance between words rather than their absolute position in the sequence. This approach may be more effective when processing longer sequences.

[0108] Hybrid position embedding: A hybrid method that combines absolute position and relative position encoding, which can not only retain the absolute position information in the sequence, but also consider the relative position relationship between words, improving the model's ability to understand position dependence.

[0109] Using different word embedding and position embedding methods can make the model better adapt to the needs of specific tasks. Static word embedding is suitable for simple semantic tasks, while dynamic word embedding can better capture contextual information. For position embedding, using learnable or relative position encoding can provide the model with more flexible and accurate position information.

[0110] Example description:

[0111] Suppose there is a sentence "AI is changing the world". First, each word "AI", "is", "change", and "world" is converted into a 512-dimensional word embedding vector. Then, through the position embedding mechanism, position encoding is added to each word embedding vector.

[0112] These text embedding features are input into a network consisting of a 6-layer Transformer decoder. The decoder captures the dependencies between words through a multi-head self-attention mechanism and processes the features in combination with contextual information. After being processed by the decoder, the generated text feature vector not only contains the semantic information of each word, but also comprehensively considers the context of the entire sentence.

[0113] By combining word embedding with position embedding, the semantic and sequence information in the text data is retained, ensuring that the Transformer decoder can fully understand its context when processing text, significantly improving the performance and recognition accuracy of the model in text processing tasks.

[0114] Step S30, processing the speech feature vector through the CTC layer to obtain a speech entity vector, processing the text feature vector through the CRF layer to obtain a text entity vector, taking the text entity vector and the speech entity vector as input, training an initial model through a comparative learning method, and obtaining a target multimodal entity recognition model.

[0115] In this embodiment, the CTC (Connectionist Temporal Classification) layer is used to process the speech feature vector to generate the corresponding speech entity vector. The CTC layer is mainly used to process speech sequence data, and can generate corresponding output sequences for variable-length input sequences and solve the problem of inconsistent lengths between speech features and entity labels.

[0116] The CRF (Conditional Random Fields) layer is used to process text feature vectors and generate text entity vectors. The CRF layer is good at handling sequence labeling problems and generates the optimal entity label sequence by considering contextual information.

[0117] After the text entity vector and speech entity vector are generated by the CRF layer and CTC layer respectively, the initial model is trained by combining the contrastive learning method. Contrastive learning optimizes the feature representation ability of the model by minimizing the distance between similar samples and maximizing the distance between heterogeneous samples.

[0118] After training, the final target model can better identify and classify entities in multimodal data, improving the accuracy of the model in multimodal entity recognition tasks.

[0119] Specific implementations may be:

[0120] Speech feature processing: The speech feature vector generated by the Transformer encoder is input into the CTC layer. The CTC layer processes these features and generates the corresponding speech entity vector. The CTC layer can process variable-length input sequences and generate output sequences aligned with entity labels, which is suitable for tasks such as speech recognition.

[0121] Text feature processing: The text feature vector generated by the Transformer decoder is input to the CRF layer. The CRF layer processes these features and generates text entity vectors. The CRF layer generates sequence labeling results by considering context information, which is suitable for text entity recognition tasks.

[0122] Model training: Combine the generated speech entity vectors and text entity vectors and input them into the initial model. Use contrastive learning to train the model. Contrastive learning optimizes model parameters by shortening the distance between similar entity vectors and increasing the distance between heterogeneous entity vectors. During the training process, the model parameters are continuously adjusted through the back propagation algorithm, and finally an optimized multimodal entity recognition model is obtained.

[0123] Example description:

[0124] Assume that the system processes a piece of speech data and generates a corresponding speech feature vector, and a piece of text data generates a corresponding text feature vector. The speech feature vector is processed by the CTC layer to generate a speech entity vector, and the text feature vector is processed by the CRF layer to generate a text entity vector.

[0125] These entity vectors are then fed into the initial model and trained using a contrastive learning approach. During the training process, the model is gradually optimized, ultimately generating a target multimodal entity recognition model that can accurately identify and classify entities in multimodal data.

[0126] By applying the CTC layer and CRF layer to the speech and text feature vectors respectively, the problem of sequence labeling in multimodal data is effectively solved. Combined with the contrastive learning method, the feature representation ability of the model is optimized, which significantly improves the accuracy of multimodal entity recognition and the generalization ability of the model. This method ensures that the model can generate accurate entity recognition results when processing multimodal data such as speech and text.

[0127] The present invention relates to the field of artificial intelligence, and discloses a multimodal entity recognition model training method, which obtains voice data and text data, and performs feature processing on the voice data and text data respectively. The voice data is subjected to frame processing and Fourier transform to obtain voice features, and is encoded by an encoder to generate a voice feature vector; the text data is subjected to word embedding and position embedding to generate text embedding features, and then processed by a decoder to generate a text feature vector. The voice feature vector is processed by a CTC layer to generate a voice entity vector, and the text feature vector is processed by a CRF layer to generate a text entity vector. By training the text entity vector, the voice entity vector and the contrast learning method, the initial model is optimized to obtain a target multimodal entity recognition model. The present invention processes the voice data and the text data simultaneously, and the multimodal entity recognition model can capture the complementary information between different modes, and improve the model's recognition ability of entities.

[0128] Furthermore, a second embodiment of the multimodal entity recognition model training method of the present invention is proposed. In the above step S30, the initial model is trained by using text entity vectors, speech entity vectors and a contrastive learning method to obtain a target multimodal entity recognition model, including:

[0129] Step a1, the text entity vector and the speech entity vector obey Gaussian distribution, and the difference between two tokens belonging to the same entity tag in the text entity vector and the speech entity vector is calculated;

[0130] Step a2, respectively calculate the difference between the feature vectors from the first token to the second token and from the second token to the first token, and take the average of the difference values ​​in the two directions to measure the symmetric difference;

[0131] Step a3: In the contrastive learning process, minimize the difference between token pairs with the same label, maximize the difference between token pairs with different labels, and combine CRF loss and CTC loss to obtain the total loss function;

[0132] Step a4, optimize the parameters of the initial model through the back propagation algorithm until the total loss function converges, complete the training of the initial model, and obtain the target multimodal entity recognition model.

[0133] In this embodiment, it is assumed that the generated text and speech entity vectors obey Gaussian distribution. The feature vector of each entity tag can be represented as Gaussian distribution, and its mean and covariance matrix are used to describe the distribution characteristics of the vector.

[0134] For two tokens belonging to the same entity label, the difference between their corresponding Gaussian distributions can be calculated, such as the symmetric KL divergence. KL divergence measures the difference between two probability distributions, and the symmetric processing ensures the bidirectionality of the distance metric.

[0135] By minimizing the divergence of token pairs with the same label, maximizing the divergence of token pairs with different labels, combining CRF loss and CTC loss, and finally optimizing the parameters of the initial model through the back propagation algorithm until the total loss function converges, the final target multimodal entity recognition model is obtained.

[0136] Specific implementations may be:

[0137] Gaussian distribution assumption: Assume that the text entity vector and speech entity vector obey the Gaussian distribution N(μ,Σ), where μ represents the mean and Σ represents the covariance matrix. The feature vector of each entity label obeys this Gaussian distribution and is used to describe the feature space of the label.

[0138] KL divergence calculation: For two tokens belonging to the same entity label, the KL divergence KL (N p ||N q ) and KL(N q ||N p ), and the symmetric KL divergence is obtained by the formula:

[0139]

[0140] Among them, N p Indicates the first tokenX belonging to the same entity tag p The eigenvector of the Gaussian distribution, N q Indicates the second tokenX belonging to the same entity tag q Gaussian distribution of the eigenvector, KL(N p ||N q ) represents the distribution N p To N q The Kullback-Leibler divergence between q ||N p ) represents the distribution N q To N p The Kullback-Leibler divergence between them, d(p,q) is the symmetric KL divergence.

[0141] Contrastive learning: During the training process, the divergence between token pairs with the same label is minimized to make the feature vectors of the same label more similar, while the divergence between token pairs with different labels is maximized to increase the distance between different entities.

[0142] Loss function optimization: Combine CRF loss and CTC loss to get the total loss function. Optimize the model parameters through the back propagation algorithm, and continuously adjust the parameters until the total loss function converges to complete the model training.

[0143] Obtain the target model: After the above steps, the trained target multimodal entity recognition model has the ability to perform entity recognition on voice and text data, and can accurately distinguish different entities and output corresponding labels.

[0144] Example

[0145] Assume that the system processes a piece of speech and the corresponding text data. After the previous processing steps, the speech entity vector and the text entity vector are generated respectively. Assume that these two vectors follow the Gaussian distribution N(μ p ,Σ p ) and N(μ q ,Σ q ).

[0146] For two tokens of the same entity label, the KL divergence between their corresponding Gaussian distributions is calculated, and the divergence value d(p,q) is obtained through symmetric processing. During the training process, the divergence between these pairs of tokens with the same label is minimized, while the divergence between pairs of tokens with different labels is maximized.

[0147] By combining CRF loss and CTC loss, the total loss function of the model is continuously optimized. The parameters of the model are adjusted using the back propagation algorithm until the total loss function converges, and finally an optimized multimodal entity recognition model is obtained.

[0148] This embodiment optimizes the model by treating text entity vectors and speech entity vectors as Gaussian distributions, using symmetric KL divergence to measure the differences between similar and heterogeneous entity vectors, and combining contrastive learning methods to train the model. By minimizing the differences between similar entity vectors and maximizing the differences between heterogeneous entity vectors, the model can accurately distinguish different entity labels, significantly improving the accuracy of multimodal entity recognition and the generalization ability of the model.

[0149] Furthermore, a third embodiment of the multimodal entity recognition model training method of the present invention is proposed. In the above step a1, before calculating the difference between two tokens belonging to the same entity label in the text entity vector and the speech entity vector, the method further includes:

[0150] Step b1, standardize the text entity vector and the speech entity vector, reduce the dimensions of the processed text entity vector and the speech entity vector to the target dimension range through the principal component analysis algorithm, and obtain low-dimensional text entity vector and speech entity vector for calculating the symmetric difference between two tokens of the same entity label.

[0151] In this embodiment, before calculating the symmetric KL divergence between the text entity vector and the speech entity vector, these vectors are first normalized. The purpose of normalization is to eliminate the dimensional differences between different feature dimensions so that each dimension of the feature vector has the same scale, thereby improving the accuracy of subsequent calculations.

[0152] After normalization, the principal component analysis (PCA) algorithm is used to reduce the dimensionality of the text entity vectors and the speech entity vectors, compressing these high-dimensional vectors into a lower-dimensional space. PCA reduces the computational complexity by retaining the most informative principal components in the data while still retaining the main information of the original features.

[0153] After dimensionality reduction, the low-dimensional text entity vector and speech entity vector are used to calculate the symmetric KL divergence between two tokens belonging to the same entity label. Dimensionality reduction helps improve computational efficiency and reduce noise and redundancy caused by high-dimensional data.

[0154] Specific implementations may be:

[0155] Standardization: The generated text entity vectors and speech entity vectors are standardized. Standardization is to adjust the dimensions of each vector to the same scale so that the values ​​of all dimensions are in the same range, such as 0 to 1 or -1 to 1. This step eliminates the differences between different feature dimensions and ensures that the subsequent dimensionality reduction and calculation process are more stable.

[0156] Dimensionality reduction by principal component analysis: After standardization, the principal component analysis (PCA) algorithm is used to reduce the dimensionality of text entity vectors and speech entity vectors. PCA compresses high-dimensional vectors into a lower-dimensional space by identifying the principal components in the data. For example, if the dimension of the original vector is 512, it can be reduced to 50 or 100 principal components through PCA. The reduced-dimensional vector still retains most of the information of the original data, but reduces the dimensions required for calculation, thereby improving the efficiency of subsequent calculations.

[0157] KL divergence calculation: Use the reduced low-dimensional text entity vector and speech entity vector to calculate the symmetric KL divergence between two tokens belonging to the same entity label. The low-dimensional representation not only reduces the computational complexity, but also reduces the impact of noise on the divergence calculation, thereby improving the accuracy of the divergence calculation.

[0158] Example description:

[0159] Assume that there are two high-dimensional entity vectors, one for text and one for speech, with a dimension of 512. First, these vectors are normalized so that the values ​​of each dimension are in the same range. Then, these vectors are reduced to a 50-dimensional space through principal component analysis (PCA).

[0160] In the low-dimensional space after dimensionality reduction, the symmetric KL divergence between two tokens belonging to the same entity label is calculated. Assume that the vectors corresponding to the two tokens are more concentrated and easier to compare after being standardized and reduced in dimensionality. By calculating the symmetric KL divergence, the similarity of the feature vectors of the two tokens can be measured more accurately.

[0161] This embodiment effectively improves the efficiency and accuracy of calculating the symmetric KL divergence by standardizing and reducing the dimensionality of the text entity vector and the speech entity vector. Standardization ensures that the feature vectors are compared on the same scale, and dimensionality reduction further optimizes the divergence calculation process by reducing the noise and redundant information in the dimensional space. This processing method not only reduces the computational complexity, but also retains the key information of the feature vector, which is helpful for the training and optimization of the final model, thereby improving the performance of multimodal entity recognition.

[0162] Furthermore, a fourth embodiment of the multimodal entity recognition model training method of the present invention is proposed. In the above step a2, the parameters of the initial model are optimized by a back propagation algorithm, including:

[0163] Step c1, calculate the gradient of the total loss function relative to each model parameter through the back propagation algorithm, dynamically adjust the learning rate of each model parameter according to the gradient of each model parameter by applying the adaptive learning rate algorithm, and update and optimize each model parameter using the learning rate adjusted by the adaptive learning rate.

[0164] In this embodiment, the model can adjust its parameters based on the calculated loss function to minimize the prediction error through the back propagation algorithm. The back propagation algorithm gradually optimizes the model by calculating the gradient of the loss function with respect to each model parameter.

[0165] The first step of the back propagation algorithm is to calculate the gradient of the loss function for each parameter of the model. This gradient reflects how the loss function changes with the changes in the model parameters and is an important basis for parameter updating.

[0166] Adaptive learning rate algorithms (such as AdaGrad, RMSProp, Adam, etc.) can dynamically adjust the learning rate according to the change of gradient, so that each parameter can have a suitable learning rate at different training stages. This helps to accelerate the convergence of the model while avoiding instability caused by too large update step size.

[0167] Based on the calculated gradient and the adjusted learning rate, each model parameter is updated. This step gradually adjusts the model parameters to a more optimal state, thereby reducing the value of the loss function.

[0168] Specific implementations may be:

[0169] Application of back propagation algorithm: After each forward propagation, the model forms a loss function by calculating the error between the predicted output and the actual label. The back propagation algorithm calculates the gradient of the loss function relative to each parameter of the model through the chain rule.

[0170] Calculate the gradient: For each parameter in the model, calculate its corresponding gradient, that is, the partial derivative of the loss function with respect to the parameter. The gradient indicates the direction of change of the loss function in the parameter space.

[0171] Adaptive learning rate algorithm: Adaptive learning rate algorithm (such as Adam) is used to dynamically adjust the learning rate. The adaptive learning rate algorithm adjusts the learning rate of each parameter based on the gradient information of the previous iterations. For example, the Adam algorithm combines the advantages of momentum and adaptive learning rate to accelerate convergence and reduce oscillation.

[0172] Parameter update: Update each model parameter based on the calculated gradient and adaptive learning rate. The new parameter value is equal to the old parameter value minus the product of the learning rate and the gradient. This step enables the model to continuously optimize in the direction of minimizing the loss function.

[0173] Example description:

[0174] The loss function of the model is a combination of contrastive learning loss, CRF loss, and CTC loss. The following is a description of each part and their role in back propagation:

[0175] Contrastive learning loss: Contrastive learning loss is used to adjust the similarity between text entity vectors and speech entity vectors of the same entity label. For two tokens with the same label, their feature vectors should be closer; for two tokens with different labels, their feature vectors should be farther apart. The loss function is in the form of symmetric KL divergence, which calculates the distance between two tokens belonging to the same entity label and minimizes this loss through the back-propagation algorithm.

[0176] CRF loss: The loss of the CRF layer is used for the sequence labeling task of text feature vectors. Through CRF loss, the model can learn a better sequence labeling scheme, combined with context information, to generate the optimal entity label sequence. During back propagation, the gradient of CRF loss is used to update the parameters involved in text processing in the model.

[0177] CTC loss: CTC loss is used for the task of sequence labeling of speech feature vectors. The CTC layer solves the problem of inconsistent input and output lengths by adjusting the alignment of the sequence to generate the optimal speech entity label sequence. During backpropagation, the gradient of the CTC loss is used to update the parameters involved in speech processing in the model.

[0178] Backpropagation and optimization:

[0179] Calculate gradients: In each training iteration, the contrastive learning loss, CRF loss, and CTC loss are first calculated. The contrastive learning loss is obtained by calculating the symmetric KL divergence of the text and speech entity vectors, and the CRF and CTC losses are obtained by the difference between their corresponding label sequences and the model output sequences. The gradients of these loss functions relative to the model parameters are calculated layer by layer through the back-propagation algorithm.

[0180] Adaptive learning rate and parameter update: Use an adaptive learning rate algorithm (such as the Adam algorithm) to dynamically adjust the learning rate of each parameter. Based on the gradient history of each parameter, the adaptive learning rate algorithm can adjust the learning rate so that the parameter can be updated with the most appropriate step size. The model parameters are updated through the adjusted learning rate, and the model is gradually optimized.

[0181] Convergence of loss function: During the training process, the loss function gradually converges with each round of iteration. Finally, through the comprehensive optimization of contrastive learning, CRF and CTC losses, the model can effectively distinguish different entity labels and achieve the expected recognition effect.

[0182] This embodiment combines contrastive learning loss, CRF loss and CTC loss to effectively improve the accuracy and generalization ability of the multimodal entity recognition model. The combination of the back propagation algorithm and the adaptive learning rate ensures the stable optimization of the model parameters during the training process, accelerates the convergence speed of the model, and reduces the oscillation phenomenon during the training process. Ultimately, the target model obtained can achieve higher recognition accuracy and better application effects when processing multimodal data of speech and text.

[0183] Furthermore, a fifth embodiment of the multimodal entity recognition model training method of the present invention is proposed. In the above step a2, after obtaining the target multimodal entity recognition model, the method further includes:

[0184] Step d1, obtaining the data to be predicted, inputting the data to be predicted into the target multimodal entity recognition model, and outputting the target entity prediction result through the CRF layer of the target multimodal entity recognition model.

[0185] In this embodiment, after the model training is completed and the target multimodal entity recognition model is obtained, the system needs to process new data. This data may be obtained in real time from an external source, or it may be read from a file or database. The data to be predicted includes multimodal data, such as voice and text, waiting for the model to perform entity recognition.

[0186] The acquired data to be predicted will be passed as input to the trained target multimodal entity recognition model. This process involves preprocessing the speech and text data and inputting the preprocessed data into the corresponding layers of the model.

[0187] During the prediction process of the model, the text data is processed by the CRF layer and the final entity labels are output. These labels represent the entity categories recognized by the model in the input data and are the final output of the entire multimodal entity recognition process.

[0188] Specific implementations may be:

[0189] Get the data to be predicted: The data to be predicted can be obtained through real-time recording, file reading or API. This data can contain voice commands, text commands or other forms of multimodal data. Assuming that the system receives a new voice and corresponding text information, this data will be input into the target multimodal entity recognition model.

[0190] Data input and processing: The acquired speech and text data are first preprocessed, such as framing and Fourier transform of speech data, word embedding and position embedding of text data. Then, the processed data is passed as input to the trained target model.

[0191] Output of the CRF layer: The final step of the model processing text features is to generate prediction results through the CRF layer. The CRF layer generates prediction results for the target entity by analyzing the text feature sequence and combining it with contextual information. These prediction results are output in the form of entity labels, which represent the model's recognition results for the input data.

[0192] Example description:

[0193] Assume that the system obtains a newly recorded speech and the corresponding text data. After preprocessing, these data are input into the target multimodal entity recognition model. The speech data is processed by the CTC layer of the model to generate speech entity vectors, and the text data is processed by the CRF layer of the model to generate text entity vectors.

[0194] The CRF layer finally outputs the entity prediction results of the text data. For example, for a sentence "John went to New York", the model may predict that "John" is a person name entity and "New York" is a place name entity. These results represent the entity types recognized by the model and can be directly applied to various downstream tasks, such as information extraction, question-answering systems, etc.

[0195] Through the above steps, this embodiment can achieve efficient and accurate multimodal entity recognition. This process is automated and highly accurate, and is suitable for various application scenarios that require extracting entity information from multimodal data, significantly improving the practicality and effectiveness of the model in practical applications.

[0196] Furthermore, a sixth embodiment of the multimodal entity recognition model training method of the present invention is proposed. In the above step S10, the speech feature is encoded by a multi-layer Transformer encoder to obtain a speech feature vector, including:

[0197] Step e1, the multi-layer Transformer encoder is a 6-layer Transformer encoder, each layer of the Transformer encoder includes a multi-head attention mechanism, a first residual connection and normalization, a feedforward neural network, a second residual connection and normalization, and the speech features are encoded by the 6-layer Transformer encoder to obtain the speech feature vector.

[0198] In this embodiment, a multi-layer Transformer encoder is used to encode the preprocessed speech features to obtain a speech feature vector. The Transformer encoder can capture long-distance dependencies and context information in the input speech data through its multi-layer structure.

[0199] In this embodiment, the encoder consists of 6 layers of Transformer encoders. Each layer of Transformer encoder contains the following main components:

[0200] Multi-head attention mechanism: used to calculate the attention distribution in parallel on multiple different "heads" to capture multiple dependencies in speech features.

[0201] First residual connection and normalization: After the multi-head attention mechanism, the input features and attention results are added and normalized through the residual connection and normalization steps to ensure model stability.

[0202] Feedforward neural network: Through a fully connected network, the features processed by the attention mechanism are further transformed nonlinearly to enhance the expressive power of the features.

[0203] Second residual connection and normalization: After the feedforward neural network, residual connection and normalization operations are performed again to further stabilize the model training.

[0204] After being processed by a 6-layer Transformer encoder, the final generated speech feature vector integrates various information in the speech data and can be used for subsequent model training or reasoning.

[0205] Specific implementations may be:

[0206] Hierarchical structure of the Transformer encoder: After preprocessing, the speech features are input into the 6-layer Transformer encoder. Each layer of the encoder contains a multi-head attention mechanism, a first residual connection and normalization, a feedforward neural network, and a second residual connection and normalization. The multi-head attention mechanism can process multiple attention distributions in parallel and capture the complex dependencies between different parts of the speech data. Through the multi-head mechanism, the model can pay attention to multiple aspects of the input speech features.

[0207] Residual connection and normalization of each layer: After the multi-head attention mechanism, the first residual connection and normalization operation is performed. This step adds the output of the attention mechanism to the input features, and ensures the stability of the model through normalization operations to prevent gradient explosion or disappearance. Then, through the feedforward neural network, the normalized features are further processed nonlinearly to enhance the richness of feature expression. The second residual connection and normalization operation is performed after the feedforward neural network to further improve the stability of the model and ensure that the output of each layer can be stably passed to the next layer.

[0208] Final output speech feature vector: After the speech features are processed by the 6-layer Transformer encoder, the final speech feature vector is generated. These vectors contain rich information of the original speech data, and the encoded representation is more compact and meaningful, which is suitable for multimodal entity recognition model training.

[0209] Example description:

[0210] Suppose the system is processing a Fourier transformed speech feature, which is input into a network consisting of a 6-layer Transformer encoder. Each layer of the encoder contains 8 multi-head attention mechanisms, which can simultaneously capture the dependencies between multiple different parts of the speech data.

[0211] After the multi-head attention mechanism of each layer, the first residual connection and normalization operation is performed, and then the data is processed nonlinearly through the feedforward neural network. Finally, the output features of this layer are obtained through the second residual connection and normalization operation. After 6 layers of such processing, the output speech feature vector can accurately represent the content and context information of the input speech.

[0212] This embodiment encodes speech features through a 6-layer Transformer encoder, which can effectively capture the complex dependencies and semantic information in speech data. The combination of the multi-head attention mechanism, multiple residual connections and normalization operations, and feedforward neural network in each layer of encoder ensures the expressiveness of feature vectors and the stability of model training.

[0213] Furthermore, a seventh embodiment of the multimodal entity recognition model training method of the present invention is proposed. In the above step S20, the text data is processed by word embedding and position embedding to obtain text embedding features, including:

[0214] Step f1, converting each word in the text data into a vector representation of a fixed dimension, i.e., a word embedding vector;

[0215] Step f2, generating the position code of each word through sine and cosine functions, adding the position code to each word embedding vector, and obtaining the text embedding feature.

[0216] In this embodiment, word embedding is a technique that converts each word in text data into a fixed-dimensional vector. This vector representation contains the semantic information of the word, enabling the machine learning model to understand and process natural language. Each word embedding vector is usually generated by a pre-trained word embedding model (such as Word2Vec, GloVe, or BERT), which captures the semantic relationship between words.

[0217] Positional encoding is used to capture the order information of words in a sentence. In the Transformer model, since the order information of words is not obtained through the recursive structure, it is necessary to explicitly provide the position information through positional embedding. Positional encoding is usually generated by sine and cosine functions, which take the position of the word in the sequence as input and generate the corresponding encoding vector. These encoding vectors are added to the word embedding vector to form a text embedding feature containing order information.

[0218] Specific implementations may be:

[0219] Word embedding processing: Each word in the text data is converted into a fixed-dimensional vector representation through the word embedding model. For example, if the dimension of the word embedding vector is 512, each word will be represented as a 512-dimensional vector. This vector representation contains the semantic information of the word, allowing the model to understand the relationship between words.

[0220] Position embedding processing: The position information encoding with the same dimension as the word embedding vector is generated through sine and cosine functions. Specifically, the sine function is used to generate the encoding of the even positions, and the cosine function is used to generate the encoding of the odd positions. The position information encoding generated in this way is added to the word embedding vector, so that each word embedding vector contains both the semantic information and the position information of the word.

[0221] Text embedding feature generation: The word embedding vector with added position information is used as the final text embedding feature. These features will be used in subsequent Transformer model processing to capture the semantic and structural information of the text.

[0222] Example description:

[0223] Suppose there is a sentence "Machine learning changes the world". Through the word embedding model, each word is converted into a 512-dimensional word embedding vector. For example, "machine learning" may be represented as a vector [0.5, -0.2, 0.3, ...]. Next, through the position encoding generated by the sine and cosine functions, assuming that the position of "machine learning" in the sentence is the first, its position encoding vector may be [0.1, 0.05, 0.2, ...]. Add these two vectors to obtain the text embedding feature [0.6, -0.15, 0.5, ...] containing the position information.

[0224] This embodiment combines word embedding with position embedding to effectively capture the semantic information and word order information in the text, so that the Transformer model can process and understand the order information of natural language without relying on recursive structure. This embedding method ensures that the model can accurately understand the semantic relationship and position dependency between words when processing text data, thereby improving the performance of the model in natural language processing tasks.

[0225] The invention also provides a multimodal entity recognition model training device.

[0226] Reference Figure 3 , Figure 3 Schematic diagram of functional modules of the first embodiment of the multimodal entity recognition model training device of the present invention. The multimodal entity recognition model training device comprises:

[0227] A speech processing and encoding module 10 is used to obtain speech data and text data, obtain speech features by performing frame processing and Fourier transform on the speech data, and encode the speech features by a multi-layer Transformer encoder 20 to obtain a speech feature vector;

[0228] A text processing and decoding module 30, used to process the text data through word embedding and position embedding to obtain text embedding features, and process the text embedding features through a Transformer decoder to obtain a text feature vector;

[0229] The feature fusion and model training module 40 is used to process the speech feature vector through the CTC layer to obtain a speech entity vector, process the text feature vector through the CRF layer to obtain a text entity vector, take the text entity vector and the speech entity vector as input, train the initial model through a comparative learning method, and obtain a target multimodal entity recognition model.

[0230] In addition, an embodiment of the present invention further provides a computer storage medium.

[0231] A multimodal entity recognition model training program is stored on the storage medium, and when the multimodal entity recognition model training program is executed by the processor, the steps of the multimodal entity recognition model training method described above are implemented.

[0232] The specific implementation of the storage medium of the present invention is basically the same as the various embodiments of the above-mentioned multimodal entity recognition model training method, and will not be repeated here.

[0233] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims. All equivalent structures or equivalent process changes made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are protected by the present invention.

[0234] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flow chart, and the combination of boxes in the block diagram and / or the flow chart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0235] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), a disk or an optical disk.

[0236] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0237] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0238] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.

Claims

1. A multimodal entity recognition model training method, characterized in that: The following steps are involved: Acquire speech data and text data, obtain speech features by performing frame processing and Fourier transform on the speech data, and encode the speech features by a multi-layer Transformer encoder to obtain a speech feature vector; Processing the text data by word embedding and position embedding to obtain text embedding features, and processing the text embedding features by a Transformer decoder to obtain a text feature vector; The speech feature vector is processed by the CTC layer to obtain a speech entity vector, and the text feature vector is processed by the CRF layer to obtain a text entity vector. The text entity vector and the speech entity vector are used as input, and an initial model is trained by a comparative learning method to obtain a target multimodal entity recognition model.

2. The multimodal entity recognition model training method according to claim 1, characterized in that: The initial model is trained by text entity vectors, voice entity vectors and contrastive learning methods to obtain the target multimodal entity recognition model, including: The text entity vector and the speech entity vector obey Gaussian distribution, and the difference between two tokens belonging to the same entity label in the text entity vector and the speech entity vector is calculated; Calculate the difference between the feature vectors from the first token to the second token and from the second token to the first token respectively, and take the average of the difference values ​​in the two directions to measure the symmetric difference; In the contrastive learning process, the difference between token pairs with the same label is minimized, the difference between token pairs with different labels is maximized, and the total loss function is obtained by combining CRF loss and CTC loss. The parameters of the initial model are optimized through the back-propagation algorithm until the total loss function converges, completing the training of the initial model and obtaining the target multimodal entity recognition model.

3. The multimodal entity recognition model training method according to claim 2, characterized in that: Before calculating the difference between two tokens belonging to the same entity tag in the text entity vector and the speech entity vector, the method further includes: The text entity vector and the speech entity vector are standardized, and the dimensions of the processed text entity vector and the speech entity vector are reduced to a target dimension range through a principal component analysis algorithm to obtain low-dimensional text entity vectors and speech entity vectors, which are used to calculate the symmetric difference between two tokens of the same entity label.

4. The multimodal entity recognition model training method as described in claim 2, characterized in that: The parameters of the initial model are optimized through the back-propagation algorithm, including: The gradient of the total loss function relative to each model parameter is calculated through the back-propagation algorithm. The learning rate of each model parameter is dynamically adjusted by applying the adaptive learning rate algorithm according to the gradient of each model parameter. Each model parameter is updated and optimized using the learning rate adjusted by the adaptive learning rate.

5. The multimodal entity recognition model training method according to claim 2, characterized in that: After obtaining the target multimodal entity recognition model, it also includes: The data to be predicted is obtained, the data to be predicted is input into the target multimodal entity recognition model, and the target entity prediction result is output through the CRF layer of the target multimodal entity recognition model.

6. The multimodal entity recognition model training method according to claim 1, characterized in that: The speech feature is encoded by a multi-layer Transformer encoder to obtain a speech feature vector, including: The multi-layer Transformer encoder is a 6-layer Transformer encoder, each layer of the Transformer encoder includes a multi-head attention mechanism, a first residual connection and normalization, a feedforward neural network, a second residual connection and normalization, and the speech features are encoded by the 6-layer Transformer encoder to obtain the speech feature vector.

7. The multimodal entity recognition model training method according to claim 1, characterized in that: The text data is processed by word embedding and position embedding to obtain text embedding features, including: Convert each word in the text data into a vector representation of a fixed dimension, i.e., a word embedding vector; The position code of each word is generated by sine and cosine functions, and the position code is added to each word embedding vector to obtain the text embedding feature.

8. A multimodal entity recognition model training device, characterized in that: The multimodal entity recognition model training device comprises: A speech processing and encoding module is used to obtain speech data and text data, obtain speech features by performing frame processing and Fourier transform on the speech data, and encode the speech features by a multi-layer Transformer encoder to obtain a speech feature vector; A text processing and decoding module, used to process the text data through word embedding and position embedding to obtain text embedding features, and process the text embedding features through a Transformer decoder to obtain a text feature vector; The feature fusion and model training module is used to process the speech feature vector through the CTC layer to obtain a speech entity vector, process the text feature vector through the CRF layer to obtain a text entity vector, take the text entity vector and the speech entity vector as input, train the initial model through a comparative learning method, and obtain a target multimodal entity recognition model.

9. A multimodal entity recognition model training device, characterized in that: The multimodal entity recognition model training device includes a memory, a processor, and a multimodal entity recognition model training program stored in the memory and executable on the processor. When the multimodal entity recognition model training program is executed by the processor, the steps of the multimodal entity recognition model training method as described in any one of claims 1 to 7 are implemented.

10. A computer storage medium, characterized in that: The storage medium stores a multimodal entity recognition model training program, and when the multimodal entity recognition model training program is executed by the processor, the steps of the multimodal entity recognition model training method as described in any one of claims 1 to 7 are implemented.

Images