Multi-modal entity relationship extraction method applied to audio data

Abstract

The invention discloses a multi-modal entity relationship extraction method applied to audio data. The method comprises the following steps: constructing an audio entity relationship data set; performing large model training through the audio entity relational data set to obtain an optimized large model, including: performing Whisper model conversion processing on the audio entity relational data set to obtain corresponding text data, optimizing the text data in combination with a connection module to obtain optimized text data, and obtaining the optimized large model based on the optimized text data. Performing entity relationship extraction and text prediction by using a GPT-3 model, and respectively generating probability distribution of an entity relationship triple and text prediction; and inputting to-be-processed audio data into the optimized large model to obtain a corresponding entity relationship extraction result. According to the method, efficient and accurate entity relationship extraction of the audio data is realized, the capability of obtaining valuable information from unstructured data is improved, and powerful support is provided for intelligent upgrading in the fields of intelligent interaction and the like.

Description

Technical Field

[0001] The present invention relates to the field of information extraction, and more specifically to a multimodal entity relationship extraction method applied to audio data. Background Art

[0002] With the rapid development of artificial intelligence technology, the demand for data in the field of natural language processing is growing. Especially in the context of the rapid accumulation of massive open domain and unstructured data, how to effectively obtain valuable information from it and reduce the interference of noise on the results has become a major challenge in information extraction tasks. As one of the core tasks of information extraction, entity relationship extraction aims to identify entities and their relationships from text or other forms of data. These relationship triplets are of foundational significance for downstream tasks such as knowledge graph construction, machine reading, text summarization, and question-answering systems.

[0003] The existing technology has several significant defects when dealing with entity relationship extraction tasks, especially in multimodal data processing. First, the traditional pipeline method relies on decomposing the entity relationship extraction task into two independent steps: entity recognition and relationship classification. This approach is prone to error propagation problems. Once a misjudgment occurs in the entity recognition stage, the subsequent relationship classification will inevitably be affected, and ultimately affect the overall extraction accuracy. In addition, this method fails to fully utilize the intrinsic relationship between entities and relationships, which causes information loss to a certain extent and limits its application effect in complex scenarios.

[0004] Secondly, although the joint extraction method attempts to solve some of the drawbacks of the pipeline method, such as avoiding the errors caused by step-by-step processing by directly outputting entity relationship triples from the source data, it mainly focuses on a single task and lacks adaptability to other tasks in the field of natural language processing. With the development of large models, these models have demonstrated stronger language understanding and processing capabilities and have achieved excellent results in a variety of NLP tasks. However, the existing joint extraction methods fail to fully utilize the advantages of large models, and thus appear insufficient in terms of versatility and performance improvement.

[0005] Finally, there is a relative lack of research on entity relationship extraction from audio data. Most methods still remain at the stage of converting audio into text and then processing it with a small language model, or designing an end-to-end small language model to directly process audio data, while ignoring the powerful potential of large models. This not only limits the effective use of potential information in audio data, but also hinders the advancement of multimodal data analysis technology. The ability to obtain valuable information from structured data.

[0006] Therefore, how to achieve more efficient and accurate entity relationship extraction for multimodal data such as audio and improve the ability to obtain valuable information from unstructured data is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0007] In view of this, the present invention provides a multimodal entity relationship extraction method applied to audio data, which aims to improve the accuracy and efficiency of entity relationship extraction in multimodal situations by constructing an audio entity relationship dataset, designing a large model for audio-to-text processing, combining an optimization module to reduce information loss, and using a large model for entity relationship extraction, thereby providing new ideas and technical support for the development of the field of information extraction.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A multimodal entity relationship extraction method applied to audio data, comprising:

[0010] S1, build audio entity relationship dataset;

[0011] S2. Performing large model training using the audio entity relationship dataset to obtain an optimized large model; the large model training includes:

[0012] S21, performing Whisper model conversion processing on the audio entity relationship data set to obtain corresponding text data;

[0013] S22, optimizing the text data in combination with the connection module to obtain optimized text data;

[0014] S23. Based on the optimized text data, the GPT-3 model is used to extract entity relationships and predict text, and the probability distribution of entity relationship triples and text predictions are generated respectively;

[0015] S3. Input the audio data to be processed into the optimized large model to obtain the corresponding entity relationship extraction results.

[0016] Furthermore, the S1 includes:

[0017] The Duie dataset, which includes the original text and the corresponding entity relationship triple list, is selected as the data source. In the Duie dataset, the data is stored in json format, including the text field and the spo_list field.

[0018] Use Python's json library to parse the json format file in the Duie dataset, extract the content in the text field as the text to be synthesized, and save the text to be synthesized as a txt format file; and extract the entity relationship information included in the spo_list field as label data;

[0019] Combined with the run function in the Elevenlabs model, the text to be synthesized is converted into audio data, and the audio entity relationship dataset is obtained corresponding to the label data.

[0020] Furthermore, in S21, the audio data to be processed is converted into a Whisper model, including:

[0021] The audio data in the audio entity relationship data set is converted into an audio feature vector through a feature extraction module, and the audio feature vector is position-encoded;

[0022] The position-encoded audio feature vector is processed through a multi-head attention module to capture long-distance dependencies.

[0023] Combined with the feedforward neural network, the audio feature vector processed by the multi-head attention module is nonlinearly transformed to obtain the enhanced audio feature representation;

[0024] Performing residual connection processing on the enhanced audio feature representation and the audio feature vector after position encoding to obtain a result after residual connection processing;

[0025] The result after residual connection processing is processed by multiple consecutive multi-head attention modules and feedforward neural network to obtain the audio data processing result.

[0026] Further, in S21, the audio data to be processed is converted into a Whisper model, and further includes:

[0027] The audio-text pairs in the audio entity relationship dataset are converted into text feature vectors through a feature extraction module, and the text feature vectors are positionally encoded;

[0028] The position-encoded text feature vector is processed through a multi-head attention module to capture long-distance dependencies.

[0029] Cross-processing the text feature vector processed by the multi-head attention module and the audio data processing result through the cross-attention module to obtain a cross-processing result;

[0030] Combined with the feedforward neural network, the cross-processing results are nonlinearly transformed to obtain enhanced audio feature representation;

[0031] The enhanced audio feature representation is processed by multiple consecutive multi-head attention modules, cross attention modules and feedforward neural networks to obtain text data.

[0032] Furthermore, the multi-head attention module processes the feature vector, including:

[0033] Perform layer normalization on the feature vector to obtain a layer normalized vector;

[0034] Input the layer-normalized vector into the multi-head attention structure to obtain the fused information; in the multi-head attention structure, each attention head performs attention calculation independently, and multiple attention heads are processed in parallel;

[0035] Perform residual connection processing on the fused information and embedded features, and perform layer normalization processing to obtain the result after layer normalization processing;

[0036] Input the result after layer normalization into the multi-layer perceptron to obtain the output of the multi-layer perceptron;

[0037] The output of the multi-layer perceptron is processed with the result after layer normalization through residual connection, and layer normalization is performed to obtain the output result of the multi-head attention module.

[0038] Furthermore, the cross-attention module performs cross-processing, including:

[0039] Process the audio data and text data separately through independent fully connected layers to generate query vectors, key vectors, and value vectors corresponding to the audio data and text data;

[0040] Calculate the dot product of the query vector corresponding to the text data and the key vector corresponding to the audio data and scale them to obtain the attention score;

[0041] Normalize the attention score and multiply it with the value vector corresponding to the audio data to obtain the first attention calculation result;

[0042] Calculate the dot product of the query vector corresponding to the audio data and the key vector corresponding to the text data and scale them to obtain the attention score;

[0043] Normalize the attention score and multiply it with the value vector corresponding to the text data to obtain the second attention calculation result;

[0044] Based on the first attention calculation result and the second attention calculation result, the output result of the cross attention module is obtained.

[0045] Furthermore, the S22 includes:

[0046] Construct a prompt word template that includes task description, output format requirements, and example entity-relationship triples;

[0047] Insert the text data converted by the Whisper model into the prompt word template to generate prompt words;

[0048] Add context information and example entity relationship triples to the prompt words and optimize the format;

[0049] Based on the prompt words with optimized format, combined with the few-sample prompt strategy, and clarifying the task instructions and output format requirements, the optimized text data is obtained.

[0050] Furthermore, the S23 includes:

[0051] The optimized text data is combined with position embedding and the data is normalized through the first normalization layer;

[0052] The multi-head shielded self-attention module is used to process the data normalized by the first normalization layer to capture the dependency between words at different positions in the text; and the first residual connection processing is performed with the optimized text data;

[0053] Data normalization is performed through the second normalization layer;

[0054] The data normalized by the second normalization layer is subjected to nonlinear transformation in combination with a feedforward neural network to obtain an enhanced feature representation; and the data processed by the first residual connection is subjected to a second residual connection processing;

[0055] The data processed by the second residual connection is processed by multiple consecutive first normalization layers, multi-head shielded self-attention modules, second normalization layers, and feedforward neural networks to obtain the corresponding hidden states;

[0056] Based on the hidden state, the probability distribution of entity relationship triples and text prediction is generated respectively through the output layer.

[0057] Furthermore, entity relationship triples are generated through the output layer, including:

[0058] Perform named entity recognition through the softmax layer to mark all entities in the text and construct potential entity pairs;

[0059] For each identified entity pair, extract the corresponding hidden state vector, and obtain the comprehensive feature vector of each entity pair through the pooling layer;

[0060] Combining the fully connected layer and the softmax layer, the comprehensive feature vector of each entity pair is mapped to the predefined relationship category space to generate entity relationship triples.

[0061] Furthermore, the probability distribution of text prediction is generated through the output layer, including:

[0062] Map the hidden state to the dimension of the vocabulary size through a fully connected layer to obtain a high-dimensional vector;

[0063] Using a softmax layer to convert the high-dimensional vector into a probability distribution;

[0064] Based on the probability distribution, a sampling algorithm is combined to generate text continuation.

[0065] It can be seen from the above technical solution that compared with the prior art, the technical solution of the present invention has the following advantages:

[0066] Beneficial effects:

[0067] 1. In view of the lack of specialized datasets in the field of multimodal entity relationship extraction, a new audio entity relationship dataset was successfully created by converting the existing text entity relationship dataset into audio data using speech synthesis technology. This not only fills the gap in research resources, but also provides a solid foundation for subsequent exploration of related fields.

[0068] 2. The Whisper model is used to convert audio data, and the carefully designed connection module is combined to optimize text data, ensuring the quality and efficiency of audio-to-text conversion. The Whisper model can provide high-quality text input with its excellent audio-to-text conversion capabilities; and the connection module clarifies the instructions and output formats through prompt word engineering, reducing errors caused by ambiguity, greatly improving the large model's understanding and adaptability to specific tasks, thereby enhancing the effect of entity relationship extraction.

[0069] 3. The GPT-3 model can directly predict the entity relationship triples in the target sentence based on the given prompts, and give the corresponding probability distribution, providing a quantitative evaluation of the reliability of the results. It integrates the latest deep learning technology and models, solves the problem that traditional methods are difficult to handle unstructured multimodal data, and significantly promotes the technical development of multimodal entity relationship extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0071] Figure 1 A flow chart of a multimodal entity relationship extraction method applied to audio data provided by an embodiment of the present invention;

[0072] Figure 2 A schematic diagram of a large model training process provided by an embodiment of the present invention;

[0073] Figure 3A schematic diagram of a process for obtaining corresponding text data based on a Whisper model provided in an embodiment of the present invention;

[0074] Figure 4 A schematic diagram of the probability distribution process of generating entity relationship triples and text prediction using the GPT-3 model provided in an embodiment of the present invention;

[0075] Figure 5 A schematic diagram of a multimodal entity relationship extraction process for audio data in the financial field provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] Embodiment 1;

[0078] like Figure 1 As shown, this embodiment provides a multimodal entity relationship extraction method applied to audio data, including:

[0079] S1, build audio entity relationship dataset;

[0080] S2. Train the big model using the audio entity relationship dataset to obtain an optimized big model;

[0081] like Figure 2 As shown, large model training includes:

[0082] S21, performing Whisper model conversion processing on the audio entity relationship data set to obtain corresponding text data;

[0083] S22, optimizing the text data in combination with the connection module to obtain optimized text data;

[0084] S23. Based on the optimized text data, the GPT-3 model is used to extract entity relationships and predict text, and the probability distribution of entity relationship triples and text predictions are generated respectively;

[0085] S3. Input the audio data to be processed into the optimized large model to obtain the corresponding entity relationship extraction results.

[0086] This method effectively processes multimodal data such as audio by introducing a large model, breaking through the limitation of traditional methods that mainly rely on text data and greatly expanding the application scope of entity relationship extraction. It uses the Whisper model to efficiently convert audio into text, and combines the powerful language understanding and generation capabilities of GPT-3 to significantly improve the accuracy and efficiency of entity relationship extraction. In addition, the specially designed connection module optimizes the prompt word engineering and reduces errors caused by unclear instructions.

[0087] The above steps and related technical features are further described in detail below:

[0088] In this embodiment S1, constructing an audio entity relationship dataset specifically includes:

[0089] The Duie dataset, which includes original text and corresponding entity relationship triple lists, is selected as the data source. In the Duie dataset, data is stored in json format, including a text field and a spo_list field. The text field stores the original text, and the spo_list field stores multiple triples including predicate, object type, subject type, object, and subject.

[0090] Use Python's json library to parse the json format file in the Duie dataset, extract the content in the text field as the text to be synthesized, and save the text to be synthesized as a txt format file; and extract the entity relationship information included in the spo_list field as label data;

[0091] Combined with the run function in the Elevenlabs model, the text to be synthesized is converted into audio data, and the audio entity relationship dataset is obtained corresponding to the label data.

[0092] After completing the above steps, the obtained audio-entity relationship dataset is divided into training set, test set and validation set in a ratio of 8:1:1 to support the subsequent large model training and evaluation process.

[0093] During the construction process, you can also consider introducing data enhancement technology, including changing speech synthesis parameters or adding background noise, to generate more diverse audio samples. In addition, a strict data review mechanism should be established to ensure that all generated audio files and corresponding label data are verified by manual or automated tools to ensure the consistency and accuracy of the data set.

[0094] Through the above steps, a high-quality audio entity relationship dataset is created, which lays a good foundation for subsequent research and application. In particular, for multimodal entity relationship extraction tasks, this dataset will become an indispensable and important resource.

[0095] In this embodiment S2, a large model is trained using an audio entity relationship dataset to obtain an optimized large model; specifically, the large model training includes:

[0096] S21, performing Whisper model conversion processing on the audio entity relationship data set to obtain corresponding text data;

[0097] S22, optimizing the text data in combination with the connection module to obtain optimized text data;

[0098] S23. Based on the optimized text data, the GPT-3 model is used to extract entity relationships and predict text, and the probability distribution of entity relationship triples and text predictions are generated respectively;

[0099] like Figure 3 As shown, in S21, the audio data to be processed is converted into a Whisper model, including:

[0100] The audio data in the audio entity relationship data set is converted into an audio feature vector through a feature extraction module, and the audio feature vector is position-encoded;

[0101] The position-encoded audio feature vector is processed through a multi-head attention module to capture long-distance dependencies; this mechanism allows the model to work in parallel in different representation subspaces, enhancing its ability to understand complex patterns.

[0102] Combined with the feedforward neural network, the audio feature vector processed by the multi-head attention module is nonlinearly transformed to obtain the enhanced audio feature representation;

[0103] The enhanced audio feature representation and the position-encoded audio feature vector are subjected to residual connection processing to obtain the result after residual connection processing; the residual connection processing can ensure the integrity of information transmission and help alleviate the gradient vanishing problem in deep networks.

[0104] The result after residual connection processing is processed by multiple consecutive multi-head attention modules and feedforward neural network to obtain the audio data processing result; there are at least 3 consecutive multi-head attention modules and feedforward neural networks in this step.

[0105] Further, the audio data to be processed is converted into a Whisper model, and further includes:

[0106] The audio-text pairs in the audio entity relationship dataset are converted into text feature vectors through a feature extraction module, and the text feature vectors are positionally encoded;

[0107] The position-encoded text feature vector is processed through a multi-head attention module to capture long-distance dependencies.

[0108] Cross-processing the text feature vector processed by the multi-head attention module and the audio data processing result through the cross-attention module to obtain a cross-processing result;

[0109] Combined with the feedforward neural network, the cross-processing results are nonlinearly transformed to obtain enhanced audio feature representation;

[0110] The enhanced audio feature representation is processed by multiple consecutive multi-head attention modules, cross attention modules and feedforward neural networks to obtain text data; in this step, there are at least 3 consecutive multi-head attention modules, cross attention modules and feedforward neural networks.

[0111] Furthermore, the multi-head attention module processes the feature vector, including:

[0112] Perform layer normalization on the feature vector to obtain a layer normalized vector;

[0113] Input the layer-normalized vector into the multi-head attention structure to obtain the fused information; in the multi-head attention structure, each attention head performs attention calculation independently, and multiple attention heads are processed in parallel;

[0114] Perform residual connection processing on the fused information and embedded features, and perform layer normalization processing to obtain the result after layer normalization processing;

[0115] Input the result after layer normalization into the multi-layer perceptron to obtain the output of the multi-layer perceptron;

[0116] The output of the multi-layer perceptron and the result after layer normalization are processed by residual connection, and layer normalization is performed to obtain the output result of the multi-head attention module. Through layer normalization, multi-head attention structure, residual connection and multi-layer perceptron processing, the multi-head attention module achieves efficient fusion and enhancement of feature vectors, significantly improving the comprehensiveness and accuracy of information processing.

[0117] Furthermore, the cross-attention module performs cross-processing, including:

[0118] Process the audio data and text data separately through independent fully connected layers to generate query vectors, key vectors, and value vectors corresponding to the audio data and text data;

[0119] Calculate the dot product of the query vector corresponding to the text data and the key vector corresponding to the audio data and scale them to obtain the attention score;

[0120] Normalize the attention score and multiply it with the value vector corresponding to the audio data to obtain the first attention calculation result;

[0121] Calculate the dot product of the query vector corresponding to the audio data and the key vector corresponding to the text data and scale them to obtain the attention score;

[0122] Normalize the attention score and multiply it with the value vector corresponding to the text data to obtain the second attention calculation result;

[0123] Based on the calculation results of the first attention and the second attention, the output results of the cross attention module are obtained. It generates query, key, and value vectors of audio and text data through independent fully connected layers, and calculates the bidirectional attention scores for scaling and normalization respectively. The cross attention module realizes the information interaction and fusion between audio and text data, effectively enhancing the comprehensiveness and relevance of multimodal data processing, and improving the accuracy and efficiency of entity relationship extraction.

[0124] The above Whisper model conversion process not only effectively captures the long-distance dependencies in audio and text features through a series of carefully designed steps, including feature extraction, position encoding, multi-head attention mechanism, nonlinear transformation, and residual connection, but also realizes the effective fusion of information between audio and text modalities through the cross-attention mechanism. It significantly improves the quality of audio-to-text conversion and enhances the ability to understand audio content, ultimately providing richer and more accurate text input for multimodal entity relationship extraction, greatly improving the accuracy and efficiency of the task.

[0125] like Figure 4 As shown, S22, optimizing the text data in combination with the connection module to obtain optimized text data; including:

[0126] Construct a prompt word template that includes task description, output format requirements, and example entity relationship triples; this template is used to clearly indicate the specific requirements of the task, the expected output format, and provide some example data to the subsequent large model to help the model better understand the task and generate results that meet the requirements.

[0127] The text data converted by the Whisper model is inserted into the prompt word template to generate prompt words; this step ensures that each text sample has targeted instructions to guide the large model on how to process and parse.

[0128] Add contextual information and example entity relationship triples to the prompt words and optimize the format; Adding additional contextual information to the generated prompt words, such as background knowledge or terms in related fields, helps improve the understanding ability of large models. At the same time, adding more diverse example entity relationship triples again allows the model to have a more comprehensive understanding of various situations that may be encountered. In addition, the prompt words are optimized in format to ensure that they are clear and easy to read and easy for large models to parse.

[0129] Based on the optimized prompt words, combined with the few-sample prompt strategy, and clarifying the task instructions and output format requirements, we can obtain optimized text data. Few-sample prompts guide the model to complete the task by providing a small but representative number of examples. This method is particularly suitable for tasks with limited training data. On this basis, we clarify the task instructions and output format requirements again to ensure that the large model can operate accurately according to the instructions.

[0130] This step achieves deep optimization of text data, so that the prompt words finally passed to the large model not only contain clear task instructions and output format requirements, but also incorporate contextual information and rich examples, greatly improving the model's ability to understand tasks and the accuracy of responses. This optimization method effectively avoids errors caused by unclear instructions or formats that do not meet expectations, and significantly improves the effect and reliability of entity relationship extraction.

[0131] In this embodiment S23, based on the optimized text data, the GPT-3 model is used to perform entity relationship extraction and text prediction, and the probability distribution of entity relationship triples and text prediction is generated respectively; including:

[0132] The optimized text data is combined with position embedding and the data is normalized through the first normalization layer;

[0133] The multi-head shielded self-attention module is used to process the data normalized by the first normalization layer to capture the dependency between words at different positions in the text; and the first residual connection processing is performed with the optimized text data;

[0134] Data normalization is performed through the second normalization layer;

[0135] The data normalized by the second normalization layer is subjected to nonlinear transformation in combination with a feedforward neural network to obtain an enhanced feature representation; and the data processed by the first residual connection is subjected to a second residual connection processing;

[0136] The data processed by the second residual connection is processed by multiple consecutive first normalization layers, multi-head shielded self-attention modules, second normalization layers, and feedforward neural networks to obtain the corresponding hidden states;

[0137] Based on the hidden state, the probability distribution of entity relationship triples and text prediction is generated respectively through the output layer.

[0138] Furthermore, entity relationship triples are generated through the output layer, including:

[0139] Perform named entity recognition through the softmax layer to mark all entities in the text and construct potential entity pairs;

[0140] For each identified entity pair, extract the corresponding hidden state vector, and obtain the comprehensive feature vector of each entity pair through the pooling layer;

[0141] Combining the fully connected layer and the softmax layer, the comprehensive feature vector of each entity pair is mapped to the predefined relationship category space to generate entity relationship triples.

[0142] Furthermore, the probability distribution of text prediction is generated through the output layer, including:

[0143] Map the hidden state to the dimension of the vocabulary size through a fully connected layer to obtain a high-dimensional vector;

[0144] Using a softmax layer to convert the high-dimensional vector into a probability distribution;

[0145] Based on the probability distribution, a sampling algorithm is combined to generate text continuation.

[0146] In this step, the multi-head shielded self-attention module and multi-layer architecture are used to significantly improve the model's understanding of the complex dependencies within the text, enhance the richness and flexibility of feature representation, ensure the effective transmission of information and avoid the gradient vanishing problem. The combination of softmax layer and fully connected layer optimizes the generation quality of entity relationship triples and the probability distribution of text prediction, supporting more realistic text continuation generation.

[0147] The multimodal entity relationship extraction method applied to audio data in this embodiment not only fully utilizes the powerful functions of the large model, but also greatly improves the quality and efficiency of the entity relationship extraction task, breaks through the limitation of traditional methods that mainly rely on text data, and greatly expands the application scope of entity relationship extraction.

[0148] Embodiment 2;

[0149] In the financial field, audio materials such as meeting minutes, customer consultation calls, and internal discussions contain a lot of information about market dynamics, company financial status, investment advice, etc. This information is crucial for risk assessment, investment decisions, and compliance reviews. However, extracting valuable entity relationships directly from audio data (such as "Company A acquired Company B", "A bank increased loan interest rates") is a complex and time-consuming process. Traditional text processing methods are difficult to apply directly to unstructured audio data. Therefore, this embodiment provides a multimodal entity relationship extraction method applied to audio data, which aims to efficiently and accurately extract entity relationships in the financial field from audio.

[0150] like Figure 5 As shown, the specific implementation steps include:

[0151] 1) Prepare audio data to be processed;

[0152] Collect original audio files from financial meetings, telephone consultations or other relevant channels. Ensure the audio quality is clear and minimize background noise to improve the accuracy of subsequent processing.

[0153] 2) Data preprocessing;

[0154] Perform preliminary cleaning on the collected audio data, such as removing silent segments, standardizing volume, etc. The audio can be segmented for more efficient batch processing.

[0155] 3) Feature extraction and conversion;

[0156] Use the trained Whisper model to convert audio data into text data. This process includes:

[0157] Audio feature extraction: Convert audio data into audio feature vectors through feature extraction modules and add position encoding. Multi-head attention mechanism: Capture long-distance dependencies in audio feature vectors. Feedforward neural network transformation: Enhance audio feature representation. Residual connection and multiple continuous processing: Ensure effective information transmission and avoid gradient vanishing problem.

[0158] For audio-text pairs, the cross-attention mechanism is further used to fuse audio and text information to obtain optimized text data.

[0159] 4) Optimize prompt words;

[0160] Construct a prompt word template that includes task instructions, output format requirements, and example entity relationship triples. Insert the text data converted from the Whisper model into the prompt word template to generate specific prompt words. Add context information and example entity relationship triples, and optimize the format to ensure that the prompt words are clear and easy to read.

[0161] 5) Entity relationship extraction;

[0162] The optimized text data is combined with position embedding and processed through a normalization layer. The normalized data is processed using a multi-headed shielded self-attention module to capture the dependencies between words at different positions in the text. Nonlinear transformations are performed through a feedforward neural network to obtain enhanced feature representations. Multiple continuous processing ensures the depth and richness of feature representations.

[0163] Further, entity relationship triples are generated;

[0164] Named entity recognition is performed through the softmax layer to mark all entities in the text and construct potential entity pairs. For each recognized entity pair, the corresponding hidden state vector is extracted, and the comprehensive feature vector is obtained through the pooling layer. Combined with the fully connected layer and the softmax layer, the comprehensive feature vector is mapped to the predefined relationship category space to generate the final entity relationship triple.

[0165] Further, text prediction;

[0166] The hidden state is mapped to the dimension of the vocabulary size through the fully connected layer to obtain a high-dimensional vector. The high-dimensional vector is converted into a probability distribution using the softmax layer. Based on the probability distribution, the possible continuation of the text is generated in combination with the sampling algorithm to assist in understanding and explaining entity relationships.

[0167] 6) Result output and application;

[0168] Output entity-relationship triples for subsequent analysis, report generation, or automated decision support systems. The generated text continuation can be used as a reference to help understand the logic behind complex financial transactions or policy changes.

[0169] Through the above steps, this method can not only extract entity relationships in the financial field directly from audio data efficiently and accurately, but also generate relevant text predictions, providing valuable information resources for financial analysts, investors and other stakeholders. This method significantly improves the value mining ability of audio data in the financial field and enhances the scientificity and timeliness of decision-making.

[0170] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0171] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal entity relationship extraction method applied to audio data, characterized in that: include: S1, build audio entity relationship dataset; S2. Train the big model using the audio entity relationship dataset to obtain an optimized big model; The large model training includes: S21, performing Whisper model conversion processing on the audio entity relationship data set to obtain corresponding text data; S22, optimizing the text data in combination with the connection module to obtain optimized text data; S23. Based on the optimized text data, the GPT-3 model is used to extract entity relationships and predict text, and the probability distribution of entity relationship triples and text predictions are generated respectively; S3. Input the audio data to be processed into the optimized large model to obtain the corresponding entity relationship extraction results.

2. The multimodal entity relationship extraction method applied to audio data according to claim 1, characterized in that: Said S1 comprises: The Duie dataset, which includes the original text and the corresponding entity relationship triple list, is selected as the data source. In the Duie dataset, the data is stored in json format, including the text field and the spo_list field. Use Python's json library to parse the json format file in the Duie dataset, extract the content in the text field as the text to be synthesized, and save the text to be synthesized as a txt format file; and extract the entity relationship information included in the spo_list field as label data; Combined with the run function in the Elevenlabs model, the text to be synthesized is converted into audio data, and the audio entity relationship dataset is obtained corresponding to the label data.

3. The multimodal entity relationship extraction method applied to audio data according to claim 1, characterized in that: In S21, the audio data to be processed is converted into a Whisper model, including: The audio data in the audio entity relationship data set is converted into an audio feature vector through a feature extraction module, and the audio feature vector is position-encoded; The position-encoded audio feature vector is processed through a multi-head attention module to capture long-distance dependencies. Combined with the feedforward neural network, the audio feature vector processed by the multi-head attention module is nonlinearly transformed to obtain the enhanced audio feature representation; Performing residual connection processing on the enhanced audio feature representation and the audio feature vector after position encoding to obtain a result after residual connection processing; The result after residual connection processing is processed by multiple consecutive multi-head attention modules and feedforward neural network to obtain the audio data processing result.

4. The multimodal entity relationship extraction method applied to audio data according to claim 3 is characterized in that: In S21, the audio data to be processed is converted into a Whisper model, and further includes: The audio-text pairs in the audio entity relationship dataset are converted into text feature vectors through a feature extraction module, and the text feature vectors are positionally encoded; The position-encoded text feature vector is processed through a multi-head attention module to capture long-distance dependencies. Cross-processing the text feature vector processed by the multi-head attention module and the audio data processing result through the cross-attention module to obtain a cross-processing result; Combined with the feedforward neural network, the cross-processing results are nonlinearly transformed to obtain enhanced audio feature representation; The enhanced audio feature representation is processed by multiple consecutive multi-head attention modules, cross attention modules and feedforward neural networks to obtain text data.

5. The multimodal entity relationship extraction method applied to audio data according to claim 4, characterized in that: The multi-head attention module processes the feature vector, including: Perform layer normalization on the feature vector to obtain a layer normalized vector; Input the layer-normalized vector into the multi-head attention structure to obtain the fused information; in the multi-head attention structure, each attention head performs attention calculation independently, and multiple attention heads are processed in parallel; Perform residual connection processing on the fused information and embedded features, and perform layer normalization processing to obtain the result after layer normalization processing; Input the result after layer normalization into the multi-layer perceptron to obtain the output of the multi-layer perceptron; The output of the multi-layer perceptron is processed with the result after layer normalization through residual connection, and layer normalization is performed to obtain the output result of the multi-head attention module.

6. The multimodal entity relationship extraction method applied to audio data according to claim 4, characterized in that: The cross-attention module performs cross-processing, including: Process the audio data and text data separately through independent fully connected layers to generate query vectors, key vectors, and value vectors corresponding to the audio data and text data; Calculate the dot product of the query vector corresponding to the text data and the key vector corresponding to the audio data and scale them to obtain the attention score; Normalize the attention score and multiply it with the value vector corresponding to the audio data to obtain the first attention calculation result; Calculate the dot product of the query vector corresponding to the audio data and the key vector corresponding to the text data and scale them to obtain the attention score; Normalize the attention score and multiply it with the value vector corresponding to the text data to obtain the second attention calculation result; Based on the first attention calculation result and the second attention calculation result, the output result of the cross attention module is obtained.

7. The multimodal entity relationship extraction method applied to audio data according to claim 1, characterized in that: The S22 includes: Construct a prompt word template that includes task description, output format requirements, and example entity-relationship triples; Insert the text data converted by the Whisper model into the prompt word template to generate prompt words; Add context information and example entity relationship triples to the prompt words and optimize the format; Based on the prompt words with optimized format, combined with the few-sample prompt strategy, and clarifying the task instructions and output format requirements, the optimized text data is obtained.

8. The multimodal entity relationship extraction method applied to audio data according to claim 1, characterized in that: The S23 comprises: The optimized text data is combined with position embedding and the data is normalized through the first normalization layer; The multi-head shielded self-attention module is used to process the data normalized by the first normalization layer to capture the dependency between words at different positions in the text; and the first residual connection processing is performed with the optimized text data; Data normalization is performed through the second normalization layer; The data normalized by the second normalization layer is subjected to nonlinear transformation in combination with a feedforward neural network to obtain an enhanced feature representation; and the data processed by the first residual connection is subjected to a second residual connection processing; The data processed by the second residual connection is processed by multiple consecutive first normalization layers, multi-head shielded self-attention modules, second normalization layers, and feedforward neural networks to obtain the corresponding hidden states; Based on the hidden state, the probability distribution of entity relationship triples and text prediction is generated respectively through the output layer.

9. A multimodal entity relationship extraction method applied to audio data according to claim 8, characterized in that: The output layer generates entity-relationship triples, including: Perform named entity recognition through the softmax layer to mark all entities in the text and construct potential entity pairs; For each identified entity pair, extract the corresponding hidden state vector, and obtain the comprehensive feature vector of each entity pair through the pooling layer; Combining the fully connected layer and the softmax layer, the comprehensive feature vector of each entity pair is mapped to the predefined relationship category space to generate entity relationship triples.

10. The multimodal entity relationship extraction method applied to audio data according to claim 8, characterized in that: The probability distribution of text prediction is generated through the output layer, including: Map the hidden state to the dimension of the vocabulary size through a fully connected layer to obtain a high-dimensional vector; Using a softmax layer to convert the high-dimensional vector into a probability distribution; Based on the probability distribution, a sampling algorithm is combined to generate text continuation.

Images