Joint extraction method, device and equipment of entity and relation and medium

Abstract

The application relates to the fields of artificial intelligence and financial technology, and discloses a joint extraction method for entities and relations, which comprises the following steps: acquiring audio data, extracting voice features, compressing and reducing the dimensions of the voice features, decoding the compressed voice features to generate linearized text sequences, using a pre-trained recognition model to perform entity recognition and relation classification on the text sequences, and generating triad information containing entities and relations. By omitting the intermediate conversion step from audio to text, the application directly extracts semantic information from audio data, reduces the information loss that may occur in the modal conversion process, and improves the processing efficiency and accuracy. In addition, by using the pre-trained recognition model to perform entity recognition and relation classification on the text sequences, the application ensures that key information can be efficiently extracted from a large amount of audio data under low-cost conditions, and realizes the joint recognition of entities and relations.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and the technical field of financial technology, and in particular to an entity and relationship joint extraction method, device and equipment and storage medium. BACKGROUND

[0002] With the rapid development of the financial industry, customer service systems have become one of the core channels for banks to interact with customers. Especially in large financial institutions, telephone customer service systems accumulate rich audio data through a large number of voice interactions. These audio data contain key business information between customers and banks. In-depth mining and analysis of these data has important commercial value, especially in entity recognition and relationship extraction, which can provide direct support for customer management and business optimization.

[0003] In the prior art, audio data is usually converted into text through automatic speech recognition (ASR) technology, and then text data is processed through text information extraction technology to identify entities and their relationships. This processing procedure can effectively obtain semantic information from text, but has the following significant shortcomings:

[0004] The cost of ASR technology is high, especially when processing a large amount of audio data. The process of transcribing text not only consumes resources, but also requires a long processing time. This method is not efficient for processing large-scale financial voice data.

[0005] In the process of converting audio to text, information loss or inaccuracy often occurs, especially the emotions, intonations and background information unique to the voice that are difficult to fully express through text. These lost information can adversely affect subsequent entity recognition and relationship extraction.

[0006] Audio data and text data are two different modalities. The existing technology needs to process across modalities, i.e. from audio to text, and then from text to information mining. This modal conversion increases the complexity of processing, and in the conversion process between different modalities, noise and errors are easily introduced, resulting in a decrease in recognition accuracy. SUMMARY

[0007] The main purpose of the present application is to provide an entity and relationship joint extraction method, device, equipment and storage medium, which aims to solve the technical problem that in the prior art, when audio data is converted into text through automatic speech recognition, key information in the voice is often lost, resulting in a decrease in the accuracy of entity recognition and relationship extraction.

[0008] To achieve the above purpose, the present application provides an entity and relationship joint extraction method, comprising:

[0009] acquire audio data, the audio data containing speech information;

[0010] input the audio data into an encoder module, extract speech features from the audio data through the encoder module;

[0011] compress and reduce dimensions of the speech features through a convolutional neural network to generate compressed speech features;

[0012] input the compressed speech features into the decoder module to generate linearized text sequences;

[0013] perform entity recognition and relationship classification operations on the text sequences through a pre-trained recognition model to generate triple information containing a first entity, a second entity and a relationship between the first entity and the second entity.

[0014] Further, to achieve the above object, the present application provides an entity and relationship joint extraction device, comprising:

[0015] an audio data acquisition module for acquiring audio data, the audio data containing speech information;

[0016] an encoder module for inputting the audio data into the encoder module and extracting speech features from the audio data through the encoder module;

[0017] a convolutional neural network module for compressing and reducing dimensions of the speech features through a convolutional neural network to generate compressed speech features;

[0018] a decoder module for inputting the compressed speech features into the decoder module to generate linearized text sequences;

[0019] a semantic information extraction module for performing entity recognition and relationship classification operations on the text sequences through a pre-trained recognition model to generate triple information containing a first entity, a second entity and a relationship between the first entity and the second entity.

[0020] Further, to achieve the above object, the present application also provides a computer device, comprising a memory, a processor and an entity and relationship joint extraction program stored on the memory and executable on the processor, the entity and relationship joint extraction program being executed by the processor to implement the steps of the above-mentioned entity and relationship joint extraction method.

[0021] Further, to achieve the above object, the present application also provides a computer readable storage medium, wherein the storage medium stores an entity and relation joint extraction program, and the entity and relation joint extraction program implements the steps of the entity and relation joint extraction method when executed by a processor.

[0022] Beneficial effects: The present application relates to the fields of artificial intelligence and financial technology, and discloses an entity and relation joint extraction method. The method obtains audio data, extracts speech features, compresses and reduces the dimensions of the speech features, decodes the compressed speech features to generate linearized text sequences, uses a pre-trained recognition model to perform entity recognition and relation classification on the text sequences, and generates triple information containing entities and relations. The present application omits the intermediate conversion step from audio to text, directly extracts semantic information from audio data, reduces information loss that may occur during modal conversion, and improves processing efficiency and accuracy. In addition, the pre-trained recognition model is used to perform entity recognition and relation classification on the text sequences, ensuring that key information can be efficiently extracted from a large amount of audio data under low-cost conditions, and realizing joint recognition of entities and relations. BRIEF DESCRIPTION OF DRAWINGS

[0023] The present application will be further described below with reference to the accompanying drawings and embodiments. In the drawings:

[0024] Figure 1 An application environment diagram of the entity and relation joint extraction method in an embodiment of the present application;

[0025] Figure 2 A flow diagram of the entity and relation joint extraction method in an embodiment of the present application;

[0026] Figure 3 A functional module diagram of the entity and relation joint extraction device in a preferred embodiment of the present application;

[0027] Figure 4 A structure diagram of a computer device in an embodiment of the present application;

[0028] Figure 5 Another structure diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0029] It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0030] The entity and relation joint extraction method provided by the embodiments of the present application can be applied in, for example, Figure 1The application environment of the application is a user terminal communicating with a server terminal through a network. The server terminal can obtain audio data through the user terminal, extract speech features, compress and reduce the dimensions of the speech features, decode the compressed speech features to generate linearized text sequences, use a pre-trained recognition model to perform entity recognition and relationship classification on the text sequences, and generate triple information containing entities and relationships. By omitting the intermediate conversion step from audio to text, the application directly extracts semantic information from audio data, reduces the information loss that may occur during modal conversion, and improves processing efficiency and accuracy. In addition, by using a pre-trained recognition model to perform entity recognition and relationship classification on the text sequences, the application ensures that key information can be efficiently extracted from a large amount of audio data under low-cost conditions, and realizes joint recognition of entities and relationships. The user terminal can be, but is not limited to, various personal computers, notebook computers, smartphones, tablet computers, and portable wearable devices. The server terminal can be implemented by an independent server or a server cluster composed of multiple servers. The application will be described in detail below through specific embodiments.

[0031] Please refer to Figure 2 , Figure 2 The flowchart of an embodiment of the joint extraction method of entities and relationships provided by the application is shown in the figure. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that shown here.

[0032] As Figure 2 shown, the joint extraction method of entities and relationships provided by the application includes the following steps:

[0033] S10, obtaining audio data, the audio data containing speech information;

[0034] In this embodiment, audio data acquisition is the starting point of the entire system processing flow, which refers to obtaining raw audio data from external input sources (such as telephone systems, recording devices or other audio capture devices). These data can include audio files of various formats (such as.wav,.mp3, etc.), and it is necessary to ensure that the quality of the audio is high enough for subsequent processing. These audio data mainly contain speech information of customer interaction, which can include customer questions, consultations, feedback, etc.

[0035] The source of audio data can be diverse, mainly including telephone call recordings between customers and financial institutions, telephone customer service interaction data, etc. In the application of the financial field, audio data is often derived from the customer service system of a bank, containing key information such as customer consultation on business, account information query, transaction behavior confirmation, etc.

[0036] The so-called voice information specifically refers to the customer voice and its corresponding semantic content contained in the audio data. For example, in the financial field, the customer's voice data may contain information such as account balance inquiry, transfer operation, loan consultation, etc. These voice information will be converted into semantic features in subsequent steps for further analysis and processing.

[0037] The system needs to access external audio data sources, which can be achieved by automatically receiving customer service recordings through the telephone system, or by manually uploading audio files for processing. The collected audio data may be stored in high-quality audio formats, such as 16kHz sampling rate.wav files, to ensure the clarity of the sound and facilitate subsequent feature extraction and analysis.

[0038] The acquired audio data is transmitted to the central processing unit of the system through the network interface, and a distributed storage system can be selected for storage. The storage system needs to support multiple formats of audio files and can receive and store data in real time to ensure that the interaction content between the customer and the financial institution is not lost.

[0039] Example: In the financial field, the telephone customer service system of a bank will handle thousands of customer requests every day. For example, customers may inquire about their account balance or apply for a loan through the phone. Acquiring the audio data of these calls can provide important basic data for the bank's subsequent customer analysis and customer relationship management. Through an automated system, the bank can collect customer voice interaction data in real time and conduct in-depth analysis and processing in subsequent steps to identify customer needs and business relationships, improving service quality and customer satisfaction.

[0040] Through automated audio data acquisition, the interaction information between customers and financial institutions can be effectively collected. Compared with manual recording or text recording, directly acquiring audio data can preserve the complete context of customer interaction, ensuring the integrity and accuracy of the information.

[0041] S20, input the audio data into an encoder module, and extract voice features from the audio data through the encoder module;

[0042] In this embodiment, the main task of the encoder module is to convert the input audio data into feature vectors for subsequent processing. This module generates voice features containing rich information through segmentation processing and feature extraction of audio data. These features capture the timing changes in the audio and the content information of the voice.

[0043] Voice features are the core information representation in audio data, including two key information:

[0044] Timing information: refers to the characteristics of audio data that change over time, manifested as fluctuations and changes in frequency of the speech signal. This part of information is crucial for capturing the rhythm, speech rate, and pronunciation time of the speech.

[0045] Content information: refers to the actual semantic content expressed by the speech in the audio. Content information can reflect the specific expression meaning of the speaker in the speech, such as the customer's request or stated business problem.

[0046] The encoder module is usually based on deep learning techniques such as Wav2Vec 2.0, Transformer encoder, etc. Its main function is to feature the original audio data and extract high-dimensional speech feature representation. Through the learned speech feature model, the module can automatically capture the timing dynamics and content of the audio signal.

[0047] The encoder module not only extracts the time structure of the speech, but also captures the semantic content of the speech expression from the signal. These information is crucial for subsequent speech processing and semantic extraction. Timing information provides the structural basis of the speech signal, while content information concerns the actual semantic expression.

[0048] Audio data is input into the encoder module in frames. Usually, the audio data has been segmented in the preprocessing stage, so when inputting into the encoder, each segment of audio represents a part of complete speech information. These audio segments enter the encoder in time sequence for processing.

[0049] The encoder extracts features from the audio segments layer by layer through convolutional layers or other neural network structures. Convolutional neural networks can capture the frequency characteristics of speech, while encoders based on self-attention mechanisms (such as Transformer) can effectively capture long-distance dependencies in time series, thus extracting timing information.

[0050] The final output of the encoder is a multi-dimensional feature vector, which contains timing information and content information in the audio. This feature vector can be used for subsequent tasks such as decoding or classification. These feature representations not only contain the physical properties of the speech such as pitch, speech rate, etc., but also contain the actual semantic content in the customer's speech.

[0051] Example: In the financial field, when the customer interacts with the bank customer service through the phone, the encoder module extracts speech features from the call recording. For example, when the customer asks "What is my account balance?", the encoder module not only captures the time variation of the speech (such as the customer's speech rate, tone), but also extracts the semantic content (such as "account balance", "query" keywords) through the deep learning model. In this way, the system can quickly understand the customer's intention and provide a basis for subsequent semantic analysis and business processing.

[0052] By inputting the audio data into the encoder module and extracting the speech features, the system can effectively capture the timing changes and speech content in the audio data. This not only improves the representation ability of the speech signal, but also provides high-quality feature input for subsequent decoding and semantic processing. Compared with traditional manual feature extraction methods, the encoder module can automatically, quickly and accurately generate complex speech feature vectors, greatly reducing the error of human intervention and improving processing efficiency.

[0053] S30, compressing and reducing the dimension of the speech feature through a convolutional neural network to generate a compressed speech feature;

[0054] In this embodiment, the convolutional neural network (CNN) is widely used in processing image, audio and other data with spatial or temporal structure. In this technology, CNN is used to further process the speech features output by the encoder module. The convolutional neural network extracts local patterns in the feature representation through multiple convolutional layers, compresses the spatial dimension of the speech feature, and retains the key timing and semantic information.

[0055] Feature extraction of audio data usually generates high-dimensional feature vectors, which contain rich information, but in actual processing and calculation process, too high dimension will lead to the increase of calculation complexity and will introduce redundant information. Through the compression and dimension reduction of convolutional neural network, the dimension of feature vector can be reduced, the most important speech features can be retained, and the redundant information can be reduced, so as to improve the calculation efficiency.

[0056] In the convolutional neural network, the convolution operation can extract feature patterns from local regions, and the pooling operation is used to downsample the extracted features to further compress the data dimension. Through the superposition of multiple convolution and pooling, the compressed speech features are finally generated, which still contain the core information of the audio, such as semantic content and timing pattern.

[0057] After the processing of the convolutional neural network, the dimension of the speech feature vector has been greatly reduced. These compressed features not only have strong expression ability, but also provide more efficient input for subsequent decoding and semantic processing. The compressed speech features retain the most important information in the original speech signal, such as pronunciation features, pitch changes, etc., which are convenient for subsequent module processing.

[0058] Each convolutional layer is responsible for extracting different speech features, such as frequency information, time information, etc. Convolution operation slides on the feature map by moving window to extract local patterns (such as pitch, syllable change, etc.). The size and number of convolution kernel determine the accuracy and range of feature extraction.

[0059] After the convolutional layers, a pooling operation is often used to reduce the dimensionality of the features. Pooling operations take the maximum or average value of a local region according to a set pooling window, reducing the dimensionality of the data. Pooling operations can effectively reduce the redundant information of the feature vector while preserving the global key information.

[0060] Multi-layer convolutional networks allow the system to extract information from different feature levels. The initial layers may capture simple speech patterns (such as pitch variations), while deeper convolutional networks can extract more complex semantic features. Through the stacking of multi-layer convolutional networks, the system can achieve efficient compression of speech features and preserve key information.

[0061] Example: In the financial field, when the bank customer service system obtains a large number of call recordings, a convolutional neural network can be used to compress and reduce the dimensionality of these audio features. For example, when a customer calls the bank hotline to inquire about loan information, the system first extracts the frequency variation, pitch feature, and key speech pattern in the speech through the convolutional layer, and then reduces the dimensionality of these features through the pooling layer for fast processing and analysis. Through this method, the bank can efficiently analyze a large amount of customer speech data, accurately identify customer intent, improve service quality, and support business decisions.

[0062] By compressing and reducing the dimensionality of speech features through convolutional neural networks, redundant information can be effectively reduced, key features in the speech signal can be preserved, and the complexity of the calculation can be reduced. Compared with high-dimensional features that have not been processed, compressed features processed by convolutional networks not only improve processing speed, but also improve the recognition accuracy of subsequent models.

[0063] S40, input the compressed speech features into the decoder module to generate a linearized text sequence;

[0064] In this embodiment, after processing by the convolutional neural network, the speech features have been compressed and key timing and content information has been preserved. This compressed speech feature vector contains the core semantic expression of the speech, facilitating subsequent decoding processing. These features can be represented as low-dimensional feature vectors that condense key information such as pitch, speech rate, and semantics in the audio.

[0065] The decoder module is responsible for converting the input speech features into text output. This step usually involves decoding the speech features into text representations through some neural network structure (such as a decoder based on recurrent neural networks (RNN), long short-term memory networks (LSTM), or self-attention mechanisms). The task of the decoder is to generate text output corresponding to the speech features, forming a linearized text sequence.

[0066] A linearized text sequence is a chronological arrangement of the decoded text information, typically a set of strings or symbols that sequentially represent the semantic content of the audio. This text sequence is the final semantic representation, encompassing the complete semantic content of the original audio data and converting it into a text representation suitable for further machine processing.

[0067] The decoder takes the compressed speech features as input and gradually generates the corresponding text sequence based on a specific model architecture. This process involves capturing the contextual relationships in the features and using the decoder's prediction mechanism to generate the optimal semantic match. The decoder generates a character or word at each step, gradually building a complete linearized text sequence.

[0068] The compressed speech feature vector is first used as input to the decoder. In the neural network decoder, these feature vectors are processed in time series order. Through each decoding step, the model generates the corresponding text symbol or word. The decoder module uses a recurrent network (such as LSTM or GRU) or a self-attention mechanism (such as Transformer) to gradually decode the input features. For each time step, the decoder generates a word or character based on the current feature input and combines it with the previous context until a complete text sequence is generated. Throughout the decoding process, the decoder outputs a gradually growing text sequence. This text sequence is linearized, indicating that the speech content in the audio data is arranged in time order. For example, from "account balance" to "query successful", the system gradually generates complete sentences that can be directly used for subsequent semantic analysis or display.

[0069] By feeding compressed speech features into a decoder module and generating a linearized text sequence, the system can generate corresponding text output directly from the audio data, avoiding the potential information loss during the audio-to-text conversion process. Compared to traditional ASR systems, this decoding method more accurately preserves the semantic information in the speech when generating text, while reducing processing complexity and improving overall efficiency.

[0070] S50, performing entity recognition operations and relationship classification operations on the text sequence through a pre-trained recognition model to generate triple information including a first entity, a second entity, and a relationship between the first entity and the second entity.

[0071] In this embodiment, the pre-trained recognition model is a natural language processing model based on deep learning. It has been trained on a large amount of data and can identify entities and their relationships within text sequences. This model is typically pre-trained with data from a specific domain and can effectively process text data from fields such as finance and customer service. The model's primary task is to automatically identify key information from the input linearized text sequence.

[0072] Named Entity Recognition (NER) is the process of detecting and extracting meaningful words or phrases from text, called entities. Entities can be people, organizations, locations, events, etc. The task of Named Entity Recognition is to identify these specific entities from a text sequence. For example, in a customer interaction in the financial domain, key entities such as "customer name", "account number", "transaction amount" might be involved.

[0073] Relation Classification refers to identifying the semantic connections between entities. The Relation Classification module infers specific relationships between the identified entities by analyzing their context. For example, in a customer query, the system identifies "customer" and "account", and the Relation Classification model can infer a "query" relationship between them. Relation Classification can help the system understand the contextual relationships between entities, such as "customer-query-account balance".

[0074] Through the operations of entity recognition and relation classification, the system can generate semantic structures in the form of triples. The format of triples is <entity1, relation, entity2>, which represents the specific relationship between entity1 and entity2. For example, in a customer interaction text, "customer A" and "account X" are identified, and their relationship is "query", then the generated triple information can be <customer A, query, account X>.

[0075] The pre-trained recognition model first processes the linearized text sequence word by word, identifying entities with specific meanings. This can be done through tagging, i.e., marking each word or phrase in the text as belonging to a certain pre-defined entity category. The model determines whether the word is a person's name, a location, or other specific entity type based on semantic context. After entity recognition is complete, the system performs relationship inference on the identified entities. The relation classification operation, based on the context relationship between the identified entities, uses a relation classification model to clarify the semantic connection between entity pairs. The system analyzes the adjacent words or context information of these entities to determine their relationship categories (such as "belongs to", "query", "belongs to", etc.). After entity recognition and relation classification are complete, the system organizes the generated entities and their relationships into triples. Each triple consists of two entities and their relationship, representing the core semantic information of the text. The generated triples can be used for subsequent knowledge graph construction, information retrieval, or automated analysis.

[0076] Example explanation: In a practical application scenario in the financial field, when a customer queries "I want to query the account balance" through the phone, the pre-trained recognition model can automatically identify key entities such as "customer A" and "account balance" after the system generates a linear text sequence through the decoder. Then, the relationship classification model further infers the "query" relationship between the two entities. Finally, the system generates the triple information <customer A, query, account balance>. This information can be directly used for processing the customer account query request in the bank system, realizing the automation and intelligentization of customer service.

[0077] Performing entity recognition and relationship classification operations through the pre-trained recognition model can greatly improve the efficiency of the system in extracting key information from text. Compared with traditional manual analysis or rule matching methods, the pre-trained model can automatically extract important entities and their relationships from text, reducing human intervention while improving the accuracy and consistency of recognition.

[0078] The present application relates to the fields of artificial intelligence technology and financial technology, and discloses a joint extraction method of entities and relationships, which obtains audio data, extracts speech features, compresses and reduces the dimensions of the speech features, decodes the compressed speech features to generate a linear text sequence, uses a pre-trained recognition model to perform entity recognition and relationship classification on the text sequence, and generates triple information containing entities and relationships. The present application directly extracts semantic information from audio data by omitting the intermediate conversion step from audio to text, reduces the information loss that may occur in the modal conversion process, and improves the processing efficiency and accuracy. In addition, by using a pre-trained recognition model to perform entity recognition and relationship classification on the text sequence, the present application ensures that key information can be efficiently extracted from a large amount of audio data under low-cost conditions, realizing joint recognition of entities and relationships.

[0079] In one embodiment, in S20 described above, the audio data is input into an encoder module, and speech features are extracted from the audio data by the encoder module, including:

[0080] S201, segmenting the audio data according to a preset frame length to generate a plurality of audio segments;

[0081] S202, inputting the plurality of audio segments into an encoder module, and extracting time-frequency features of each audio segment by a feature extractor based on a convolutional neural network;

[0082] S203, in the process of extracting the time-frequency features, using an attention mechanism to enhance the feature extraction of key frequency bands and time periods;

[0083] S204, the enhanced time-frequency features are globally modeled through a self-attention mechanism to capture the contextual dependency between audio segments and generate final speech feature vectors containing global temporal information.

[0084] In this embodiment, the audio data is a continuous signal, and for processing convenience, it is usually divided into multiple segments according to fixed time intervals (frames). Each segment represents a small part of the audio data, used to capture speech information in a short time. The selection of frame length depends on the processing requirements, and is usually set to vary between 20ms and 50ms. Each segment after segmentation can be independently processed for subsequent feature extraction.

[0085] The audio segment as input data enters the convolutional neural network (CNN) feature extractor in the encoder module. Convolutional neural networks are suitable for processing data with time series or spatial structure, such as frequency and timing information in audio. The feature extractor will extract time-frequency features from each audio segment, mainly including:

[0086] Frequency response: refers to the performance of the speech signal at different frequencies, reflecting tone, pronunciation characteristics, etc.

[0087] Energy variation: represents the intensity variation of the speech signal, which can distinguish high and low pitch passages and capture changes in volume and pitch.

[0088] Time series pattern: contains changes in speech over time, reflecting speaking speed, pauses, etc.

[0089] Attention mechanism is a technique that can focus on important parts of data. In speech feature extraction, the attention mechanism can help the model focus on specific frequencies and time periods in the audio segment to obtain more important speech information. By enhancing the feature extraction of these key areas, the model can more accurately capture important semantic or sound changes and avoid interference from redundant information.

[0090] Self-attention mechanism is a method that can capture global dependencies in sequence data. It can use all the context information when processing the current audio segment, not just local information. In this way, the model can learn the complex association between audio segments and capture more rich temporal dependencies, thereby generating speech feature vectors containing global temporal information. These feature vectors represent the complete content of the audio segment and preserve the semantic information in the speech.

[0091] The input audio data is first segmented according to a preset frame length, and each segment of audio represents a short-time speech signal. The frame length can be adjusted according to actual needs, and a typical frame length is 20 ms or 25 ms. Each audio segment is input into a convolutional neural network, and a convolutional layer extracts time-frequency features from each segment. Convolutional operations can capture features such as frequency, energy, and timing changes. As the convolutional layers deepen, the network gradually extracts more complex speech patterns, generating a set of feature vectors for further processing. During the extraction of time-frequency features, the system applies an attention mechanism to enhance the focus on key frequency bands and time periods. The attention mechanism allows the network to concentrate on highly relevant parts of the speech signal, such as key pronunciation segments of the speaker, through weight distribution. The extracted time-frequency features are processed through a self-attention mechanism to capture context dependencies in the entire sequence of audio segments. The self-attention mechanism can identify relationships between audio segments and generate global speech features. These global features represent the temporal structure and content of the entire audio segment and are used for subsequent decoding or semantic analysis.

[0092] This embodiment processes audio data by segmentation and extracts time-frequency features using a convolutional neural network. The system can quickly capture key frequencies and timing information in the audio. Combined with the attention mechanism and the self-attention mechanism, the system's ability to capture key information in the audio signal is further enhanced, and global modeling is achieved. Compared with traditional processing methods that only use convolutional neural networks, this method not only accurately captures speech features but also fully utilizes global context information between audio segments, resulting in more accurate and expressive speech feature representations.

[0093] In one embodiment, S30 described above includes:

[0094] S301, performing convolutional processing on the speech features through multi-layer convolutional operations;

[0095] S302, after each convolutional operation, performing dimension reduction processing on the speech features in combination with a pooling operation to compress the feature dimensions of the speech features;

[0096] S303, performing channel compression on the channels of the speech features using a channel compression convolutional operation;

[0097] S304, performing activation function processing on the speech feature vectors after the multi-layer convolutional operations, the pooling operations, and the channel compression to generate final speech feature vectors.

[0098] In this embodiment, convolutional layers in a convolutional neural network (CNN) are used to extract local features. Through multiple layers of convolutional operations, the system progressively extracts different levels of features in the speech signal, including pitch, volume, speech rate, and other information. Each layer of convolutional operation extracts a different dimension of feature representation from the input speech features. Multiple layers of convolution can progressively focus on complex patterns in the speech signal, improving the expressive ability of the speech content.

[0099] After the convolutional operation, the system uses a pooling operation (such as max pooling or average pooling) to downsample the convolved features. The purpose of the pooling operation is to reduce the feature dimension while retaining the most important information. Pooling reduces the computational complexity by reducing the dimension of the convolutional feature map, reducing the size of the input data, and ensuring that local information is retained while redundant information is discarded.

[0100] Each layer of convolutional operation generates multiple feature channels (feature maps) representing different feature types. Through channel compression technology, the system further compresses the feature maps generated by convolution, reducing the number of channels, thereby reducing the network computing cost. The purpose of channel compression is to further compress the size of the speech features while keeping the core features unchanged, reducing redundant data. This method is suitable for processing larger-scale audio feature maps.

[0101] After convolution, pooling, and channel compression, the system applies an activation function (such as ReLU or Leaky ReLU) to the processed feature vector. The activation function is a nonlinear operation in the neural network that can introduce nonlinear factors, allowing the model to better represent complex speech features. The speech feature vector processed by the activation function is the final feature representation after multiple layers of processing, ready for subsequent decoding or further analysis.

[0102] After multiple layers of convolution operations, each layer extracts higher-order features from the previous layer's features. The initial layer may extract simple frequency and energy information, while deeper convolutions can extract complex speech patterns such as specific speech units or semantic associations. This process makes the speech features more compact and suitable for subsequent dimensionality reduction. After each convolution operation, a pooling operation is performed on the feature map. Pooling operation reduces the size of the feature map by retaining the maximum or average value, thereby reducing the complexity and computational load of the network. The feature representation after pooling is more compact, but still retains the key information in the speech signal. Channel compression reduces the number of channels by performing convolution operations on multiple channels of the feature map. This reduces the size of the feature map while retaining the most important speech features. Channel compression convolution operations can fuse information from different channels to form a compact representation suitable for large-scale data processing scenarios. The final step is to input the processed feature vector to an activation function. Common activation functions such as ReLU can handle negative numbers, retain positive information, and introduce nonlinearity. This step helps the model handle complex nonlinear data, allowing the generated feature vector to better represent the complex relationships in speech.

[0103] The embodiment effectively compresses the dimensionality of speech features, retains key information, and reduces computational complexity through multiple layers of convolution operations, pooling operations, and channel compression techniques. Not only can it improve processing efficiency, but it can also ensure the expressiveness of speech features, reducing feature redundancy while retaining the core features of speech. The final generated feature vector is more compact and has strong semantic expressiveness, making it suitable for subsequent decoding or semantic analysis.

[0104] In one embodiment, the above S40 comprises:

[0105] S401, input the compressed speech features into a decoder module;

[0106] S402, through a decoder module based on a recurrent neural network or self-attention mechanism, layer-by-layer decoding the speech features to generate an intermediate semantic representation corresponding to the speech features;

[0107] S403, the decoder module extracts semantic information from the intermediate semantic representation according to the context relationship of the speech features;

[0108] S404, convert the semantic information into a linearized text sequence.

[0109] In this embodiment, after processing through the convolutional neural network, the speech features have been compressed and reduced in dimensionality, and these feature vectors need to be further decoded into text sequences. The task of the decoder module is to convert these speech features into linearized text representations step by step. This step is a critical part of the speech-to-text conversion, responsible for converting complex feature vectors into natural language text.

[0110] The decoder can be implemented based on different structures, the most common being recurrent neural networks (RNNs) and self-attention mechanisms. RNNs are suitable for processing sequential data because they can maintain the temporal dependencies of input sequences. Through step-by-step decoding, the decoder extracts the corresponding semantic information from each time step. The self-attention mechanism (such as Transformer) can simultaneously focus on the dependencies of different parts based on the entire sequence, enhancing the accuracy of semantic extraction. Layer-by-layer decoding refers to the decoder processing each layer of the feature vector in turn, generating the corresponding intermediate semantic representation.

[0111] During the layer-by-layer decoding process, the decoder not only relies on individual speech features, but also combines context information. Through the context relationship, the decoder can better capture the dependencies between speech segments and extract more complete semantic information. The context relationship can be the connection between different syllables or words in the speech, which is captured and analyzed through the attention mechanism or recursive structure in the decoder.

[0112] The final output of the decoder is a linearized text sequence. This text sequence is a semantic expression arranged in chronological order, usually a sentence or phrase in natural language. The decoder generates a set of text symbols or words arranged in order based on the extracted semantic information, representing the meaning of each speech segment in the audio. This linearized text sequence can be directly used for further text processing or display.

[0113] The system inputs the compressed speech feature vectors into a decoder module. The input order of speech features is determined by the time axis, i.e. each feature vector represents a certain segment in the speech sequence. The decoder receives these compressed features and starts the decoding operation layer by layer. The decoder module can be based on RNN or Transformer architecture. For RNN, the decoder processes each input feature step by step and generates the corresponding intermediate semantic representation. For the Transformer-based decoder, the system can process the speech features in parallel, combine the context information in the entire sequence through the self-attention mechanism, and generate the intermediate semantic representation. These intermediate semantic representations contain the preliminary understanding of the input speech features. The decoder module further processes the intermediate semantic representations, combines the context relationships in the input speech features, and comprehensively extracts the semantics of the entire speech sequence. The system uses recursive mechanisms or attention mechanisms to identify the dependencies between different speech segments to ensure that the generated semantic information can reflect the complete audio content. The last step is to convert the extracted semantic information into text form. The decoder generates a text sequence step by step based on the speech features, and the order of the text is consistent with the time order of the audio sequence. This text sequence can be used for further natural language processing, such as intent recognition, entity extraction, etc.

[0114] By inputting compressed speech features into the decoder module, the system can directly generate the corresponding linearized text sequence from the speech data, avoiding information loss in the audio-to-text modal conversion process. The layer-by-layer decoding of the decoder based on the context relationship can capture important semantic information in the audio and present it in the form of natural language. Not only can it improve the accuracy of speech recognition, but it can also greatly improve the processing efficiency, especially in real-time speech recognition tasks.

[0115] In one embodiment, S50 comprises:

[0116] S501, input the linearized text sequence into a pre-trained recognition model;

[0117] S502, the recognition model processes the text sequence, detects and extracts entities belonging to predefined entity categories;

[0118] S503, analyze the recognized entities, determine the semantic relationships between the entities based on the context relationships between the entities, and assign a corresponding relationship category to each pair of entities;

[0119] S504, based on the relationship category between each pair of entities, generate triple information containing the first entity, the second entity, and the relationship between the first entity and the second entity.

[0120] In this embodiment, the previously generated linearized text sequence is input into a pre-trained entity recognition and relation classification model. The pre-trained model has typically been learned through a large amount of labeled data and is capable of identifying key entities and their relationships in the text. The input to the model is the text sequence, which is usually processed word by word, but can also be processed by characters or phrases.

[0121] The entity recognition (NER, named entity recognition) model scans the input text sequence, detects and extracts entities belonging to specific categories. Each entity category is predefined and typically covers common entity types in a specific domain. In this solution, entity categories include but are not limited to person names, locations, organizations, etc. The model determines whether each word belongs to a certain entity category based on contextual information, and if it meets the predefined category, the word is labeled as an entity.

[0122] Relation classification refers to analyzing the semantic relationship between entities after they have been identified. Through contextual relationships, the model infers the association between two entities and assigns a corresponding relationship category based on this association. The relationship classification model is usually based on contextual dependency information and a predefined set of relationships, and common relationship categories include "belongs to", "queries", "belongs to", etc. This step can be understood as the model inferring the logical or semantic association between entities from the text.

[0123] After entity recognition and relation classification are complete, the system organizes the generated entities and their relationships into triplets in the form of <entity1, relationship, entity2>. Each triplet consists of two entities and their semantic relationship, and these triplets are used to represent the core meaning of the text. The generated triplet information can be used in applications such as building knowledge graphs, automated analysis, question and answer systems, etc.

[0124] The system inputs the generated linearized text sequence into a pre-trained model. This model is usually a natural language processing model based on deep learning technology (such as BERT, RoBERTa, etc.), which can efficiently process text and extract semantic information. The model analyzes the input sequence word by word and labels and classifies the text according to predefined categories.

[0125] The recognition model detects and extracts entities that meet predefined categories by scanning the input text sequence. Each word is labeled as a specific category (such as person name, location, organization, etc.), and when a group of consecutive words is identified as the same category, the system extracts it as a complete entity. For example, in the sentence "Customer A queries account balance", "Customer A" and "account balance" are identified as two different categories of entities.

[0126] After entity recognition is complete, the model begins to analyze the relationships between the identified entities. By using contextual information, the model infers the semantic associations between two entities and categorizes them into certain relationship classes based on these associations. Relationship classification typically involves multiple categories, such as "query," "association," "belongs to," and so on. Relationship classification can help the system understand the specific connection between entities, such as the "query" relationship between "Customer A" and "account balance."

[0127] The results of entity recognition and relationship classification will be output in the form of triples. Each triple contains two entities and the relationship between them. The final generated triple structure is like <Entity1, Relationship, Entity2>, which represents the explicit relationship between the two entities. These triple information can be used for further natural language processing applications, such as knowledge graph construction or automated question answering.

[0128] This embodiment uses a pre-trained recognition model, so the system can automatically extract entities from text sequences and analyze the relationships between entities. Compared with traditional manual annotation or rule matching, the pre-trained model can more accurately and quickly extract key information and generate structured triple data. This is particularly important for scenarios that require processing large amounts of text data, especially in the financial, legal, and other fields. The system can automatically identify entities such as customers and institutions and infer their relationships, thereby significantly improving the efficiency and accuracy of data processing.

[0129] In one embodiment, after the triple information containing the first entity, the second entity, and the relationship between the first entity and the second entity is generated in S50, the method further includes:

[0130] S601, performing semantic consistency verification on the entities and relationships of each triple based on semantic verification rules;

[0131] S602, verifying the triple information based on structural integrity rules to confirm whether the entities and relationships in the triple information are complete;

[0132] S603, comparing the triple information with known entities and relationships in the knowledge graph, verifying whether the entities and relationships in the triple information are consistent with the knowledge graph based on knowledge graph rules, and recording the conflicting entities or relationships;

[0133] S604, modifying the triple information that is inconsistent with the knowledge graph according to the results of the knowledge graph consistency verification;

[0134] S605, formatting the triple information that passes the semantic consistency verification, structural integrity verification, and knowledge graph consistency verification to generate structured data.

[0135] In this embodiment, semantic consistency verification is checking whether the entities and relationships in the generated triples are semantically reasonable. Semantic verification rules are standards established based on linguistic or domain knowledge to judge whether entities and their relationships conform to real-world scenarios. For example, "bank" as an entity and "deposit" are reasonable, while "bank" and "color" may not conform to semantic logic. This step ensures that the semantic association of each triple is accurate and correct.

[0136] Structural integrity verification refers to checking whether the structure of triples is complete, i.e. each triple must contain two entities and their relationship. If a certain triple lacks an entity or relationship, it will be considered incomplete. The system checks the format of each triple through the structural integrity rule to ensure that there is no missing entity or relationship information.

[0137] Compare the generated triples with the existing knowledge graph to verify whether the entities and relationships in the triples match the known knowledge. The knowledge graph is a structured data representation that contains predefined rules of entities and their mutual relationships. If the entities or relationships in the triples are inconsistent with the known entities and relationships in the knowledge graph, the system will mark the conflict and record the problematic entity or relationship for subsequent processing.

[0138] Once the system detects that the triples are inconsistent with the information in the knowledge graph, it will correct the triples according to the existing rules in the knowledge graph. The correction process may include replacing incorrect entities, modifying incorrect relationships, or supplementing missing entities. This step ensures that the final generated triples conform to the established facts in the knowledge graph and maintain data consistency.

[0139] Format all triples information that passes the verification. Formatting will convert triples into standardized structured data formats such as XML, JSON or relational database format. The generated structured data can be further used for data analysis, knowledge graph construction or other application scenarios.

[0140] The system first checks the semantic consistency of each generated triple using predefined semantic verification rules. If an entity or relationship in a triple is semantically unreasonable, the system will mark the triple and either correct it or remove it. Next, the system checks the structural integrity of the triples, ensuring that each triple contains two entities and one relationship. If a triple is incomplete, the system will attempt to supplement the missing parts through context or mark it as an incomplete triple. The system compares the generated triples with the existing knowledge graph. By matching with known entities and relationships in the knowledge graph, the system can detect and record any conflicts. If an entity or relationship in a triple does not match the knowledge graph, the system will correct it according to the knowledge graph rules. For triples that are inconsistent with the knowledge graph, the system will correct them according to the information in the knowledge graph, ensuring that the entities and relationships in each triple conform to the established knowledge system. After correction, the system will re-verify the corrected triples to ensure their consistency with the knowledge graph. Triples that pass all verification steps will be converted into a standardized structured data format. Formatting processing includes organizing data into machine-readable and analyzable formats such as JSON or XML formats. These structured data can be used for subsequent analysis, knowledge graph updates, or other applications.

[0141] By performing semantic consistency, structural integrity, and knowledge graph consistency verification, the system can ensure that the generated triple data is accurate and reliable. Compared to traditional manual verification methods, automated verification and correction greatly improve processing efficiency and reduce errors. In addition, the generated structured data can be directly applied to the expansion and update of the knowledge graph, further enhancing the usability and consistency of the data.

[0142] In one embodiment, after S10, the above also includes:

[0143] S101, performing volume normalization processing on the audio data;

[0144] S102, filtering and removing background noise in the normalized audio data through a filter;

[0145] S103, detecting and deleting silent segments from the audio data.

[0146] In this embodiment, volume normalization refers to adjusting audio signals of different volumes to a unified standard, ensuring that the overall volume of the audio remains consistent during processing. This step can balance the volume of different parts of the audio, making it more stable and consistent in subsequent processing (such as feature extraction). By normalizing the volume of audio data, information loss or processing errors caused by excessively high or low volume can be avoided.

[0147] Noise filtering is an important step in audio processing. Audio data that has been volume-normalized can still contain background noise, such as environmental noise, mechanical noise, or other interference signals. By applying filters such as high-pass, low-pass, or band-pass filters, background noise in the audio can be effectively removed, preserving the main part of the audio signal and enhancing the clarity of the speech features. The choice and setting of the filter depend on the frequency range of the noise and the specific characteristics of the audio.

[0148] Silent segments refer to parts of the audio signal that do not contain valid information. Removing these silent segments can reduce the transmission of invalid information, save processing time and computing resources, and improve the efficiency of speech recognition. Silent segment detection is usually achieved by setting a threshold for the audio signal, that is, detecting parts with a volume below a certain threshold, and then cutting or deleting these parts. This step can ensure that subsequent processing steps focus on valid speech content.

[0149] First, the system volume-normalizes the acquired audio data. Volume normalization adjusts the overall volume of the audio signal to meet the pre-set standard range. This step can balance high and low volume parts, ensuring consistent overall volume of the audio. Common methods of normalization include RMS (Root Mean Square) value-based normalization or peak normalization. After volume normalization, the system filters the background noise in the audio data. Noise filtering can be achieved in various ways, including using different types of filters such as high-pass, low-pass, and band-pass. These filters selectively remove low or high frequency noise based on the frequency range of the noise, while preserving the main speech information, thereby improving the clarity and quality of the speech signal. The system then detects silent segments in the audio data to remove invalid information. Silent segment detection can be achieved by setting a volume threshold, when the audio signal is below the threshold, the part will be marked as a silent segment. After detecting silent segments, the system deletes these parts, retaining valid speech content. This operation can reduce processing time and data storage requirements, while improving the efficiency of speech recognition.

[0150] This embodiment significantly improves the processing quality of audio data through volume normalization, background noise removal, and silent segment deletion. Volume normalization ensures the stability of the audio signal, noise filtering enhances the clarity of the speech signal, and silent segment deletion reduces the interference of invalid information. By integrating these processing steps, the system can provide more accurate and clear audio data for subsequent speech feature extraction, speech recognition, and natural language processing.

[0151] In an embodiment, an entity and relationship joint extraction device is provided, which corresponds to the entity and relationship joint extraction method in the above embodiment. Referring to Figure 3 ,Figure 3 A functional module schematic diagram of a preferred embodiment of the joint extraction device of entities and relationships of the present application. The audio data acquisition module 10, the encoder module 20, the convolutional neural network module 30, the decoder module 40 and the semantic information extraction module 50. The detailed description of each functional module is as follows:

[0152] The audio data acquisition module 10 is used to acquire audio data, and the audio data contains speech information;

[0153] The encoder module 20 is used to input the audio data into the encoder module, and extract speech features from the audio data through the encoder module;

[0154] The convolutional neural network module 30 is used to compress and reduce the dimension of the speech features through the convolutional neural network, to generate compressed speech features;

[0155] The decoder module 40 is used to input the compressed speech features into the decoder module, to generate linearized text sequences;

[0156] The semantic information extraction module 50 is used to perform entity recognition operation and relationship classification operation on the text sequence through the pre-trained recognition model, to generate triple information containing the first entity, the second entity and the relationship between the first entity and the second entity.

[0157] In an embodiment, the encoder module 20 is specifically used for:

[0158] Segmenting the audio data according to a preset frame length to generate a plurality of audio segments;

[0159] Inputting the plurality of audio segments into the encoder module, and extracting time-frequency features of each audio segment through a feature extractor based on a convolutional neural network;

[0160] In the process of extracting the time-frequency features, the attention mechanism is used to enhance the feature extraction of key frequency bands and time periods;

[0161] The enhanced time-frequency features are globally modeled through a self-attention mechanism to capture the context dependency relationship between the audio segments, and to generate a final speech feature vector containing global timing information.

[0162] In an embodiment, the convolutional neural network module 30 is specifically used for:

[0163] Convolutionally processing the speech features through a plurality of convolutional operations;

[0164] After each convolutional operation, the speech features are reduced in dimension through a pooling operation to compress the feature dimension of the speech features;

[0165] The convolution operation of channel compression is used to compress the channels of the speech feature;

[0166] The speech feature vector after the multi-layer convolution operation, the pooling operation and the channel compression is processed by an activation function to generate a final speech feature vector.

[0167] In an embodiment, the decoder module 40 is specifically configured to:

[0168] input the compressed speech feature into the decoder module;

[0169] decode the speech feature layer by layer by the decoder module based on a recurrent neural network or a self-attention mechanism to generate an intermediate semantic representation corresponding to the speech feature;

[0170] The decoder module extracts semantic information from the intermediate semantic representation according to the context relationship of the speech feature;

[0171] convert the semantic information into a linearized text sequence.

[0172] In an embodiment, the semantic information extraction module 50 is specifically configured to:

[0173] input the linearized text sequence into a pre-trained recognition model;

[0174] The recognition model processes the text sequence, detects and extracts entities belonging to a pre-defined entity category;

[0175] analyze the recognized entities, determine the semantic relationship between the entities based on the context relationship between the entities, and assign a corresponding relationship category to each pair of entities;

[0176] generate triple information containing a first entity, a second entity and a relationship between the first entity and the second entity based on the relationship category between each pair of entities.

[0177] In an embodiment, the semantic information extraction module 50 is specifically configured to:

[0178] verify the semantic consistency of the entities and the relationship of each triple based on semantic verification rules;

[0179] verify the triple information based on the structural integrity rules to confirm whether the entities and the relationship in the triple information are complete;

[0180] compare the triple information with known entities and relationships in the knowledge graph, verify whether the entities and the relationship in the triple information are consistent with the knowledge graph based on the knowledge graph rules, and record the conflicting entities or relationships;

[0181] According to the result of the knowledge graph consistency verification, the triple information inconsistent with the knowledge graph is corrected.

[0182] The triple information passing the semantic consistency verification, the structure integrity verification and the knowledge graph consistency verification is formatted to generate structured data.

[0183] In an embodiment, the audio data acquisition module 10 is specifically configured to:

[0184] The audio data is subjected to volume normalization processing.

[0185] The background noise in the normalized audio data is filtered and removed through a filter.

[0186] Silent segments are detected and deleted from the audio data.

[0187] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in Figure 4 The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with an external user terminal through a network connection. The computer program is executed by the processor to implement the functions or steps of the entity and relationship joint extraction method on the server side.

[0188] In one embodiment, a computer device is provided, which can be a user terminal, and an internal structure diagram thereof can be as shown in Figure 5 The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with an external server through a network connection. The computer program is executed by the processor to implement the functions or steps of the entity and relationship joint extraction method on the user terminal side

[0189] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0190] obtaining audio data, wherein the audio data comprises speech information;

[0191] inputting the audio data into an encoder module, and extracting speech features from the audio data through the encoder module;

[0192] compressing and reducing dimensions of the speech features through a convolutional neural network to generate compressed speech features;

[0193] inputting the compressed speech features into the decoder module to generate a linearized text sequence;

[0194] performing entity recognition and relationship classification operations on the text sequence through a pre-trained recognition model to generate triple information comprising a first entity, a second entity, and a relationship between the first entity and the second entity.

[0195] In one embodiment, a computer readable storage medium is provided, which stores a computer program, wherein the computer program is executed by a processor to implement the following steps:

[0196] obtaining audio data, wherein the audio data comprises speech information;

[0197] inputting the audio data into an encoder module, and extracting speech features from the audio data through the encoder module;

[0198] compressing and reducing dimensions of the speech features through a convolutional neural network to generate compressed speech features;

[0199] inputting the compressed speech features into the decoder module to generate a linearized text sequence;

[0200] performing entity recognition and relationship classification operations on the text sequence through a pre-trained recognition model to generate triple information comprising a first entity, a second entity, and a relationship between the first entity and the second entity.

[0201] It should be noted that the functions or steps that the computer readable storage medium or the computer device can implement correspond to the descriptions of the server side and the user side in the foregoing method embodiments, and will not be described again here to avoid repetition.

[0202] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0203] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0204] It should be noted that if non-company software tools or components appear in the embodiments of the present application, they are only used for example introduction and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.​​

Claims

1. A method for joint extraction of entities and relations, characterized in that, The method comprises the following steps: obtaining audio data containing speech information; segmenting the audio data according to a preset frame length to generate a plurality of audio segments; inputting the plurality of audio segments into an encoder module, and extracting time-frequency features of each audio segment through a feature extractor based on a convolutional neural network; in the process of extracting the time-frequency features, using an attention mechanism to enhance feature extraction of key frequency bands and time periods; performing global modeling on the enhanced time-frequency features through a self-attention mechanism to capture context-dependent relationships between the audio segments and generate final speech feature vectors containing global timing information; compressing and reducing the dimensions of the speech feature vectors through a convolutional neural network to generate compressed speech feature vectors; inputting the compressed speech feature vectors into a decoder module to generate linearized text sequences; performing entity recognition and relationship classification operations on the text sequences through a pre-trained recognition model to generate triple information containing a first entity, a second entity, and a relationship between the first entity and the second entity; performing semantic consistency verification on the entities and relationships of each triple based on semantic verification rules; verifying the triple information based on structural integrity rules to confirm whether the entities and relationships in the triple information are complete; comparing the triple information with known entities and relationships in a knowledge graph, verifying whether the entities and relationships in the triple information are consistent with the knowledge graph based on knowledge graph rules, and recording conflicting entities or relationships; modifying the triple information inconsistent with the knowledge graph according to the results of the knowledge graph consistency verification; formatting the triple information verified through semantic consistency verification, structural integrity verification, and knowledge graph consistency verification to generate structured data.

2. The method of claim 1, wherein, The compression and dimension reduction of the speech feature vectors through the convolutional neural network to generate the compressed speech feature vectors comprise: performing convolution processing on the speech feature vectors through multi-layer convolution operations; after each convolution operation, performing dimension reduction processing on the speech feature vectors in combination with a pooling operation to compress the feature dimensions of the speech feature vectors; performing channel compression on the channels of the speech feature vectors through channel compression convolution operations; performing activation function processing on the speech feature vectors after the multi-layer convolution operations, the pooling operations, and the channel compression to generate the final speech feature vectors.

3. The method of claim 1, wherein the method comprises: The inputting of the compressed speech feature vectors into the decoder module to generate linearized text sequences comprises: inputting the compressed speech feature vectors into the decoder module; performing layer-by-layer decoding on the speech feature vectors through a decoder module based on a recurrent neural network or a self-attention mechanism to generate intermediate semantic representations corresponding to the speech feature vectors; the decoder module extracts semantic information from the intermediate semantic representations according to the context relationships of the speech feature vectors; the semantic information is converted into linearized text sequences.

4. The method of claim 1, wherein, The entity recognition and relationship classification operations on the text sequences through the pre-trained recognition model to generate triple information containing a first entity, a second entity, and a relationship between the first entity and the second entity comprise: inputting the linearized text sequence into a pre-trained recognition model; the recognition model processes the text sequence, detects and extracts entities belonging to predefined entity categories; analyzing the recognized entities, determining semantic relationships between entities based on contextual relationships between entities, and assigning each pair of entities a corresponding relationship category; based on the relationship category between each pair of entities, generating triple information containing the first entity, the second entity, and the relationship between the first entity and the second entity.

5. The method of claim 1, wherein, After obtaining the audio data, it further includes: volume normalization processing on the audio data; filtering and removing background noise in the normalized audio data through a filter; detecting and deleting silent segments from the audio data.

6. An entity and relation joint extraction apparatus characterized by comprising: The joint extraction device of entities and relationships includes: an audio data acquisition module for acquiring audio data containing voice information; an encoder module for segmenting the audio data into multiple audio segments according to a preset frame length, inputting the multiple audio segments into the encoder module, extracting time-frequency features of each audio segment through a feature extractor based on a convolutional neural network, enhancing feature extraction of key frequency bands and time periods using an attention mechanism during the extraction of time-frequency features, and generating a final speech feature vector containing global timing information through a self-attention mechanism to capture contextual dependency between audio segments; a convolutional neural network module for compressing and reducing the dimensions of the speech feature vector through a convolutional neural network to generate a compressed speech feature vector; a decoder module for inputting the compressed speech feature vector into the decoder module to generate a linearized text sequence; a semantic information extraction module for performing entity recognition and relationship classification operations on the text sequence through a pre-trained recognition model to generate triple information containing a first entity, a second entity, and a relationship between the first entity and the second entity, performing semantic consistency verification on the entities and relationships of each triple based on semantic verification rules, verifying the triple information based on structure integrity rules to confirm whether the entities and relationships in the triple information are complete, comparing the triple information with known entities and relationships in a knowledge graph, verifying whether the entities and relationships in the triple information are consistent with the knowledge graph based on knowledge graph rules, and recording conflicting entities or relationships, modifying triple information inconsistent with the knowledge graph based on the results of the knowledge graph consistency verification, and formatting triple information that passes semantic consistency verification, structure integrity verification, and knowledge graph consistency verification to generate structured data.

7. A computer device, comprising: The computer device includes a memory, a processor, and an entity and relationship joint extraction program stored on the memory and executable on the processor, which implements the steps of the entity and relationship joint extraction method of any one of claims 1-5 when executed by the processor.

8. A computer-readable storage medium, characterized in that, The storage medium has stored thereon a joint extraction program of entities and relations, the joint extraction program of entities and relations, when executed by the processor, implements the steps of the joint extraction method of entities and relations in any one of claims 1-5.

Images