Entity and relation joint extraction method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses an entity and relation joint extraction method, which comprises the following steps of: acquiring audio data, extracting voice features, compressing and reducing dimensions of the voice features, and decoding the compressed voice features to generate a linearized text sequence; and performing entity recognition and relationship classification on the text sequence by using a pre-trained recognition model to generate triple information containing entities and relationships. According to the method, the intermediate conversion step from the audio to the text is omitted, the semantic information is directly extracted from the audio data, information loss possibly occurring in the modal conversion process is reduced, and the processing efficiency and accuracy are improved. Besides, entity recognition and relation classification are carried out on the text sequence through the pre-trained recognition model, it is ensured that key information can be efficiently extracted from a large amount of audio data under the condition of low cost, and combined recognition of entities and relations is achieved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a method, device, equipment and storage medium for jointly extracting entities and relationships. Background Art

[0002] With the rapid development of the financial industry, customer service systems have become one of the core channels for banks to interact with their customers. Especially in large financial institutions, telephone customer service systems have accumulated a wealth of audio data through a large number of voice interactions. These audio data contain key business information between customers and banks. In-depth mining and analysis of these data has important commercial value, especially in terms of entity recognition and relationship extraction, which can provide direct support for customer management, business optimization, etc.

[0003] In the existing technology, audio data is usually converted into text through automatic speech recognition (ASR) technology, and then the text data is processed through text information extraction technology to identify entities and their relationships. Although this processing flow can effectively obtain semantic information from text, it has the following significant shortcomings:

[0004] ASR technology is expensive, especially when dealing with large amounts of audio data, and the process of transcribing text is both resource-intensive and time-consuming. This approach is not efficient for large-scale financial speech data.

[0005] In the process of audio-to-text conversion, information loss or inaccuracy often occurs, especially the unique emotions, intonation and background information in speech are difficult to fully express through text. This lost information may have an adverse effect on subsequent entity recognition and relationship extraction.

[0006] Audio data and text data are two different modalities. Existing technologies require cross-modal processing, that is, converting audio to text and then mining information from the text. This modality conversion increases the complexity of processing and easily introduces noise and errors during the conversion between different modalities, resulting in a decrease in recognition accuracy. Summary of the invention

[0007] The main purpose of the present invention is to provide a method, device, equipment and storage medium for the joint extraction of entities and relationships, aiming to solve the technical problem in the prior art that when audio data is converted into text through automatic speech recognition, key information in the speech is often lost, resulting in a decrease in the accuracy of entity recognition and relationship extraction.

[0008] To achieve the above object, the present invention provides a method for jointly extracting entities and relations, comprising:

[0009] Acquire audio data, wherein the audio data includes voice information;

[0010] Inputting the audio data into an encoder module, and extracting speech features from the audio data by the encoder module;

[0011] Compressing and reducing the dimension of the speech features through a convolutional neural network to generate compressed speech features;

[0012] Inputting the compressed speech features into the decoder module to generate a linearized text sequence;

[0013] The pre-trained recognition model is used to perform entity recognition and relationship classification operations on the text sequence to generate triple information including a first entity, a second entity, and a relationship between the first entity and the second entity.

[0014] Furthermore, to achieve the above object, the present invention provides a device for jointly extracting entities and relations, comprising:

[0015] An audio data acquisition module, used to acquire audio data, wherein the audio data includes voice information;

[0016] An encoder module, configured to input the audio data into an encoder module, and extract speech features from the audio data through the encoder module;

[0017] A convolutional neural network module, used for compressing and reducing the dimension of the speech features through a convolutional neural network to generate compressed speech features;

[0018] A decoder module, used for inputting the compressed speech features into the decoder module to generate a linearized text sequence;

[0019] The semantic information extraction module is used to perform entity recognition operations and relationship classification operations on the text sequence through a pre-trained recognition model to generate triple information including a first entity, a second entity, and a relationship between the first entity and the second entity.

[0020] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a program for joint extraction of entities and relationships stored in the memory and executable on the processor, wherein the program for joint extraction of entities and relationships, when executed by the processor, implements the steps of the method for joint extraction of entities and relationships as described above.

[0021] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a program for joint extraction of entities and relationships is stored, and when the program for joint extraction of entities and relationships is executed by a processor, the steps of the method for joint extraction of entities and relationships as described above are implemented.

[0022] Beneficial effects: The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a method for the joint extraction of entities and relationships, which obtains audio data, extracts speech features, compresses and reduces the dimension of the speech features, decodes the compressed speech features to generate a linearized text sequence, and uses a pre-trained recognition model to perform entity recognition and relationship classification on the text sequence to generate triple information containing entities and relationships. The present invention omits the intermediate conversion step from audio to text, directly extracts semantic information from audio data, reduces information loss that may occur during modal conversion, and improves processing efficiency and accuracy. In addition, by performing entity recognition and relationship classification on text sequences through a pre-trained recognition model, it ensures that key information can be efficiently extracted from a large amount of audio data under low-cost conditions, thereby achieving joint recognition of entities and relationships. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0024] Figure 1 A schematic diagram of an application environment of a method for jointly extracting entities and relationships in an embodiment of the present invention;

[0025] Figure 2 It is a flowchart of an embodiment of a method for joint extraction of entities and relations of the present invention;

[0026] Figure 3 It is a functional module diagram of a preferred embodiment of the joint extraction device of entities and relations of the present invention;

[0027] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0028] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0029] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0030] The entity and relationship joint extraction method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, the user end communicates with the server end through a network. The server end can obtain audio data through the user end, extract voice features, compress and reduce the dimension of the voice features, decode the compressed voice features to generate a linearized text sequence, use a pre-trained recognition model to perform entity recognition and relationship classification on the text sequence, and generate triple information containing entities and relationships. The present invention omits the intermediate conversion step from audio to text, directly extracts semantic information from audio data, reduces information loss that may occur during modal conversion, and improves processing efficiency and accuracy. In addition, the text sequence is subjected to entity recognition and relationship classification by a pre-trained recognition model, ensuring that key information can be efficiently extracted from a large amount of audio data under low-cost conditions, and realizing the joint recognition of entities and relationships. Among them, the user end can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server end can be implemented by an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0031] See also Figure 2 , Figure 2 This is a flowchart of an embodiment of a method for joint extraction of entities and relations provided by the present invention. It should be noted that although a logical sequence is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0032] like Figure 2 As shown, the entity and relationship joint extraction method proposed in the present invention includes the following steps:

[0033] S10, acquiring audio data, where the audio data includes voice information;

[0034] In this embodiment, audio data acquisition is the starting point of the entire system processing flow, which refers to obtaining raw audio data from an external input source (such as a telephone system, a recording device, or other audio capture device). These data may include audio files in various formats (such as .wav, .mp3, etc.), and the audio quality needs to be high enough for subsequent processing. These audio data mainly contain voice information interacting with customers, which may include customer questions, inquiries, feedback, etc.

[0035] The sources of audio data can be diverse, mainly including call recordings between customers and financial institutions, telephone customer service interaction data, etc. In financial applications, audio data often comes from the bank's customer service system, including key information such as customer consultation on business, account information query, and transaction behavior confirmation.

[0036] The so-called voice information refers to the customer's voice and its corresponding semantic content contained in the audio data. For example, in the financial field, the customer's voice data may contain information such as account balance inquiry, transfer operation, loan consultation, etc. These voice information will be converted into semantic features in the subsequent steps for further analysis and processing.

[0037] The system needs to access an external audio data source, which can be achieved by automatically receiving customer service recordings through the telephone system, or by manually uploading recording files for processing. The collected audio data may be stored in a high-quality audio format, such as a .wav file with a 16kHz sampling rate, to ensure the clarity of the sound and facilitate subsequent feature extraction and analysis.

[0038] The acquired audio data is transmitted to the central processing unit of the system through a network interface, and can be saved using a distributed storage system. The storage system needs to support audio files in multiple formats and can receive and store data in real time to ensure that the interaction between the customer and the financial institution is not lost.

[0039] Example: In the financial sector, a bank's telephone customer service system handles thousands of customer requests every day. For example, a customer may inquire about an account balance or apply for a loan over the phone. Obtaining the audio data of these calls can provide important basic data for the bank's subsequent customer analysis and customer relationship management. Through the automated system, the bank can collect customers' voice interaction data in real time, and conduct in-depth analysis and processing in subsequent steps to identify customer needs and business relationships, and improve service quality and customer satisfaction.

[0040] Automated audio data capture can effectively collect information about interactions between customers and financial institutions. Compared with manual or text recording methods, direct audio data capture can retain the full context of customer interactions and ensure the integrity and accuracy of the information.

[0041] S20, inputting the audio data into an encoder module, and extracting speech features from the audio data through the encoder module;

[0042] In this embodiment, the main task of the encoder module is to convert the input audio data into feature vectors for subsequent processing. The module generates speech features containing rich information by segmenting and extracting features from the audio data. These features capture the temporal changes in the audio and the content information of the speech.

[0043] Speech features are the core information representation in audio data, including two key information:

[0044] Timing information: refers to the characteristics of audio data that change over time, which are manifested as fluctuations in speech signals and changes in frequency. This information is crucial for capturing elements such as speech rhythm, speaking speed, and pronunciation time.

[0045] Content information: refers to the actual semantic content of the speech in the audio. Content information can reflect the specific meaning of the speaker's speech, such as the customer's request or the business problem stated.

[0046] The encoder module is usually based on deep learning technology, such as Wav2Vec 2.0, Transformer encoder, etc. Its main function is to characterize the original audio data and extract high-dimensional speech feature representation. Through the learned speech feature model, this module can automatically capture the temporal dynamics and content in the audio signal.

[0047] The encoder module not only extracts the temporal structure of speech, but also captures the semantic content of speech expression from the signal. This information is crucial for subsequent speech processing and semantic extraction. The timing information provides the structural basis of the speech signal, while the content information is related to the actual semantic expression.

[0048] The audio data is fed into the encoder module in frames. Usually, the audio data has been segmented in the preprocessing stage, so when it is fed into the encoder, each audio segment represents a part of the complete speech information. These audio segments are fed into the encoder in chronological order for processing.

[0049] The encoder extracts features from the audio clip layer by layer through convolutional layers or other neural network structures. Convolutional neural networks can capture the frequency characteristics of speech, while encoders based on self-attention mechanisms (such as Transformer) can effectively capture long-distance dependencies in time series, thereby extracting timing information.

[0050] The final output of the encoder is a multi-dimensional feature vector that contains the timing information and content information in the audio. This feature vector can be used for subsequent tasks such as decoding or classification. These feature representations not only include the physical properties of the voice, such as pitch and speaking speed, but also the actual semantic content of the customer's voice.

[0051] Example: In the financial field, when a customer interacts with a bank customer service representative over the phone, the encoder module extracts speech features from the call recording. For example, when a customer asks, "What's my account balance?", the encoder module not only captures the temporal changes in the speech (such as the customer's speaking speed and tone), but also extracts semantic content (such as keywords such as "account balance" and "query") through a deep learning model. In this way, the system can quickly understand the customer's intentions and provide basic support for subsequent semantic analysis and business processing.

[0052] By inputting audio data into the encoder module and extracting speech features, the system can effectively capture the timing changes and speech content in the audio data. This not only improves the representation ability of speech signals, but also provides high-quality feature input for subsequent decoding and semantic processing. Compared with traditional manual feature extraction methods, the encoder module can automatically, quickly and accurately generate complex speech feature vectors, greatly reducing errors caused by human intervention and improving processing efficiency.

[0053] S30, compressing and reducing the dimension of the speech feature through a convolutional neural network to generate compressed speech features;

[0054] In this embodiment, convolutional neural networks (CNNs) are widely used to process data with spatial or temporal structures such as images and audio. In this technology, CNNs are used to further process the speech features output by the encoder module. Through multiple convolutional layers, convolutional neural networks extract local patterns in feature representations, compress the spatial dimensions of speech features, and retain key temporal and semantic information.

[0055] Feature extraction of audio data usually generates high-dimensional feature vectors, which contain rich information. However, in actual processing and calculation, too high a dimension will increase the computational complexity and introduce redundant information. Compression and dimensionality reduction through convolutional neural networks can reduce the dimension of feature vectors, maintain the most important speech features, reduce redundant information, and thus improve computational efficiency.

[0056] In convolutional neural networks, convolution operations can extract feature patterns from local areas, while pooling operations are used to downsample the extracted features to further compress the data dimension. Through the superposition of multiple layers of convolution and pooling, compressed speech features are finally generated. These features still contain the core information in the audio, such as semantic content and timing patterns.

[0057] The dimension of the speech feature vector after being processed by the convolutional neural network has been greatly reduced. These compressed features not only have strong expressive power, but also provide more efficient input for subsequent decoding and semantic processing. The compressed speech features retain the most important information in the original speech signal, such as pronunciation characteristics, pitch changes, etc., which is convenient for subsequent module processing.

[0058] Each convolution layer is responsible for extracting different speech features, such as frequency information, time information, etc. The convolution operation slides on the feature map by moving the window to extract local patterns (such as pitch, syllable changes, etc.). The size and number of convolution kernels determine the accuracy and range of feature extraction.

[0059] After the convolution layer, the pooling operation is usually used to reduce the dimension of the feature. The pooling operation uses the maximum value or average value of the local area as a new feature representation according to the set pooling window, thereby reducing the dimension of the data. The pooling operation can effectively reduce the redundant information of the feature vector while retaining the global key information.

[0060] Multi-layer convolutional networks allow the system to extract information from different feature levels. The initial layer may capture simple speech patterns (such as pitch changes), while deeper convolutional networks can extract more complex semantic features. By stacking multiple layers of convolutional networks, the system can achieve efficient compression of speech features and retain key information.

[0061] Example: In the financial sector, when a bank's customer service system receives a large number of call recordings, convolutional neural networks can be used to compress and reduce the dimensionality of these audio features. For example, when a customer calls a bank hotline to inquire about loan information, the system first extracts frequency changes, pitch features, and key speech patterns in the voice through the convolution layer, and then reduces the dimensionality of these features through the pooling layer for rapid processing and analysis. In this way, banks can efficiently analyze large amounts of customer voice data, accurately identify customer intent, improve service quality, and support business decisions.

[0062] Compressing and reducing the dimension of speech features through convolutional neural networks can effectively reduce redundant information, retain key features in speech signals, and reduce computational complexity. Compared with unprocessed high-dimensional features, compressed features processed by convolutional networks can not only increase processing speed, but also improve the recognition accuracy of subsequent models.

[0063] S40, inputting the compressed speech features into the decoder module to generate a linearized text sequence;

[0064] In this embodiment, after the convolutional neural network processing, the speech features have been compressed and the key timing and content information has been retained. The compressed speech feature vector contains the core semantic expression of the speech, which is convenient for subsequent decoding processing. These features can be represented as low-dimensional feature vectors, which condense the key information in the audio, such as pitch, speech speed, semantics, etc.

[0065] The decoder module is responsible for converting the input speech features into text output. This step usually involves gradually decoding the speech features into text representations through some kind of neural network structure (such as a decoder based on a recurrent neural network (RNN), a long short-term memory network (LSTM), or a self-attention mechanism). The task of the decoder is to generate text outputs corresponding to the speech features, forming a linearized text sequence.

[0066] The linearized text sequence refers to the decoded text information arranged in chronological order, usually a set of strings or symbols, which sequentially represent the semantic content expressed in the audio. This text sequence is the final semantic representation, which contains the complete semantic content expressed in the original audio data and converts it into a text representation that can be further processed by the machine.

[0067] The decoder inputs the compressed speech features and gradually generates the corresponding text sequence according to a specific model architecture. This process involves capturing the contextual relationship in the features and generating the optimal semantic match through the decoder's prediction mechanism. The decoder generates a character or word at each step, gradually building a complete linearized text sequence.

[0068] The compressed speech feature vector is first used as the input of the decoder. In the neural network decoder, these feature vectors are processed in the order of the time series. Through each step of decoding, the model generates the corresponding text symbol or word. The decoder module uses a recurrent network (such as LSTM or GRU) or a self-attention mechanism (such as Transformer) to gradually decode the input features. For each time step, the decoder generates a word or character based on the current feature input and combines it with the previous context until a complete text sequence is generated. Throughout the decoding process, the decoder outputs a gradually growing text sequence. This text sequence is linearized, indicating that the speech content in the audio data is arranged in chronological order. For example, from "account balance" to "query successful", the system gradually generates complete sentences and can be directly used for subsequent semantic analysis or display.

[0069] By inputting the compressed speech features into the decoder module and generating a linearized text sequence, the system can directly generate the corresponding text output from the audio data, avoiding the information loss that may occur during the audio-to-text modality conversion process. Compared with the traditional ASR system, this decoding method can more accurately maintain the semantic information in the speech when generating text, while reducing the complexity of processing and improving overall efficiency.

[0070] S50, performing entity recognition operations and relationship classification operations on the text sequence through a pre-trained recognition model to generate triple information including a first entity, a second entity, and a relationship between the first entity and the second entity.

[0071] In this embodiment, the pre-trained recognition model is a natural language processing model based on deep learning, which is trained with a large amount of data and can recognize entities in text sequences and their relationships. This model is usually pre-trained with data from a specific field and can effectively process text data in the fields of finance, customer service, etc. The main task of the model is to automatically identify key information from the input linearized text sequence.

[0072] Entity recognition (NER) is the process of detecting and extracting words or phrases with specific meanings from text, called entities. Entities can be people, organizations, places, events, etc. The task of entity recognition is to identify these specific entities from text sequences. For example, in customer interactions in the financial field, key entities such as "customer name", "account number", and "transaction amount" may be involved.

[0073] Relation classification refers to identifying semantic connections between entities. The relationship classification module analyzes the identified entities and infers specific relationships between these entities. For example, in a customer query, the system identifies "customer" and "account", and the relationship classification model can infer the "query" relationship between the two. Relation classification can help the system understand the contextual relationship between entities, such as "customer-query-account balance".

[0074] Through entity recognition and relationship classification operations, the system can generate a semantic structure in the form of triples. The format of a triple is <entity 1, relationship, entity 2>, which represents the specific relationship between entity 1 and entity 2. For example, in a customer interaction text, "Customer A" and "Account X" are identified, and the relationship between them is "query", then the generated triple information can be <Customer A, query, account X>.

[0075] The pre-trained recognition model first processes the linearized text sequence word by word to identify entities with specific meanings. This can be done by marking whether each word or phrase in the text belongs to a predefined entity category. The model determines whether the word is a person's name, a place, or other specific entity type through semantic context. After entity recognition is completed, the system infers the relationship between the identified entities. The relationship classification operation is based on the contextual connection between the identified entities. Through the relationship classification model, the semantic connection between entity pairs is clarified. The system analyzes the adjacent words or context information of these entities to determine the relationship category between them (such as "belonging", "query", "affiliation", etc.). After entity recognition and relationship classification are completed, the system organizes the generated entities and the relationships between them into triples. Each triple consists of two entities and the relationship between them, representing the core semantic information of the text. The generated triples can be used for subsequent knowledge graph construction, information retrieval or automated analysis.

[0076] Example description: In an actual application scenario in the financial field, when a customer inquires "I want to check my account balance" by phone, the system generates a linearized text sequence through the decoder, and the pre-trained recognition model can automatically identify key entities such as "Customer A" and "Account Balance". Then, the relationship classification model further infers the "query" relationship between the two entities. Finally, the triple information generated by the system is <Customer A, Query, Account Balance>. This information can be directly used to process customer account query requests in the banking system, realizing the automation and intelligence of customer service.

[0077] Using pre-trained recognition models to perform entity recognition and relationship classification operations can greatly improve the efficiency of the system in extracting key information from text. Compared with traditional manual analysis or rule matching methods, pre-trained models can automatically extract important entities and their relationships from text, reducing manual intervention while improving recognition accuracy and consistency.

[0078] The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a method for jointly extracting entities and relationships, which obtains audio data, extracts speech features, compresses and reduces the dimension of the speech features, decodes the compressed speech features to generate a linearized text sequence, and uses a pre-trained recognition model to perform entity recognition and relationship classification on the text sequence to generate triple information containing entities and relationships. The present invention omits the intermediate conversion step from audio to text, directly extracts semantic information from audio data, reduces information loss that may occur during modal conversion, and improves processing efficiency and accuracy. In addition, by performing entity recognition and relationship classification on text sequences through a pre-trained recognition model, it is ensured that key information can be efficiently extracted from a large amount of audio data under low-cost conditions, and the joint recognition of entities and relationships is achieved.

[0079] In one embodiment, in the above S20, the audio data is input into an encoder module, and the speech features are extracted from the audio data by the encoder module, including:

[0080] S201, segmenting the audio data according to a preset frame length to generate multiple audio segments;

[0081] S202, inputting multiple audio clips into an encoder module, and extracting time-frequency features of each audio clip through a feature extractor based on a convolutional neural network;

[0082] S203, in the process of extracting time-frequency features, using the attention mechanism to enhance feature extraction of key frequency bands and time periods;

[0083] S204, globally modeling the enhanced time-frequency features through the self-attention mechanism, capturing the contextual dependencies between audio clips, and generating a final speech feature vector containing global timing information.

[0084] In this embodiment, the audio data is a continuous signal. For the convenience of processing, it is usually divided into multiple segments according to fixed time intervals (frames). Each segment represents a small part of the audio data and is used to capture voice information in a short period of time. The selection of frame length depends on the processing requirements and is usually set to 20ms to 50ms. Each segmented segment can be independently subjected to subsequent feature extraction.

[0085] The audio clip is used as input data to the Convolutional Neural Network (CNN) feature extractor in the encoder module. Convolutional neural networks are suitable for processing data with time series or spatial structures, such as frequency and timing information in audio. The feature extractor extracts time-frequency features for each audio clip, mainly including:

[0086] Frequency response: refers to the performance of speech signals at different frequencies, reflecting the pitch, pronunciation characteristics, etc.

[0087] Energy change: represents the intensity change of speech signals, can distinguish between high-pitched and low-pitched passages, and capture the volume and pitch changes of speech.

[0088] Time series pattern: contains the changes in speech over time, which can reflect information such as speaking speed and pauses.

[0089] The attention mechanism is a technology that can focus on important parts of the data. In speech feature extraction, the attention mechanism can help the model focus on specific frequencies and time periods in the audio clip to obtain more important speech information. By enhancing the feature extraction of these key areas, the model can more accurately capture important semantic or sound changes and avoid interference from redundant information.

[0090] The self-attention mechanism is a method that can capture global dependencies in sequence data. It can use all the information of the context, not just the local information, when processing the current audio segment. In this way, the model can learn the complex associations between audio segments and capture richer temporal dependencies, thereby generating speech feature vectors that contain global temporal information. These feature vectors represent the complete content of the audio segment and retain the semantic information in the speech.

[0091] The input audio data is first segmented according to the preset frame length, and each audio segment represents a speech signal in a short period of time. The frame length can be adjusted according to actual needs, and the typical frame length is 20ms or 25ms. Each audio segment is input into the convolutional neural network, and the convolution layer extracts time-frequency features from each segment. The convolution operation can capture features such as frequency, energy, and timing changes. As the convolution layer deepens, the network gradually extracts more complex speech patterns and generates a set of feature vectors for further processing. In the process of extracting time-frequency features, the system applies an attention mechanism to enhance the focus on key frequency bands and time periods. The attention mechanism enables the network to focus on the highly relevant parts of the speech signal, such as the speaker's key pronunciation paragraphs, through weight distribution. The extracted time-frequency features are processed by the self-attention mechanism to capture the contextual dependencies in the entire sequence of audio segments. The self-attention mechanism can identify the connection between audio segments and generate global speech features. These global features represent the temporal structure and content of the entire audio segment and are used for subsequent decoding or semantic analysis.

[0092] This embodiment processes audio data in segments and uses convolutional neural networks to extract time-frequency features, so that the system can quickly capture key frequency and timing information in the audio. Combining the attention mechanism and the self-attention mechanism, the system's ability to capture key information in the audio signal is further enhanced, and global modeling is achieved. Compared with the traditional processing method that only uses convolutional neural networks, this method can not only accurately capture speech features, but also make full use of the global context information between audio segments, thereby generating a more accurate and expressive speech feature representation.

[0093] In one embodiment, the above S30 includes:

[0094] S301, performing convolution processing on the speech feature through multi-layer convolution operations;

[0095] S302, after each convolution operation, performing dimensionality reduction processing on the speech feature in combination with a pooling operation to compress the feature dimension of the speech feature;

[0096] S303, compressing the channel of the speech feature by using a channel compression convolution operation;

[0097] S304, performing activation function processing on the speech feature vector after multi-layer convolution operation, pooling operation and channel compression to generate a final speech feature vector.

[0098] In this embodiment, the convolutional layer in the convolutional neural network (CNN) is used to extract local features. Through multi-layer convolution operations, the system gradually extracts different levels of features in the speech signal, including pitch, volume, speech rate and other information. Each layer of convolution operation extracts feature representations of different dimensions from the input speech features. Multi-layer convolution can gradually focus on the complex patterns in the speech signal and improve the ability to express the speech content.

[0099] After the convolution operation, the system uses pooling operations (such as maximum pooling or average pooling) to downsample the convolutional features. The purpose of the pooling operation is to reduce the feature dimension while retaining the most important information. Pooling reduces the computational complexity and the size of the input data by reducing the dimensionality of the convolutional feature map, and ensures that local information is retained while discarding redundant information.

[0100] Each layer of convolution operation generates multiple feature channels (feature maps), representing different feature types. Through channel compression technology, the system further compresses the feature maps generated by convolution, reducing the number of channels and thus reducing the network computing cost. The purpose of channel compression is to further compress the size of speech features and reduce redundant data while keeping the core features unchanged. This method is suitable for processing larger-scale audio feature maps.

[0101] After convolution, pooling, and channel compression, the system applies an activation function (such as ReLU or Leaky ReLU) to the processed feature vector. The activation function is a nonlinear operation in the neural network, which can introduce nonlinear factors so that the model can better represent complex speech features. The speech feature vector processed by the activation function is the final feature representation after multiple layers of processing, ready for subsequent decoding or further analysis.

[0102] After the speech features are processed through multiple layers of convolution, each convolution extracts higher-order features from the features of the previous layer. The initial layer may extract simple frequency and energy information, while deeper convolutions can extract complex speech patterns, such as specific speech units or semantic associations. This process makes the speech features more compact and suitable for subsequent dimensionality reduction. After each convolution operation, the pooling operation downsamples the feature map. The pooling operation reduces the size of the feature map by retaining the maximum value or the average value, thereby reducing the complexity and computation of the network. The feature representation after pooling is more compact, but still retains the key information in the speech signal. Channel compression reduces the number of channels by convolving multiple channels of the feature map. This can reduce the size of the feature map while retaining the most important speech features. The convolution operation of channel compression can fuse the information of different channels together to form a compact representation, which is suitable for scenarios where large-scale data is processed. The last step is to input the processed feature vector into the activation function. Common activation functions such as ReLU can process negative numbers, retain positive information, and introduce nonlinearity. This step helps the model handle complex nonlinear data and enables the generated feature vectors to better represent complex relationships in speech.

[0103] In this embodiment, through multi-layer convolution operation, pooling operation and channel compression technology, the system can effectively compress the dimension of speech features, retain key information, and reduce computational complexity. It can not only improve processing efficiency, but also ensure the expressiveness of speech features, while reducing feature redundancy and retaining the core features of speech. The feature vector finally generated is more compact and has a strong semantic expression ability, which is suitable for subsequent decoding or semantic analysis.

[0104] In one embodiment, the above S40 includes:

[0105] S401, inputting the compressed speech features into a decoder module;

[0106] S402, decoding the speech features layer by layer through a decoder module based on a recurrent neural network or a self-attention mechanism to generate an intermediate semantic representation corresponding to the speech features;

[0107] S403, the decoder module extracts semantic information from the intermediate semantic representation according to the contextual relationship of the speech feature;

[0108] S404: Convert the semantic information into a linearized text sequence.

[0109] In this embodiment, after being processed by the convolutional neural network, the speech features have been compressed and reduced in dimension, and these feature vectors need to be further decoded into text sequences. The task of the decoder module is to gradually convert these speech features into linearized text representations. This step is the key part of speech-to-text conversion, which is responsible for converting complex feature vectors into natural language text.

[0110] The decoder can be implemented based on different structures, the most common of which are recurrent neural networks (RNNs) and self-attention mechanisms. RNNs are suitable for processing sequence data because they can maintain the temporal dependencies of the input sequence. By decoding step by step, the decoder extracts the corresponding semantic information from each time step. Self-attention mechanisms (such as Transformers) can simultaneously focus on the dependencies of different parts based on the entire sequence, enhancing the accuracy of semantic extraction. Layer-by-layer decoding means that the decoder processes each layer of the feature vector in turn to generate the corresponding intermediate semantic representation.

[0111] In the process of layer-by-layer decoding, the decoder not only relies on a single speech feature, but also combines contextual information. Through the contextual relationship, the decoder can better capture the dependencies between speech segments and extract more complete semantic information. The contextual relationship can be the connection between different syllables and words in the speech, which is captured and analyzed by the attention mechanism or recursive structure in the decoder.

[0112] The final output of the decoder is a linearized text sequence. This text sequence is a semantic expression arranged in time order, usually a sentence or phrase in natural language. Based on the extracted semantic information, the decoder generates a set of text symbols or words arranged in order to represent the meaning of each speech segment in the audio. This linearized text sequence can be directly used for further text processing or display.

[0113] The system inputs the convolutional and compressed speech feature vectors into the decoder module. The order of speech features input is determined by the time axis, that is, each feature vector represents a certain segment in the speech sequence. The decoder receives these compressed features and starts decoding layer by layer. The decoder module can be based on RNN or Transformer architecture. For RNN, the decoder processes each input feature in time step and generates the corresponding intermediate semantic representation. For the Transformer-based decoder, the system can process speech features in parallel and combine the contextual information in the entire sequence through the self-attention mechanism to generate intermediate semantic representations. These intermediate semantic representations contain a preliminary understanding of the input speech features. The decoder module further processes the intermediate semantic representation and extracts the semantics of the entire speech sequence in combination with the contextual relationship in the input speech features. The system uses recursive mechanisms or attention mechanisms to identify the dependencies between different speech segments to ensure that the generated semantic information can reflect the complete audio content. The last step is to convert the extracted semantic information into text form. The decoder gradually generates a text sequence based on the speech features, and the order of the text is consistent with the time order of the audio sequence. This text sequence can be used for further natural language processing, such as intent recognition, entity extraction, etc.

[0114] In this embodiment, by inputting the compressed speech features into the decoder module, the system can directly generate the corresponding linearized text sequence from the speech data, avoiding the information loss in the audio-to-text mode conversion process. The decoder can capture the important semantic information in the audio and present it in the form of natural language based on the layer-by-layer decoding of the context relationship. It can not only improve the accuracy of speech recognition, but also greatly improve the processing efficiency, especially in real-time speech recognition tasks.

[0115] In one embodiment, the above S50 includes:

[0116] S501, inputting the linearized text sequence into the pre-trained recognition model;

[0117] S502, the recognition model processes the text sequence to detect and extract entities belonging to a predefined entity category;

[0118] S503, analyzing the identified entities, determining the semantic relationship between the entities based on the contextual relationship between the entities, and assigning a corresponding relationship category to each pair of entities;

[0119] S504: Generate triple information including the first entity, the second entity, and the relationship between the first entity and the second entity based on the relationship category between each pair of entities.

[0120] In this embodiment, the previously generated linearized text sequence is passed as input to a pre-trained entity recognition and relationship classification model. The pre-trained model has usually been learned through a large amount of annotated data and can identify key entities and their relationships in the text. The input of the model is a text sequence, which is usually processed word by word, but can also be processed by character or phrase.

[0121] The entity recognition (NER) model scans the input text sequence to detect and extract entities belonging to a specific category. Each entity category is pre-defined and usually covers common entity types in a specific field. In this solution, entity categories include but are not limited to names, places, organizations, etc. The model determines whether each word belongs to a certain type of entity based on the context information. If it meets the predefined categories, the word is marked as an entity.

[0122] Relation classification refers to analyzing the semantic relationship between entities after identifying them. Through the contextual relationship, the model infers the association between two entities and assigns the corresponding relationship category based on this association. Relation classification models are usually performed based on contextual dependency information and a predefined set of relationships. Common relationship categories include "belonging to", "query", "subordinate to", etc. This step can be understood as the model inferring the logical or semantic association between entities from the text.

[0123] After entity recognition and relationship classification are completed, the system organizes the generated entities and their relationships into triples, i.e. <entity 1, relationship, entity 2>. Each triple consists of two entities and the semantic relationship between them, and these triples are used to represent the core meaning of the text. The generated triple information can be used in applications such as building knowledge graphs, automated analysis, and question-answering systems.

[0124] The system inputs the generated linearized text sequence into a pre-trained model. This model is usually a natural language processing model based on deep learning technology (such as BERT, RoBERTa, etc.), which can efficiently process text and extract semantic information. The model analyzes the input sequence word by word and annotates and classifies the text according to predefined categories.

[0125] The recognition model detects and extracts entities that match predefined categories by scanning the input text sequence. Each word is labeled as a specific category (such as a person's name, place, organization, etc.), and when a group of consecutive words are identified as the same category, the system extracts it as a complete entity. For example, in the sentence "Customer A inquires about the account balance", "Customer A" and "Account Balance" are identified as two entities of different categories.

[0126] After entity recognition is complete, the model begins to analyze the relationships between the recognized entities. By using contextual information, the model infers the semantic associations between two entities and classifies them into a certain relationship category based on these associations. Relationship classification usually involves multiple categories, such as "query", "association", "belonging to", etc. Relationship classification can help the system understand the specific connection between entities, such as the "query" relationship between "Customer A" and "Account Balance".

[0127] The results of entity recognition and relationship classification will be output in the form of triples. Each triple contains two entities and the relationship between them. The final generated triple structure is like <entity 1, relationship, entity 2>, which represents the clear relationship between the two entities. This triple information can be used for further natural language processing applications, such as knowledge graph construction or automated question answering.

[0128] In this embodiment, through the pre-trained recognition model, the system can automatically extract entities from text sequences and analyze the relationships between entities. Compared with traditional manual annotation or rule matching, the pre-trained model can extract key information more accurately and quickly and generate structured triple data. This is particularly important for scenarios that require processing large amounts of text data, especially in the fields of finance and law. The system can automatically identify entities such as customers and institutions and infer the relationships between them, thereby significantly improving the efficiency and accuracy of data processing.

[0129] In one embodiment, in the above S50, after generating triple information including the first entity, the second entity and the relationship between the first entity and the second entity, the following is further included:

[0130] S601, verifying the semantic consistency of entities and relations of each triple based on semantic verification rules;

[0131] S602, verifying the triple information based on the structural integrity rule to confirm whether the entities and relationships in the triple information are complete;

[0132] S603, comparing the triple information with known entities and relationships in the knowledge graph, verifying whether the entities and relationships in the triple information are consistent with the knowledge graph based on the knowledge graph rules, and recording conflicting entities or relationships;

[0133] S604, according to the result of the knowledge graph consistency verification, correct the triple information inconsistent with the knowledge graph;

[0134] S605, formatting the triple information that has passed the semantic consistency verification, structural integrity verification and knowledge graph consistency verification to generate structured data.

[0135] In this embodiment, semantic consistency verification is to check whether the entities and relations in the generated triples are semantically reasonable. Semantic verification rules are standards based on linguistics or domain knowledge, which are used to determine whether the entities and the relations between them conform to the real scene. For example, the relationship between "bank" as an entity and "deposit" is reasonable, while the relationship between "bank" and "color" may not conform to the semantic logic. This step ensures that the semantic association of each triple is accurate.

[0136] Structural integrity verification refers to checking whether the structure of the triple is complete, that is, each triple must contain two entities and the relationship between them. If a triple is missing an entity or a relationship, it will be considered incomplete. The system checks whether the format of each triple is correct through structural integrity rules to ensure that there is no missing entity or relationship information.

[0137] The generated triples are compared with the existing knowledge graph to verify whether the entities and relations in the triples match the known knowledge. The knowledge graph is a structured data representation that contains predefined rules for entities and their relationships. If the entities or relations in the triples are inconsistent with the known entities and relations in the knowledge graph, the system will mark the conflict and record the problematic entities or relations for subsequent processing.

[0138] Once the system detects that a triple is inconsistent with the information in the knowledge graph, it will correct the triple according to the existing rules in the knowledge graph. The correction process may include replacing incorrect entities, modifying incorrect relations, or supplementing missing entities. This step ensures that the final generated triple conforms to the established facts in the knowledge graph and maintains data consistency.

[0139] All triple information that passes verification is formatted. Formatting converts triples into standardized structured data forms, such as XML, JSON, or relational database formats. The generated structured data can be further used for data analysis, knowledge graph construction, or other application scenarios.

[0140] The system first performs a semantic consistency check on each generated triple using predefined semantic verification rules. If an entity or relationship in a triple is semantically unreasonable, the system will mark the triple and correct or remove it. Next, the system checks the structural integrity of the triples to ensure that each triple contains two entities and one relationship. If a triple is incomplete, the system will try to supplement the missing part through context or mark it as an incomplete triple. The system compares the generated triples with the existing knowledge graph. By matching with known entities and relationships in the knowledge graph, the system can detect and record any conflicts. If the entities or relationships in the triples do not match the knowledge graph, the system will correct them according to the knowledge graph rules. For triples that are inconsistent with the knowledge graph, the system will correct them based on the information in the knowledge graph to ensure that the entities and relationships in each triple conform to the established knowledge system. After correction, the system will re-verify the corrected triples to ensure that they are consistent with the knowledge graph. Triples that pass all verification steps are converted into a standardized structured data format. Formatting includes organizing the data into a format that can be read and analyzed by machines, such as JSON or XML. These structured data can be used for subsequent analysis, knowledge graph updates, or other applications.

[0141] In this embodiment, the system can ensure that the generated triple data is accurate and reliable by verifying semantic consistency, structural integrity, and knowledge graph consistency. Compared with the traditional manual verification method, automated verification and correction greatly improves processing efficiency and reduces the occurrence of errors. In addition, the generated structured data can be directly applied to the expansion and update of the knowledge graph, further enhancing the availability and consistency of the data.

[0142] In one embodiment, after the above S10, the method further includes:

[0143] S101, performing volume normalization processing on the audio data;

[0144] S102, filtering and removing background noise in the normalized audio data through a filter;

[0145] S103: Detect and delete silent segments from the audio data.

[0146] In this embodiment, volume normalization refers to adjusting audio signals of different volumes to a unified standard to ensure that the overall volume of the audio remains consistent during the processing. This step can balance the volume of different parts of the audio, making it more stable and consistent in subsequent processing (such as feature extraction). By normalizing the volume of audio data, information loss or processing errors caused by excessive or low volume can be avoided.

[0147] Noise filtering is an important step in audio processing. Audio data that has been volume normalized may still contain background noise, such as environmental noise, mechanical noise, or other interfering signals. By applying filters (such as high-pass filters, low-pass filters, or band-pass filters), background noise in the audio can be effectively removed, the main part of the audio signal can be retained, and the clarity of speech features can be enhanced. The selection and setting of filters depends on the frequency range of the noise and the specific characteristics of the audio.

[0148] Silence segments refer to the parts of the audio signal that do not contain valid information. Removing these silence segments can reduce the transmission of invalid information, save processing time and computing resources, and improve the efficiency of speech recognition. Silence segment detection is usually achieved by setting a threshold for the audio signal, that is, detecting the parts with a volume below a certain threshold, and then cutting or deleting these parts. This step ensures that the subsequent processing steps focus on valid speech content.

[0149] First, the system normalizes the volume of the acquired audio data. Volume normalization adjusts the overall volume of the audio signal to meet a preset standard range. This step can balance the high-volume and low-volume parts to ensure that the overall volume of the audio is consistent. Common methods of normalization include normalization based on RMS (root mean square) values ​​or peak normalization. After volume normalization, the system filters the background noise in the audio data. Noise filtering can be achieved in a variety of ways, including using different types of filters (such as high-pass, low-pass, band-pass, etc.). These filters selectively remove low-frequency or high-frequency noise based on the frequency range of the noise while retaining the main voice information, thereby improving the clarity and quality of the voice signal. The system then removes invalid information by detecting silent segments in the audio data. The detection of silent segments can be achieved by setting a volume threshold. When the audio signal is below this threshold, the section will be marked as a silent segment. After detecting silent segments, the system will delete these sections and retain valid voice content. This operation can reduce processing time and data storage requirements while improving the efficiency of speech recognition.

[0150] In this embodiment, the system can significantly improve the processing quality of audio data by normalizing the volume, removing background noise, and deleting silent segments. Volume normalization ensures the stability of the audio signal, noise filtering enhances the clarity of the voice signal, and silent segment deletion reduces the interference of invalid information. Combining these processing steps, the system can provide more accurate and clear audio data for subsequent voice feature extraction, voice recognition, and natural language processing.

[0151] In one embodiment, a device for jointly extracting entities and relationships is provided, and the device for jointly extracting entities and relationships corresponds one-to-one to the method for jointly extracting entities and relationships in the above embodiment. Figure 3 , Figure 3 The figure is a functional module diagram of a preferred embodiment of the joint extraction device of entities and relations of the present invention. The audio data acquisition module 10, the encoder module 20, the convolutional neural network module 30, the decoder module 40 and the semantic information extraction module 50. The detailed description of each functional module is as follows:

[0152] An audio data acquisition module 10, used to acquire audio data, wherein the audio data includes voice information;

[0153] An encoder module 20, configured to input the audio data into an encoder module, and extract speech features from the audio data through the encoder module;

[0154] A convolutional neural network module 30, used to compress and reduce the dimension of the speech features through a convolutional neural network to generate compressed speech features;

[0155] A decoder module 40, configured to input the compressed speech features into the decoder module to generate a linearized text sequence;

[0156] The semantic information extraction module 50 is used to perform entity recognition operations and relationship classification operations on the text sequence through a pre-trained recognition model to generate triple information including a first entity, a second entity, and a relationship between the first entity and the second entity.

[0157] In one embodiment, the encoder module 20 is specifically configured to:

[0158] The audio data is segmented according to a preset frame length to generate multiple audio segments;

[0159] Input multiple audio clips into the encoder module, and extract the time-frequency features of each audio clip through a feature extractor based on a convolutional neural network;

[0160] In the process of extracting time-frequency features, the attention mechanism is used to enhance the feature extraction of key frequency bands and time periods;

[0161] The enhanced time-frequency features are globally modeled through the self-attention mechanism to capture the contextual dependencies between audio clips and generate the final speech feature vector containing global timing information.

[0162] In one embodiment, the convolutional neural network module 30 is specifically used to:

[0163] Performing convolution processing on the speech features through multi-layer convolution operations;

[0164] After each convolution operation, a dimensionality reduction process is performed on the speech feature in combination with a pooling operation to compress the feature dimension of the speech feature;

[0165] The channel of speech features is compressed by using the convolution operation of channel compression;

[0166] The speech feature vector after multi-layer convolution operation, pooling operation and channel compression is processed by activation function to generate the final speech feature vector.

[0167] In one embodiment, the decoder module 40 is specifically configured to:

[0168] Input the compressed speech features into the decoder module;

[0169] Decoding the speech features layer by layer through a decoder module based on a recurrent neural network or a self-attention mechanism to generate an intermediate semantic representation corresponding to the speech features;

[0170] The decoder module extracts semantic information from the intermediate semantic representation based on the contextual relationship of the speech features;

[0171] The semantic information is converted into a linearized text sequence.

[0172] In one embodiment, the semantic information extraction module 50 is specifically used to:

[0173] Input the linearized text sequence into the pre-trained recognition model;

[0174] The recognition model processes the text sequence to detect and extract entities belonging to predefined entity categories;

[0175] Analyze the identified entities, determine the semantic relationship between the entities based on the contextual relationship between the entities, and assign corresponding relationship categories to each pair of entities;

[0176] Based on the relationship category between each pair of entities, triple information including the first entity, the second entity and the relationship between the first entity and the second entity is generated.

[0177] In one embodiment, the semantic information extraction module 50 is specifically used to:

[0178] Verify the semantic consistency of entities and relations of each triple based on semantic verification rules;

[0179] Verify the triple information based on the structural integrity rules to confirm whether the entities and relationships in the triple information are complete;

[0180] Compare the triple information with the known entities and relationships in the knowledge graph, verify whether the entities and relationships in the triple information are consistent with the knowledge graph based on the knowledge graph rules, and record the conflicting entities or relationships;

[0181] According to the results of the knowledge graph consistency verification, the triple information that is inconsistent with the knowledge graph is corrected;

[0182] The triple information that has passed the semantic consistency verification, structural integrity verification and knowledge graph consistency verification is formatted to generate structured data.

[0183] In one embodiment, the audio data acquisition module 10 is specifically used for:

[0184] Performing volume normalization processing on the audio data;

[0185] Filtering and removing background noise in the normalized audio data through a filter;

[0186] Silence segments are detected and removed from the audio data.

[0187] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a joint extraction method of entities and relationships on the server side.

[0188] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a user-side method for joint extraction of entities and relationships

[0189] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0190] Acquire audio data, wherein the audio data includes voice information;

[0191] Inputting the audio data into an encoder module, and extracting speech features from the audio data by the encoder module;

[0192] Compressing and reducing the dimension of the speech features through a convolutional neural network to generate compressed speech features;

[0193] Inputting the compressed speech features into the decoder module to generate a linearized text sequence;

[0194] The pre-trained recognition model is used to perform entity recognition and relationship classification operations on the text sequence to generate triple information including a first entity, a second entity, and a relationship between the first entity and the second entity.

[0195] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0196] Acquire audio data, wherein the audio data includes voice information;

[0197] Inputting the audio data into an encoder module, and extracting speech features from the audio data by the encoder module;

[0198] Compressing and reducing the dimension of the speech features through a convolutional neural network to generate compressed speech features;

[0199] Inputting the compressed speech features into the decoder module to generate a linearized text sequence;

[0200] The pre-trained recognition model is used to perform entity recognition and relationship classification operations on the text sequence to generate triple information including a first entity, a second entity, and a relationship between the first entity and the second entity.

[0201] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0202] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0203] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0204] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A joint extraction method of entities and relations, characterized in that: The following steps are involved: Acquire audio data, wherein the audio data includes voice information; Inputting the audio data into an encoder module, and extracting speech features from the audio data by the encoder module; Compressing and reducing the dimension of the speech features through a convolutional neural network to generate compressed speech features; Inputting the compressed speech features into the decoder module to generate a linearized text sequence; The pre-trained recognition model is used to perform entity recognition and relationship classification operations on the text sequence to generate triple information including a first entity, a second entity, and a relationship between the first entity and the second entity.

2. The method for joint extraction of entities and relations as claimed in claim 1, characterized in that: Inputting the audio data into an encoder module, and extracting speech features from the audio data by the encoder module, comprising: The audio data is segmented according to a preset frame length to generate multiple audio segments; Input multiple audio clips into the encoder module, and extract the time-frequency features of each audio clip through a feature extractor based on a convolutional neural network; In the process of extracting time-frequency features, the attention mechanism is used to enhance the feature extraction of key frequency bands and time periods; The enhanced time-frequency features are globally modeled through the self-attention mechanism to capture the contextual dependencies between audio clips and generate the final speech feature vector containing global timing information.

3. The method for joint extraction of entities and relations as claimed in claim 1, characterized in that: The speech features are compressed and dimensionally reduced by a convolutional neural network to generate compressed speech features, including: Performing convolution processing on the speech features through multi-layer convolution operations; After each convolution operation, a dimensionality reduction process is performed on the speech feature in combination with a pooling operation to compress the feature dimension of the speech feature; The channel of speech features is compressed by using the convolution operation of channel compression; The speech feature vector after multi-layer convolution operation, pooling operation and channel compression is processed by activation function to generate the final speech feature vector.

4. The method for joint extraction of entities and relations as claimed in claim 1, characterized in that: The compressed speech features are input into the decoder module to generate a linearized text sequence, including: Input the compressed speech features into the decoder module; Decoding the speech features layer by layer through a decoder module based on a recurrent neural network or a self-attention mechanism to generate an intermediate semantic representation corresponding to the speech features; The decoder module extracts semantic information from the intermediate semantic representation based on the contextual relationship of the speech features; The semantic information is converted into a linearized text sequence.

5. The method for joint extraction of entities and relations as claimed in claim 1, characterized in that: By using a pre-trained recognition model, an entity recognition operation and a relationship classification operation are performed on the text sequence to generate triple information including a first entity, a second entity, and a relationship between the first entity and the second entity, including: Input the linearized text sequence into the pre-trained recognition model; The recognition model processes the text sequence to detect and extract entities belonging to predefined entity categories; Analyze the identified entities, determine the semantic relationship between the entities based on the contextual relationship between the entities, and assign corresponding relationship categories to each pair of entities; Based on the relationship category between each pair of entities, triple information including the first entity, the second entity and the relationship between the first entity and the second entity is generated.

6. The method for joint extraction of entities and relations as claimed in claim 1, characterized in that: After generating triple information including the first entity, the second entity, and the relationship between the first entity and the second entity, the method further includes: Verify the semantic consistency of entities and relations of each triple based on semantic verification rules; Verify the triple information based on the structural integrity rules to confirm whether the entities and relationships in the triple information are complete; Compare the triple information with the known entities and relationships in the knowledge graph, verify whether the entities and relationships in the triple information are consistent with the knowledge graph based on the knowledge graph rules, and record the conflicting entities or relationships; According to the results of the knowledge graph consistency verification, the triple information that is inconsistent with the knowledge graph is corrected; The triple information that has passed the semantic consistency verification, structural integrity verification and knowledge graph consistency verification is formatted to generate structured data.

7. The method for joint extraction of entities and relations as claimed in claim 1, characterized in that: After getting the audio data, it also includes: Performing volume normalization processing on the audio data; Filtering and removing background noise in the normalized audio data through a filter; Silence segments are detected and removed from the audio data.

8. A device for joint extraction of entities and relations, characterized in that: The entity and relationship joint extraction device comprises: An audio data acquisition module, used to acquire audio data, wherein the audio data includes voice information; An encoder module, configured to input the audio data into an encoder module, and extract speech features from the audio data through the encoder module; A convolutional neural network module, used for compressing and reducing the dimension of the speech features through a convolutional neural network to generate compressed speech features; A decoder module, used for inputting the compressed speech features into the decoder module to generate a linearized text sequence; The semantic information extraction module is used to perform entity recognition operations and relationship classification operations on the text sequence through a pre-trained recognition model to generate triple information including a first entity, a second entity, and a relationship between the first entity and the second entity.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a program for joint extraction of entities and relationships stored in the memory and executable on the processor. When the program for joint extraction of entities and relationships is executed by the processor, the steps of the method for joint extraction of entities and relationships as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program for jointly extracting entities and relationships, which, when executed by a processor, implements the steps of the method for jointly extracting entities and relationships as described in any one of claims 1-7.

Images