Voice-based semantic extraction method and device, electronic device, and storage medium

Abstract

The embodiment of the application provides a kind of based on voice's semantic extraction method and device, electronic equipment and storage medium, belong to artificial intelligence technical field, it is applicable to the field of financial technology and medical science and technology scene.The method comprises: obtaining original audio data;Original audio data is preprocessed to audio, and target audio data is obtained;Text conversion is carried out to target audio data, and original audio text is obtained;Audio category detection is carried out based on target audio data and original audio text, and non-language label is obtained;Non-language label is used to indicate that there is non-language audio in original audio data;Original audio text is labeled based on non-language label, and target semantic data of original audio data is obtained.The embodiment of the application can improve the accuracy of semantic extraction.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and is applicable to the fields of financial technology and medical technology. In particular, it relates to a speech-based semantic extraction method and device, electronic device and storage medium. Background Technology

[0002] Speech semantic extraction technology refers to identifying and understanding the underlying meaning of speech signals, accurately capturing the speaker's intent and the specific content they express. This technology can be applied to various scenarios. For example, in fintech, it can extract semantic meaning from customer-input speech signals to provide more accurate and personalized services. In medical technology, it can extract semantic meaning from patient-input speech signals to help understand the patient's condition more comprehensively and accurately, providing more targeted medical advice and services.

[0003] Currently, speech semantic extraction methods mainly focus on semantic text content. However, in actual speech communication, non-verbal information in speech, such as intonation, speech rate, pauses, laughter, and sighs, may also contain the speaker's emotions. Speech semantic extraction methods cannot accurately extract non-verbal information, which affects the accuracy of semantic extraction.

[0004] Therefore, improving the accuracy of semantic extraction has become an urgent technical problem to be solved. Summary of the Invention

[0005] The main objective of this application is to propose a speech-based semantic extraction method, apparatus, electronic device, and storage medium, aiming to solve the technical problem of low accuracy in speech semantic extraction due to the inability of speech semantic extraction methods to accurately extract non-linguistic information, and to improve the accuracy of semantic extraction.

[0006] To achieve the above objectives, a first aspect of this application proposes a speech-based semantic extraction method, the method comprising: Obtain the raw audio data; The original audio data is preprocessed to obtain the target audio data; The target audio data is converted into text to obtain the original audio text; Based on the target audio data and the original audio text, audio category detection is performed to obtain non-verbal labels; wherein, the non-verbal labels are used to indicate the presence of non-verbal audio in the original audio data; The original audio text is annotated based on the non-linguistic tags to obtain the target semantic data of the original audio data.

[0007] In some embodiments, the step of performing audio category detection based on the target audio data and the original audio text to obtain non-linguistic labels includes: Feature extraction is performed on the target audio data to obtain initial audio features; The original audio text is text-encoded to obtain audio text encoding features; The initial audio features and the audio text encoding features are concatenated to obtain concatenated features; Obtain multiple preset non-language query vectors; wherein, the non-language query vectors are used to query non-language types in audio; The non-linguistic tag is obtained by performing non-linguistic detection on the concatenated features based on the non-linguistic query vector.

[0008] In some embodiments, the step of performing non-linguistic detection on the concatenated features based on the non-linguistic query vector to obtain the non-linguistic label includes: The spliced ​​features are then encoded to obtain the original encoded features; Based on each of the non-linguistic query vectors, a non-linguistic query is performed on the original encoded features to obtain non-linguistic query features; The non-linguistic query features and the original encoded features are classified to obtain a non-linguistic classification matrix; The non-linguistic labels are determined based on the non-linguistic classification matrix.

[0009] In some embodiments, classifying the non-linguistic query features and the original encoded features to obtain a non-linguistic classification matrix includes: For each of the non-linguistic query features, activation calculation is performed based on the non-linguistic query features and the original encoded features to obtain the non-linguistic type probability; The non-language type probabilities are filtered based on a preset probability threshold to obtain the identification type data; The non-language classification matrix is ​​obtained by constructing a matrix based on the recognition type data.

[0010] In some embodiments, the non-linguistic tags include tag types and non-linguistic timestamps; the step of annotating the original audio text based on the non-linguistic tags to obtain the target semantic data of the original audio data includes: Based on the non-linguistic timestamp, the tag type is timestamped with the original audio text to obtain an aligned timestamp; Extract the target text segment from the original audio text based on the aligned timestamp; If the target text fragment does not contain speech information, the tag type is embedded into the original audio text based on the alignment timestamp to obtain the target semantic data.

[0011] In some embodiments, after extracting the target text segment from the original audio text based on the aligned timestamp, the method further includes: If the target text segment contains speech information, perform an integrity check on the target text segment and the original audio text to obtain a complete text segment; Obtain the start and end timestamps of the complete text segment; The tag type is embedded into the original audio text based on the start timestamp, and the tag type is embedded into the original audio text based on the end timestamp to obtain the target semantic data.

[0012] In some embodiments, the audio preprocessing of the original audio data to obtain the target audio data includes: The original audio data is denoised to obtain denoised audio data; Speaker separation is performed on the denoised audio data to obtain the initial audio data; Speech activity detection is performed on the initial audio data to obtain labels for non-silent segments; Based on the non-silent segment tags, audio is extracted from the initial audio data to obtain the target audio data.

[0013] To achieve the above objectives, a second aspect of this application provides a speech-based semantic extraction apparatus, the apparatus comprising: The audio acquisition module is used to acquire raw audio data; The audio processing module is used to perform audio preprocessing on the raw audio data to obtain target audio data; The text conversion module is used to convert the target audio data into text to obtain the original audio text; An audio detection module is used to perform audio category detection based on the target audio data and the original audio text to obtain non-verbal tags; wherein, the non-verbal tags are used to indicate the presence of non-verbal audio in the original audio data; The speech annotation module is used to annotate the original audio text based on the non-linguistic tags to obtain the target semantic data of the original audio data.

[0014] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0015] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0016] This application proposes a speech-based semantic extraction method, apparatus, electronic device, and storage medium. It acquires raw audio data and performs audio preprocessing to remove noise and other interference, resulting in cleaner target audio data. The target audio data is then converted into text to obtain the original audio text, improving the accuracy of subsequent processing. Next, audio category detection is performed based on the target audio data and the original audio text to obtain non-verbal tags indicating the presence of non-verbal audio in the raw audio data, enabling accurate extraction of non-verbal information. Finally, the original audio text is annotated based on the non-verbal tags to obtain the target semantic data of the raw audio data. This comprehensive and accurate mining of audio semantic information improves the accuracy of semantic extraction from audio. Attached Figure Description

[0017] Figure 1 This is a flowchart of a speech-based semantic extraction method provided in an embodiment of this application; Figure 2 yes Figure 1 The flowchart of step S102 in the document; Figure 3 yes Figure 1 The flowchart of step S104 in the process; Figure 4 yes Figure 3 The flowchart of step S305 in the text; Figure 5 yes Figure 4 The flowchart of step S403 in the process; Figure 6 yes Figure 1 The flowchart of step S105 in the process; Figure 7 This is a schematic diagram of the structure of the speech-based semantic extraction device provided in the embodiments of this application; Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0022] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). It is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0023] Speech semantic extraction (SSE) is a technology that identifies and understands the underlying meaning of speech signals, accurately capturing the speaker's intent and the specific content they express. SSE technology first uses speech recognition to convert speech into text, then employs natural language processing techniques, including syntactic analysis and semantic role labeling, to deeply analyze word collocations, sentence structures, and contextual information in the text to accurately capture the speaker's intent and the specific content they express. This process involves not only understanding the literal meaning but also inferring implicit information from context and common sense, ultimately transforming unstructured speech data into a structured semantic representation, providing crucial support for applications such as intelligent question answering, machine translation, and voice interaction.

[0024] Speech semantic extraction technology can be applied to multiple scenarios. For example, in fintech scenarios such as intelligent customer service and fraud detection, semantic extraction of customer-inputted speech signals can provide more accurate and personalized services. In medical technology scenarios such as remote medical consultations and health consultations, semantic extraction of patient-inputted speech signals helps to understand the patient's condition more comprehensively and accurately, providing more targeted medical advice and services.

[0025] Nonverbal communication, also known as non-verbal cues, refers to communication methods that convey information, express emotions, and convey intentions without relying on written language. These methods utilize non-verbal symbols such as body language, facial expressions, eye contact, spatial distance, tone of voice, smell, touch, and environmental arrangement. Nonverbal cues often complement and synergize with verbal communication in interpersonal interactions, sometimes even reflecting inner thoughts more truthfully and directly than language. For example, a smile may convey friendliness and acceptance, while crossed arms may imply defensiveness or resistance. Nonverbal cues play a crucial role in understanding others' true attitudes and emotions.

[0026] Currently, speech semantic extraction methods mainly focus on semantic text content. However, in actual speech communication, non-verbal information in speech, such as intonation, speech rate, pauses, laughter, and sighs, may also contain the speaker's emotions. Speech semantic extraction methods cannot accurately extract non-verbal information, which affects the accuracy of semantic extraction.

[0027] Based on this, embodiments of this application provide a speech-based semantic extraction method and apparatus, electronic device and storage medium, aiming to solve the technical problem that the accuracy of speech semantic extraction is low due to the inability of speech semantic extraction methods to accurately extract non-linguistic information, and to improve the accuracy of semantic extraction.

[0028] The speech-based semantic extraction method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the speech-based semantic extraction method in this application is described.

[0029] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0030] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0031] The speech-based semantic extraction method provided in this application relates to the field of artificial intelligence technology. This speech-based semantic extraction method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the speech-based semantic extraction method, but is not limited to the above forms.

[0032] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0033] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0034] Figure 1 This is an optional flowchart of a speech-based semantic extraction method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0035] Step S101: Obtain the raw audio data; Step S102: Perform audio preprocessing on the original audio data to obtain the target audio data; Step S103: Perform text conversion on the target audio data to obtain the original audio text; Step S104: Perform audio category detection based on the target audio data and the original audio text to obtain non-verbal labels; wherein, the non-verbal labels are used to indicate the presence of non-verbal audio in the original audio data; Step S105: Annotate the original audio text based on non-linguistic tags to obtain the target semantic data of the original audio data.

[0036] Steps S101 to S105, as illustrated in this embodiment, involve acquiring raw audio data and performing audio preprocessing to remove noise and other interference, resulting in cleaner target audio data. Text conversion of the target audio data yields the original audio text, improving the accuracy of subsequent processing. Next, audio category detection is performed based on the target audio data and the original audio text to obtain non-verbal tags indicating the presence of non-verbal audio in the raw audio data, enabling accurate extraction of non-verbal information. Finally, the original audio text is annotated based on the non-verbal tags to obtain the target semantic data of the raw audio data, allowing for comprehensive and accurate mining of audio semantic information and improving the accuracy of semantic extraction from the audio.

[0037] In step S101 of some embodiments, raw audio data refers to audio signal data acquired directly from an audio acquisition device (such as a microphone, recorder, etc.) without any processing. Raw audio data contains the original waveform information of the sound and may include various types of sounds, such as human voices, environmental noise, and non-verbal sounds (such as coughs, laughter, etc.). In addition, the raw audio data supports multiple languages, including but not limited to Chinese, English, Japanese, Korean, and Cantonese.

[0038] Please see Figure 2 In some embodiments, step S102 may include, but is not limited to, steps S201 to S204: Step S201: Denoise the original audio data to obtain denoised audio data; Step S202: Speaker separation is performed on the denoised audio data to obtain initial audio data; Step S203: Perform speech activity detection on the initial audio data to obtain non-silent segment labels; Step S204: Extract audio from the initial audio data based on the non-silent segment tags to obtain the target audio data.

[0039] Steps S201 to S204, as illustrated in this embodiment, denoise the original audio data to obtain denoised audio data, which reduces noise interference and improves audio quality. Next, speaker separation is performed on the denoised audio data to obtain initial audio data, and audio segments from different speakers are distinguished and extracted. Furthermore, speech activity detection is performed on the initial audio data to accurately label non-silent segments, and audio extraction is performed on the initial audio data based on these labels. This efficiently and accurately obtains the required target audio data, removes silent segments, and improves audio processing efficiency and result accuracy.

[0040] In step S201 of some embodiments, since the original audio data is unprocessed audio and contains noise such as environmental noise (such as wind noise, traffic noise) and equipment noise (such as circuit noise), it is necessary to first perform noise reduction processing on the original audio data to make the speech clearer and more distinguishable, reduce the interference of noise on subsequent audio analysis and processing, and improve the accuracy and effect of subsequent steps.

[0041] Specifically, denoising algorithms (such as spectral subtraction, Wiener filtering, wavelet denoising, etc.) can be used to analyze and process the original audio data, remove the noise, and obtain denoised audio data.

[0042] In step S202 of some embodiments, since the original audio data may contain the audio of multiple speakers, it is also necessary to perform speaker separation on the denoised audio data to accurately separate the speech of different speakers, so as to facilitate subsequent audio processing and analysis for each speaker. For example, speech recognition and emotion analysis can be performed on the speech of different speakers respectively.

[0043] Specifically, speaker separation algorithms (such as those based on Gaussian mixture models (GMM) or deep neural networks (DNN)) can be used to analyze the denoised audio data, distinguish the speech signals of different speakers, and obtain the initial audio data.

[0044] In step S203 of some embodiments, speech activity detection is the process of determining whether speech activity exists in the audio signal, i.e., distinguishing between speech segments and silent segments. Specifically, speech activity detection algorithms (such as those based on energy, zero-crossing rate, or a combination of short-time energy and zero-crossing rate) can be used to analyze the initial audio data, determine whether speech activity exists in each time period, and generate corresponding non-silent segment labels. These non-silent segment labels are used to mark which parts of the initial audio data are speech (non-silent), including the start and end times of the speech.

[0045] In step S204 of some embodiments, after obtaining the non-silent segment tags, the corresponding speech segments are extracted from the initial audio data based on the non-silent segment tags, the silent parts are removed, and the target audio data is obtained. This makes the target audio data more concise and accurate, improves the efficiency and accuracy of subsequent audio extraction, and also saves storage.

[0046] In some embodiments, after step S102 and before step S103, the speech-based semantic extraction method further includes: The filtering threshold is dynamically adjusted based on the signal-to-noise ratio (SNR) of the audio to further filter the target audio data obtained in step S204. For example: when SNR > 20dB (quiet scene, such as radio drama), the word error rate (WER) threshold is set to 15% and the depth noise suppression mean opinion score (DNSMOS) threshold is set to 1.2; when SNR < 10dB (noisy scene, such as variety show), the error rate (WER) threshold is relaxed to 25% and the depth noise suppression mean opinion score (DNSMOS) threshold is set to 0.9. By further filtering the target audio data obtained in step S204, higher quality target audio data is obtained.

[0047] In step S103 of some embodiments, the spectral features of the target audio data are analyzed by automatic speech recognition technology (such as Whisper model, Wav2Vec model, HuBERT model, etc.), and the speech in the audio is identified as text word by word to obtain the original audio text. Furthermore, word-level timestamps of each character / word are identified and generated to facilitate subsequent semantic analysis.

[0048] In some embodiments, if the word-level timestamps of the original audio text are misaligned (e.g., error > 100ms), the Montreal Forced Alignment Tool (MFA) is used to re-align them and reduce timestamp errors.

[0049] Please see Figure 3 In some embodiments, step S104 may include, but is not limited to, steps S301 to S305: Step S301: Extract features from the target audio data to obtain initial audio features; Step S302: Perform text encoding on the original audio text to obtain audio text encoding features; Step S303: Concatenate the initial audio features and audio text encoding features to obtain concatenated features; Step S304: Obtain multiple preset non-language query vectors; wherein, the non-language query vectors are used to query non-language types in audio; Step S305: Perform non-linguistic detection on the concatenated features based on the non-linguistic query vector to obtain non-linguistic labels.

[0050] Steps S301 to S305, as illustrated in this embodiment, involve extracting features from the target audio data to obtain initial audio features, text encoding the original audio text to obtain audio-text encoded features, and concatenating the initial audio features and audio-text encoded features to obtain concatenated features, thus integrating various aspects of audio and text information. Next, multiple preset non-linguistic query vectors are obtained; these non-linguistic query vectors are used to query non-linguistic types in the audio. Finally, non-linguistic detection is performed on the concatenated features based on the non-linguistic query vectors to obtain non-linguistic labels. By comprehensively utilizing audio and text information, the non-linguistic types in the audio are identified comprehensively and accurately, providing a strong basis for subsequent semantic extraction.

[0051] In step S301 of some embodiments, the target audio data is feature extracted using a pre-trained Wav2vec 2.0 model to obtain initial audio features, which can reduce the amount of data while retaining key information.

[0052] In step S302 of some embodiments, the original audio text is encoded using a pre-trained XLM-RoBERTa model, converting each word or sentence in the text into a vector to obtain audio text encoding features.

[0053] In step S303 of some embodiments, the initial audio features and audio text encoding features are concatenated sequentially to obtain concatenated features, forming a new vector, namely concatenated features. This can comprehensively utilize the information of audio and text, give full play to the advantages of both, and improve the accuracy and comprehensiveness of understanding audio content.

[0054] In step S304 of some embodiments, the preset non-linguistic query vector is a predefined vector used to detect the type of non-linguistic information in the audio. Each vector corresponds to a specific non-linguistic type, which can detect specific non-linguistic information in the audio in a targeted manner, thereby improving the accuracy and efficiency of non-linguistic detection.

[0055] Specifically, non-linguistic types include, but are not limited to: Laughter (indicating happiness, relaxation, and humor); Coughing (indicates interruption, attracting attention, or masking nervousness); Sighing (expressing exhaustion, helplessness, frustration, or disappointment); Clearing one's throat (to attract attention, express seriousness, or conceal unease); Crying (expressing sadness, pain, extreme emotion, or feeling wronged); Smacking one's lips / making a clicking sound (indicating dissatisfaction, contemplation, or a reminder); A gasping sound (indicating surprise, preparation to speak, or a deep breath); Exhalation sound (indicating relaxation, fatigue, or emphasis); A humming sound (indicating agreement, effort, doubt, or dissatisfaction).

[0056] It should be noted that the types included in non-language types need to be set according to the actual application scenario, and are not limited to this.

[0057] Please see Figure 4 In some embodiments, step S305 may include, but is not limited to, steps S401 to S404: Step S401: Perform feature encoding on the concatenated features to obtain the original encoded features; Step S402: Perform non-linguistic queries on the original encoded features based on each non-linguistic query vector to obtain non-linguistic query features; Step S403: Classify the non-linguistic query features and the original encoded features to obtain the non-linguistic classification matrix; Step S404: Determine non-linguistic labels based on the non-linguistic classification matrix.

[0058] Steps S401 to S404, as illustrated in this embodiment, involve encoding the concatenated features to obtain the original encoded features. Next, each non-linguistic query vector is used to perform a non-linguistic query on the original encoded features to obtain non-linguistic query features, which can accurately extract non-linguistic related features. Furthermore, the non-linguistic query features and the original encoded features are classified to effectively distinguish different categories of features, resulting in a non-linguistic classification matrix. Finally, non-linguistic labels are determined based on the non-linguistic classification matrix, improving the accuracy of non-linguistic label recognition.

[0059] In step S401 of some embodiments, the spliced ​​features are encoded by a Transformer encoder to extract key information from the spliced ​​features, obtain the original encoded features, unify the format and dimension of features from different sources, and eliminate the dimensional differences between different features.

[0060] In step S402 of some embodiments, for each non-language query vector, a non-language query is performed on the original encoded features based on the non-language query vector, and the features most relevant to the non-language query vector are selected from the original encoded features to obtain the non-language query features.

[0061] It should be noted that the target audio data includes multiple speech frames, and the original encoded features are the data corresponding to each speech frame. Furthermore, the number of non-linguistic query features is the same as the number of non-linguistic query vectors; that is, for the original encoded features of each speech frame, queries are performed based on each non-linguistic query vector, resulting in non-linguistic query features corresponding to multiple non-linguistic types.

[0062] Please see Figure 5In some embodiments, step S403 may include, but is not limited to, steps S501 to S503: Step S501: For each non-linguistic query feature, activation calculation is performed based on the non-linguistic query feature and the original encoded feature to obtain the non-linguistic type probability; Step S502: Filter the non-language type probabilities based on a preset probability threshold to obtain the recognition type data; Step S503: Construct a matrix based on the recognition type data to obtain a non-language classification matrix.

[0063] Steps S501 to S503, as illustrated in this embodiment, involve performing activation calculations on each non-verbal query feature based on the non-verbal query feature and the original encoded feature to obtain the non-verbal type probability, accurately quantifying the probability that the original encoded feature belongs to each non-verbal type. Next, the non-verbal type probabilities are filtered based on a preset probability threshold to obtain identification type data, eliminating low-confidence results and ensuring the accuracy and reliability of the identification type data. Finally, a matrix is ​​constructed based on the identification type data to obtain a non-verbal classification matrix, thereby determining non-verbal information and improving the accuracy of non-verbal recognition and classification.

[0064] In step S501 of some embodiments, for each non-language type corresponding to the non-language query feature, the non-language query feature and the original encoded feature are aggregated to obtain aggregated features. Then, the aggregated features are activated by the sigmoid activation function to achieve binary classification and obtain the non-language type probability. The non-language type probability is used to characterize the probability value that the non-language query feature of a certain speech frame belongs to a certain non-language type.

[0065] In step S502 of some embodiments, the preset probability threshold is a probability value pre-set according to the actual application scenario, used to determine whether the probability of a non-language type meets the standard for being identified as a certain non-language type, such as 0.7, 0.8, etc., and is not limited thereto. By setting a probability threshold for filtering, results with low probability and low reliability can be eliminated, improving the accuracy and reliability of the recognition results and avoiding misjudgments.

[0066] Next, based on a preset probability threshold, the probability of non-language type corresponding to each non-language type is filtered. If the probability of non-language type is greater than or equal to the probability threshold, the non-language query feature is determined to belong to the corresponding non-language type, and the relevant data is retained; if the probability of non-language type is less than the probability threshold, the non-language query feature is determined not to belong to the corresponding non-language type, and the relevant data is discarded.

[0067] In step S503 of some embodiments, for each speech frame, a matrix is ​​constructed based on the recognition type data corresponding to the speech frame to obtain the non-language classification matrix of the speech frame. The non-language classification matrix is ​​used to characterize the non-language type to which a certain speech frame belongs.

[0068] For example: Suppose there are 5 non-verbal types: laughter, cough, sigh, cry and tsk. For a certain speech frame, the recognition type data corresponding to the speech frame is laughter, then the non-verbal classification matrix of the speech frame is [1,0,0,0,0]; in the next speech frame, the recognition type data corresponding to the speech frame is tsk, then the non-verbal classification matrix of the speech frame is [0,0,0,0,1].

[0069] In step S404 of some embodiments, for each speech frame, non-language labels are determined based on the non-language classification matrix, transforming the complex classification matrix results into intuitive non-language labels for easy understanding and application.

[0070] It is understandable that when a speaker is speaking, at any given moment, they usually only carry one type of non-verbal information (for example, laughter and sighs cannot be produced at the same time, and breathing sounds and coughs cannot be produced at the same time). Therefore, for each speech frame, if there is a corresponding non-verbal label, there is only one non-verbal label.

[0071] It should be noted that there are multiple non-verbal tags. Each non-verbal tag includes a tag type and a non-verbal timestamp. The tag type is used to identify which non-verbal type it belongs to, such as laughter, sighing, or throat clearing. The non-verbal timestamp is used to mark the time point when the non-verbal information appears in the audio.

[0072] Please see Figure 6 In some embodiments, step S105 includes, but is not limited to, steps S601 to S603: Step S601: Align the tag type with the original audio text based on the non-linguistic timestamp to obtain the aligned timestamp; Step S602: Extract the target text segment from the original audio text based on the aligned timestamp; Step S603: If the target text fragment does not contain speech information, embed the tag type into the original audio text based on the aligned timestamp to obtain the target semantic data.

[0073] Steps S601 to S603, as illustrated in this embodiment, align the tag type with the original audio text based on non-linguistic timestamps to obtain an aligned timestamp, ensuring a precise temporal correspondence between the two and providing a temporal basis for subsequent annotation. Next, the target text segment is extracted from the original audio text based on the aligned timestamp, accurately locating the content to be inserted. If the target text segment does not contain linguistic information, the tag type is embedded into the original audio text based on the aligned timestamp to obtain target semantic data. This process fully preserves non-linguistic information, resulting in more comprehensive and accurate semantic expression and improved information processing quality.

[0074] In step S601 of some embodiments, for each non-language tag, based on the non-language timestamp corresponding to the non-language tag, the speech text region closest to the non-language tag is found from the original audio text, and the timestamp of the speech text region is confirmed as the alignment timestamp.

[0075] Specifically, if the original audio text contains a speech text region at the non-verbal timestamp, the non-verbal timestamp is confirmed as the alignment timestamp. If the original audio text does not contain a speech text region at the non-verbal timestamp, the speech text region closest to the non-verbal tag is found in the original audio text. If the time difference between the timestamp of the speech text region closest to the non-verbal tag and the non-verbal timestamp exceeds a preset time threshold (e.g., one second), the non-verbal tag is discarded. This provides accurate time information for subsequent extraction of relevant text segments and embedding of tag types, avoiding unrelated tags and preventing information extraction errors or semantic confusion caused by time mismatch.

[0076] In step S602 of some embodiments, after obtaining the alignment timestamp, the target text segment is extracted from the original audio text based on the alignment timestamp. The target text segment may contain speech information (a word or a phrase) or may not contain speech information (i.e., no spoken content).

[0077] Therefore, it is necessary to check whether there is actual speech content in the target text fragment, such as text corresponding to words or speech.

[0078] In step S603 of some embodiments, if the target text fragment does not contain speech information, the tag type is embedded into the corresponding position of the original audio text based on the position determined by the alignment timestamp to obtain the target semantic data.

[0079] For example, the original audio text is "He laughed so happily," and the tag type is "laugh." The non-verbal timestamp of the tag type "laugh" is between the timestamps of the words "laughed" and "so happy," indicating that the non-verbal action of laughing occurred between "laughed" and "so happy." Therefore, after embedding the tag type "laugh," the target semantic data is "He laughed so happily."

[0080] In other embodiments, if the target text segment contains speech information, the integrity of the target text segment and the original audio text is checked to obtain the complete text segment. Get the start and end timestamps of the complete text segment; The tag type is embedded into the original audio text based on the start timestamp, and the tag type is also embedded into the original audio text based on the end timestamp to obtain the target semantic data.

[0081] Understandably, if the target text fragment contains speech information, that speech information may only be a part of a word. Therefore, it is necessary to perform integrity verification on the target text fragment and the original audio text to obtain the complete text fragment, ensuring that the text content is complete and accurate, and avoiding missing or incorrect information.

[0082] Next, by obtaining the start and end timestamps of the complete text segment, the time range of the text within the audio can be accurately located. Finally, based on the start and end timestamps, the tag types are embedded into the original audio text, that is, the tag types are embedded before and after the complete text segment. This allows the tag types to perfectly match the corresponding speech and non-language information in time, so that the generated target semantic data can more completely and accurately reflect the semantic content of the audio, thus improving the quality of information processing.

[0083] For example: The original audio text is "He laughed so happily," and the tag type is "laugh." The non-verbal timestamp of the tag type "laugh" occurs on the timestamp of the word "laugh." Therefore, the word "laugh" is the target text segment. Integrity checks are performed on the target text segment and the original audio text to obtain the complete text segment "laughed to the point of laughter," indicating that the non-verbal action of laughing and the phonological sound of "laughed to the point of laughter" occur simultaneously. Further, the start and end timestamps of the complete text segment "laughed to the point of laughter" are obtained. The tag type "laugh" is embedded into the original audio text based on the start timestamp, and the tag type "laugh" is also embedded into the original audio text based on the end timestamp, resulting in the target semantic data: "He..." <laugh>Laughing <laugh>"So happy."

[0084] Following step S105 in some embodiments, the speech-based semantic extraction method provided in this application extracts semantics from audio data. The resulting target semantic data can be input into a large model for dialogue generation and applied to intelligent customer service, guidance robots, etc. in fintech and medical technology scenarios. It can also be used for digital image synthesis in fintech and medical technology scenarios to synthesize promotional materials, advertisements, teaching videos, etc. It can also be used to construct multilingual nonverbal datasets to provide high-quality data support for speech model training and can be widely applied in many scenarios.

[0085] Please see Figure 7 This application also provides a speech-based semantic extraction apparatus, which can implement the above-described speech-based semantic extraction method. The apparatus includes: The audio acquisition module 701 is used to acquire raw audio data; The audio processing module 702 is used to perform audio preprocessing on the raw audio data to obtain the target audio data; The text conversion module 703 is used to convert the target audio data into text to obtain the original audio text. The audio detection module 704 is used to perform audio category detection based on the target audio data and the original audio text to obtain non-language labels; wherein, the non-language labels are used to indicate the presence of non-language audio in the original audio data; The speech annotation module 705 is used to annotate the original audio text based on non-linguistic tags to obtain the target semantic data of the original audio data.

[0086] The specific implementation of this speech-based semantic extraction device is basically the same as the specific implementation of the speech-based semantic extraction method described above, and will not be repeated here.

[0087] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described speech-based semantic extraction method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0088] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 to execute the speech-based semantic extraction method of the embodiments of this application. The 803 input / output interface is used to implement information input and output. The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804); The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0089] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described speech-based semantic extraction method.

[0090] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0091] The speech-based semantic extraction method, apparatus, electronic device, and storage medium provided in this application acquire raw audio data and perform audio preprocessing to remove noise and other interference, resulting in cleaner target audio data. The target audio data is then converted into text to obtain the original audio text, improving the accuracy of subsequent processing. Next, audio category detection is performed based on the target audio data and the original audio text to obtain non-verbal tags indicating the presence of non-verbal audio in the raw audio data, enabling accurate extraction of non-verbal information. Finally, the original audio text is labeled based on the non-verbal tags to obtain the target semantic data of the raw audio data, enabling comprehensive and accurate mining of audio semantic information and improving the accuracy of audio semantic extraction.

[0092] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0093] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0094] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0095] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0096] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0097] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0099] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0100] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0101] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] The software tools or components not belonging to our company that appear in the embodiments of this application are for illustrative purposes only and do not represent actual use.

[0103] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.< / laugh> < / laugh>

Claims

1. A speech-based semantic extraction method, characterized in that, The method includes: Obtain the raw audio data; The original audio data is preprocessed to obtain the target audio data; The target audio data is converted into text to obtain the original audio text; Based on the target audio data and the original audio text, audio category detection is performed to obtain non-verbal labels; wherein, the non-verbal labels are used to indicate the presence of non-verbal audio in the original audio data; The original audio text is annotated based on the non-linguistic tags to obtain the target semantic data of the original audio data.

2. The method according to claim 1, characterized in that, The step of performing audio category detection based on the target audio data and the original audio text to obtain non-linguistic labels includes: Feature extraction is performed on the target audio data to obtain initial audio features; The original audio text is text-encoded to obtain audio text encoding features; The initial audio features and the audio text encoding features are concatenated to obtain concatenated features; Obtain multiple preset non-language query vectors; wherein, the non-language query vectors are used to query non-language types in audio; The non-linguistic tag is obtained by performing non-linguistic detection on the concatenated features based on the non-linguistic query vector.

3. The method according to claim 2, characterized in that, The non-linguistic detection of the concatenated features based on the non-linguistic query vector to obtain the non-linguistic label includes: The spliced ​​features are then encoded to obtain the original encoded features; Based on each of the non-linguistic query vectors, a non-linguistic query is performed on the original encoded features to obtain non-linguistic query features; The non-linguistic query features and the original encoded features are classified to obtain a non-linguistic classification matrix; The non-linguistic labels are determined based on the non-linguistic classification matrix.

4. The method according to claim 3, characterized in that, The process of classifying the non-linguistic query features and the original encoded features to obtain a non-linguistic classification matrix includes: For each of the non-linguistic query features, activation calculation is performed based on the non-linguistic query features and the original encoded features to obtain the non-linguistic type probability; The non-language type probabilities are filtered based on a preset probability threshold to obtain the identification type data; The non-language classification matrix is ​​obtained by constructing a matrix based on the recognition type data.

5. The method according to claim 1, characterized in that, The non-linguistic tags include tag types and non-linguistic timestamps; the annotation of the original audio text based on the non-linguistic tags to obtain the target semantic data of the original audio data includes: Based on the non-linguistic timestamp, the tag type is timestamped with the original audio text to obtain an aligned timestamp; Extract the target text segment from the original audio text based on the aligned timestamp; If the target text fragment does not contain speech information, the tag type is embedded into the original audio text based on the alignment timestamp to obtain the target semantic data.

6. The method according to claim 5, characterized in that, After extracting the target text segment from the original audio text based on the aligned timestamp, the method further includes: If the target text segment contains speech information, perform an integrity check on the target text segment and the original audio text to obtain a complete text segment; Obtain the start and end timestamps of the complete text segment; The tag type is embedded into the original audio text based on the start timestamp, and the tag type is embedded into the original audio text based on the end timestamp to obtain the target semantic data.

7. The method according to any one of claims 1 to 6, characterized in that, The step of preprocessing the original audio data to obtain the target audio data includes: The original audio data is denoised to obtain denoised audio data; Speaker separation is performed on the denoised audio data to obtain the initial audio data; Speech activity detection is performed on the initial audio data to obtain labels for non-silent segments; Based on the non-silent segment tags, audio is extracted from the initial audio data to obtain the target audio data.

8. A speech-based semantic extraction device, characterized in that, The device includes: The audio acquisition module is used to acquire raw audio data; The audio processing module is used to perform audio preprocessing on the raw audio data to obtain target audio data; The text conversion module is used to convert the target audio data into text to obtain the original audio text; An audio detection module is used to perform audio category detection based on the target audio data and the original audio text to obtain non-verbal tags; wherein, the non-verbal tags are used to indicate the presence of non-verbal audio in the original audio data; The speech annotation module is used to annotate the original audio text based on the non-linguistic tags to obtain the target semantic data of the original audio data.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images