Audio-to-text conversion method and device, electronic equipment and storage medium

Abstract

The invention provides an audio-to-text conversion method and device, electronic equipment and a storage medium, and belongs to the technical field of voice recognition, and the method comprises the steps: carrying out the feature extraction of a target audio, and obtaining an audio feature used for representing the content of the target voice; performing segmentation processing based on the time sequence of the audio features to obtain a plurality of audio segments; generating a plurality of text segments based on the plurality of audio segments; and adding corresponding punctuation marks to the plurality of text segments based on the semantic model. According to the audio-to-text conversion method and device, the electronic equipment and the storage medium provided by the invention, high-precision text output can be realized.

Description

Technical Field

[0001] The present disclosure belongs to the field of speech recognition technology, and more specifically, relates to a method and device for converting audio to text, an electronic device, and a storage medium. Background Art

[0002] With the continuous advancement of speech recognition technology, people are increasingly looking forward to the efficient and accurate conversion of audio to text. However, traditional audio-to-text conversion methods face many challenges. Among them, the most prominent problem is the trade-off between accuracy and efficiency. These methods may be able to cope with simple speech signals, but once they encounter complex speech environments or need to output text containing punctuation marks, their limitations are exposed. It is often impossible to achieve high-precision text output while ensuring real-time performance. Summary of the invention

[0003] The purpose of the present disclosure is to provide an audio-to-text conversion method and device, electronic device, and storage medium to achieve high-precision text output.

[0004] A first aspect of an embodiment of the present disclosure provides a method for converting audio to text, comprising: Extracting features of the target audio to obtain audio features for representing the target speech content; Perform segmentation processing based on the time series of the audio features to obtain multiple audio segments; generating a plurality of text segments based on the plurality of audio segments; Corresponding punctuation marks are added to the multiple text segments based on the semantic model to generate text information.

[0005] A second aspect of the embodiments of the present disclosure provides an audio to text conversion device, including: A feature extraction module is used to extract features from the target audio to obtain audio features for representing the target speech content; A feature segmentation module, used for performing segmentation processing based on the time series of the audio features to obtain multiple audio segments; A text generation module, configured to generate a plurality of text segments based on the plurality of audio segments; The punctuation mark adding module is used to add corresponding punctuation marks to the multiple text segments based on the semantic model to generate text information.

[0006] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned audio-to-text conversion method when executing the computer program.

[0007] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned audio-to-text conversion method are implemented.

[0008] The beneficial effects of the audio-to-text conversion method and device, electronic device, and storage medium provided by the embodiments of the present disclosure are as follows: the embodiments of the present disclosure firstly extract features from the target audio, thereby being able to dig out key information from complex audio signals, making it easier for computers to understand and process, effectively overcoming the difficulty of directly processing the original audio, and laying the foundation for subsequent accurate conversion. Secondly, based on the audio feature time series segmentation, the processing complexity is reduced and the conversion accuracy is improved, and the speech units are accurately divided according to the speech pauses. Then, based on multiple audio segments, text segments are generated, and the speech information can be accurately converted into text expressions, thereby improving the effectiveness of the conversion. Finally, punctuation marks are added based on the semantic model, so that the generated text is more in line with human language habits, the readability and comprehensibility of the text are enhanced, and appropriate punctuation marks can be accurately added according to the sentence type and semantic relationship, thereby achieving high-precision text output while ensuring real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A flowchart of a method for converting audio to text provided in one embodiment of the present disclosure; Figure 2 A structural block diagram of an audio to text conversion device provided by an embodiment of the present disclosure; Figure 3 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0011] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0012] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0013] Please refer to Figure 1 , Figure 1 A flowchart of a method for converting audio to text provided in an embodiment of the present disclosure is provided, and the method comprises: S101: extracting features from the target audio to obtain audio features for representing the target speech content.

[0014] In this embodiment, the initial audio is preprocessed to obtain the target audio.

[0015] In this embodiment, the initial audio is the most original audio data, and the initial audio can be live voice recorded by a microphone, audio files (such as .wav, .mp3 and other formats) read from various storage devices, or audio streams obtained from network streaming media. These initial audios vary greatly in content and quality. For example, the recording environment may be very noisy, the audio files may have been compressed to varying degrees, resulting in reduced sound quality, or the audio stream may have problems such as packet loss and jitter caused by network transmission. The impact of these adverse factors on audio quality can be reduced or eliminated through preprocessing. For example, noise reduction processing can remove background noise and enhance the clarity of speech signals, so that the audio features extracted subsequently can more accurately reflect the speech content instead of being interfered by noise features.

[0016] Preprocessing of the initial audio includes noise reduction, format conversion, and data screening.

[0017] For example, the initial audio is recorded on a noisy street, which contains background noise such as vehicle driving sound and crowd noise. Through noise reduction processing (such as spectrum subtraction), the speaker's voice can be highlighted, so that the subsequent extracted audio features can focus more on the voice part, such as the formant and fundamental frequency of the voice. Features can be extracted more clearly. If the sampling rate of the initial audio is 44.1kHz, and the sampling rate required by the system is 16kHz, it can be converted to a suitable sampling rate through resampling technology. This ensures that the frequency characteristics of the audio data meet the requirements of the subsequent feature extraction algorithm. For example, some audio feature extraction methods based on a specific frequency range can work more accurately at a suitable sampling rate. Calculate the energy of the initial audio. If the energy of the initial audio is at a very low level (close to silence) for a long time, it can be judged that the audio may not have valid voice content. For the verification of the duration, a minimum and maximum duration range can be set according to the requirements of the system. Audio beyond this range can be regarded as not meeting the requirements. Finally, the target audio is obtained.

[0018] In this embodiment, feature extraction is to dig out key information that can effectively represent the target speech content from a complex audio signal. The target audio is a complex waveform, and directly processing and analyzing it is difficult for subsequent text conversion. By extracting audio features, the audio signal can be converted into a form that is easier for a computer to understand and process.

[0019] In this embodiment, the target audio is feature extracted by using Mel-frequency cepstral coefficients to obtain audio features for characterizing the target speech content.

[0020] The target audio is pre-emphasized and framed and windowed to obtain first feature data; the first data is fast Fourier transformed and filtered to obtain second data; the second data is discrete cosine transformed to obtain audio features for characterizing the target speech content.

[0021] Exemplarily, the high frequency part of the speech signal contains more speech detail information. First, the target audio is pre-emphasized to enhance the high frequency part, and then a frame windowing operation is performed to divide the audio signal into multiple short frames, and each frame is multiplied by a window function (such as a Hamming window) to reduce spectrum leakage. Then a fast Fourier transform is performed to convert the time domain signal into a frequency domain signal to obtain the spectrum of the audio signal. After that, the spectrum is filtered by a Mel filter bank. The Mel filter bank has a higher resolution in the low frequency part and a lower resolution in the high frequency part, which is consistent with the auditory characteristics of the human ear. Finally, the result after filtering by the Mel filter bank is logarithmic and discrete cosine transformed to obtain the Mel frequency cepstral coefficient (ie, audio feature). The Mel frequency cepstral coefficient can well reflect important speech features such as the resonance peak of speech.

[0022] S102: Perform segmentation processing based on the time series of audio features to obtain multiple audio segments.

[0023] In this embodiment, the audio features obtained after feature extraction are arranged in chronological order, which forms a time series of audio features. This time series contains various acoustic feature information of speech at different time points, such as the frequency feature and energy feature of speech at a certain moment. Segmenting the time series of the entire audio can decompose a complex and long speech sequence into multiple relatively simple subsequences (i.e., audio segments). This reduces the complexity and difficulty of processing and can improve the accuracy of audio-to-text conversion.

[0024] In this embodiment, segmentation processing can be performed based on speech pauses, speech prosodic features, and boundary detection of an acoustic model to obtain multiple audio segments.

[0025] For example, in Chinese, in the sentence "I am very happy today", there will be short pauses between "I" and "today", and between "today" and "very happy". In the audio feature time series, segmentation can be performed by detecting the energy troughs corresponding to these pauses, and the audio corresponding to this sentence can be segmented into three audio segments, corresponding to "I", "today" and "very happy" respectively.

[0026] For example, in the English sentence "Do you like it?", the word "like" has a stress. In terms of audio features, the energy of the stressed part is higher than other parts, and the fundamental frequency may also change accordingly. By detecting these changes, the audio can be segmented according to the prosodic features, which helps to more accurately recognize the semantics of words and sentences in the future.

[0027] For example, for an audio containing multiple words, the trained DNN model can predict the probability of each time point belonging to a word boundary based on the time series of audio features. When the probability exceeds a certain threshold, the position can be determined as the boundary of the audio segment, thereby segmenting the audio into multiple audio segments, each of which may correspond to a word or a phrase.

[0028] S103: Generate multiple text segments based on the multiple audio segments.

[0029] In this embodiment, each audio segment contains specific speech information, which needs to be converted into corresponding text content to generate a text segment. An audio segment can be regarded as a local segment of speech, and a text segment is a textual description of this segment. For example, if an audio segment contains the pronunciation of a word, the corresponding text segment is the spelling of the word.

[0030] In this embodiment, the text generation model can map the acoustic features in the audio signal to the phoneme or subphoneme level. The text generation model establishes the probabilistic relationship between different acoustic features and speech units by learning a large amount of speech data. For example, for a specific audio feature vector, the text generation model can calculate the probability of it corresponding to each phoneme.

[0031] For example, when processing an audio segment, the acoustic features of the audio segment, such as Mel-frequency cepstral coefficients, are first extracted. These features are then input into the text generation model, which can give the most likely phoneme sequence corresponding to the audio segment based on the learned probability distribution. For example, for an audio segment containing a vowel pronunciation, the acoustic model may output the corresponding vowel phoneme.

[0032] S104: Add corresponding punctuation marks to multiple text segments based on the semantic model to generate text information.

[0033] In the present embodiment, the semantic model can understand the semantic content of the text segment so as to accurately add the corresponding punctuation marks. The semantic model can capture the semantic information, grammatical structure and contextual relationship in the text, so as to determine where to add punctuation marks most appropriately. The semantic model can be trained by a large amount of text data. These text data contain various types of sentences and correct punctuation mark annotations. By learning from these data, the semantic model can gradually grasp the corresponding relationship between different text segments and punctuation marks.

[0034] In this embodiment, multiple text segments are analyzed as input semantic models to obtain semantic analysis results; and corresponding punctuation marks are added to corresponding positions in the text segments based on the semantic analysis results.

[0035] Exemplarily, for declarative sentences, the semantic model can determine whether the text segment expresses a complete statement, and if so, add a period to indicate the end of the sentence. For interrogative sentences, the semantic model can detect whether there are interrogative words (such as "who", "what", "where", etc.) or expressions of interrogative tone (such as "maybe", "where", etc.) in the text segment. If these features exist, a question mark will be added to the end of the sentence. For exclamatory sentences, the semantic model can analyze the vocabulary and tone in the text segment to determine whether strong emotions or emotions such as surprise are expressed. If so, an exclamation mark will be added to emphasize this emotion. In addition to sentence-end punctuation, the semantic model can also add punctuation such as commas based on the internal structure of the text segment. For example, when listing multiple things, commas can be added between each thing; in complex sentences, commas can be added at appropriate locations to separate different components or clauses based on grammatical structure and semantic relationships.

[0036] It can be concluded from the above that this embodiment firstly extracts the features of the target audio, and can dig out key information from the complex audio signal, making it easier for the computer to understand and process, effectively overcoming the difficulty of direct processing of the original audio, and laying the foundation for subsequent accurate conversion. Secondly, based on the audio feature time series segmentation, the processing complexity is reduced and the conversion accuracy is improved, and the speech units are accurately divided according to the speech pauses. Then, based on multiple audio segments, text segments are generated, and the speech information can be accurately converted into text expressions, which improves the effectiveness of the conversion. Finally, punctuation marks are added based on the semantic model, so that the generated text is more in line with human language habits, the readability and comprehensibility of the text are enhanced, and appropriate punctuation marks can be accurately added according to the sentence type and semantic relationship, thereby achieving high-precision text information output while ensuring real-time performance.

[0037] In one embodiment of the present disclosure, segmentation processing is performed based on the time series of audio features to obtain multiple audio segments, including: If the time interval of the audio feature is greater than the first time, the audio feature is segmented to obtain multiple audio segments.

[0038] In this embodiment, the time series of audio features is a series of audio feature data points arranged in chronological order. The time interval refers to the time length corresponding to two adjacent audio feature data points. For example, if the audio is collected at a certain number of audio features per second, then the time interval between two adjacent feature points is 1 second divided by the collection frequency.

[0039] In this embodiment, the first time is a pre-set time threshold. This threshold is determined based on the general characteristics of speech and actual application requirements. For example, based on the statistical analysis of a large amount of speech data, it is found that the pauses between words or the natural intervals between speech segments are usually within a certain time range, and the average value or typical value of this range is set as the first time. When the time interval of the audio feature is greater than the first time, it is considered that there may be a boundary of the speech content at this time interval. By detecting this time interval, the audio feature time series can be segmented at these possible boundaries, thereby obtaining multiple relatively independent audio segments.

[0040] Exemplarily, assuming that the first time is set to 0.5 seconds, when processing a speech, it is found that the time interval between two adjacent audio feature points is 0.8 seconds, which is greater than the set threshold of 0.5 seconds. Then the segmentation can be performed between these two feature points, and the previous audio part is divided into one audio segment, and the subsequent audio part is processed as another audio segment. In this way, if the speech is "Hello, the weather is so nice today. I want to go for a walk in the park.", there may be a long pause between "It's so nice." and "I want to", and the corresponding time interval exceeds the first time, then the audio can be divided into two audio segments here, corresponding to the two sentence fragments "Hello, the weather is so nice today." and "I want to go for a walk in the park.", so that they can be converted to text and other operations separately later.

[0041] From the above, it can be concluded that this embodiment can accurately identify and segment the boundaries of speech content in the audio by performing intelligent segmentation processing based on the time interval of audio features and a pre-set time threshold (first time), and divide the continuous audio stream into multiple relatively independent audio segments, thereby effectively improving the accuracy and efficiency of audio-to-text conversion.

[0042] In one embodiment of the present disclosure, it also includes: The speaking speed of the target speech is determined based on the audio feature, and the first time is determined based on the speaking speed.

[0043] In this embodiment, the audio features are extracted from the original audio signal, which can reflect various properties of the audio signal. These properties contain rich voice information, including the speaker's speaking speed. The speaking speed, that is, the number of syllables or words uttered by the speaker per minute, is an important indicator reflecting the speed of the speaker. There are significant differences in the speaking speed of different people. For people with fast speaking speed, the time interval of their audio features is relatively short, because they utter more syllables or words per unit time, resulting in the voice segments in the audio signal being arranged more closely together. On the contrary, for people with slow speaking speed, the time interval of their audio features is relatively long, because they utter fewer syllables or words per unit time, and the interval between the voice segments is correspondingly increased. If the same first time is used for segmentation of people with fast speaking speed and people with slow speaking speed, inaccurate segmentation may result. For example, for people with fast speaking speed, the audio that should belong to one semantic unit may be mistakenly segmented; while for people with slow speaking speed, some suitable segmentation points may be missed, resulting in unclear division of semantic units.

[0044] In this embodiment, the first time is determined based on a mapping relationship of the speech rate, and the mapping relationship is a mapping relationship between the speech rate and the first time.

[0045] For example, a mapping relationship between speech rate and first time can be established and determined by a function or a lookup table. For example, an empirical formula between speech rate and first time can be obtained based on statistical analysis of a large amount of speech data. According to this formula, when the speech rate of the target speech is determined, a suitable first time value can be calculated for audio segmentation.

[0046] From the above, it can be concluded that this embodiment can more accurately identify and segment the boundaries of speech content at different speaking speeds by dynamically adjusting the first time according to the speaking speed of the target speech, effectively avoiding the problem of inaccurate segmentation caused by differences in speaking speed, and improving the accuracy and applicability of audio segmentation.

[0047] In one embodiment of the present disclosure, it also includes: The text information is adjusted based on multimodal information fusion to obtain the target text information.

[0048] In this embodiment, in the task of audio-to-text conversion, relying solely on audio features may not be able to fully capture all semantic information. Introducing information from other modalities can provide additional clues and context, thereby more accurately understanding the audio content and generating better quality text information. For example, if the audio comes from a video, the visual information in the video frame may include the speaker's mouth shape, expression, action, etc., which can help determine the content and emotional tendency of the speech. If the audio is associated with a specific scene, the image information can provide background knowledge of the scene and help understand the objects or events mentioned in the audio. Keywords or contextual information entered by the user can provide clear guidance and constraints for text generation.

[0049] In this embodiment, multimodal information fusion can be implemented by a multimodal attention mechanism model, which can automatically focus on the modal information that is most valuable to the current task by learning the attention weights between different modalities. For example, in audio-to-text conversion, the model can dynamically adjust the weights of audio features, video frame features, image features, and user input features in the fusion process based on their importance. For some audio clips, the lip information in the video frame may be very important, and the model will give the video frame a higher attention weight; for other clips, the keywords entered by the user may be more critical, and the model will adjust the weight distribution accordingly.

[0050] In this embodiment, feature extraction is performed on video information, audio information and image information to obtain multimodal feature information; the multimodal feature information is input into a multimodal attention mechanism model to calculate the attention weight; the multimodal feature information is weighted fused to obtain comprehensive semantic information; the comprehensive semantic information is input into a text generation model to obtain target text information.

[0051] In this embodiment, first, feature extraction is performed on information of different modalities. For video frames, convolutional neural networks can be used to extract visual features; for images, similar methods can be used to extract image features; for user input, natural language processing techniques (such as word vector representation) can be used to convert it into numerical features. Then, these features of different modalities are input into the multimodal attention mechanism model. The model calculates the attention weights and performs weighted fusion of features of different modalities to obtain a comprehensive semantic representation. Finally, this fused semantic representation is input into the text generation model to generate optimized and adjusted target text information.

[0052] From the above, it can be concluded that this embodiment achieves more accurate and comprehensive semantic understanding by capturing multi-dimensional information such as audio, video, and images, and using a multimodal attention mechanism model to dynamically adjust the weight of each modal information, thereby generating a more accurate, fluent, and context-rich target text, greatly enhancing the user experience and the effects of subsequent text processing tasks.

[0053] In one embodiment of the present disclosure, it also includes: Calculating the timbre change rate of adjacent audio segments among the multiple audio segments; if the timbre change rate of the adjacent audio segments is greater than a first preset value, determining that the scene corresponding to the audio segment is a dialogue scene; In response to a dialogue scenario, labels are added to corresponding multiple text segments based on the timbre of the multiple audio segments.

[0054] In this embodiment, timbre is a characteristic of sound, which is mainly determined by the spectral structure of the sound. Different people speaking, different musical instruments playing or different environmental sounds all have different timbres. In audio processing, timbre can be characterized by analyzing the spectral characteristics of the audio signal.

[0055] For adjacent audio segments in multiple audio segments, their timbre features are first extracted. Mel frequency cepstral coefficients, etc. can be used to obtain feature vectors that can reflect the timbre. Then, the difference between the timbre features of adjacent audio segments is calculated, and the difference is divided by the time interval to obtain the timbre change rate. For example, the difference in timbre features can be calculated using methods such as Euclidean distance and cosine similarity, and then the change rate is calculated based on the time interval between adjacent audio segments.

[0056] The first preset value is a preset threshold value used to determine whether the audio segment is in a dialogue scene. The first preset value can be obtained by analyzing and counting a large number of audio data of dialogue scenes and non-dialogue scenes. For example, the distribution of the timbre change rate of adjacent audio segments in different scenes can be calculated, and then a suitable value can be selected as the first preset value, so that the dialogue scene and the non-dialogue scene can be better distinguished. If the timbre change rate of the adjacent audio segment is greater than the first preset value, it is determined that the scene corresponding to the audio segment is a dialogue scene. This is because in a dialogue scene, different people take turns to speak, and their timbre usually changes greatly. In non-dialogue scenes, such as one person talking continuously or background music, the change in timbre is relatively small.

[0057] In a conversation scenario, the target audio is usually emitted by different people. By analyzing the timbre of the audio, the audio signals emitted by each person can be distinguished. For example, different people's voices have different characteristics in terms of frequency distribution, resonance peaks, etc. These characteristics are reflected in the timbre characteristics. Based on these timbre characteristics, different labels can be assigned to different people.

[0058] For each audio segment, the corresponding speaker label is determined based on its timbre characteristics. Then, when a text segment is generated, this label is added to the text segment. For example, if the timbre characteristics of audio segment A correspond to speaker A, then when generating the text segment corresponding to audio segment A, a label "A:" is added to distinguish the audio signals emitted by different people. In this way, when reading the generated text, you can clearly know who spoke each text segment, which improves the readability and comprehensibility of the text.

[0059] For example, suppose there is a target audio recording of two people talking. The audio is divided into multiple audio segments. First, the timbre features of each audio segment are extracted. Assuming that the MFCC method is used, a set of feature vectors for each audio segment is obtained.

[0060] For two adjacent audio segments, such as audio segment 1 and audio segment 2, the difference in their timbre feature vectors is calculated. The Euclidean distance can be used for calculation, and it is assumed that the obtained distance is d.

[0061] It is known that the time interval between these two audio segments is t seconds, so the rate of timbre change is d / t.

[0062] Assume that the first preset value is P. If the calculated timbre change rate of adjacent audio segments is greater than P, it is determined that the scene corresponding to the audio segment is a dialogue scene.

[0063] For example, assuming that the calculated timbre change rate of a certain adjacent audio segment is 0.8, and the first preset value P=0.5, since 0.8>0.5, it is determined that this is a dialogue scene.

[0064] Different labels are assigned to different people based on the timbre characteristics of the audio. Suppose that through analysis, it is found that the timbre characteristics of audio segments 1 to 5 correspond to speaker A, and the timbre characteristics of audio segments 6 to 10 correspond to speaker B.

[0065] When a text segment is generated, add the corresponding label to it. If the text segment corresponding to audio segment 2 is "The weather is really nice today.", then after adding the label, it becomes "A: The weather is really nice today."; if the text segment corresponding to audio segment 8 is "It's really nice.", then it becomes "B: It's really nice.".

[0066] This allows the generated text segments to be labeled in conversational scenarios in order to distinguish between audio signals from different people.

[0067] It can be concluded from the above that this embodiment determines the dialogue scene by calculating the timbre change rate of adjacent audio segments in multiple audio segments, and adds labels to the corresponding text segments based on the timbre features, thereby effectively distinguishing between dialogue and non-dialogue scenes, and accurately marking the text segments of different speakers, thereby improving the readability and comprehensibility of the text, and making the generated text clearer and easier to read.

[0068] In one embodiment of the present disclosure, generating multiple text segments based on multiple audio segments includes: A text generation model is obtained by training based on a plurality of first audio segments and a plurality of first text segments corresponding to the plurality of first audio segments, wherein the plurality of first audio segments are a plurality of historical audio segments, and the plurality of first text segments are text segments corresponding to the plurality of historical audio segments; Multiple audio segments are input into the text generation model to obtain multiple text segments.

[0069] In this embodiment, first, multiple historical audio segments are collected as training data. These historical audio segments can come from various sources, such as existing voice databases, past audio recordings, etc. At the same time, for each historical audio segment, there needs to be a corresponding correct text segment. For example, these data can be obtained from speech recognition annotation data sets, voice recordings in specific fields, and manually annotated texts. The collected historical audio segments and corresponding text segments are preprocessed. For audio segments, noise reduction, format conversion, etc. can be performed to ensure the consistency of audio quality and format. For text segments, text cleaning can be performed to remove special symbols, punctuation errors, etc. to make them more standardized.

[0070] In this embodiment, a recurrent neural network, a long short-term memory network, a Transformer, etc. can be selected as the text generation model architecture, and the audio features are extracted for each historical audio segment. The extracted audio features are used as input, and the corresponding text segment is used as the output target, so that the model learns the mapping relationship between the audio features and the text. The text generation model is trained using training data. During the training process, the model continuously adjusts its own parameters to minimize the difference between the predicted text and the real text. For example, a loss function (such as cross entropy loss) can be used to measure the gap between the predicted text and the real text, and the model parameters can be updated through an optimization algorithm (such as stochastic gradient descent, etc.) so that the loss function gradually decreases.

[0071] When multiple new audio segments need to be converted into text segments, these audio segments are input into the trained text generation model. The text generation model processes the input audio segments based on the mapping relationship between the learned audio features and text, and predicts the corresponding text content. The model can generate accurate text segments based on the features of the input audio segments and the patterns and rules learned in the previous training process.

[0072] For example, suppose we have collected some historical audio segments and corresponding text segments from a speech recognition research project. Some of the historical audio segments are as follows: Historical audio segment 1: A clear voice with the content "Hello world". The corresponding text segment is "Helloworld".

[0073] Historical audio segment 2: The voice content is "This is a good day". The corresponding text segment is "This is a good day".

[0074] Historical audio segment 3: "I am happy today". The corresponding text segment is "I am happy today".

[0075] These audio segments are processed for noise reduction to remove possible background noise and ensure the quality of the audio. At the same time, the text segments are simply cleaned up to remove extra spaces and irregular punctuation.

[0076] The LSTM-based neural network model is selected as the text generation model. For each historical audio segment, MFCC features are extracted. For example, for historical audio segment 1, the extracted audio features are input into the model, and the goal of the model is to output the corresponding text "Hello world".

[0077] During the training process, the model continuously adjusts its parameters. Assume that initially, the model's prediction for historical audio segment 1 may be "Hello wordl", which is somewhat different from the real text. By calculating the cross entropy loss, the model knows that its prediction is inaccurate, and then uses an optimization algorithm (such as the Adam optimizer) to adjust the model's parameters such as weights and biases. After multiple iterations of training, the model gradually reduces the loss, making the prediction results closer and closer to the real text.

[0078] Now there are multiple new audio segments that need to be converted into text segments. For example, the new audio segment is a voice with the content "Today is sunny".

[0079] The audio features of this audio segment are extracted and input into the trained text generation model. The model makes predictions based on the mapping relationship between audio features and text learned previously. After processing, the model outputs the text segment "Today is sunny".

[0080] It can be concluded from the above that this embodiment constructs an efficient text generation model by using historical audio segments and their corresponding text segments for training, which not only simplifies the audio-to-text conversion process, but also improves the accuracy and efficiency of the conversion.

[0081] In one embodiment of the present disclosure, training a text generation model based on a plurality of first audio segments and a plurality of first text segments corresponding to the plurality of first audio segments includes: determining a target loss function based on the plurality of first text segments; The text generation model is obtained based on the target loss function.

[0082] In this embodiment, the loss function is used in model training to measure the difference between the model's predicted results and the actual results. By minimizing the loss function, the model can continuously adjust its own parameters to improve the accuracy of the prediction. In the task of audio-to-text conversion, the purpose of the target loss function is to ensure that the text generation model can accurately convert the audio segment into the corresponding text segment.

[0083] In this embodiment, the cross entropy loss function can be selected as the target loss function. The cross entropy loss function can regard the prediction of each character or word as a classification problem. For a given audio segment, the model predicts the probability distribution of each possible character or word, and the real text segment can be regarded as a standard category label. The cross entropy loss function calculates the difference between the probability distribution predicted by the model and the real label.

[0084] Taking into account the particularities of audio-to-text conversion, the loss function can be adjusted. For example, if there are specific language structures or grammatical rules in the text segment, some regularization terms can be added to the loss function to encourage the model to generate text that conforms to these rules. In addition, if certain words or characters appear more frequently in the text segment, the loss of these words or characters can be given a higher weight, so that the model pays more attention to these important parts.

[0085] In this embodiment, a text generation model is obtained based on a target loss function. Before training begins, the parameters of the text generation model are randomly initialized. These parameters include weights and biases of a neural network, etc. Iterative training is performed using training data. In each iteration, an audio segment is input into the text generation model, the text generation model predicts the corresponding text, and then the loss value between the predicted result and the real text is calculated according to the target loss function. According to the loss value, an optimization algorithm (such as a stochastic gradient descent algorithm) is used to update the parameters of the model. The purpose of the optimization algorithm is to find a set of parameters that minimize the loss function. Through continuous iterative training and parameter updating, the text generation model gradually learns the mapping relationship between audio features and text, and improves the accuracy of prediction. After multiple iterative training and parameter adjustment, when the performance of the text generation model on the validation set reaches a certain standard or the loss function converges to a smaller value, it can be considered that the model has been trained. The text generation model obtained at this time can more accurately convert the new audio segment into the corresponding text segment.

[0086] From the above, we can conclude that by designing the target loss function and iteratively optimizing the model parameters, we can effectively improve the accuracy of audio-to-text conversion, enable the model to learn the complex mapping relationship between audio features and text, and thus achieve more accurate speech recognition and text conversion in practical applications.

[0087] In one embodiment of the present disclosure, determining a target loss function based on a plurality of first text segments includes: Determining a first loss function based on text features of the plurality of first text segments; Determining a second loss function based on semantic features of the plurality of first text segments; The first loss function and the second loss function are weighted to obtain the target loss function.

[0088] In this embodiment, in order to more comprehensively and accurately measure the performance of the text generation model, so as to effectively guide the model training and improve the quality of audio-to-text conversion. The first loss function is determined based on the text features, and the difference between the generated text and the real text can be measured from the information at the text level such as character distribution and word frequency, and its advantage is that it can automatically handle the time alignment problem between audio and text, which is convenient for processing heterogeneous data. The second loss function is determined based on the semantic features, which directly considers the semantic difference between the generated text and the real text through the edit distance from the semantic level. This method is similar to the intuitive feeling of human beings on text errors, and can focus on spelling errors, word omissions and the like. Using a single loss function alone cannot take into account all aspects of the text. The target loss function is obtained by weighted calculation, which can comprehensively consider the text features and semantic features, so that the model can be fully optimized during the training process, so that the generated text is closer to the real text in multiple dimensions.

[0089] In this embodiment, the first loss function is:

[0090] in, Represented as the vector of the i-th real text segment; It represents the i-th text segment generated by the text generation model, and n represents the number of text segments.

[0091] By representing both the real text and the generated text in vector form, their differences in the vector space can be compared. For audio-to-text conversion tasks, text features can include information such as character distribution and word frequency. When the text segment generated by the model differs greatly from the real text segment in these features, the squared error will be large. During the training process, the model will strive to reduce this error, thereby adjusting its own parameters so that the generated text is closer to the real text in terms of text features.

[0092] The first loss function can automatically handle the time alignment problem between audio and text because it does not rely on explicit alignment operations. In audio and text conversion, since the lengths of audio and text may not correspond exactly, traditional methods may require complex time alignment steps. This loss function can gradually optimize the model to generate more accurate text without explicit alignment by comparing the overall text feature differences, which is very convenient for processing heterogeneous data such as audio and text.

[0093] In this embodiment, the second loss function is:

[0094] in, represents the text segment generated by the text generation model, Represented as a real text segment, Expressed as edit distance, and Respectively represent the text segment generated by the text generation model and the real text segment Length.

[0095] The second loss function can directly measure the difference between the generated text and the real text from a semantic level. For example, if there are spelling errors, missing words, etc. in the generated text, the edit distance will increase, which will lead to an increase in the loss function value. By minimizing this loss function, the model will strive to reduce these semantic errors, making the generated text closer to the semantics of the real text.

[0096] The second loss function directly measures the difference between the generated text and the real text from a semantic level, which is closer to human’s intuitive feeling about text errors, such as spelling errors, word omissions, etc.

[0097] In this embodiment, the first loss function mainly measures the difference from the perspective of text features, and the second loss function mainly measures the difference from the perspective of semantic features. Using any one loss function alone may not be able to fully reflect the gap between the generated text and the real text. Therefore, it is necessary to combine the two and comprehensively consider different aspects of the text.

[0098] By performing weighted calculation on the first loss function and the second loss function, the target loss function is obtained, and the calculation formula is:

[0099] in, Expressed as the first loss function, Expressed as the second loss function, is expressed as the adjustment parameter ( ), Expressed as an adjustable exponential parameter ( , you can adjust its value according to the actual situation The degree of its role in the loss function), In this embodiment, the adjustment parameters The proportion of the first loss function and the second loss function in the target loss function can be adjusted more flexibly. When L1 and L2 are given equal importance; when When it is close to 0, more emphasis is placed on L2; when When it is close to 1, more emphasis is placed on L1.

[0100] L2 was The power operation can further adjust the impact on the overall loss. For example, when When , it is the original weighted form; when When , the influence of larger L2 values ​​can be amplified, allowing the text generation model to adjust more quickly when the semantic difference is large; when When , the influence of L2 can be relatively smoothed, preventing the model from focusing too much on small semantic differences and ignoring other aspects.

[0101] For example, assume that there are three historical audio segments and their corresponding real text segments: Audio segment 1: A voice segment, the corresponding real text segment is "Today is a beautiful day.".

[0102] Audio segment 2: Another audio segment, the corresponding real text segment is "I love this place."

[0103] Audio segment 3: Another audio segment. The corresponding real text segment is "The sky is blue."

[0104] The real text segment and the generated text segment are represented as vectors. For example, each word can be represented as a vector by word vectors, and then the entire text segment is represented as a combination of these word vectors.

[0105] For audio segment 1, assume that the text segment generated by the text generation model is "Today is a nice day." Calculate the vector difference between the two text segments. Assume that the vector difference is obtained by calculation. Similarly, similar calculations are performed for audio segments 2 and 3, assuming that and . Then the result of the first loss function is 0.2.

[0106] For audio segment 1, the generated text segment is "Today is a nice day." and the true text segment is "Today isa beautiful day.". Calculate the edit distance , assuming it is 2 (for example, "beautiful" may be mistakenly generated as "nice"). Generated text segment length (corresponding to the length of the words "Today", "is", "a", "nice", and "day", respectively), the length of the actual text segment .

[0107] Then the second loss function Similarly, similar calculations are performed for audio segments 2 and 3.

[0108] Assume that the tuning parameters , adjustable exponential parameter .

[0109] Then the target loss function .

[0110] The previously calculated and Substitute into the formula to calculate the value of the objective loss function.

[0111] Through the target loss function, the text generation model can comprehensively consider the differences between text features and semantic features during the training process, and continuously adjust parameters to reduce losses and improve the accuracy of audio-to-text conversion.

[0112] From the above, it can be concluded that this embodiment can more comprehensively optimize the text generation model of audio-to-text conversion during the training process by weighted calculation of the first loss function and the second loss function, comprehensively consider the differences in text features and semantic features, thereby improving the accuracy of the conversion, making the generated text closer to the real text in multiple dimensions, and effectively improving the quality of audio-to-text conversion.

[0113] Corresponding to the audio to text conversion method of the above embodiment, Figure 2 This is a structural block diagram of an audio to text conversion device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 2 The audio-to-text conversion device 20 includes: a feature extraction module 21, a feature segmentation module 22, a text generation module 23, and a punctuation mark adding module 24. The feature extraction module 21 is used to extract features from the target audio to obtain audio features for representing the target speech content; A feature segmentation module 22, used for performing segmentation processing based on the time series of audio features to obtain multiple audio segments; A text generation module 23, configured to generate a plurality of text segments based on a plurality of audio segments; The punctuation mark adding module 24 is used to add corresponding punctuation marks to multiple text segments based on the semantic model to generate text information.

[0114] In one embodiment of the present disclosure, the feature segmentation module 22 is specifically used to: If the time interval of the audio feature is greater than the first time, the audio feature is segmented to obtain multiple audio segments.

[0115] In one embodiment of the present disclosure, the feature segmentation module 22 is further configured to: The speaking speed of the target speech is determined based on the audio feature, and the first time is determined based on the speaking speed.

[0116] In one embodiment of the present disclosure, the audio to text conversion device further includes: an optimization and adjustment module; The optimization and adjustment module is specifically used to adjust the text information based on multimodal information fusion to obtain the target text information.

[0117] In one embodiment of the present disclosure, the text generation module 23 is specifically used to: A text generation model is obtained by training based on a plurality of first audio segments and a plurality of first text segments corresponding to the plurality of first audio segments, wherein the plurality of first audio segments are a plurality of historical audio segments, and the plurality of first text segments are text segments corresponding to the plurality of historical audio segments; Multiple audio segments are input into the text generation model to obtain multiple text segments.

[0118] In one embodiment of the present disclosure, the text generation module 23 is further configured to: determining a target loss function based on the plurality of first text segments; The text generation model is obtained based on the target loss function.

[0119] In one embodiment of the present disclosure, the text generation module 23 is further configured to: Determining a first loss function based on text features of the plurality of first text segments; Determining a second loss function based on semantic features of the plurality of first text segments; The first loss function and the second loss function are weighted to obtain the target loss function.

[0120] See also Figure 3 , Figure 3 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 3The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of the modules in the above-mentioned device embodiments, such as Figure 2 The functions of modules 21 to 24 are shown.

[0121] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0122] The input device 302 may include a touch panel, a fingerprint collection sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.

[0123] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0124] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of the audio-to-text conversion method provided in the embodiments of the present disclosure, and can also execute the implementation methods of the electronic device described in the embodiments of the present disclosure, which will not be repeated here.

[0125] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by the processor, all or part of the processes in the above-mentioned embodiment method are implemented, and the computer program can also be completed by instructing the relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0126] The computer-readable storage medium may be an internal storage unit of the electronic device of any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0127] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0128] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0129] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0130] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0131] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0132] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for converting audio to text, characterized in that: include: Extracting features of the target audio to obtain audio features for representing the target speech content; Perform segmentation processing based on the time series of the audio features to obtain multiple audio segments; generating a plurality of text segments based on the plurality of audio segments; Corresponding punctuation marks are added to the multiple text segments based on the semantic model to generate text information.

2. The method for converting audio to text as claimed in claim 1, characterized in that The segmentation process is performed based on the time series of the audio features to obtain multiple audio segments, including: If the time interval of the audio feature is greater than the first time, the audio feature is segmented to obtain multiple audio segments.

3. The method for converting audio to text as claimed in claim 2, characterized in that: Also includes: The speaking speed of the target speech is determined based on the audio features, and the first time is determined based on the speaking speed.

4. The method for converting audio to text according to claim 1, wherein: Also includes: The text information is adjusted based on multimodal information fusion to obtain target text information.

5. The method for converting audio to text according to claim 1, wherein: The generating a plurality of text segments based on the plurality of audio segments comprises: A text generation model is obtained by training based on a plurality of first audio segments and a plurality of first text segments corresponding to the plurality of first audio segments, wherein the plurality of first audio segments are a plurality of historical audio segments, and the plurality of first text segments are text segments corresponding to the plurality of historical audio segments; The multiple audio segments are input into the text generation model to obtain the multiple text segments.

6. The method for converting audio to text as claimed in claim 5, characterized in that: The step of training a text generation model based on a plurality of first audio segments and a plurality of first text segments corresponding to the plurality of first audio segments comprises: determining a target loss function based on the plurality of first text segments; The text generation model is obtained based on the target loss function.

7. The method for converting audio to text as claimed in claim 6, characterized in that: The determining of the target loss function based on the plurality of first text segments comprises: determining a first loss function based on text features of the plurality of first text segments; determining a second loss function based on the semantic features of the plurality of first text segments; The first loss function and the second loss function are weightedly calculated to obtain the target loss function.

8. An audio to text conversion device, characterized in that: include: A feature extraction module is used to extract features from the target audio to obtain audio features for representing the target speech content; A feature segmentation module, used for performing segmentation processing based on the time series of the audio features to obtain multiple audio segments; A text generation module, configured to generate a plurality of text segments based on the plurality of audio segments; The punctuation mark adding module is used to add corresponding punctuation marks to the multiple text segments based on the semantic model to generate text information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images