Speech processing method and system, and storage medium

Abstract

A speech processing method and system, and a storage medium. The method comprises: acquiring speech information to be processed, wherein said speech information comprises a plurality of speech segments (S21); performing feature conversion on the plurality of speech segments, to obtain a conversion result (S22); performing feature decoding on the conversion result, to obtain a decoding result (S23); and determining a target speech processing result on the basis of the decoding result (S24). By means of the method, the technical problems of low processing efficiency and poor recognition accuracy when a speech data stream is processed in the relevant art are solved.

Description

Voice processing method, storage medium and system Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a speech processing method, storage medium, and system. Background Art

[0002] Speech processing is the use of computers to analyze, recognize, synthesize, and understand human speech. With the advent of the intelligent age, speech processing is finding increasingly widespread application, playing a crucial role in providing a more intelligent human-computer interaction experience. Related technologies employ different speech processing models to process speech data streams. However, due to differing construction principles, these models all exhibit varying degrees of flaws in speech processing, resulting in inefficient processing of speech data streams and poor recognition accuracy.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far.

[0004] Summary of the Invention

[0005] The embodiments of the present disclosure provide a voice processing method, a storage medium, and a system to at least solve the technical problems of low processing efficiency and poor recognition accuracy when processing voice data streams in related technologies.

[0006] According to one aspect of an embodiment of the present disclosure, a speech processing method is provided, comprising: obtaining speech information to be processed, wherein the speech information to be processed comprises: a plurality of speech segments; performing feature conversion on the plurality of speech segments to obtain conversion results; performing feature decoding on the conversion results to obtain decoding results; and determining a target speech processing result based on the decoding results.

[0007] According to another aspect of an embodiment of the present disclosure, a speech processing method is also provided, including: obtaining speech information to be recognized, wherein the speech information to be recognized includes: multiple speech segments; performing feature conversion on the multiple speech segments to obtain multiple speech feature vectors; performing feature decoding on the multiple speech feature vectors to obtain multiple text segments; and determining a target speech recognition result based on the multiple text segments.

[0008] According to another aspect of an embodiment of the present disclosure, a speech processing method is also provided, including: obtaining speech information to be processed, wherein the speech information to be processed includes: multiple speech segments; using a large speech processing model to perform feature conversion on the multiple speech segments to obtain conversion results, and performing feature decoding on the conversion results to obtain decoding results; and determining a target speech processing result based on the decoding result.

[0009] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is also provided, which includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the speech processing method described in any one of the embodiments of the present disclosure.

[0010] According to another aspect of an embodiment of the present disclosure, a speech processing system is also provided, including: a processor; and a memory, connected to the processor, and configured to provide the processor with instructions for processing the following processing steps: obtaining speech information to be processed, wherein the speech information to be processed includes: multiple speech segments; performing feature conversion on the multiple speech segments to obtain conversion results; performing feature decoding on the conversion results to obtain decoding results; and determining a target speech processing result based on the decoding results.

[0011] In the embodiment of the present disclosure, by obtaining multiple voice segments in the voice information to be processed, and then performing feature conversion on the multiple voice segments to obtain conversion results, then performing feature decoding on the conversion results to obtain decoding results, and finally determining the target voice processing results based on the decoding results, the purpose of efficiently processing the voice information to obtain voice processing results corresponding to the multiple voice segments is achieved, thereby achieving the technical effect of improving the processing efficiency and recognition accuracy when processing the voice data stream, and thus solving the technical problems of low processing efficiency and poor recognition accuracy when processing the voice data stream in the related art.

[0012] It is easy to note that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present disclosure, and do not constitute a limitation of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:

[0014] FIG1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a voice processing method;

[0015] FIG2 is a flow chart of a speech processing method according to Embodiment 1 of the present disclosure;

[0016] FIG3 is a schematic diagram of a speech processing method according to Embodiment 1 of the present disclosure;

[0017] FIG4 is a schematic diagram of another speech processing method according to Embodiment 1 of the present disclosure;

[0018] FIG5 is a flow chart of a speech processing method according to Embodiment 2 of the present disclosure;

[0019] FIG6 is a flow chart of a speech processing method according to Embodiment 3 of the present disclosure;

[0020] FIG7 is a flow chart of a speech processing method according to Embodiment 4 of the present disclosure;

[0021] FIG8 is a structural block diagram of a speech processing device according to Embodiment 1 of the present disclosure;

[0022] FIG9 is a structural block diagram of a speech processing device according to Embodiment 2 of the present disclosure;

[0023] FIG10 is a structural block diagram of a speech processing device according to Embodiment 3 of the present disclosure;

[0024] FIG11 is a structural block diagram of a speech processing device according to Embodiment 4 of the present disclosure;

[0025] FIG12 is a structural block diagram of a computer terminal according to Embodiment 6 of the present disclosure. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present disclosure.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] The technical solution provided by the present disclosure is mainly implemented using large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. The large model can also be called a cornerstone model / foundation model (Foundation Model). The large model is pre-trained by large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as large-scale language models (LLMs) and multi-modal pre-training models.

[0029] It should be noted that when the large model is actually used, the pre-trained model can be fine-tuned through a small number of samples, so that the large model can be applied to different tasks. For example, the large model can be widely used in fields such as natural language processing (NLP), computer vision, etc., and can be specifically applied to computer vision tasks such as visual question answering (VQA), image description (IC), and image generation. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present disclosure, data processing by a large voice processing model in a voice processing scenario is used as an example for explanation.

[0030] First, some nouns or terms that appear in the description of the embodiments of the present disclosure are subject to the following explanations:

[0031] The Connectionist Temporal Classification (CTC) model is a deep learning model for sequence classification. It uses an end-to-end training approach and can handle variable-length input and output sequences. The CTC model performs classification tasks by mapping input sequences to output sequences. During the inference phase, the CTC model uses a forward algorithm to calculate all possible label combinations and their probabilities at each time step and select an output result from these combinations.

[0032] Attention-based Encoder-Decoder (AED) model: It is a sequence-to-sequence model used to map an input sequence of variable length to another output sequence of variable length. The model mainly consists of an encoder and a decoder. The encoder converts the input sequence into a fixed-length vector representation and can pass information to the decoder through this vector. The decoder uses this vector as the initial state to generate the output sequence.

[0033] Recurrent Neural Network-Transducer (RNN-T) model: An end-to-end speech recognition model that uses sequence modeling and transcription techniques to convert speech signals into text.

[0034] Related technologies use different speech processing models to process speech data streams. However, due to their different construction principles, these various speech processing models all have varying degrees of defects when performing speech processing, resulting in low processing efficiency and poor recognition accuracy. Specifically, the CTC streaming model framework has an independent frame assumption, resulting in relatively weak modeling context capabilities; the AED model requires the use of an additional designed block attention mechanism to achieve streaming recognition, which is not suitable for processing speech data streams; the RNN-T model has poor long-term dependencies, and its recognition accuracy may decrease as the length of the input audio increases.

[0035] It can be seen from this that the relevant technologies have technical problems of low processing efficiency and poor recognition accuracy when processing voice data streams. Currently, no effective solutions have been proposed to address the above problems.

[0036] Example 1

[0037] According to an embodiment of the present disclosure, a method embodiment of speech processing is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] Taking into account the huge amount of model parameters of the large model and the limited computing resources of the mobile terminal, the above-mentioned speech processing method provided by the embodiment of the present disclosure can be applied to the application scenario shown in Figure 1, but is not limited thereto. In the application scenario shown in Figure 1, the large speech processing model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client devices 20 here can include but are not limited to: smart phones, tablet computers, laptops, PDAs, personal computers, smart home devices, car-mounted devices, etc. The client device 20 can interact with the user through a graphical user interface to implement the call to the large model, thereby implementing the method provided by the embodiment of the present disclosure.

[0039] In an embodiment of the present disclosure, a system consisting of a client device and a server can perform the following steps: the server obtains voice information to be processed, wherein the voice information to be processed includes: multiple voice segments; performs feature conversion on the multiple voice segments to obtain conversion results; performs feature decoding on the conversion results to obtain decoding results; and determines a target voice processing result based on the decoding results. In the client device, in response to a first touch operation on a graphical user interface, a target voice processing type is selected from a plurality of candidate voice processing types, wherein the plurality of candidate voice processing types include: voice recognition type, voice translation type, and voice interaction type; in response to a second touch operation on the graphical user interface, a plurality of voice segments corresponding to the target voice processing type are determined; in response to a third touch operation on the graphical user interface, the plurality of voice segments are converted into a plurality of voice feature vectors, feature decoding is performed on the plurality of voice feature vectors to obtain a plurality of decoded segments, and the plurality of decoded segments are spliced ​​to obtain a target voice processing result; and the target voice processing result is displayed in the graphical user interface. It should be noted that if the operating resources of the client device can meet the deployment and operating conditions of the large voice processing model, the embodiment of the present disclosure can be performed in the client device.

[0040] In the above operating environment, the present disclosure provides a speech processing method as shown in Figure 2. Figure 2 is a flow chart of a speech processing method according to embodiment 1 of the present disclosure. As shown in Figure 2, the method includes the following steps:

[0041] Step S21, obtaining voice information to be processed, wherein the voice information to be processed includes: multiple voice segments;

[0042] Step S22, performing feature conversion on the multiple speech segments to obtain conversion results;

[0043] Step S23, feature decoding is performed on the conversion result to obtain a decoding result;

[0044] Step S24: determining the target speech processing result based on the decoding result.

[0045] The above-mentioned voice information to be processed can be obtained by collecting the user's voice through the audio component of the terminal device, such as using the mobile phone microphone component to obtain multiple user voice clips; the voice information to be processed can also be extracted from pre-recorded audio files, such as voice messages in application software; the voice information to be processed can also be obtained from audio resources on the Internet, such as live broadcasts and online music radio stations; the voice information to be processed can also be obtained from voice commands or conversation information obtained from smart devices. The voice information to be processed obtained through the above channels can be used for applications such as speech recognition, speech synthesis, sentiment analysis, and speaker recognition.

[0046] It should be noted that the multiple voice segments in the above-mentioned voice information to be processed may include continuous multi-person conversation content or discrete personal voice content, which is not limited in the embodiment of the present disclosure.

[0047] Furthermore, feature conversion is performed on multiple speech segments in the processed speech information to obtain multiple speech feature vectors. Feature conversion refers to the process of converting the processed speech data into a new feature representation through a certain method. Feature conversion can reduce the data dimension, remove redundant information, extract useful features, and improve the robustness and generalization ability of the algorithm.

[0048] Multiple speech feature vectors are decoded to obtain decoding results, which are then used to determine the target speech processing result. In the process of decoding multiple speech feature vectors, autoregressive decoding, classification decoding, generative decoding, sequence-to-sequence decoding, path decoding, and attention decoding can be used. These decoding methods can be selected based on specific tasks and application scenarios.

[0049] Autoregressive decoding is a common decoding method, often used in sequence generation tasks such as natural language processing and speech recognition. The basic idea of ​​autoregressive decoding is that during a generation process, each element in the sequence is generated sequentially, with the results of each generation influencing the next generation. This can be represented using a conditional probability model. For example, when using an RNN to generate a text sequence, characters are generated one at a time, and the probability of each character's occurrence depends on the previously generated characters and the generation conditions. Categorical decoding maps speech feature vectors to a fixed set of categories, with each category representing a phoneme, syllable, or word. This decoding method is commonly used in speech recognition tasks. Generative decoding uses generative models, such as generative adversarial networks and variational autoencoders, to map speech feature vectors into a sound waveform space, thereby generating a speech signal corresponding to the input features. Sequence-to-sequence decoding uses recurrent neural networks, such as long short-term memory networks and gated recurrent units, to map a sequence of speech feature vectors to another sequence of speech feature vectors, enabling tasks such as speech conversion and speech synthesis. Path decoding maps speech feature vectors to a specific path, which can represent the duration and boundaries of a phoneme, syllable, word, or speech. Attention decoding uses the attention mechanism to align the input speech feature vector sequence with another speech feature vector sequence to generate the corresponding output sequence.

[0050] Based on the above steps S21 to S24, by obtaining multiple voice segments in the voice information to be processed, and then performing feature conversion on the multiple voice segments to obtain a conversion result, then performing feature decoding on the conversion result to obtain a decoding result, and finally determining the target voice processing result based on the decoding result, the purpose of efficiently processing the voice information to obtain voice processing results corresponding to multiple voice segments is achieved, thereby achieving the technical effect of improving the processing efficiency and recognition accuracy when processing the voice data stream, and thus solving the technical problems of low processing efficiency and poor recognition accuracy when processing the voice data stream in the related technology.

[0051] It should be noted that the embodiments of the present disclosure can be applied, but not limited to, to any scenario involving voice interaction in the fields of health, science, society, language, and art, especially in application scenarios such as voice recognition and voice translation in the fields of smart homes, smart customer service, and smart conferences. The voice processing method in the embodiments of the present disclosure can more accurately and efficiently realize the function of converting voice into text, further providing users with a more intelligent interactive experience.

[0052] The following further introduces the speech processing method in the embodiment of the present disclosure.

[0053] In an optional embodiment, in step S22, feature conversion is performed on multiple speech segments, and the conversion results include: at multiple consecutive moments, a large speech processing model is used to perform speech encoding or speech segmentation on the multiple speech segments in sequence to obtain multiple speech feature vectors, wherein the multiple moments correspond one-to-one to the multiple speech segments.

[0054] In an optional embodiment, the speech processing model includes: a feature conversion model and a feature decoding model. The main function of the feature conversion model is to convert the speech signal from the time domain or frequency domain to the feature domain. Common feature conversion models include Fourier transform, discrete cosine transform, wavelet transform, etc., which can extract the spectral features, energy features, etc. of the speech signal. The feature decoding model (is another important model in speech processing, and its main function is to decode the speech signal in the feature domain into the original time domain or frequency domain signal. The feature decoding model can restore the feature signal to the original speech signal by inverse transformation, thereby realizing the restoration of the speech signal.

[0055] Feature conversion models and feature decoding models are often used in conjunction with each other. The former converts speech signals into feature representations, while the latter decodes the feature representations back into the original signal. This process extracts useful information from speech signals and reduces signal redundancy, facilitating subsequent tasks such as speech recognition, speech synthesis, and speech enhancement.

[0056] Specifically, at multiple consecutive moments, a large speech processing model is used to sequentially perform speech encoding on multiple speech segments using a speech encoder or perform speech segmentation using a speech segmenter to obtain multiple speech feature vectors.

[0057] When using a speech encoder to encode multiple speech segments, the segments can be preprocessed, such as by truncating fixed-length segments, performing speech signal enhancement, and removing noise. This process then extracts speech features, converting the segments into feature representations that can be understood by the speech encoder. These features can include Mel-Frequency Cepstral Coefficients (MFCCs), spectrograms, and Linear Predictive Cepstral Coding (LPCCs). A speech encoder model is then constructed, using a neural network model that accepts speech features as input and generates encoded speech feature vector representations as output. These models include autoencoders and variational autoencoders. The speech features corresponding to the speech segment at each moment are input into the speech encoder model to obtain the corresponding encoded speech feature vectors. These feature vectors can be used to perform tasks such as recognition, classification, and clustering of the speech segments.

[0058] It should be noted that different speech encoder models may have differences in input and output parameter settings and network structure design. The specific implementation method needs to be adjusted and optimized according to specific tasks and data, and this disclosure does not limit it.

[0059] When using a speech segmenter to segment multiple speech segments, the segments can first be preprocessed, such as by truncating fixed-length segments, performing speech signal enhancement, and removing noise. This allows for the extraction of speech features, converting the segments into feature representations that can be understood by a speech encoder. These features can include MFCCs, spectrograms, and LPCCs. A speech segmentation model is then constructed. This model uses a neural network to accept speech features as input and generate a segmented speech feature vector. Commonly used speech segmentation models include convolutional neural networks, recurrent neural networks, and convolutional-recurrent neural networks. The speech features corresponding to each speech segment at each moment are input into the speech segmentation model to obtain the corresponding segmentation results. The output format of the speech segmentation results can include annotated text and a timestamp sequence. The segmentation results can also be post-processed and optimized, such as removing duplicate words, merging adjacent words, and performing phoneme-level segmentation. Finally, the segmentation results and related information, such as the original speech data, segmented text, and a timestamp sequence, are stored as the speech feature vector corresponding to the speech segment.

[0060] It is important to note that for different speech segmentation tasks, targeted processing and model design must be performed based on the characteristics and requirements of the data. Furthermore, the establishment of the speech segmentation model, the evaluation of the segmentation results, and post-processing also require fine-tuning to achieve more accurate and satisfactory speech segmentation results, which are not limited in this disclosure.

[0061] Based on the above optional embodiment, by using a large speech processing model to perform speech encoding or speech segmentation on multiple speech segments in sequence at multiple consecutive moments, multiple speech feature vectors are obtained, which can be used for tasks such as streaming speech recognition and streaming speech translation, thereby further improving speech processing efficiency.

[0062] In an optional embodiment, in step S23, feature decoding is performed on the conversion result to obtain the decoding result, which includes: at multiple consecutive moments, using a large speech processing model to perform feature decoding on multiple speech feature vectors in sequence to obtain multiple decoding segments.

[0063] In an optional embodiment, the feature decoding model is a multimodal language model. A multimodal language model is a language model that can process input from multiple different modalities (such as text, images, audio, etc.). It can simultaneously understand and generate multiple different types of data, and can effectively integrate and interact with these different modal data. The goal of a multimodal language model is to improve the performance and performance of the language model in understanding and generation tasks by combining information from different modalities.

[0064] Specifically, at multiple consecutive moments, a multimodal language model (MLM) based on a decoder structure is used to sequentially decode multiple speech feature vectors, obtaining multiple decoded segments. During the feature decoding process, the output of each decoding is used as part of the input for the next decoding. Therefore, the decoding output can be used to repeatedly obtain decoded segments at multiple consecutive moments until a specific termination condition is met or the maximum number of rounds is reached. After obtaining multiple decoded segments, they can also be post-processed, such as deduplication, sorting, and filtering of invalid information.

[0065] Based on the above optional embodiment, by using a large speech processing model to sequentially decode multiple speech feature vectors at multiple consecutive moments to obtain multiple decoding segments, it is possible to achieve rapid decoding of the speech feature vectors, thereby improving speech processing efficiency.

[0066] In an optional embodiment, at multiple consecutive moments, a large speech processing model is used to sequentially perform feature decoding on multiple speech feature vectors to obtain multiple decoded segments, including: at multiple consecutive moments, a large speech processing model is used to sequentially perform feature decoding on multiple speech feature vectors and context information to obtain multiple decoded segments, wherein the context information is determined by the historical moment speech segment and the historical moment decoding segment associated with the current moment speech feature vector in the multiple speech feature vectors.

[0067] This contextual information refers to information associated with the speech segment and decoded segment at the previous moment. In speech recognition or speech synthesis tasks, contextual information includes information such as the speech segment at the previous moment, the decoded segment, and their associated feature vectors. This contextual information provides more context, helping to accurately understand the current speech information and generate more fluent and natural speech segments.

[0068] Based on the above optional embodiment, by using a large speech processing model to sequentially decode multiple speech feature vectors and context information at multiple consecutive moments, multiple decoding segments are obtained, thereby further improving the accuracy of speech recognition.

[0069] In an optional embodiment, at multiple consecutive moments, a large speech processing model is used to sequentially perform feature decoding on multiple speech feature vectors and context information to obtain multiple decoded segments, including: at multiple consecutive moments, a large speech processing model is used to sequentially perform feature decoding on the current moment speech feature vector, the speech segment at the previous moment adjacent to the current moment speech feature vector, and the decoded segment at the previous moment adjacent to the current moment speech feature vector to obtain the current moment decoded segment, until all multiple speech feature vectors are decoded to obtain multiple decoded segments.

[0070] Specifically, at each moment, the current speech feature vector depends on the speech segments at previous moments. These historical segments are also contextual information. Similarly, during the decoding process of speech recognition or speech synthesis, the decoded segment at the current moment is also influenced by the decoded segments at previous moments. Therefore, these decoded segments at previous moments also constitute part of the contextual information.

[0071] Taking the first moment and the second moment in multiple consecutive moments as an example, the previous moment adjacent to the second moment is the first moment, and the speech feature vector of the second moment, the speech segment of the first moment and the decoded segment of the first moment are feature decoded in turn to obtain the decoded segment of the second moment, until all multiple speech feature vectors are decoded to obtain multiple decoded segments.

[0072] Based on the above optional embodiment, by using a large speech processing model to perform feature decoding on the speech feature vector of the current moment, the speech segment of the previous moment adjacent to the speech feature vector of the current moment, and the decoding segment of the previous moment adjacent to the speech feature vector of the current moment at multiple consecutive moments to obtain the decoding segment of the current moment, until all multiple speech feature vectors are decoded, multiple decoding segments are quickly obtained, which can improve decoding efficiency and decoding accuracy, and further improve the accuracy of speech recognition.

[0073] In an optional embodiment, feature decoding is performed on the conversion result to obtain a decoding result, including: using a multimodal language model to perform feature decoding on the conversion result to obtain a decoding result; or, using a multimodal language model to perform non-feature decoding on the conversion result to obtain a decoding result.

[0074] Specifically, feature decoding uses speech feature vectors to assist in decoding the conversion results using a multimodal language model. This helps more accurately predict the probability of sentence generation, improving decoding accuracy and effectiveness. Feature-free decoding, on the other hand, does not use speech feature vectors during decoding, relying solely on the multimodal language model to decode the conversion results.

[0075] Based on the above optional embodiment, a multimodal language model can be used to flexibly decode the conversion result to further improve the speech recognition efficiency and recognition accuracy.

[0076] In an optional embodiment, a graphical user interface is provided by a terminal device, and content displayed by the graphical user interface at least partially includes a speech processing scenario. The speech processing method further includes:

[0077] Step S31, in response to a first touch operation on the graphical user interface, selecting a target speech processing type from a plurality of candidate speech processing types, wherein the plurality of candidate speech processing types include: a speech recognition type, a speech translation type, and a speech interaction type;

[0078] Step S32, in response to a second touch operation on the graphical user interface, determining a plurality of voice segments corresponding to the target voice processing type;

[0079] Step S33, in response to a third touch operation on the graphical user interface, converting the plurality of speech segments into a plurality of speech feature vectors, performing feature decoding on the plurality of speech feature vectors to obtain a plurality of decoded segments, and concatenating the plurality of decoded segments to obtain a target speech processing result;

[0080] Step S34: Display the target speech processing result in the graphical user interface.

[0081] In the above optional embodiment, the above graphical user interface is a voice processing interface, and the voice processing scenario displayed in the voice processing interface may be a voice recognition scenario, a voice translation scenario, or a voice interaction scenario.

[0082] The above-mentioned graphical user interface also includes a first control (or a first touch area). When a first touch operation acting on the first control (or the first touch area) is detected, a target voice processing type is selected from a plurality of candidate voice processing types, wherein the plurality of candidate voice processing types include: voice recognition type, voice translation type, and voice interaction type.

[0083] The above-mentioned graphical user interface also includes a second control (or a second touch area). When a second touch operation acting on the second control (or the second touch area) is detected, multiple voice segments corresponding to the target voice processing type are determined.

[0084] The above-mentioned graphical user interface also includes a third control (or a third touch area). When a third touch operation acting on the third control (or the third touch area) is detected, multiple voice segments are converted into multiple voice feature vectors, the multiple voice feature vectors are feature decoded to obtain multiple decoded segments, and the multiple decoded segments are spliced ​​to obtain the target voice processing result.

[0085] The target speech processing results are displayed in the graphical user interface so that users can intuitively view the speech processing results and speech processing progress, further enhancing the user's intelligent interactive experience.

[0086] It should be noted that the first touch operation, the second touch operation, and the third touch operation can all be operations in which a user touches the display screen of the terminal device with a finger and touches the terminal device. The touch operation can include single-point touch and multi-point touch, wherein the touch operation of each touch point can include clicking, long pressing, pressing hard, swiping, etc. The first touch operation, the second touch operation, and the third touch operation can also be touch operations implemented through input devices such as a mouse and keyboard.

[0087] FIG3 is a schematic diagram of a speech processing method according to Embodiment 1 of the present disclosure. As shown in FIG3 , the method includes the following steps:

[0088] Step S301: obtaining voice information to be processed, wherein the voice information to be processed includes: multiple voice segments;

[0089] Step S302: At a plurality of consecutive moments, a large speech processing model is used to sequentially perform speech encoding or speech segmentation on the plurality of speech segments to obtain a plurality of speech feature vectors, wherein the plurality of moments correspond one-to-one to the plurality of speech segments;

[0090] Step S303, at multiple consecutive moments, using the large speech processing model, sequentially perform feature decoding on the speech feature vector at the current moment, the speech segment at the previous moment adjacent to the speech feature vector at the current moment, and the decoded segment at the previous moment adjacent to the speech feature vector at the current moment to obtain a decoded segment at the current moment, until all the speech feature vectors are decoded to obtain multiple decoded segments;

[0091] Step S304: determining a target speech processing result based on the decoding result.

[0092] Based on the above steps S301 to S304, by obtaining multiple voice segments in the voice information to be processed, and then performing feature conversion on the multiple voice segments to obtain a conversion result, then performing feature decoding on the conversion result to obtain a decoding result, and finally determining the target voice processing result based on the decoding result, the purpose of efficiently processing the voice information to obtain voice processing results corresponding to the multiple voice segments is achieved, thereby achieving the technical effect of improving the processing efficiency and recognition accuracy when processing the voice data stream, and thus solving the technical problems of low processing efficiency and poor recognition accuracy when processing the voice data stream in the related art.

[0093] FIG4 is a schematic diagram of another speech processing method according to embodiment 1 of the present disclosure. As shown in FIG4 , at the first moment, speech input segment 1 (Speech chunk 1) is converted into multiple speech feature vectors after passing through a speech encoder or speech segmenter (also called speech discretization). These vectors are fed into a multimodal language model for feature decoding, and a text segment 1 (Text chunk 1) is generated based on the decoding results. Next, at the second moment, speech input segment 2 (Speech chunk 2) is also converted into speech feature vectors through a speech encoder or speech segmenter, and is input into a multimodal language model for feature decoding to generate a text segment 2 (Text chunk 2). By analogy, streaming decoding is achieved. Therefore, the speech recognition method of the embodiment of the present disclosure can be used for tasks such as streaming speech recognition, streaming speech translation, and streaming speech interaction.

[0094] From the perspective of actual functional value, the speech recognition method of the embodiment of the present disclosure can realize the function of converting speech into text more accurately and efficiently in scenarios such as speech recognition and speech translation, thereby improving the user experience and providing better services for users. At the same time, the speech recognition method of the embodiment of the present disclosure can also be applied to the field of voice interaction, such as smart home, smart customer service and other fields, to provide users with a more intelligent interactive experience. From the perspective of problem solving, the speech recognition method of the embodiment of the present disclosure solves the contradiction between the newly input speech features and the generated text in the traditional streaming model framework, and improves the accuracy. At the same time, the streaming model framework based on the decoder is more unified in model structure, which is convenient for optimization and expansion. In addition, the speech recognition method of the embodiment of the present disclosure proposes a new type of decoder-based streaming model framework, which is different from traditional streaming model frameworks such as CTC, AED, RNN-T, and has better applicability and accuracy.

[0095] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0096] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present disclosure.

[0098] Example 2

[0099] According to an embodiment of the present disclosure, a speech processing method is further provided. FIG5 is a flow chart of a speech processing method according to Embodiment 2 of the present disclosure. As shown in FIG5 , the method includes the following steps:

[0100] Step S51, obtaining voice information to be recognized, wherein the voice information to be recognized includes: multiple voice segments;

[0101] Step S52, performing feature conversion on the multiple speech segments to obtain multiple speech feature vectors;

[0102] Step S53, performing feature decoding on the multiple speech feature vectors to obtain multiple text segments;

[0103] Step S54: determining a target speech recognition result based on the multiple text segments.

[0104] Based on the above steps S51 to S54, by obtaining multiple voice segments in the voice information to be recognized, and then performing feature conversion on the multiple voice segments to obtain a conversion result, then performing feature decoding on the conversion result to obtain a decoding result, and finally determining the target voice recognition result based on the decoding result, the purpose of efficiently processing the voice information to obtain voice recognition results corresponding to multiple voice segments is achieved, thereby achieving the technical effect of improving the recognition efficiency and recognition accuracy when processing the voice data stream, and thus solving the technical problems of low recognition efficiency and poor recognition accuracy when processing the voice data stream in the related technology.

[0105] It should be noted that the embodiments of the present disclosure can be applied, but not limited to, to any scenario involving voice interaction in the fields of health, science, society, language, and art, especially in application scenarios such as voice recognition and voice translation in the fields of smart homes, smart customer service, and smart conferences. The voice processing method in the embodiments of the present disclosure can more accurately and efficiently realize the function of converting voice into text, further providing users with a more intelligent interactive experience.

[0106] For the parts not described in detail in the above embodiments of the present disclosure, please refer to the relevant description of Embodiment 1 and will not be described in detail here.

[0107] Example 3

[0108] According to an embodiment of the present disclosure, a speech processing method is further provided. FIG6 is a flow chart of a speech processing method according to Embodiment 3 of the present disclosure. As shown in FIG6 , the method includes the following steps:

[0109] Step S61: obtaining voice information to be translated, wherein the voice information to be translated includes: a plurality of first voice segments;

[0110] Step S62, performing feature conversion on the plurality of first speech segments to obtain a plurality of speech feature vectors;

[0111] Step S63, performing feature decoding on the plurality of speech feature vectors to obtain a plurality of second speech segments;

[0112] Step S64: determining a target speech translation result based on the multiple second speech segments.

[0113] Based on the above steps S61 to S64, by obtaining multiple voice segments in the voice information to be translated, and then performing feature conversion on the multiple voice segments to obtain conversion results, then performing feature decoding on the conversion results to obtain decoding results, and finally determining the target voice translation result based on the decoding results, the purpose of efficiently processing the voice information to obtain voice translation results corresponding to the multiple voice segments is achieved, thereby achieving the technical effect of improving the translation efficiency and translation accuracy when processing the voice data stream, and thus solving the technical problems of low translation efficiency and poor translation accuracy when processing the voice data stream in the related art.

[0114] It should be noted that the embodiments of the present disclosure can be applied, but not limited to, to any scenario involving voice interaction in the fields of health, science, society, language, and art, especially in application scenarios such as voice recognition and voice translation in the fields of smart homes, smart customer service, and smart conferences. The voice processing method in the embodiments of the present disclosure can more accurately and efficiently realize the function of converting voice into text, further providing users with a more intelligent interactive experience.

[0115] For the parts not described in detail in the above embodiments of the present disclosure, please refer to the relevant description of Embodiment 1 and will not be repeated here.

[0116] Example 4

[0117] According to an embodiment of the present disclosure, a speech processing method is further provided. FIG7 is a flow chart of a speech processing method according to Embodiment 4 of the present disclosure. As shown in FIG7 , the method includes the following steps:

[0118] Step S71, obtaining voice information to be processed, wherein the voice information to be processed includes: multiple voice segments;

[0119] Step S72, performing feature conversion on the plurality of speech segments using the speech processing large model to obtain conversion results, and performing feature decoding on the conversion results to obtain decoding results;

[0120] Step S73: Determine the target speech processing result based on the decoding result.

[0121] Based on the above steps S71 to S73, by obtaining the voice information to be processed, and then using a large voice processing model to perform feature conversion on multiple voice segments to obtain conversion results, and feature decoding the conversion results to obtain decoding results, and finally determining the target voice processing results based on the decoding results, the purpose of efficiently processing the voice information to obtain voice recognition results corresponding to multiple voice segments is achieved, thereby achieving the technical effect of improving the recognition efficiency and recognition accuracy when processing voice data streams, and thus solving the technical problems of low recognition efficiency and poor recognition accuracy when processing voice data streams in related technologies.

[0122] It should be noted that the embodiments of the present disclosure can be applied, but not limited to, to any scenario involving voice interaction in the fields of health, science, society, language, and art, especially in application scenarios such as voice recognition and voice translation in the fields of smart homes, smart customer service, and smart conferences. The voice processing method in the embodiments of the present disclosure can more accurately and efficiently realize the function of converting voice into text, further providing users with a more intelligent interactive experience.

[0123] In an optional embodiment, performing feature conversion on a plurality of speech segments using a large speech processing model to obtain conversion results, and performing feature decoding on the conversion results to obtain decoding results includes:

[0124] At multiple consecutive moments, a large speech processing model is used to sequentially perform speech encoding or speech segmentation on multiple speech segments to obtain multiple speech feature vectors, wherein the multiple moments correspond one-to-one to the multiple speech segments;

[0125] At multiple consecutive moments, a large speech processing model is used to sequentially perform feature decoding on multiple speech feature vectors and context information to obtain multiple decoded segments, wherein the context information is determined by the historical moment speech segments and historical moment decoding segments associated with the current moment speech feature vector in the multiple speech feature vectors.

[0126] For the parts not described in detail in the above embodiments of the present disclosure, please refer to the relevant description of Embodiment 1 and will not be repeated here.

[0127] Example 5

[0128] According to an embodiment of the present disclosure, a speech processing device for implementing the above-mentioned speech processing method is also provided. FIG8 is a structural block diagram of a speech processing device according to Embodiment 1 of the present disclosure. As shown in FIG8 , the speech processing device includes:

[0129] The acquisition module 801 is configured to acquire voice information to be processed, wherein the voice information to be processed includes: multiple voice segments;

[0130] The conversion module 802 is configured to perform feature conversion on the plurality of speech segments to obtain conversion results;

[0131] The decoding module 803 is configured to perform feature decoding on the conversion result to obtain a decoding result;

[0132] The determination module 804 is configured to determine the target speech processing result based on the decoding result.

[0133] Optionally, the conversion module 802 is further configured to use a large speech processing model to perform speech encoding or speech segmentation on multiple speech segments in sequence at multiple consecutive moments to obtain multiple speech feature vectors, wherein the multiple moments correspond one-to-one to the multiple speech segments.

[0134] Optionally, the decoding module 803 is further configured to use a large speech processing model to perform feature decoding on multiple speech feature vectors in sequence at multiple consecutive moments to obtain multiple decoding segments.

[0135] Optionally, the decoding module 803 is also configured to use a large speech processing model to perform feature decoding on multiple speech feature vectors and context information in sequence at multiple consecutive moments to obtain multiple decoded segments, wherein the context information is determined by the historical moment speech segment and the historical moment decoding segment associated with the current moment speech feature vector in the multiple speech feature vectors.

[0136] Optionally, the decoding module 803 is also configured to use a large speech processing model to perform feature decoding on the speech feature vector at the current moment, the speech segment at the previous moment adjacent to the speech feature vector at the current moment, and the decoded segment at the previous moment adjacent to the speech feature vector at the current moment at multiple consecutive moments to obtain the decoded segment at the current moment, until all multiple speech feature vectors are decoded and multiple decoded segments are obtained.

[0137] Optionally, the speech processing large model includes: a feature conversion model and a feature decoding model.

[0138] Optionally, the feature decoding model is a multimodal language model.

[0139] Optionally, the decoding module 803 is further configured to: perform feature decoding on the conversion result using a multimodal language model to obtain a decoding result; or perform non-feature decoding on the conversion result using a multimodal language model to obtain a decoding result.

[0140] Optionally, the speech processing device also includes: a response module 805, which is configured to respond to a first touch operation applied to the graphical user interface and select a target speech processing type from a plurality of candidate speech processing types, wherein the plurality of candidate speech processing types include: speech recognition type, speech translation type, and speech interaction type; the response module 805 is also configured to respond to a second touch operation applied to the graphical user interface and determine a plurality of speech segments corresponding to the target speech processing type; the response module 805 is also configured to respond to a third touch operation applied to the graphical user interface and convert the plurality of speech segments into a plurality of speech feature vectors, perform feature decoding on the plurality of speech feature vectors to obtain a plurality of decoded segments, and splice the plurality of decoded segments to obtain a target speech processing result; and a display module 806 is configured to display the target speech processing result in the graphical user interface.

[0141] In the embodiment of the present disclosure, by obtaining multiple voice segments in the voice information to be processed, and then performing feature conversion on the multiple voice segments to obtain conversion results, then performing feature decoding on the conversion results to obtain decoding results, and finally determining the target voice processing results based on the decoding results, the purpose of efficiently processing the voice information to obtain voice processing results corresponding to the multiple voice segments is achieved, thereby achieving the technical effect of improving the processing efficiency and recognition accuracy when processing the voice data stream, and thus solving the technical problems of low processing efficiency and poor recognition accuracy when processing the voice data stream in the related art.

[0142] It should be noted that the acquisition module 801, conversion module 802, decoding module 803, and determination module 804 correspond to steps S21 to S24 in Example 1. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors.

[0143] FIG9 is a structural block diagram of a speech processing device according to Embodiment 2 of the present disclosure. As shown in FIG9 , the speech processing device includes:

[0144] The acquisition module 901 is configured to acquire voice information to be recognized, wherein the voice information to be recognized includes: multiple voice segments;

[0145] The conversion module 902 is configured to perform feature conversion on the plurality of speech segments to obtain a plurality of speech feature vectors;

[0146] A decoding module 903 is configured to perform feature decoding on the plurality of speech feature vectors to obtain a plurality of text segments;

[0147] The determination module 904 is configured to determine a target speech recognition result based on the multiple text segments.

[0148] In the embodiment of the present disclosure, by obtaining multiple voice segments in the voice information to be recognized, and then performing feature conversion on the multiple voice segments to obtain conversion results, then performing feature decoding on the conversion results to obtain decoding results, and finally determining the target voice recognition result based on the decoding results, the purpose of efficiently processing the voice information to obtain voice recognition results corresponding to the multiple voice segments is achieved, thereby achieving the technical effect of improving the recognition efficiency and recognition accuracy when processing the voice data stream, and thus solving the technical problems of low recognition efficiency and poor recognition accuracy when processing the voice data stream in the related art.

[0149] It should be noted that the acquisition module 901, conversion module 902, decoding module 903, and determination module 904 correspond to steps S51 to S54 in Example 2. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 2. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors.

[0150] FIG10 is a structural block diagram of a speech processing device according to Embodiment 3 of the present disclosure. As shown in FIG10 , the speech processing device includes:

[0151] The acquisition module 1001 is configured to acquire voice information to be translated, wherein the voice information to be translated includes: a plurality of first voice segments;

[0152] The conversion module 1002 is configured to perform feature conversion on the plurality of first speech segments to obtain a plurality of speech feature vectors;

[0153] The decoding module 1003 is configured to perform feature decoding on the plurality of speech feature vectors to obtain a plurality of second speech segments;

[0154] The determination module 1004 is configured to determine a target speech translation result based on the multiple second speech segments.

[0155] In the embodiment of the present disclosure, by obtaining multiple voice segments in the voice information to be translated, and then performing feature conversion on the multiple voice segments to obtain conversion results, then performing feature decoding on the conversion results to obtain decoding results, and finally determining the target voice translation result based on the decoding results, the purpose of efficiently processing the voice information to obtain voice translation results corresponding to the multiple voice segments is achieved, thereby achieving the technical effect of improving the translation efficiency and translation accuracy when processing the voice data stream, and thus solving the technical problems of low translation efficiency and poor translation accuracy when processing the voice data stream in the related art.

[0156] It should be noted that the acquisition module 1001, conversion module 1002, decoding module 1003, and determination module 1004 correspond to steps S61 to S64 in Example 3. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 3. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors.

[0157] FIG11 is a structural block diagram of a speech processing device according to Embodiment 4 of the present disclosure. As shown in FIG11 , the speech processing device includes:

[0158] The acquisition module 1101 acquires voice information to be processed, wherein the voice information to be processed includes: multiple voice segments;

[0159] The processing module 1102 performs feature conversion on the plurality of speech segments using a large speech processing model to obtain conversion results, and performs feature decoding on the conversion results to obtain decoding results;

[0160] The determination module 1103 determines the target speech processing result based on the decoding result.

[0161] Optionally, the processing module 1102 is also configured to: at multiple consecutive moments, use a large speech processing model to perform speech encoding or speech segmentation on multiple speech segments in turn to obtain multiple speech feature vectors, wherein multiple moments correspond one-to-one to multiple speech segments; at multiple consecutive moments, use a large speech processing model to perform feature decoding on multiple speech feature vectors and context information in turn to obtain multiple decoded segments, wherein the context information is determined by the historical moment speech segment and the historical moment decoding segment associated with the current moment speech feature vector in the multiple speech feature vectors.

[0162] In the embodiment of the present disclosure, by obtaining the voice information to be processed, and then using a large voice processing model to perform feature conversion on multiple voice segments to obtain conversion results, and feature decoding the conversion results to obtain decoding results, and finally determining the target voice processing results based on the decoding results, the purpose of efficiently processing the voice information to obtain voice recognition results corresponding to multiple voice segments is achieved, thereby achieving the technical effect of improving the recognition efficiency and recognition accuracy when processing voice data streams, and thus solving the technical problems of low recognition efficiency and poor recognition accuracy when processing voice data streams in related technologies.

[0163] It should be noted that the acquisition module 1101, processing module 1102, and determination module 1103 correspond to steps S71 to S73 in Example 4. The examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 4. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors.

[0164] Example 6

[0165] The embodiment of the present disclosure may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0166] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0167] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the speech processing method: obtaining speech information to be processed, wherein the speech information to be processed includes: multiple speech segments; performing feature conversion on the multiple speech segments to obtain conversion results; performing feature decoding on the conversion results to obtain decoding results; and determining the target speech processing result based on the decoding results.

[0168] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the speech processing method: obtaining speech information to be recognized, wherein the speech information to be recognized includes: multiple speech segments; performing feature conversion on the multiple speech segments to obtain multiple speech feature vectors; performing feature decoding on the multiple speech feature vectors to obtain multiple text segments; and determining the target speech recognition result based on the multiple text segments.

[0169] In this embodiment, the computer terminal can execute the program code of the following steps in the speech processing method: obtaining speech information to be translated, wherein the speech information to be translated includes: multiple first speech segments; performing feature conversion on the multiple first speech segments to obtain multiple speech feature vectors; performing feature decoding on the multiple speech feature vectors to obtain multiple second speech segments; and determining a target speech translation result based on the multiple second speech segments.

[0170] Optionally, Figure 12 is a block diagram of a computer terminal according to Embodiment 6 of the present disclosure. As shown in Figure 12, the computer terminal may include: one or more (only one is shown in the figure) processors 122, a memory 124, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0171] Among them, the memory can be configured to store software programs and modules, such as the program instructions / modules corresponding to the speech processing method and device in the embodiments of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned speech processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0172] The processor can call the information and application stored in the memory through the transmission device to execute the following steps: obtain the voice information to be processed, wherein the voice information to be processed includes: multiple voice segments; perform feature conversion on the multiple voice segments to obtain conversion results; perform feature decoding on the conversion results to obtain decoding results; determine the target voice processing result based on the decoding result.

[0173] Optionally, the processor may also execute the following program code: at multiple consecutive moments, use a large speech processing model to perform speech encoding or speech segmentation on multiple speech segments in sequence to obtain multiple speech feature vectors, wherein the multiple moments correspond one-to-one to the multiple speech segments.

[0174] Optionally, the processor may further execute program code of the following steps: at a plurality of consecutive moments, using a large speech processing model to sequentially perform feature decoding on a plurality of speech feature vectors to obtain a plurality of decoded segments.

[0175] Optionally, the processor may also execute the program code of the following steps: at multiple consecutive moments, a large speech processing model is used to sequentially perform feature decoding on multiple speech feature vectors and context information to obtain multiple decoded segments, wherein the context information is determined by the historical moment speech segments and historical moment decoded segments associated with the current moment speech feature vector in the multiple speech feature vectors.

[0176] Optionally, the processor may also execute the program code of the following steps: at multiple consecutive moments, using a large speech processing model to sequentially perform feature decoding on the speech feature vector at the current moment, the speech segment at the previous moment adjacent to the speech feature vector at the current moment, and the decoded segment at the previous moment adjacent to the speech feature vector at the current moment to obtain the decoded segment at the current moment, until all the speech feature vectors are decoded and multiple decoded segments are obtained.

[0177] Optionally, the speech processing large model includes: a feature conversion model and a feature decoding model.

[0178] Optionally, the feature decoding model is a multimodal language model.

[0179] Optionally, the processor may further execute program code of the following steps: performing feature decoding on the conversion result using a multimodal language model to obtain a decoding result; or performing non-feature decoding on the conversion result using a multimodal language model to obtain a decoding result.

[0180] Optionally, the processor may also execute the program code of the following steps: in response to a first touch operation on the graphical user interface, selecting a target speech processing type from a plurality of candidate speech processing types, wherein the plurality of candidate speech processing types include: speech recognition type, speech translation type, and speech interaction type; in response to a second touch operation on the graphical user interface, determining a plurality of speech segments corresponding to the target speech processing type; in response to a third touch operation on the graphical user interface, converting the plurality of speech segments into a plurality of speech feature vectors, performing feature decoding on the plurality of speech feature vectors to obtain a plurality of decoded segments, and splicing the plurality of decoded segments to obtain a target speech processing result; and displaying the target speech processing result in the graphical user interface.

[0181] Optionally, the processor may also execute the program code of the following steps: obtaining speech information to be recognized, wherein the speech information to be recognized includes: multiple speech segments; performing feature conversion on the multiple speech segments to obtain multiple speech feature vectors; performing feature decoding on the multiple speech feature vectors to obtain multiple text segments; and determining the target speech recognition result based on the multiple text segments.

[0182] Optionally, the processor may further execute program code for the following steps: obtaining speech information to be translated, wherein the speech information to be translated includes: multiple first speech segments; performing feature conversion on the multiple first speech segments to obtain multiple speech feature vectors; performing feature decoding on the multiple speech feature vectors to obtain multiple second speech segments; and determining a target speech translation result based on the multiple second speech segments.

[0183] Optionally, the processor may also execute the program code of the following steps: obtaining speech information to be processed, wherein the speech information to be processed includes: multiple speech segments; performing feature conversion on the multiple speech segments using a large speech processing model to obtain conversion results, and performing feature decoding on the conversion results to obtain decoding results; and determining the target speech processing result based on the decoding result.

[0184] Optionally, the processor may also execute the program code of the following steps: at a plurality of consecutive moments, using a large speech processing model to sequentially perform speech encoding or speech segmentation on a plurality of speech segments to obtain a plurality of speech feature vectors, wherein the plurality of moments correspond one-to-one to the plurality of speech segments; at a plurality of consecutive moments, using a large speech processing model to sequentially perform feature decoding on a plurality of speech feature vectors and context information to obtain a plurality of decoded segments, wherein the context information is determined by the historical moment speech segment and the historical moment decoding segment associated with the current moment speech feature vector in the plurality of speech feature vectors.

[0185] By adopting the embodiment of the present disclosure, multiple voice segments in the voice information to be processed are obtained, and then the multiple voice segments are feature converted to obtain conversion results, and then the conversion results are feature decoded to obtain decoding results. Finally, the target voice processing result is determined based on the decoding result, thereby achieving the purpose of efficiently processing the voice information to obtain voice processing results corresponding to the multiple voice segments, thereby achieving the technical effect of improving the processing efficiency and recognition accuracy when processing the voice data stream, and thus solving the technical problems of low processing efficiency and poor recognition accuracy when processing the voice data stream in the related art.

[0186] Those skilled in the art will appreciate that the structure shown in FIG12 is merely illustrative, and the computer terminal may also be a terminal device such as a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, or a mobile internet device (MID). FIG12 does not limit the structure of the above-mentioned electronic devices. For example, the computer terminal may include more or fewer components (such as a network interface, a display device, etc.) than those shown in FIG12, or may have a configuration different from that shown in FIG12.

[0187] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0188] Example 7

[0189] The embodiment of the present disclosure further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium may store program codes executed by the speech processing method provided in the embodiment.

[0190] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0191] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining voice information to be processed, wherein the voice information to be processed includes: multiple voice segments; performing feature conversion on the multiple voice segments to obtain conversion results; performing feature decoding on the conversion results to obtain decoding results; and determining the target voice processing result based on the decoding result.

[0192] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: at multiple consecutive moments, a large speech processing model is used to perform speech encoding or speech segmentation on multiple speech segments in sequence to obtain multiple speech feature vectors, wherein the multiple moments correspond one-to-one to the multiple speech segments.

[0193] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: at multiple consecutive moments, using a large speech processing model to sequentially perform feature decoding on multiple speech feature vectors to obtain multiple decoded segments.

[0194] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: at multiple consecutive moments, a large speech processing model is used to sequentially perform feature decoding on multiple speech feature vectors and context information to obtain multiple decoded segments, wherein the context information is determined by the historical moment speech segment and the historical moment decoding segment associated with the current moment speech feature vector in the multiple speech feature vectors.

[0195] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: at multiple consecutive moments, using a large speech processing model, feature decoding is performed on the speech feature vector at the current moment, the speech segment at the previous moment adjacent to the speech feature vector at the current moment, and the decoded segment at the previous moment adjacent to the speech feature vector at the current moment in turn to obtain the decoded segment at the current moment, until all multiple speech feature vectors are decoded to obtain multiple decoded segments.

[0196] Optionally, the speech processing large model includes: a feature conversion model and a feature decoding model.

[0197] Optionally, the feature decoding model is a multimodal language model.

[0198] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: performing feature decoding on the conversion result using a multimodal language model to obtain a decoding result; or performing non-feature decoding on the conversion result using a multimodal language model to obtain a decoding result.

[0199] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: in response to a first touch operation applied to a graphical user interface, selecting a target speech processing type from a plurality of candidate speech processing types, wherein the plurality of candidate speech processing types include: speech recognition type, speech translation type, and speech interaction type; in response to a second touch operation applied to the graphical user interface, determining a plurality of speech segments corresponding to the target speech processing type; in response to a third touch operation applied to the graphical user interface, converting the plurality of speech segments into a plurality of speech feature vectors, performing feature decoding on the plurality of speech feature vectors to obtain a plurality of decoded segments, and splicing the plurality of decoded segments to obtain a target speech processing result; and displaying the target speech processing result in the graphical user interface.

[0200] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining speech information to be recognized, wherein the speech information to be recognized includes: multiple speech segments; performing feature conversion on the multiple speech segments to obtain multiple speech feature vectors; performing feature decoding on the multiple speech feature vectors to obtain multiple text segments; and determining the target speech recognition result based on the multiple text segments.

[0201] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining speech information to be translated, wherein the speech information to be translated includes: multiple first speech segments; performing feature conversion on the multiple first speech segments to obtain multiple speech feature vectors; performing feature decoding on the multiple speech feature vectors to obtain multiple second speech segments; and determining a target speech translation result based on the multiple second speech segments.

[0202] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining speech information to be processed, wherein the speech information to be processed includes: multiple speech segments; performing feature conversion on the multiple speech segments using a large speech processing model to obtain conversion results, and performing feature decoding on the conversion results to obtain decoding results; and determining the target speech processing result based on the decoding result.

[0203] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: at multiple consecutive moments, using a large speech processing model to perform speech encoding or speech segmentation on multiple speech segments in turn to obtain multiple speech feature vectors, wherein the multiple moments correspond one-to-one to the multiple speech segments; at multiple consecutive moments, using a large speech processing model to perform feature decoding on multiple speech feature vectors and context information in turn to obtain multiple decoded segments, wherein the context information is determined by the historical moment speech segment and the historical moment decoding segment associated with the current moment speech feature vector in the multiple speech feature vectors.

[0204] By adopting the embodiment of the present disclosure, multiple voice segments in the voice information to be processed are obtained, and then the multiple voice segments are feature converted to obtain conversion results, and then the conversion results are feature decoded to obtain decoding results. Finally, the target voice processing result is determined based on the decoding result, thereby achieving the purpose of efficiently processing the voice information to obtain voice processing results corresponding to the multiple voice segments, thereby achieving the technical effect of improving the processing efficiency and recognition accuracy when processing the voice data stream, and thus solving the technical problems of low processing efficiency and poor recognition accuracy when processing the voice data stream in the related art.

[0205] Example 8

[0206] An embodiment of the present disclosure further provides a speech processing system, comprising:

[0207] A processor; and a memory, connected to the processor, configured to provide the processor with instructions for performing the following processing steps:

[0208] Acquire speech information to be processed, wherein the speech information to be processed includes: multiple speech segments; perform feature conversion on the multiple speech segments to obtain conversion results; perform feature decoding on the conversion results to obtain decoding results; and determine a target speech processing result based on the decoding results.

[0209] By adopting the embodiment of the present disclosure, multiple voice segments in the voice information to be processed are obtained, and then the multiple voice segments are feature converted to obtain conversion results, and then the conversion results are feature decoded to obtain decoding results. Finally, the target voice processing result is determined based on the decoding result, thereby achieving the purpose of efficiently processing the voice information to obtain voice processing results corresponding to the multiple voice segments, thereby achieving the technical effect of improving the processing efficiency and recognition accuracy when processing the voice data stream, and thus solving the technical problems of low processing efficiency and poor recognition accuracy when processing the voice data stream in the related art.

[0210] The serial numbers of the above-mentioned embodiments of the present disclosure are for description only and do not represent the advantages or disadvantages of the embodiments.

[0211] In the above embodiments of the present disclosure, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0212] In the several embodiments provided in the present disclosure, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0213] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0214] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0215] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0216] The above is only a preferred embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present disclosure. These improvements and modifications should also be regarded as within the scope of protection of the present disclosure.

Claims

1. A speech processing method, comprising: Acquire voice information to be processed, wherein the voice information to be processed includes: a plurality of voice segments; Performing feature conversion on the multiple speech segments to obtain conversion results; Performing feature decoding on the conversion result to obtain a decoding result; A target speech processing result is determined based on the decoding result.

2. The speech processing method according to claim 1, wherein: Performing feature conversion on the multiple voice segments to obtain the conversion results includes: At a plurality of consecutive moments, a large speech processing model is used to sequentially perform speech encoding or speech segmentation on the plurality of speech segments to obtain a plurality of speech feature vectors, wherein the plurality of moments correspond one-to-one to the plurality of speech segments.

3. The speech processing method according to claim 2, wherein: The conversion result is subjected to feature decoding, and the decoding result obtained includes: At the plurality of consecutive moments, the speech processing large model is used to sequentially perform feature decoding on the plurality of speech feature vectors to obtain a plurality of decoding segments.

4. The speech processing method according to claim 3, wherein: At the plurality of consecutive moments, the plurality of speech feature vectors are sequentially feature decoded using the speech processing large model to obtain the plurality of decoded segments, including: At the multiple consecutive moments, the speech processing large model is used to perform feature decoding on the multiple speech feature vectors and context information in turn to obtain the multiple decoded segments, wherein the context information is determined by the historical moment speech segments and historical moment decoded segments associated with the current moment speech feature vector in the multiple speech feature vectors.

5. The speech processing method according to claim 4, wherein: At the plurality of consecutive moments, the plurality of speech feature vectors and the context information are sequentially feature decoded using the speech processing large model to obtain the plurality of decoded segments, including: At the multiple consecutive moments, the speech processing large model is used to sequentially perform feature decoding on the speech feature vector at the current moment, the speech segment at the previous moment adjacent to the speech feature vector at the current moment, and the decoded segment at the previous moment adjacent to the speech feature vector at the current moment to obtain the decoded segment at the current moment, until all the multiple speech feature vectors are decoded to obtain the multiple decoded segments.

6. The speech processing method according to claim 2, wherein: The speech processing large model includes: a feature conversion model and a feature decoding model.

7. The speech processing method according to claim 6, wherein: The feature decoding model is a multimodal language model.

8. The speech processing method according to claim 7, wherein: The conversion result is subjected to feature decoding, and the decoding result obtained includes: Using the multimodal language model to perform feature decoding on the conversion result to obtain the decoding result; or, The multimodal language model is used to perform non-feature decoding on the conversion result to obtain the decoding result.

9. The speech processing method according to claim 1, wherein: A graphical user interface is provided through a terminal device, and the content displayed by the graphical user interface at least partially includes a voice processing scenario, and the voice processing method further includes: In response to a first touch operation on the graphical user interface, a target speech processing type is selected from a plurality of candidate speech processing types, wherein the plurality of candidate speech processing types include: a speech recognition type, a speech translation type, and a speech interaction type; In response to a second touch operation applied to the graphical user interface, determining the plurality of voice segments corresponding to the target voice processing type; In response to a third touch operation on the graphical user interface, converting the plurality of speech segments into the plurality of speech feature vectors, performing feature decoding on the plurality of speech feature vectors to obtain a plurality of decoded segments, and splicing the plurality of decoded segments to obtain the target speech processing result; The target speech processing result is displayed in the graphical user interface.

10. A speech processing method, comprising: Acquire voice information to be recognized, wherein the voice information to be recognized includes: a plurality of voice segments; Performing feature conversion on the multiple speech segments to obtain multiple speech feature vectors; Performing feature decoding on the multiple speech feature vectors to obtain multiple text segments; A target speech recognition result is determined based on the multiple text segments.

11. A speech processing method, comprising: Acquire voice information to be processed, wherein the voice information to be processed includes: a plurality of voice segments; Using a speech processing large model to perform feature conversion on the multiple speech segments to obtain conversion results, and performing feature decoding on the conversion results to obtain decoding results; A target speech processing result is determined based on the decoding result.

12. The speech processing method according to claim 11, wherein: Using the speech processing large model to perform feature conversion on the multiple speech segments to obtain the conversion results, and performing feature decoding on the conversion results to obtain the decoding results includes: At multiple consecutive moments, the speech processing model is used to perform speech encoding on the multiple speech segments in sequence. Code or speech segmentation, to obtain a plurality of speech feature vectors, wherein the plurality of time moments correspond one-to-one to the plurality of speech segments; At the multiple consecutive moments, the speech processing large model is used to perform feature decoding on the multiple speech feature vectors and context information in turn to obtain multiple decoded segments, wherein the context information is determined by the historical moment speech segments and historical moment decoded segments associated with the current moment speech feature vector among the multiple speech feature vectors.

13. A computer-readable storage medium, the computer-readable storage medium comprising a stored executable program, wherein: When the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the speech processing method according to any one of claims 1 to 12.

14. A speech processing system, comprising: processor; as well as The memory is connected to the processor and configured to provide the processor with instructions for performing the following processing steps: Acquire voice information to be processed, wherein the voice information to be processed includes: a plurality of voice segments; Performing feature conversion on the multiple speech segments to obtain conversion results; Performing feature decoding on the conversion result to obtain a decoding result; A target speech processing result is determined based on the decoding result.

Images