Voice processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a voice processing method and device, electronic equipment and a storage medium, and relates to the technical field of voice processing, and the method comprises the steps: extracting a language-independent intermediate representation from an input voice through a voice encoder, the voice encoder being obtained by carrying out knowledge distillation based on a neural network of a Transform structure and a cloud model; on the basis of a preset mapping relationship between the running context and the voice processing strategy, executing a target voice processing strategy corresponding to the current running context, the target voice processing strategy being to upload a language-independent intermediate representation to a cloud model and receive a standard task processing result returned by the cloud model, or to send the standard task processing result to the cloud model; and executing the voice processing task according to the language-independent intermediate representation to obtain a basic task processing result. According to the invention, the processing precision of end-cloud collaborative voice processing can be ensured, and the efficiency of end-cloud transmission can be improved at the same time.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a speech processing method, apparatus, electronic device and storage medium. Background Technology

[0002] With the development and advancement of technology, scenarios requiring speech processing (such as speech recognition and speech translation) are gradually increasing. Due to the limited resources of user terminals, most speech processing tasks need to be completed based on cloud-based speech signal processing, and then the speech processing results are returned to the user terminal.

[0003] Currently, in order to ensure the accuracy of cloud processing, user terminals need to upload complete raw audio data to the cloud. However, the raw audio data is massive, the data transmission takes a long time, and it requires high network bandwidth. If the raw audio data is compressed or converted to text on the user terminal to reduce transmission overhead, and only the text content is uploaded, the processing accuracy will be insufficient due to the limited computing power of the terminal, and the computing power of the cloud cannot be effectively utilized.

[0004] In summary, how to improve the efficiency of edge-cloud transmission while ensuring the processing accuracy of edge-cloud collaborative voice processing has become a pressing technical problem that needs to be solved in this field.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this application is to provide a voice processing method, apparatus, electronic device, and storage medium, which aims to improve the efficiency of end-to-end cloud transmission while ensuring the processing accuracy of end-to-end cloud collaborative voice processing.

[0007] To achieve the above objectives, this application proposes a voice processing method applied to a user terminal, the voice processing method comprising: The speech encoder extracts language-independent intermediate representations from the input speech, wherein the language-independent intermediate representations are feature vectors that reflect the acoustic details and semantic information of the input speech, and the speech encoder is a neural network based on the Transformer (a deep learning model structure) structure and is obtained by knowledge distillation based on a cloud model. Based on the preset mapping relationship between the running context and the speech processing strategy, the target speech processing strategy corresponding to the current running context is executed. The target speech processing strategy is to upload the language-independent intermediate representation to the cloud model and receive the standard task processing result returned by the cloud model, or to execute a speech processing task based on the language-independent intermediate representation to obtain the basic task processing result.

[0008] In one embodiment, prior to the step of extracting language-independent intermediate representations from the input speech using a speech encoder, the method further includes: The language-independent intermediate representation generated by the cloud model is used as a supervision signal; The speech encoder is trained with the training objective of minimizing the difference between the language-independent intermediate representation output by the speech encoder and the supervision signal.

[0009] In one embodiment, before the step of executing the target speech processing strategy corresponding to the current running context based on the preset mapping relationship between the running context and the speech processing strategy, the method further includes: Monitor the battery level and network bandwidth of the user terminal; Use the battery level and network bandwidth as the current operating context; The step of executing the target speech processing strategy corresponding to the current running context based on the preset mapping relationship between the running context and the speech processing strategy includes: If the battery level exceeds a preset battery level threshold and the network bandwidth exceeds a preset bandwidth threshold, then the language-independent intermediate representation is uploaded to the cloud model and the standard task processing result returned by the cloud model is received. If the battery level does not exceed the preset battery level threshold, or the network bandwidth does not exceed the preset bandwidth threshold, then the speech processing task is performed based on the language-independent intermediate representation to obtain the basic task processing result.

[0010] In one embodiment, after the step of performing a speech processing task based on the language-independent intermediate representation to obtain the basic task processing result, the method further includes: Determine whether the task type of the voice processing task is a real-time task; If not, the language-independent intermediate representation is cached, and if the battery level exceeds the preset battery threshold and the network bandwidth exceeds the preset bandwidth threshold, the cached language-independent intermediate representation is uploaded to the cloud model, and the optimization processing result returned by the cloud model is received. Replace the content of the basic task processing result with the content of the optimized processing result.

[0011] In one embodiment, the step of extracting language-independent intermediate representations from input speech using a speech encoder includes: The input speech is segmented into frames to obtain multiple speech frames; Acoustic features are extracted from the multiple speech frames, and the acoustic features are encoded by a speech encoder to obtain a language-independent intermediate representation.

[0012] In one embodiment, the method further includes: Receive encoder optimization information sent by the cloud model, wherein the encoder optimization information is generated by the cloud model based on the processing effect of the language-independent intermediate representation; The voice encoder is updated based on the encoder optimization information.

[0013] In one embodiment, the method further includes: Determine whether the task scenario of the voice processing task is a multi-terminal collaborative scenario; If the task scenario is the multi-terminal collaborative scenario, then based on the preset mapping relationship between the running context and the speech processing strategy, the target speech processing strategy corresponding to the current running context is executed. The target speech processing strategy is to upload the language-independent intermediate representation to the cloud model and receive the multi-terminal collaborative speech processing result returned by the cloud model, or to execute a speech processing task based on the language-independent intermediate representation to obtain the basic task processing result. The multi-terminal collaborative speech processing result is obtained by the cloud model executing the speech processing task based on the hybrid intermediate representation. The hybrid intermediate representation is obtained by associating the language-independent intermediate representation with the language-independent intermediate representation uploaded by other user terminals. If the task scenario is not the multi-terminal collaborative scenario, then based on the preset mapping relationship between the running context and the speech processing strategy, the target speech processing strategy corresponding to the current running context is executed. The target speech processing strategy is to upload the language-independent intermediate representation to the cloud model and receive the standard task processing result returned by the cloud model, or to execute a speech processing task based on the language-independent intermediate representation to obtain the basic task processing result.

[0014] Furthermore, to achieve the above objectives, this application also proposes a speech processing apparatus, which includes: The intermediate representation extraction module is used to extract language-independent intermediate representations from the input speech through a speech encoder. The language-independent intermediate representations are feature vectors that reflect the acoustic details and semantic information of the input speech. The speech encoder is a neural network based on the Transformer structure and is obtained by knowledge distillation based on a cloud model. The speech processing module is used to execute a target speech processing strategy corresponding to the current running context based on a preset mapping relationship between the running context and the speech processing strategy. The target speech processing strategy is to upload the language-independent intermediate representation to the cloud model and receive the standard task processing result returned by the cloud model, or to execute a speech processing task based on the language-independent intermediate representation to obtain the basic task processing result.

[0015] In addition, to achieve the above objectives, this application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the speech processing method described above.

[0016] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the speech processing method described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: In this application, a speech encoder based on a Transformer architecture and obtained through knowledge distillation extracts a language-independent intermediate representation from the input speech. This language-independent intermediate representation is a feature vector that simultaneously reflects the acoustic details and semantic information contained in the input speech. Then, based on the mapping relationship between the operating context and the speech processing strategy, the target speech processing strategy corresponding to the current operating context is executed. This strategy can either upload the language-independent intermediate representation to the cloud model to obtain the standard task processing result generated by it, or directly execute the speech processing task on the terminal side based on the language-independent intermediate representation to obtain the basic task processing result. Thus, by replacing the transmission of the original audio with the language-independent intermediate representation, network bandwidth consumption and transmission latency are reduced, and transmission efficiency is improved. At the same time, since the language-independent intermediate representation is generated by the speech encoder through knowledge distillation of the cloud model, semantic alignment between it and the cloud model is ensured, enabling the cloud model to perform high-precision processing on this basis. This ensures that the accuracy of the final processing result is no less than that of the traditional scheme of uploading the original audio, thereby achieving the effect of improving the overall efficiency of edge-cloud collaboration while ensuring processing accuracy. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an embodiment of the speech processing method of this application. Figure 2This is a schematic diagram of the implementation process of the speech processing method of this application in Embodiment 1; Figure 3 This is a flowchart illustrating Embodiment 2 of the speech processing method of this application; Figure 4 This is a schematic diagram of the module structure of the voice processing device according to an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the speech processing method in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] With the development and advancement of technology, scenarios requiring speech processing (such as speech recognition and speech translation) are gradually increasing. Due to the limited resources of user terminals, most speech processing tasks need to be completed based on cloud-based speech signal processing, and then the speech processing results are returned to the user terminal.

[0025] Currently, in order to ensure the accuracy of cloud processing, user terminals need to upload complete raw audio data to the cloud. However, the raw audio data is massive, the data transmission takes a long time, and it requires high network bandwidth. If the raw audio data is compressed or converted to text on the user terminal to reduce transmission overhead, and only the text content is uploaded, the processing accuracy will be insufficient due to the limited computing power of the terminal, and the computing power of the cloud cannot be effectively utilized.

[0026] In summary, how to improve the efficiency of edge-cloud transmission while ensuring the processing accuracy of edge-cloud collaborative voice processing has become a pressing technical problem that needs to be solved in this field.

[0027] In order to solve the above technical problems, an embodiment of the present application uses a speech encoder based on the Transformer architecture and obtained through knowledge distillation technology to extract a language-independent intermediate representation from the input speech. This language-independent intermediate representation is a feature vector that can simultaneously reflect the acoustic details and semantic information contained in the input speech. Then, according to the mapping relationship between the running context and the speech processing strategy, the target speech processing strategy corresponding to the current running context is executed. This strategy can either upload the language-independent intermediate representation to the cloud model to obtain the standard task processing result generated by it, or directly execute the speech processing task based on the language-independent intermediate representation on the terminal side to obtain the basic task processing result. Thus, by replacing the transmission of the original audio with the language-independent intermediate representation, the occupancy of network bandwidth and transmission delay are reduced, the transmission efficiency is improved. At the same time, since this language-independent intermediate representation is generated by a speech encoder that has undergone knowledge distillation of the cloud model, it ensures the semantic alignment with the cloud model, enabling the cloud model to perform high-precision processing on this basis, thereby ensuring that the accuracy of the final processing result is not inferior to the traditional scheme of uploading the original audio, and achieving the effect of improving the overall efficiency of end-cloud collaboration while ensuring the processing accuracy.

[0028] It should be noted that the execution subject of this embodiment can be an electronic device with data processing, network communication, and program running functions, such as a mobile phone, a smart wearable device, a laptop computer, etc. Hereinafter, the user terminal will be used as an example to illustrate this embodiment and the following embodiments.

[0029] It should also be noted that the "language-independent intermediate representation" in this embodiment is a feature vector learned from multilingual speech signals. It maps speech inputs of different languages or language mixtures into a unified vector space by stripping acoustic characteristics related to specific languages (such as phonology, intonation) and strengthening the acoustic and semantic commonalities shared across languages. The vector distance in this space directly reflects the semantic similarity between speech segments, rather than language similarity.

[0030] The language-independent intermediate representation can be regarded as the "semantic fingerprint" or "thought code" of speech. Whether saying "Hello" in Chinese, "Hello" in English, or "Hola" in Spanish, as long as the intention is to greet, after being processed by the user terminal, their "semantic fingerprints" will be highly similar. This enables downstream tasks (such as translation, retrieval) to be directly processed based on the "intention" without caring about which language the input is.

[0031] The first embodiment of the speech processing method of the present application is proposed below. Refer to Figure 1 , [[ID=!15]] Figure 1 is the flowchart of the first embodiment of the speech processing method of the present application.

[0032] In this embodiment, the voice processing method includes steps S10 to S20: Step S10: Extract language-independent intermediate representations from the input speech using a speech encoder. The language-independent intermediate representations are feature vectors that reflect the acoustic details and semantic information of the input speech. The speech encoder is a neural network based on a Transformer structure and is obtained by knowledge distillation based on a cloud model. The system extracts language-independent intermediate representations from input speech in real time. These representations are feature vectors reflecting the acoustic details (such as pronunciation rhythm and phoneme information) and semantic information (such as sentence logic and contextual meaning) of the input speech. This language-independent intermediate representation can be directly used for various speech processing tasks without relying on pre-processing language or semantic recognition. The speech encoder is implemented using a lightweight Transformer-based neural network. This encoder is trained using a larger-scale cloud model deployed in the cloud as the teacher model through knowledge distillation, ensuring that the speech encoder deployed on the user terminal maintains its lightweight design while aligning its output language-independent intermediate representation with the cloud model in the semantic space.

[0033] It should also be noted that the cloud model in this embodiment refers to a neural network model with large-scale parameters and powerful computing capabilities deployed on a cloud server, which has stronger feature extraction and semantic representation capabilities than the terminal speech encoder. During the training phase of the speech encoder deployed on the user terminal, the cloud model acts as a teacher model, guiding the training of the speech encoder through knowledge distillation technology; during the speech processing phase, as a processing center, it can enhance and perform deep processing on the language-independent intermediate representations uploaded by the user terminal to generate high-precision speech processing results.

[0034] Step S20: Based on the preset mapping relationship between the running context and the speech processing strategy, execute the target speech processing strategy corresponding to the current running context. The target speech processing strategy is to upload the language-independent intermediate representation to the cloud model and receive the standard task processing result returned by the cloud model, or to execute the speech processing task according to the language-independent intermediate representation to obtain the basic task processing result.

[0035] It should be noted that the runtime context is a comprehensive state parameter that can cover the user terminal's own resource status, such as battery level, as well as external environmental factors, such as network bandwidth and latency. Depending on the actual application scenario, the runtime context can include more types of state parameters.

[0036] After obtaining the language-independent intermediate representation, a decision is made based on the mapping relationship between the predefined runtime context and the speech processing strategy. Specifically, the optimal speech processing strategy for the current runtime context can be dynamically selected by querying a preset strategy mapping table, i.e., the target speech processing strategy. The speech processing strategy can be to upload the language-independent intermediate representation to the cloud model to obtain high-precision standard task processing results; or, to directly perform lightweight processing on the user terminal based on the language-independent intermediate representation to quickly obtain basic task processing results. Generally speaking, the accuracy, completeness, or richness of the basic task processing results are lower than those of the standard task processing results, but the processing efficiency is higher, and it can provide users with more timely feedback, ensuring a better user experience in different scenarios.

[0037] In one feasible embodiment, steps A10 to A20 may be included before step S20: Step A10: Monitor the user terminal's battery level and network bandwidth; Step A20: Set battery level and network bandwidth as the current operating context; Monitoring the battery level and network bandwidth of user terminals can be achieved by calling the power management interface and network diagnostic tools in the user terminal to obtain the remaining battery percentage and downlink bandwidth data of the current network connection in real time, and use the battery level and network bandwidth as the current running context.

[0038] In one feasible implementation scenario, power consumption and network bandwidth can be quantified into predefined levels, such as high, medium, and low, and a direct mapping relationship can be established between the level of the current operating context and the voice processing strategy. Alternatively, the values ​​of power consumption and network bandwidth can be compared with the corresponding thresholds, and the voice processing strategy can be selected based on the comparison results.

[0039] Based on this, step S20 may include steps S201 to S202: Step S201: If the battery level exceeds a preset battery level threshold and the network bandwidth exceeds a preset bandwidth threshold, then upload the language-independent intermediate representation to the cloud model and receive the standard task processing result returned by the cloud model. In step S202, if the battery level does not exceed the preset battery level threshold or the network bandwidth does not exceed the preset bandwidth threshold, then the speech processing task is performed based on the language-independent intermediate representation to obtain the basic task processing result.

[0040] If the battery level exceeds a preset battery threshold (e.g., 30%) and the network bandwidth exceeds a preset bandwidth threshold (e.g., 1Mbps), the language-independent intermediate representation is uploaded to the cloud model and the standard task processing result returned by the cloud model is received. If the battery level does not exceed the preset battery threshold or the network bandwidth does not exceed the preset bandwidth threshold, the speech processing task is performed based on the language-independent intermediate representation to obtain the basic task processing result. The preset battery threshold and preset bandwidth threshold can be set based on the actual application scenario, and are not specifically limited in this embodiment.

[0041] Thus, the user terminal can intelligently select the optimal voice processing path according to the actual operating status, ensuring both the voice processing quality when resources are sufficient and the functional availability when resources are limited.

[0042] In one feasible embodiment, steps S202 may be followed by steps S203 to S205: Step S203: Determine whether the task type of the voice processing task is a real-time task; It should be noted that real-time tasks refer to task types that are sensitive to processing latency, such as voice control commands and real-time dialogue translation, while non-real-time tasks refer to tasks that allow for a certain amount of processing latency, such as audio transcription and document translation.

[0043] To determine whether the voice processing task currently being performed by the user terminal is a real-time task, the following methods can be used: First, determine the task type by parsing the task type identifier contained in the task request. If the identifier matches a preset real-time task identifier, it is determined to be a real-time task. Second, determine the task by analyzing the interactive characteristics of the input voice. If the voice stream exhibits a short-term interactive mode and includes an immediate response requirement, it is determined to be a real-time task. Third, determine the task by verifying the service quality requirements configured in the task. If the task is configured for streaming processing or the required end-to-end latency is lower than a preset threshold, it is determined to be a real-time task.

[0044] Step S204: If not, cache the language-independent intermediate representation, and if the battery power exceeds a preset battery power threshold and the network bandwidth exceeds a preset bandwidth threshold, upload the cached language-independent intermediate representation to the cloud model and receive the optimization processing result returned by the cloud model.

[0045] If the task type of the voice processing task is determined to be not a real-time task, then the language-independent intermediate representation is cached. This intermediate representation can be temporarily stored in the terminal's non-volatile memory. When the battery level exceeds a preset battery threshold and the network bandwidth exceeds a preset bandwidth threshold, the cached language-independent intermediate representation is uploaded to the cloud model, and the optimized processing result returned by the cloud model is received.

[0046] Step S205: Replace the content of the basic task processing result with the content of the optimized processing result.

[0047] Upon receiving the optimized processing result from the cloud model, the system automatically replaces the locally generated basic task processing result with the optimized processing result returned by the cloud model. For example, if the basic task processing result has already been displayed as text on the user's terminal, the content on the display is dynamically updated to ensure the optimized processing result overwrites the original basic task processing result; if the basic task processing result has not yet been output, the optimized processing result is directly set as the final output content in the internal data stream. This replacement process requires no manual user intervention, achieving seamless optimization of the processing result through a silent update mechanism, while retaining the timestamp and contextual information of the original task, ensuring the continuity of user interaction and data consistency. For application scenarios requiring version management, the replacement relationship between the basic task processing result and the optimized processing result can be recorded, supporting result backtracking and comparative analysis.

[0048] Thus, through this asynchronous optimization mechanism, while ensuring immediate response, higher quality processing results can be obtained when resource conditions permit. This realizes the voice processing strategy of first solving the problem of availability and then pursuing quality, which not only ensures the smoothness of user experience but also ensures the high standard of the final processing results. At the same time, it avoids failures and resource waste caused by forcibly uploading data in weak network environments.

[0049] Furthermore, if the voice processing task is determined to be a real-time task, the subsequent cloud upload process for the current voice segment is terminated, and the basic task processing result is directly output as the final processing result. In other words, for application scenarios with extremely high requirements for response timeliness, such as voice control commands and real-time dialogue translation, unnecessary network transmission and cloud processing steps are avoided, minimizing system resource consumption while ensuring functional availability.

[0050] For example, a speech processing flow is as follows Figure 2 As shown, the input speech is fed into the speech encoder to obtain a language-independent intermediate representation. In this scenario, the user terminal detects that the current battery level exceeds a preset battery threshold and the network bandwidth exceeds a preset bandwidth threshold, which meets the conditions for cloud processing. Therefore, the generated language-independent intermediate representation is uploaded to the cloud model for deep processing, and finally, a high-precision standard task processing result is returned by the cloud model.

[0051] Another speech processing flow is as follows Figure 3As shown, the input speech is fed into a speech encoder to obtain a language-independent intermediate representation. In this scenario, the user terminal detects that the current battery level and network bandwidth are within the preset thresholds. Therefore, the language-independent intermediate representation is handed over to the local task processing unit to perform speech processing tasks, quickly obtaining the basic task processing results. After obtaining the basic task processing results, it is further determined whether the current task type is a real-time task. If it is determined to be a non-real-time task, it enters the asynchronous optimization stage, asynchronously uploading the language-independent intermediate representation to the cloud model, and finally obtaining the optimized processing results returned by the cloud. Thus, by prioritizing the availability of basic speech processing functions in a resource-constrained environment and then optimizing the results through the cloud model when conditions permit, a balance between processing efficiency and quality is achieved.

[0052] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, step S10 may include steps S101~S102: Step S101: Perform frame segmentation on the input speech to obtain multiple speech frames; First, the input speech is processed by framing. For example, a sliding window with a frame length of 25 milliseconds and a frame shift of 10 milliseconds is used to divide the continuous speech signal into a series of short speech frames, which prepares for subsequent feature extraction.

[0053] Step S102: Extract acoustic features from multiple speech frames and encode the acoustic features using a speech encoder to obtain a language-independent intermediate representation.

[0054] It should be noted that the speech encoder in this embodiment adopts a lightweight neural network architecture, with its core based on a pruned Transformer structure. In one feasible implementation, the speech encoder contains 4 to 6 encoding layers, each equipped with a 4-head attention mechanism, and the hidden layer dimension is set between 256 and 512 dimensions. To further improve the computational efficiency on the user terminal, depthwise separable convolution can be used instead of standard convolution operations in the feature extraction part at the front end of the encoder, thereby reducing the computational complexity and number of parameters of the encoder.

[0055] Acoustic features such as Mel (Mel-Frequency Spectrum) spectral coefficients are extracted from the segmented speech frames. Then, the speech encoder performs nonlinear transformation and semantic abstraction on the acoustic features through its multi-layer Transformer structure, and finally outputs a fixed-dimensional language-independent intermediate representation.

[0056] For example, in one feasible implementation, the speech encoder includes a shared encoder and a hierarchical feature fusion network. The process for generating language-independent intermediate representations is as follows: input speech in any language is received and processed by a shared encoder based on a Transformer architecture. This encoder is trained through self-supervised learning and is able to encode the input speech into a sequence of acoustic features containing common acoustic characteristics across languages. The acoustic feature sequence is then fed into a hierarchical feature fusion network for progressive processing. This network comprises a bottom-level fusion module, a middle-level fusion module, and a high-level fusion module. In the bottom-level fusion module, local attention mechanisms and convolutional attention operations are used to extract and enhance acoustic unit features shared across languages. In the middle-level fusion module, a hybrid attention mechanism is used to associate acoustic features from different languages, and layer normalization is applied to filter language-specific information, forming acoustic-semantic transition features. In the high-level fusion module, a global attention mechanism is used to capture complete semantic dependencies and, combined with semantic alignment constraints, to map the same semantic content from different languages ​​to a unified representation space. During this process, cross-level skip connections directly pass the acoustic unit features from the bottom level to the high-level fusion module, ensuring that the final language-independent intermediate representation retains acoustic details valuable for semantic understanding. Finally, the semantic encoder outputs a language-independent intermediate representation that completely removes language information, preserving only acoustic details and semantic information. This representation can be directly used for speech processing tasks such as speech recognition and speech translation.

[0057] In one feasible embodiment, steps B10 to B20 may be included before step S10: Step B10: Obtain the language-independent intermediate representation generated by the cloud model as a supervision signal; During the training phase of the speech encoder, the same training speech samples are input into the cloud model and the speech encoder to be trained respectively, and the language-independent intermediate representation output by the cloud model is collected as a high-quality supervision signal.

[0058] Step B20: Train the speech encoder with the training objective of minimizing the difference between the language-independent intermediate representation output by the speech encoder and the supervision signal.

[0059] The speech encoder is trained with the goal of minimizing the difference between the language-independent intermediate representations output by the speech encoder and the supervision signal. The loss function is calculated by the KL divergence (relative entropy) or mean square error between the two language-independent intermediate representations. The backpropagation algorithm is used to optimize the parameters of the terminal speech encoder so that its output intermediate representations are aligned with the cloud model in semantic space.

[0060] Thus, through this knowledge distillation training mechanism, it is ensured that the lightweight speech encoder on the terminal can generate intermediate representations that are semantically consistent with the cloud model.

[0061] In one feasible embodiment, the speech processing method may further include steps C10-C20: Step C10: Receive encoder optimization information sent by the cloud model, wherein the encoder optimization information is generated by the cloud model based on the processing effect of language-independent intermediate representations; Step C20: Update the voice encoder based on the encoder optimization information.

[0062] When the cloud-based model processes the language-independent intermediate representation uploaded by the user terminal, it analyzes the difference between the processing result and the expected result, generating encoder optimization information to improve the speech encoder. This encoder optimization information refers to mathematical quantities used to guide the adjustment of the terminal's speech encoder parameters, and its form includes gradient information of model weights, parameter adjustment amounts, or model difference vectors, etc. The user terminal receives the encoder optimization information sent by the cloud-based model and updates the speech encoder according to the encoder optimization information.

[0063] For example, when the cloud model processes the language-independent intermediate representation uploaded from the user terminal, a model optimization analysis process is simultaneously initiated: the cloud model compares and analyzes the speech processing result generated based on the language-independent intermediate representation with the expected optimal result, and generates model optimization information for the speech encoder of the user terminal by calculating the gradient of the loss function and analyzing feature representation bias. This information specifically represents the adjustment amount or update direction of the encoder parameters. The user terminal receives the encoder optimization information from the cloud model and updates the speech encoder accordingly. As a result, the user terminal generates a higher-quality language-independent intermediate representation using the updated speech encoder, and the cloud model can produce more accurate speech processing results based on the improved language-independent intermediate representation. Through this iterative optimization mechanism, the entire edge-cloud collaborative speech processing system can continuously improve its performance.

[0064] In one feasible embodiment, steps S10 may be followed by steps S30 to S40: Step S30: Determine whether the task scenario of the voice processing task is a multi-terminal collaborative scenario; Whether a multi-terminal collaborative scenario is being determined by parsing the scenario identifier in the task request or analyzing the current connection status of the user terminal. When it is detected that the user terminal is in a multi-party conference mode, a group voice chat state, or that other collaborative devices are identified nearby through the device discovery protocol, it can be determined that the current scenario is a multi-terminal collaborative scenario.

[0065] Step S40: If the task scenario is a multi-terminal collaborative scenario, then based on the preset mapping relationship between the running context and the speech processing strategy, the target speech processing strategy corresponding to the current running context is executed. The target speech processing strategy is to upload the language-independent intermediate representation to the cloud model and receive the multi-terminal collaborative speech processing result returned by the cloud model, or to execute the speech processing task according to the language-independent intermediate representation to obtain the basic task processing result. The multi-terminal collaborative speech processing result is obtained by the cloud model executing the speech processing task according to the hybrid intermediate representation. The hybrid intermediate representation is obtained by associating the language-independent intermediate representation with the language-independent intermediate representation uploaded by other user terminals. In multi-terminal collaborative scenarios, based on the preset mapping relationship between the runtime context and the speech processing strategy, the corresponding target speech processing strategy is selected and executed. In this scenario, the target speech processing strategy is to upload the language-independent intermediate representation to the cloud model. After receiving the language-independent intermediate representations from multiple terminals, the cloud model first performs timestamp alignment, speaker recognition, and semantic association processing on these intermediate representations to generate a hybrid intermediate representation. Based on this hybrid intermediate representation, the cloud model executes the speech processing task to generate a multi-terminal collaborative speech processing result with global context consistency, such as a complete meeting minutes or a unified cross-device translated text. This multi-terminal collaborative speech processing result is returned to each participating user terminal to ensure that all terminals receive consistent processing results.

[0066] Step S50: If the task scenario is not a multi-terminal collaborative scenario, then based on the preset mapping relationship between the running context and the speech processing strategy, the target speech processing strategy corresponding to the current running context is executed. The target speech processing strategy is to upload the language-independent intermediate representation to the cloud model and receive the standard task processing result returned by the cloud model, or to execute the speech processing task according to the language-independent intermediate representation to obtain the basic task processing result.

[0067] In independent processing scenarios without multi-terminal collaboration, the corresponding target speech processing strategy is selected and executed based on the preset mapping relationship between the runtime context and the speech processing strategy. In this scenario, the target speech processing strategy is to upload the language-independent intermediate representation to the cloud model and receive the standard task processing result returned by the cloud model, or to perform a speech processing task based on the language-independent intermediate representation to obtain the basic task processing result.

[0068] Thus, through multi-terminal collaborative scenario judgment and differentiated processing mechanism, it can intelligently adapt to different needs of single-device independent processing and multi-device collaborative processing, while ensuring the processing efficiency of single-device scenarios, and providing high-quality voice processing capabilities with global consistency for multi-terminal collaborative scenarios.

[0069] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the speech processing method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0070] This application also provides a voice processing device, please refer to... Figure 4 The voice processing device includes: The intermediate representation extraction module 10 is used to extract language-independent intermediate representations from the input speech through a speech encoder. The language-independent intermediate representations are feature vectors that reflect the acoustic details and semantic information of the input speech. The speech encoder is a neural network based on the Transformer structure and is obtained by knowledge distillation based on a cloud model. The speech processing module 20 is used to execute a target speech processing strategy corresponding to the current running context based on a preset mapping relationship between the running context and the speech processing strategy. The target speech processing strategy is to upload the language-independent intermediate representation to the cloud model and receive the standard task processing result returned by the cloud model, or to execute a speech processing task based on the language-independent intermediate representation to obtain the basic task processing result.

[0071] Optionally, the speech processing device further includes an encoder training module (not shown), which is used for: Obtain language-independent intermediate representations generated by cloud-based models as supervision signals; The speech encoder is trained with the goal of minimizing the difference between the language-independent intermediate representation output by the speech encoder and the supervision signal.

[0072] Optionally, the voice processing module 20 is also used for: Monitor the battery level and network bandwidth of user terminals; Use battery power and network bandwidth as the current operating context; If the battery level exceeds a preset battery level threshold and the network bandwidth exceeds a preset bandwidth threshold, the language-independent intermediate representation will be uploaded to the cloud model and the standard task processing result returned by the cloud model will be received. If the battery level does not exceed the preset battery level threshold, or the network bandwidth does not exceed the preset bandwidth threshold, then the speech processing task is performed based on the language-independent intermediate representation to obtain the basic task processing result.

[0073] Optionally, the voice processing module 20 is also used for: Determine whether the voice processing task is a real-time task; If not, cache the language-independent intermediate representation, and upload the cached language-independent intermediate representation to the cloud model when the battery power exceeds the preset battery power threshold and the network bandwidth exceeds the preset bandwidth threshold, and receive the optimization processing result returned by the cloud model; Replace the content of the basic task processing result with the content of the optimized processing result.

[0074] Optionally, the middle section represents the extraction module 10, which is also used for: The input speech is segmented into frames to obtain multiple speech frames; Acoustic features are extracted from multiple speech frames, and the acoustic features are encoded by a speech encoder to obtain a language-independent intermediate representation.

[0075] Optionally, the speech processing device further includes an encoder update module (not shown), which is used for: Receive encoder optimization information sent by the cloud model, wherein the encoder optimization information is generated by the cloud model based on the processing effect of language-independent intermediate representation; Update the voice encoder based on encoder optimization information.

[0076] Optionally, the voice processing module 20 is also used for: Determine whether the task scenario for the voice processing task is a multi-terminal collaborative scenario; If the task scenario is a multi-terminal collaborative scenario, then based on the preset mapping relationship between the running context and the speech processing strategy, the target speech processing strategy corresponding to the current running context is executed. The target speech processing strategy is to upload the language-independent intermediate representation to the cloud model and receive the multi-terminal collaborative speech processing result returned by the cloud model, or to execute the speech processing task according to the language-independent intermediate representation to obtain the basic task processing result. The multi-terminal collaborative speech processing result is obtained by the cloud model executing the speech processing task according to the hybrid intermediate representation. The hybrid intermediate representation is obtained by associating the language-independent intermediate representation with the language-independent intermediate representation uploaded by other user terminals. If the task scenario is not a multi-terminal collaborative scenario, then based on the preset mapping relationship between the running context and the speech processing strategy, the target speech processing strategy corresponding to the current running context is executed. The target speech processing strategy is to upload the language-independent intermediate representation to the cloud model and receive the standard task processing result returned by the cloud model, or to execute the speech processing task according to the language-independent intermediate representation to obtain the basic task processing result.

[0077] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the status indication method described above.

[0078] The following is for reference. Figure 5 The diagram illustrates an electronic device suitable for implementing embodiments of this application. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0079] like Figure 5 As shown, the electronic device may include a processing unit 1001 (e.g., a DSP processor), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: an input device 1007 including, for example, a microphone, an accelerometer, etc.; an output device 1008 including, for example, a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 allows the electronic device to exchange data wirelessly or via wired communication with other devices. Although electronic devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. It can be implemented alternatively or with more or fewer systems.

[0080] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0081] Compared with the prior art, the beneficial effects of the electronic device provided in this application embodiment are the same as those of the voice processing method provided in the above embodiment, and other technical features in the electronic device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0082] It should be understood that the various parts disclosed in the embodiments of this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0083] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0084] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the speech processing method in the above embodiments.

[0085] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0086] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0087] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to perform the functions defined in the methods of the embodiments disclosed in this application.

[0088] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0089] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0090] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0091] The readable storage medium provided in this application embodiment is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for performing the above-described speech processing method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application embodiment are the same as the beneficial effects of the speech processing method provided in the above-described embodiments, and will not be repeated here.

[0092] This application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the status indication method described above. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the status indication method provided in the above embodiments, and will not be repeated here.

[0093] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A speech processing method, characterized in that, The voice processing method, applied to a user terminal, includes: The speech encoder extracts language-independent intermediate representations from the input speech, wherein the language-independent intermediate representations are feature vectors that reflect the acoustic details and semantic information of the input speech, and the speech encoder is a neural network based on the Transformer structure and is obtained by knowledge distillation based on a cloud model; Based on the preset mapping relationship between the running context and the speech processing strategy, the target speech processing strategy corresponding to the current running context is executed. The target speech processing strategy is to upload the language-independent intermediate representation to the cloud model and receive the standard task processing result returned by the cloud model, or to execute a speech processing task based on the language-independent intermediate representation to obtain the basic task processing result.

2. The speech processing method as described in claim 1, characterized in that, Before the step of extracting language-independent intermediate representations from the input speech using a speech encoder, the method further includes: The language-independent intermediate representation generated by the cloud model is used as a supervision signal; The speech encoder is trained with the training objective of minimizing the difference between the language-independent intermediate representation output by the speech encoder and the supervision signal.

3. The speech processing method as described in claim 1, characterized in that, Before the step of executing the target speech processing strategy corresponding to the current running context based on the preset mapping relationship between the running context and the speech processing strategy, the method further includes: Monitor the battery level and network bandwidth of the user terminal; Use the battery level and network bandwidth as the current operating context; The step of executing the target speech processing strategy corresponding to the current running context based on the preset mapping relationship between the running context and the speech processing strategy includes: If the battery level exceeds a preset battery level threshold and the network bandwidth exceeds a preset bandwidth threshold, then the language-independent intermediate representation is uploaded to the cloud model and the standard task processing result returned by the cloud model is received. If the battery level does not exceed the preset battery level threshold, or the network bandwidth does not exceed the preset bandwidth threshold, then the speech processing task is performed based on the language-independent intermediate representation to obtain the basic task processing result.

4. The speech processing method as described in claim 3, characterized in that, After the step of performing speech processing tasks based on the language-independent intermediate representation to obtain the basic task processing result, the method further includes: Determine whether the task type of the voice processing task is a real-time task; If not, the language-independent intermediate representation is cached, and if the battery level exceeds the preset battery threshold and the network bandwidth exceeds the preset bandwidth threshold, the cached language-independent intermediate representation is uploaded to the cloud model, and the optimization processing result returned by the cloud model is received. Replace the content of the basic task processing result with the content of the optimized processing result.

5. The speech processing method as described in claim 1, characterized in that, The step of extracting language-independent intermediate representations from input speech using a speech encoder includes: The input speech is segmented into frames to obtain multiple speech frames; Acoustic features are extracted from the multiple speech frames, and the acoustic features are encoded by a speech encoder to obtain a language-independent intermediate representation.

6. The speech processing method as described in claim 1, characterized in that, The method further includes: Receive encoder optimization information sent by the cloud model, wherein the encoder optimization information is generated by the cloud model based on the processing effect of the language-independent intermediate representation; The voice encoder is updated based on the encoder optimization information.

7. The speech processing method as described in claim 1, characterized in that, Following the step of extracting language-independent intermediate representations from the input speech using a speech encoder, the method further includes: Determine whether the task scenario of the voice processing task is a multi-terminal collaborative scenario; If the task scenario is the multi-terminal collaborative scenario, then based on the preset mapping relationship between the running context and the speech processing strategy, the target speech processing strategy corresponding to the current running context is executed. The target speech processing strategy is to upload the language-independent intermediate representation to the cloud model and receive the multi-terminal collaborative speech processing result returned by the cloud model, or to execute a speech processing task based on the language-independent intermediate representation to obtain the basic task processing result. The multi-terminal collaborative speech processing result is obtained by the cloud model executing the speech processing task based on the hybrid intermediate representation. The hybrid intermediate representation is obtained by associating the language-independent intermediate representation with the language-independent intermediate representation uploaded by other user terminals. If the task scenario is not the multi-terminal collaborative scenario, then based on the preset mapping relationship between the running context and the speech processing strategy, the target speech processing strategy corresponding to the current running context is executed. The target speech processing strategy is to upload the language-independent intermediate representation to the cloud model and receive the standard task processing result returned by the cloud model, or to execute a speech processing task based on the language-independent intermediate representation to obtain the basic task processing result.

8. A voice processing device, characterized in that, The voice processing device includes: The intermediate representation extraction module is used to extract language-independent intermediate representations from the input speech through a speech encoder. The language-independent intermediate representations are feature vectors that reflect the acoustic details and semantic information of the input speech. The speech encoder is a neural network based on the Transformer structure and is obtained by knowledge distillation based on a cloud model. The speech processing module is used to execute a target speech processing strategy corresponding to the current running context based on a preset mapping relationship between the running context and the speech processing strategy. The target speech processing strategy is to upload the language-independent intermediate representation to the cloud model and receive the standard task processing result returned by the cloud model, or to execute a speech processing task based on the language-independent intermediate representation to obtain the basic task processing result.

9. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the speech processing method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the speech processing method as described in any one of claims 1 to 7.

Images