Speech processing method and speech processing model training method

Abstract

Embodiments of the present specification provide a speech processing method and a speech processing model training method. The speech processing method comprises: determining a speech to be translated, wherein the speech to be translated is real-time determined speech data or speech segment data; inputting the speech to be translated into a target speech processing model to obtain a speech translation result corresponding to the speech to be translated, wherein the target speech processing model is obtained by model training of an initial speech processing model according to a plurality of speech segment training data, and the initial speech processing model is obtained by model training of a speech processing model according to speech training data; thereby using the target speech processing model to perform speech translation on the speech to be translated to obtain an accurate speech translation result, improving the accuracy of the speech processing result of the neural network model, and avoiding the problem that the speech translation result is inaccurate due to complex speech data.

Description

Technical Field

[0001] This specification relates to the fields of artificial intelligence and speech technology, and particularly to a speech processing method. One or more embodiments of this specification also relate to a speech processing model training method, a speech translation model training method, a computing device, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the continuous development of artificial intelligence technology, neural network models are being applied to various scenarios to perform tasks, such as applying neural network models to speech processing scenarios.

[0003] Current neural network models often produce inaccurate results during speech processing due to the complexity of speech data. Therefore, improving the accuracy of speech processing results from neural network models has become an urgent problem to be solved. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a speech processing method. One or more embodiments of this specification also relate to a speech processing model training method, a speech translation model training method, a speech translation method, a speech processing model training device, a speech translation model training device, a speech processing device, a speech translation device, a computing device, an electronic device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a method for training a speech processing model is provided, comprising:

[0006] Determine speech training data and multiple speech segment training data, wherein the speech training data includes speech training samples and speech sample labels corresponding to the speech training samples, and each speech segment training data includes speech segment training samples and speech segment sample labels corresponding to the speech segment training samples, and the speech segment training samples are extracted from the speech training samples.

[0007] The speech processing model is trained based on the speech training data to obtain an initial speech processing model.

[0008] The initial speech processing model is trained using the training data of the multiple speech segments to obtain the target speech processing model.

[0009] According to a second aspect of the embodiments of this specification, a speech processing model training apparatus is provided, comprising:

[0010] The data determination module is configured to determine speech training data and multiple speech segment training data. The speech training data includes speech training samples and speech sample labels corresponding to the speech training samples. Each speech segment training data includes speech segment training samples and speech segment sample labels corresponding to the speech segment training samples. The speech segment training samples are extracted from the speech training samples.

[0011] The first model training module is configured to train the speech processing model based on the speech training data to obtain an initial speech processing model.

[0012] The second model training module is configured to train the initial speech processing model based on the training data of the multiple speech segments to obtain the target speech processing model.

[0013] According to a third aspect of the embodiments of this specification, a method for training a speech translation model is provided, comprising:

[0014] Determine speech training data and multiple speech segment training data, wherein the speech training data includes speech training samples and speech sample labels corresponding to the speech training samples, and each speech segment training data includes speech segment training samples and speech segment sample labels corresponding to the speech segment training samples, and the speech segment training samples are extracted from the speech training samples.

[0015] The speech translation model is trained based on the speech training data to obtain an initial speech translation model.

[0016] The initial speech translation model is trained using the training data from the multiple speech segments to obtain the target speech translation model.

[0017] According to a fourth aspect of the embodiments of this specification, a speech translation model training apparatus is provided, comprising:

[0018] The data determination module is configured to determine speech training data and multiple speech segment training data. The speech training data includes speech training samples and speech sample labels corresponding to the speech training samples. Each speech segment training data includes speech segment training samples and speech segment sample labels corresponding to the speech segment training samples. The speech segment training samples are extracted from the speech training samples.

[0019] The first model training module is configured to train the speech translation model based on the speech training data to obtain an initial speech translation model.

[0020] The second model training module is configured to train the initial speech translation model based on the training data of the multiple speech segments to obtain the target speech translation model.

[0021] According to a fifth aspect of the embodiments of this specification, a speech processing method is provided, comprising:

[0022] The speech to be translated is determined, wherein the speech to be translated is real-time determined speech data or speech segment data;

[0023] The speech to be translated is input into the target speech processing model to obtain the speech translation result corresponding to the speech to be translated. The target speech processing model is obtained by training an initial speech processing model based on training data of multiple speech segments. The initial speech processing model is obtained by training a speech processing model based on speech training data.

[0024] According to a sixth aspect of the embodiments of this specification, a voice processing apparatus is provided, comprising:

[0025] The speech determination module is configured to determine the speech to be translated, wherein the speech to be translated is speech data or speech segment data determined in real time;

[0026] The speech processing module is configured to input the speech to be translated into a target speech processing model to obtain the speech translation result corresponding to the speech to be translated. The target speech processing model is obtained by training an initial speech processing model based on training data from multiple speech segments. The initial speech processing model is obtained by training a speech processing model based on speech training data.

[0027] According to a seventh aspect of the embodiments of this specification, a speech translation method is provided, applied to a cloud-side device, comprising:

[0028] The voice to be translated sent by the receiving device;

[0029] The speech to be translated is input into a target speech translation model to obtain the speech translation corresponding to the speech to be translated. The target speech translation model is obtained by training an initial speech translation model based on training data of multiple speech segments. The initial speech translation model is obtained by training a speech translation model based on speech training data. The speech training data includes speech training samples and speech sample labels corresponding to the speech training samples. Each speech segment training data includes speech segment training samples and speech segment sample labels corresponding to the speech segment training samples. The speech segment training samples are extracted from the speech training samples.

[0030] The translated speech is sent to the terminal device.

[0031] According to an eighth aspect of the embodiments of this specification, a voice translation device is provided, comprising:

[0032] The voice receiving module is configured to receive the voice to be translated sent by the receiving device.

[0033] A speech translation module is configured to input the speech to be translated into a target speech translation model to obtain the speech translation corresponding to the speech to be translated. The target speech translation model is obtained by training an initial speech translation model based on training data of multiple speech segments. The initial speech translation model is obtained by training a speech translation model based on speech training data. The speech training data includes speech training samples and speech sample labels corresponding to the speech training samples. Each speech segment training data includes speech segment training samples and speech segment sample labels corresponding to the speech segment training samples. The speech segment training samples are extracted from the speech training samples.

[0034] The voice translation sending module is configured to send the voice translation to the end-side device.

[0035] According to a ninth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0036] Memory and processor;

[0037] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of any of the methods described above.

[0038] According to a tenth aspect of the embodiments of this specification, an electronic device is provided, comprising:

[0039] A memory and a processor, the memory and the processor being connected via a bus;

[0040] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of any of the methods described above.

[0041] According to an eleventh aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0042] According to a twelfth aspect of an embodiment of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0043] This specification provides one or more embodiments of a speech processing method. Considering the inaccuracy of speech processing results output by neural network models, this method, during the training of the speech processing model, can determine a large number of complex speech training data and multiple speech segment training data. The model is trained using these speech training data and multiple speech segment training data to obtain a target speech processing model capable of processing complex speech data and outputting accurate speech processing results. The target speech processing model is then used to perform speech translation on the speech to be translated, obtaining accurate speech translation results. This improves the accuracy of the speech processing results from the neural network model and avoids the problem of inaccurate speech translation results due to the complexity of the speech data. Attached Figure Description

[0044] Figure 1 This is a schematic diagram illustrating the application of a speech processing method provided in one embodiment of this specification;

[0045] Figure 2 This is a flowchart illustrating a speech processing model training method provided in one embodiment of this specification;

[0046] Figure 3 This is a schematic diagram of the processing procedure of a speech processing model training method provided in one embodiment of this specification;

[0047] Figure 4 This is a schematic diagram of the model prediction result in a speech processing model training method provided in one embodiment of this specification;

[0048] Figure 5 This is a flowchart illustrating a speech translation model training method provided in one embodiment of this specification;

[0049] Figure 6 This is a flowchart illustrating a speech processing method provided in one embodiment of this specification;

[0050] Figure 7 This is a flowchart illustrating a speech translation method provided in one embodiment of this specification;

[0051] Figure 8 This is a schematic diagram of the structure of a speech processing model training device provided in one embodiment of this specification;

[0052] Figure 9 This is a schematic diagram of the structure of a speech translation model training device provided in one embodiment of this specification;

[0053] Figure 10 This is a schematic diagram of the structure of a voice processing device provided in one embodiment of this specification;

[0054] Figure 11 This is a schematic diagram of the structure of a speech translation device provided in one embodiment of this specification;

[0055] Figure 12 This is a structural block diagram of a computing device provided in one embodiment of this specification;

[0056] Figure 13 This is a structural block diagram of an electronic device provided in one embodiment of this specification. Detailed Implementation

[0057] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0058] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0059] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0060] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0061] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundational model (Foundation Model 1). It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and Multi-modal Pre-training Models (MLMs).

[0062] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0063] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0064] MT (Machine Translation): refers to machine translation.

[0065] Cross-modal l: refers to cross-modal.

[0066] End-to-End Audio Translation LLM (End-to-End Audio Translation Large Language Model) refers to a large-scale end-to-end audio translation model, which is further trained from an LLM model and focuses on translation functionality.

[0067] SFT (Supervised Finetune) refers to supervised fine-tuning training, a technique for training large models to follow instructions and align with human preferences.

[0068] Tokens: refers to tokens, where a token is the state of a word after natural language has been segmented by a large model.

[0069] Input Projector: An input projector is used to transform input data (such as text, images, or other types of signals) into a format and dimension suitable for subsequent processing. In multimodal systems, data from different modalities (such as text, images, and audio) typically have different feature representations. The input projector can help map these different modalities of data to a common feature space (usually text feature tokens), enabling the model to fuse and process them more effectively.

[0070] An AUD (audio-to-speech) encoder is used to process and extract features from audio signals. It transforms the raw audio waveform into a high-level feature representation, which typically captures more important information in the audio, such as pitch, rhythm, and timbre. Audio encoders play a crucial role in large multimodal models, enabling the model to understand and utilize the meaning of audio data within its context.

[0071] S2TT (Speech-to-Text Translation): refers to speech translation, which can convert the speech signal of the source language into text in the target language.

[0072] TTS (Text To Speech) refers to speech synthesis, which can convert text information into audible sound information.

[0073] CosyVoice: A speech synthesis model.

[0074] ASR (Automatic Speech Recognition): refers to speech recognition, which can convert speech information into text information.

[0075] Stream: Streaming refers to a data processing method where, during model training or inference, data is not loaded or input all at once, but processed as a continuous data stream.

[0076] teacherforce: A teacher-guided decoding algorithm refers to a model that predicts the current token given a standard and correct prefix translation.

[0077] Cascade: refers to the combination of multiple independent modules into a complex system, where the output of the previous module serves as the input of the next module to achieve an overall goal.

[0078] Illusion problems refer to situations where the answers generated by large models are not based on any factual data.

[0079] With the continuous development of artificial intelligence technology, neural network models are being applied to various scenarios to perform tasks, such as speech processing. However, current neural network models, due to the complexity of speech data, can lead to inaccuracies in the processing results.

[0080] For example, current speech translation pipelines rely on cascading ASR and translation systems, resulting in significant overall translation latency. Utilizing large-scale end-to-end speech translation models enables cross-modal audio translation, greatly reducing overall latency and increasing performance. However, streaming translation before the audio is complete significantly increases the model's illusion problem. This illusion is particularly pronounced between languages ​​with different expression orders, such as English, Chinese, and Japanese. For instance, when translating the Chinese phrase "Blockchain is the hottest technology trend this year" into the English "Blockchain is the most trendy technology of the year," the Chinese phrase "blockchain is this year" is followed by "of the year" at the end of the English sentence. If the large-scale model attempts to generate the text at the end, the intermediate content will be generated in the wrong order because the Chinese text hasn't been fully expressed, leading to the model generating an illusion.

[0081] Based on this, a speech processing method is provided in this specification. One or more embodiments of this specification also relate to a speech processing model training method, a speech translation model training method, a speech translation method, a speech processing model training device, a speech translation model training device, a speech processing device, a speech translation device, a computing device, an electronic device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0082] See Figure 1 , Figure 1 This diagram illustrates an application of a speech processing method according to an embodiment of this specification, the speech processing method comprising:

[0083] The speech to be translated is determined, wherein the speech to be translated is real-time determined speech data or speech segment data;

[0084] The speech to be translated is input into the target speech processing model to obtain the speech translation result corresponding to the speech to be translated. The target speech processing model is obtained by training an initial speech processing model based on training data of multiple speech segments. The initial speech processing model is obtained by training a speech processing model based on speech training data.

[0085] For example, to meet users' voice translation needs, training data (i.e., audio training data and multiple audio segment training data) can be constructed for a large-scale voice translation model (i.e., a voice processing model). Specifically, a batch of high-quality voice translation data (voice training data) is collected, which includes audio (voice training samples) and corresponding bilingual translations (voice sample labels); the voice translation data is then truncated to obtain multiple audio segment translation data (multiple audio segment training data), which includes audio segments (voice segment training samples) and corresponding bilingual translations (voice segment sample labels).

[0086] The speech translation model is trained using speech translation data to obtain an initial speech translation model (i.e., the initial speech processing model); the initial speech translation model is then trained using speech segment translation data to obtain a trained speech translation model (i.e., the target speech processing model).

[0087] After training the large-scale speech translation model is completed, the server 104 can input the speech data (i.e., the data to be translated) sent in real time by the client 102 into the large-scale speech translation model for translation, obtain the translated text (i.e., the speech translation result), and send the translated text to the client 102, thereby meeting the user's speech translation needs.

[0088] See Figure 2 , Figure 2 A flowchart of a speech processing model training method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0089] Step 202: Determine the speech training data and multiple speech segment training data, wherein the speech training data includes speech training samples and speech sample labels corresponding to the speech training samples, and each speech segment training data includes speech segment training samples and speech segment sample labels corresponding to the speech segment training samples, and the speech segment training samples are extracted from the speech training samples.

[0090] The speech training data can be understood as the data used to train the speech processing model; the multiple speech segment training data can be understood as the data used to train the initial speech processing model obtained by training the speech training data.

[0091] The speech training sample can be speech data used as training samples, such as human voices, conference audio, songs, etc., without specific restrictions. The speech segment training sample can be understood as a speech segment extracted from the speech training sample, such as human voice segments, conference audio segments, song segments, etc.

[0092] In one or more embodiments of this specification, the speech segment training samples are extracted from the speech training samples, which can be understood as the speech segment training samples being obtained by speech segmentation in the speech training samples. This speech segmentation can be based on a speech segmentation time threshold. For example, this method can determine speech training data for a speech processing model; perform speech segmentation on the speech training samples in the speech training data to obtain multiple speech segment training samples; and segment the speech sample labels in the speech training data to obtain speech segment sample labels corresponding to each speech segment training sample; for each speech segment training sample and its corresponding speech segment sample label, a speech segment training sample and its corresponding speech segment sample label are treated as a single speech segment training data set, thereby obtaining multiple speech segment training data sets.

[0093] The speech training data and multiple speech segment training data can be understood as training data associated with the target speech processing task; the target speech processing task can be a speech translation task or a speech recognition task; the speech training sample can be understood as speech data that needs to be translated or recognized; the speech segment training sample can be understood as speech segment data that needs to be translated or recognized.

[0094] Speech sample labels and speech segment sample labels can be understood as sample labels associated with the target speech processing task. In the case of a speech translation task, the speech sample label can be the actual speech translation corresponding to the speech training sample, and the speech segment sample label can be the actual speech segment translation corresponding to the speech segment training sample. In the case of a speech recognition task, the speech sample label can be the actual speech text corresponding to the speech training sample, and the speech segment sample label can be the actual speech segment text corresponding to the speech segment training sample.

[0095] In one or more embodiments provided in this specification, in order to ensure the accuracy and complexity of the training data, this solution can perform speech extraction and data filtering on the speech training data during the process of determining the speech segment training data, and obtain the speech segment training data in the following specific way.

[0096] The determination of the speech training data and multiple speech segment training data includes:

[0097] Determine the speech training data for the speech processing model, and perform speech extraction on the speech training data to obtain multiple candidate speech segment training data. Each candidate speech segment training data includes a candidate speech segment training sample and a candidate speech segment sample label corresponding to the candidate speech segment training sample, and the candidate speech segment training sample is extracted from the speech training sample.

[0098] Data filtering is performed on the training data of the multiple candidate speech segments to obtain the training data of the multiple speech segments.

[0099] In this context, candidate speech segment training data can be understood as training data that requires data filtering. By filtering multiple candidate speech segment training data, the filtered speech segment training data is obtained. Candidate speech segment training samples can be understood as training samples that require data filtering; candidate speech segment sample labels can be understood as sample labels that require data filtering.

[0100] Data filtering can be understood as discarding candidate speech segments that do not meet the requirements for training data, and using candidate speech segments that meet the requirements as training data for speech segments.

[0101] Taking the application of the speech processing model training method provided in this specification in a speech translation scenario as an example, this speech processing model training method is explained. Here, the speech training data and multiple speech segment training data are the training data corresponding to the speech translation task, and the speech processing model is the speech translation model that performs the speech translation task. Based on this, this method can determine the speech training data for the speech translation model; perform speech segmentation on the speech training samples in the speech training data to obtain multiple candidate speech segment training samples, and segment the speech translations (speech sample labels) in the speech training data to obtain candidate speech segment sample labels corresponding to each candidate speech segment training sample; for each candidate speech segment training sample and its corresponding candidate speech segment sample label, one candidate speech segment training sample and its corresponding candidate speech segment sample label are used as one candidate speech segment training data, thereby obtaining multiple candidate speech segment training data.

[0102] Data filtering is performed on multiple candidate speech segment training data. Candidate speech segment training data that does not meet the requirements are discarded, and candidate speech segment training data that meets the requirements is retained as speech segment training data, thereby accurately determining the training data that meets the requirements.

[0103] In one or more embodiments provided in this specification, the attribute information of candidate speech segment training samples and the correlation between candidate speech segment training samples and corresponding candidate speech segment sample labels can be used to filter multiple candidate speech segment training data, thereby accurately determining the training data that meets the requirements. The specific implementation method is as follows.

[0104] The step of filtering the training data of the multiple candidate speech segments to obtain the speech segment training data includes steps one and two:

[0105] Step 1: Based on the attribute information of each candidate speech segment training sample, perform a first-stage data filtering on the training data of each candidate speech segment to obtain the first-stage training data, wherein the first-stage training data is the candidate speech segment training data determined after the first-stage data filtering.

[0106] In this context, the attribute information can be understood as information that characterizes specific attributes of the candidate speech segment training sample. For example, the attribute information can be the speech segment duration, speech segment size, speech segment length, etc. of the candidate speech segment training sample.

[0107] Following the previous example, during the data filtering process for candidate speech segment training data, the duration or size of the speech segment in each candidate speech segment training sample can be determined. If the duration of a candidate speech segment training sample is less than or equal to a duration threshold (e.g., 1 second) or the size is less than or equal to 1KB, the candidate speech segment training sample is discarded. Otherwise, the candidate speech segment training sample is determined to pass the data filtering and is retained. Then, the retained candidate speech segment training samples and their corresponding candidate speech segment sample labels are used as the first-stage training data. Thus, efficient filtering of candidate speech segment training data is achieved through attribute information.

[0108] In one or more embodiments provided in this specification, the attribute information is the duration of a speech segment;

[0109] The first stage of data filtering, based on the attribute information of each candidate speech segment training sample, to obtain the first stage training data includes:

[0110] From the candidate speech segment training samples included in each candidate speech segment training data, determine the target candidate speech segment training sample whose speech segment duration is greater than or equal to the first time threshold, and determine the candidate speech segment training data corresponding to the target candidate speech segment training sample as the target speech segment training data;

[0111] Using a semantic recognition model, semantic recognition is performed on the target candidate speech segment training samples in the target speech segment training data to obtain the semantic recognition result of the target speech segment training data, wherein the semantic recognition result is a semantically clear result or a semantically unclear result;

[0112] The target speech segment training data corresponding to the semantically clear result is determined as the first stage training data.

[0113] The duration of the speech segment can be understood as the length of time corresponding to the training sample of the speech segment; the first time threshold can be set according to the actual application scenario, for example, the first time threshold can be 1 second or 1 minute.

[0114] The target speech segment training data can be understood as speech segment training data after data filtering by speech segment duration and first time threshold.

[0115] A semantic recognition model can be understood as a model used to perform semantic recognition on training samples of speech segments. This model can identify whether the training samples contain semantic content. For example, it can identify whether the training samples contain clear dialogue content, clear human voice content, or empty sounds, etc. In the embodiments of this specification, the semantic recognition model can be a classification model, which can take as input classification results containing clear semantics (i.e., semantically clear results) and classification results not containing clear semantics (i.e., semantically unclear results). For example, the semantic recognition model can be an empty sound discrimination model, which is used to determine whether short speech segments (speech segment training samples) contain clear semantics. If a speech segment is classified as an empty sound segment that does not contain clear semantics (i.e., semantically unclear results), these segments will be further filtered. If a speech segment is classified as a speech segment containing clear semantics (i.e., semantically clear results), these speech segments and their corresponding translations (i.e., the first-stage training data) will be retained.

[0116] Following the previous example, this method takes into account that short speech segments (such as 0.5s, 1s, 1.5s, etc.) often cannot express clear semantics. Therefore, this method first filters out speech segments shorter than 1s (i.e., the first time threshold) and retains speech segments longer than or equal to 1s (i.e., target candidate speech segment training samples).

[0117] Then, speech segments longer than 1 second are input into the empty sound discrimination model. The empty sound discrimination model is used to determine whether shorter speech segments contain clear semantics. If the speech segment is classified as an empty sound segment that does not contain clear semantics (i.e., semantically unclear result), these segments will be further filtered. If the speech segment is classified as a speech segment that contains clear semantics (i.e., semantically clear result), these speech segments and their corresponding translations will be retained.

[0118] It should be noted that this empty sound discrimination model can be a model trained using training data.

[0119] In the above embodiments, this method uses the method of judging whether the duration of the speech segment is greater than or equal to a first time threshold and whether the semantic recognition result output by the semantic recognition model is a semantically clear result to perform the first stage of data filtering on the candidate speech segment training data, thereby obtaining the first stage training data containing clear semantic content and improving the model training effect.

[0120] Step 2: Based on the correlation between the candidate speech segment training samples included in the first stage training data and the corresponding candidate speech segment sample labels, perform a second stage data filtering on the first stage training data to obtain the multiple speech segment training data, wherein the speech segment training data is the first stage training data determined after the second stage data filtering.

[0121] The correlation can be understood as a parameter that characterizes the degree of correlation between the training samples of candidate speech segments and the labels of candidate speech segments. This correlation can be the similarity between the speech recognition text of the training samples of candidate speech segments and the corresponding labels of candidate speech segments. The speech recognition text can be the speech recognition text obtained by performing speech recognition on the training samples of candidate speech segments. Alternatively, the correlation can be the ratio between the size of the speech segment of the training samples of candidate speech segments and the size of the corresponding labels of candidate speech segments.

[0122] Following the previous example, during the second-stage data filtering process of the first-stage training data, the correlation between each candidate speech segment training sample in the first-stage training data and the corresponding candidate speech segment sample label can be determined.

[0123] If the correlation degree of a first-stage training data is greater than the preset threshold, then the first-stage training data can be discarded. Conversely, if the correlation degree of a first-stage training data is less than or equal to the preset threshold, then the first-stage training data can be retained and used as training data for speech segments.

[0124] In the process of filtering training data for multiple candidate speech segments in the above embodiments, a first-stage data filtering and a second-stage data filtering can be used to filter the training data for multiple candidate speech segments. Through two-stage data filtering, accurate and suitable speech segment training data can be obtained from the training data for multiple candidate speech segments.

[0125] In one or more embodiments provided in this specification, the correlation degree is the similarity between the speech segment recognition text of the candidate speech segment training sample and the candidate speech segment recognition text, the speech sample label includes speech text, the speech segment sample label is speech segment text, and the speech segment text is extracted from the speech text.

[0126] The step of performing a second-stage data filtering on the first-stage training data based on the correlation between the candidate speech segment training samples included in the first-stage training data and the corresponding candidate speech segment sample labels to obtain the speech segment training data includes:

[0127] From the candidate speech segment training samples included in the first stage training data, target candidate speech segment training samples with a speech segment duration less than or equal to a second time threshold are determined, and the first stage training data corresponding to the target candidate speech segment training samples is determined as the target first stage training data.

[0128] Using a speech recognition model, speech recognition is performed on the target candidate speech segment training samples in the first stage training data of the target to obtain the candidate speech segment recognition text of the target candidate speech segment training samples;

[0129] Determine the similarity between the recognized text of the speech segment and the candidate speech segment text, and determine a target similarity less than or equal to a preset similarity threshold from the similarity;

[0130] The first-stage training data of the target corresponding to the target similarity is used as the training data of the speech segment.

[0131] The second time threshold can be set according to the actual application scenario. For example, this method can use the time threshold obtained by subtracting a preset time from the speech duration of the speech training sample as the second time threshold. The speech duration can be the speech length of the speech training sample, and the preset time can be set according to the actual scenario, such as 1 second or 1 minute. Alternatively, the second time threshold can be set to 10 seconds, 2 minutes, etc.

[0132] The target first-stage training data can be understood as the first-stage training data after filtering the data through the duration of the speech segment and the second time threshold.

[0133] Here, speech text can be understood as the real speech text corresponding to the speech training sample, which records the language content contained in the speech training sample; speech segment text can be understood as the real speech segment text corresponding to the speech segment training sample, which records the language content contained in the speech segment training sample; extracting speech segment text from the speech text means obtaining speech segment text by segmenting the speech text.

[0134] A speech recognition model can be understood as a model used to perform speech recognition on training samples of speech segments. Using this speech recognition model, the corresponding speech text of the training sample can be identified. For example, the speech data "good morning" can be recognized as the text data "good morning". This semantic recognition model can be an ASR model.

[0135] The preset similarity threshold can be set according to the actual application scenario, and this manual does not impose specific limitations on it.

[0136] Following the previous example, this method considers that when the speech segments are relatively complete (e.g., the speech segment duration is 4s, 4.5s, 5s), the difference between them and the original complete speech data is insufficient. Therefore, in order to provide complexity for the training data, this method can filter out speech segments that are greater than a time threshold (i.e., the second time threshold) and retain speech segments that are less than or equal to the time threshold (i.e., target candidate speech segment training samples). The time threshold can be the speech duration of the complete speech time minus 1 second.

[0137] Then, speech segments less than or equal to the time threshold are input into the ASR model. The ASR model recognizes longer speech segments to obtain the corresponding speech segment recognition text (i.e., candidate speech segment recognition text). This speech segment recognition text records the language content in the speech segment.

[0138] The similarity between the recognized text of a speech segment and the corresponding real speech text (i.e., candidate speech segment text) is calculated. If the similarity is greater than a threshold theta (i.e., a preset similarity threshold), these speech segments will be further filtered. If the similarity is less than or equal to the threshold theta, then the speech segment contains clear semantics and differs from the original complete speech after segmentation. Therefore, these speech segments and their corresponding real speech text (i.e., speech segment training data) are retained.

[0139] In the above embodiments, this method uses the method of judging whether the duration of the speech segment is less than or equal to a second time threshold and whether the similarity between the speech segment recognition text and the candidate speech segment text is less than or equal to a preset similarity threshold to perform data filtering in the second stage, thereby obtaining multiple speech segment training data containing clear semantic content and different from the speech training data, which increases the complexity of the training data and improves the effect of subsequent model training.

[0140] Based on the above, this method can employ at least four data filtering strategies during the data filtering process for multiple candidate speech segment training data to obtain the multiple speech segment training data. These at least four data filtering strategies include: determining whether the speech segment duration of the candidate speech segment training sample is greater than or equal to a first time threshold; determining whether the semantic recognition result of the speech segment training data is semantically clear; determining whether the speech segment duration of the candidate speech segment training sample is less than or equal to a second time threshold; and determining whether the similarity between the speech segment recognition text of the candidate speech segment training sample and the candidate speech segment text (i.e., the candidate speech segment sample label) is less than or equal to a preset similarity threshold. If all four data filtering strategies result in a "yes" outcome for the candidate speech segment training data, then the candidate speech segment training data is determined to be speech segment training data.

[0141] In one or more embodiments provided in this specification, in order to increase the complexity and quantity of training data, this method can obtain multiple candidate speech segment training data by segmenting the speech training data. The specific implementation is as follows:

[0142] The step of extracting speech from the speech training data to obtain multiple candidate speech segment training data includes:

[0143] Determine the speech segmentation time threshold for the speech training samples;

[0144] Based on the speech segmentation time threshold, the speech training data is segmented to obtain multiple candidate speech segment training data.

[0145] The speech segmentation time threshold can be set according to the actual application scenario. For example, the speech segmentation time threshold can be 1 second, 0.5 seconds, etc.

[0146] Specifically, in the process of segmenting speech training data, this method needs to determine the speech segmentation time threshold for the speech training samples, and use the speech segmentation time threshold to divide the speech training data into multiple average speech segmentation ranges; based on these multiple average speech segmentation ranges, the speech training data is segmented to obtain multiple candidate speech segment training data, wherein the speech segment time of the candidate speech segment training data is consistent with the speech segmentation time threshold, thereby increasing the complexity and quantity of training data.

[0147] In one or more embodiments provided in this specification, the step of extracting speech from the speech training data to obtain multiple candidate speech segment training data includes:

[0148] The number of speech segments corresponding to the speech training sample is calculated based on the speech length of the speech training sample and the speech segmentation time threshold.

[0149] Based on the number of speech segments and the speech segmentation time threshold, multiple speech segmentation ranges are calculated, wherein the number of multiple speech segmentation ranges is the same as the number of speech segments, each speech segment segmentation range is different, and the speech segmentation ranges overlap.

[0150] Based on the segmentation range of the multiple speech segments, the speech training samples and the speech sample labels corresponding to the speech training samples are segmented to obtain the multiple speech segment training samples and the speech segment sample labels corresponding to each speech segment training sample.

[0151] The training samples of the multiple speech segments, and the speech segment sample labels corresponding to each training sample of the speech segments, are used as the training data of the multiple candidate speech segments.

[0152] The number of speech segments can be understood as the number of speech segment training samples obtained from the speech training samples. For example, 5 or 10. The speech segment segmentation range can be understood as the time range corresponding to the speech segment training samples during the process of segmenting speech segment training samples from the speech training samples. For example, the speech segmentation range can be the time range from the 0th second (0s) to 0.5s, 1s, 1.5s, 2s, 2.5s, 3s, 3.5s, 4s, 4.5s, or 5s.

[0153] Following the previous example, this method first collects a batch of high-quality speech translation data (i.e., speech training data), which includes speech (speech training samples) and corresponding bilingual translations (speech sample labels).

[0154] Secondly, an interval threshold Xs is determined for the speech, and the speech is truncated at intervals of Xs. Taking a 5-second speech with an interval threshold Xs of 0.5s as an example, 5s and 0.5s are divided to determine the number of speech segments. Then, based on the number of speech segments and the interval threshold Xs of 0.5s, the segmentation time range of multiple speech segments (i.e., the segmentation range of multiple speech segments) is calculated and truncated. After truncation, speech segments of different lengths (0.5s, 1s, 1.5s, 2s, 2.5s, 3s, 3.5s, 4s, 4.5s, 5s) and corresponding bilingual translation segments (i.e., speech segment sample labels) are generated, thereby increasing the complexity and quantity of training data.

[0155] Step 204: Train the speech processing model based on the speech training data to obtain an initial speech processing model.

[0156] Here, a speech processing model can be understood as a model used to process speech data. This speech processing model can be a speech translation model, a speech recognition model, or a large language model (LLM); for example, this speech processing model can be an end-to-end speech translation large model. The initial speech processing model is the initial speech processing model trained using speech training data.

[0157] In one or more embodiments provided in this specification, the speech sample labels include speech translations and speech texts, and the speech processing model is a speech translation model;

[0158] The step of training the speech processing model based on the speech training data to obtain an initial speech processing model includes:

[0159] The speech training samples are input into the speech translation model to perform speech translation, thereby obtaining the predicted speech translation and predicted speech text of the speech training samples;

[0160] Based on the predicted speech translation, the speech translation, the predicted speech text, and the speech text, a first loss function is determined, and the model parameters of the speech translation model are adjusted based on the first loss function to obtain the initial speech translation model.

[0161] Following the previous example, the speech sample labels in this method include speech translations and speech texts. Therefore, during model training, speech (i.e., speech training samples) can be input into the original end-to-end speech translation model (i.e., speech processing model) for translation processing to obtain the predicted speech translations and predicted speech texts corresponding to the speech. Then, based on the predicted speech translations, speech translations, predicted speech texts, and speech texts, the cross-entropy loss function (i.e., the first loss function) is calculated, and the model parameters of the original end-to-end speech translation model are adjusted based on the cross-entropy loss function to obtain the initialized end-to-end speech translation model (i.e., the initial speech processing model).

[0162] A large-scale end-to-end speech translation model is trained using high-quality speech translation data. This model is then used to translate complete speech, yielding predicted translations. A loss function is calculated based on these predicted translations, the speech-recognized text, and the translated bilingual text, thus training an initialized large-scale end-to-end speech translation model. At this point, the model possesses the ability to translate complete audio, but it is prone to producing illusions when the audio is incomplete during streaming.

[0163] Step 206: Train the initial speech processing model based on the training data of the multiple speech segments to obtain the target speech processing model.

[0164] The target speech processing model can be understood as a speech processing model trained using training data from multiple speech segments.

[0165] In one or more embodiments provided in this specification, the speech sample labels are speech translations and speech texts, the speech segment sample labels are speech segment translations and speech segment texts, the speech segment translations are extracted from the speech translations, the speech segment texts are extracted from the speech texts, and the speech processing model is a speech translation model.

[0166] The step of training the initial speech processing model based on the training data of the multiple speech segments to obtain the target speech processing model includes:

[0167] The target speech segment training sample in the target speech segment training data is input into the initial speech translation model for speech translation to obtain the predicted speech segment translation and predicted speech segment text of the target speech segment training sample, wherein the target speech segment training data is any one of the plurality of speech segment training data;

[0168] A second loss function is determined based on the speech segment translation and speech segment text in the target speech segment training data, as well as the predicted speech segment translation and predicted speech segment text.

[0169] Based on the second loss function, the model parameters of the initial speech translation model are adjusted to obtain the target speech translation model.

[0170] Following the previous example, after training the original end-to-end speech translation model using complete speech training data to obtain the initialized end-to-end speech translation model, this method requires further training using speech segments (target speech segment training samples). The specific training method is as follows: First, segmented speech segment translation data is determined from the speech segment training data (i.e., target speech segment training data), and this segmented translation data (i.e., target speech segment training data) is input into the initialized end-to-end speech translation model for translation processing, obtaining predicted speech segment translations and predicted speech segment texts. Second, sample labels are obtained from the speech segment training data; these labels represent the actual speech segment translations and actual speech segment texts. Finally, the cross-entropy loss function (i.e., the second loss function) is calculated using the predicted speech segment translations, predicted speech segment texts, actual speech segment translations, and actual speech segment texts. The model parameters are then adjusted using this cross-entropy loss function to obtain the trained end-to-end speech translation model.

[0171] Using the segmented speech fragment translation data (i.e., the target speech fragment training data), an initialized end-to-end speech translation model is trained. This model identifies speech fragments, obtaining their corresponding speech recognition text, and then translates the speech fragments to obtain their predicted translations. A loss function is calculated based on the predicted translations, the speech recognition text, and the corresponding bilingual translations, resulting in a fully trained end-to-end speech translation model. At this point, the model possesses the ability to accurately translate even incomplete streaming audio.

[0172] In one or more embodiments provided in this specification, the step of inputting the target speech segment training samples from the target speech segment training data into an initial speech translation model for speech translation to obtain the predicted speech segment translation and predicted speech segment text of the target speech segment training samples includes steps one to three:

[0173] Step 1: Input the target speech segment training samples from the target speech segment training data into the initial speech translation model.

[0174] Step 2: Use the initial speech translation model to translate the target speech segment training sample to obtain the predicted speech segment text of the target speech segment training sample, as well as the translation probability parameters and translation termination probability parameters corresponding to the target translation of the target speech segment training sample.

[0175] Following the previous example, during the retraining of the initialized end-to-end speech translation model in this method, the segmented speech segment translation data (i.e., the target speech segment training data) will be input into the initialized end-to-end speech translation model for translation processing. During the translation processing, the initialized end-to-end speech translation model first performs speech recognition on the speech segment translation data to obtain the predicted speech segment text corresponding to the speech segment translation data. Then, it performs translation processing on the predicted speech segment text to obtain the translation probability parameters and the translation termination probability parameters corresponding to the target translation of the target speech segment training sample.

[0176] In one or more embodiments provided in this specification, the step of using the initial speech translation model to translate the target speech segment training sample to obtain the predicted speech segment text of the target speech segment training sample, as well as the translation probability parameters and translation termination probability parameters corresponding to the target translation of the target speech segment training sample, includes:

[0177] In the initial speech translation model, a speech recognition network layer is used to perform text recognition on the training samples of the target speech segment to obtain the predicted speech segment text;

[0178] By using a text processing network layer, the predicted speech segment text is converted into text to obtain the speech segment text features corresponding to the predicted speech segment text;

[0179] Using a speech translation network layer, speech translation is performed based on the text features of the speech segment to obtain the translation probability parameters and the translation termination probability parameters corresponding to the target translation of the training sample of the target speech segment.

[0180] The target translation can be understood as the token predicted by the initial speech translation model during the translation process, and the translation probability parameter corresponding to the target translation can be understood as the probability corresponding to the token.

[0181] The speech translation network layer can be understood as a network layer used for speech translation, and this speech translation network layer can be an LLM; the speech recognition network layer can be understood as a network layer used for speech recognition, and this speech recognition network layer can be an audio encoder (aud io encoder).

[0182] The text processing network layer can be understood as a network layer that converts the output data of the speech recognition network layer (i.e., the predicted speech segment text) into the input data of the speech translation network layer (i.e., the speech segment text features); that is, the text processing network layer is used for data conversion between the speech recognition network layer and the speech translation network layer, so that the two network layers can transfer data; the text processing network layer can be an input projector.

[0183] The text features of this speech segment can be understood as features that can be input into the speech translation network layer for speech translation. For example, the text features of this speech segment can be a token.

[0184] Continuing with the previous example, the initial end-to-end speech translation model is trained using the segmented speech fragment translation data to obtain the trained end-to-end speech translation model. Specifically, the training method for the initial end-to-end speech translation model is as follows:

[0185] 1. Concatenate the filtered speech segments, the correct translation of the prefixes, and the prompt text to obtain the input data, and then input the input data into the initialized end-to-end speech translation model;

[0186] 2. The input data is translated using the audio encoder (aud io encoder), input projector, and large language model (LLM) in the model, and the correct token probability (i.e., translation probability parameter) and the token probability representing the end of the sentence (i.e., the end of translation probability parameter) are output.

[0187] Specifically, the audio signal (speech segment) is input into the Audio Encoder for speech encoding to obtain the speech feature code corresponding to the audio signal (i.e., the predicted speech segment text).

[0188] The speech feature encoding is input into the Input Projector for projection processing, thereby obtaining the audio representation (i.e., speech segment text features) through the Input Projector.

[0189] The audio representation is input into a Large Speech Model (LLM) for translation processing. The LLM outputs the probability of the correct token and the probability of the token representing the end of the sentence.

[0190] In the above embodiments, speech recognition processing is performed through a speech recognition network layer and speech translation processing is performed through a speech translation network layer. By dividing the work into different tasks, speech recognition and speech translation are performed accurately and efficiently, thereby obtaining accurate predicted speech segment text and the corresponding translation probability parameters and translation termination probability parameters.

[0191] Step 3: Use the translation recognition model to perform translation recognition processing on the translation probability parameters and the end translation probability parameters to obtain the translation recognition result, and based on the translation recognition result, determine the predicted speech segment translation of the target speech segment training sample.

[0192] The translation recognition model can be understood as a model that determines whether the translation is truncated at the current moment to correspond to the truncated speech; the translation recognition model can be a large model or a deep learning network model.

[0193] Specifically, in the process of retraining the initial end-to-end speech translation model, this method can improve the accuracy of the predicted speech segment translation by using a translation recognition model to perform translation recognition processing on the translation probability parameters and the translation end probability parameters. This allows for the detection of whether the translation probability parameters and the translation end probability parameters are accurate, obtaining the corresponding translation recognition results. Based on the translation recognition results, the predicted speech segment translation of the target speech segment training sample is determined, thereby achieving the goal of obtaining accurate predicted speech segment translations.

[0194] In one or more embodiments provided in this specification, the step of using a translation recognition model to perform translation recognition processing on the translation probability parameter and the end-of-translation probability parameter to obtain a translation recognition result, and determining the predicted speech segment translation of the target speech segment training sample based on the translation recognition result, includes:

[0195] Determine the ratio of the length of the translated speech to the length of the translated speech segment of the target speech segment, and determine the ratio of the duration of the speech segment of the training sample of the target speech segment to the length of the speech segment among the training samples.

[0196] The translation probability parameter, the translation end probability parameter, the translation length ratio, and the speech length ratio are input into the translation recognition model to obtain the translation recognition result, wherein the translation recognition result is the translation execution result or the translation end result;

[0197] If the translation recognition result is the translation execution result, the speech translation network layer is used to continue to perform speech translation based on the text features of the speech segment, and to obtain the translation probability parameters and the translation termination probability parameters corresponding to the target translation of the target speech segment training sample, until the translation recognition result is the translation termination result.

[0198] If the translation recognition result is the translation completion result, the target translation is used as the predicted speech segment translation.

[0199] Continuing with the previous example, the initial end-to-end speech translation model is trained using the segmented speech fragment translation data to obtain the trained end-to-end speech translation model. Specifically, the training method for the initial end-to-end speech translation model is as follows:

[0200] 1. After inputting the input data into the initialized end-to-end speech translation large model to obtain the audio representation, as well as the correct token probability and the token probability representing the end of the sentence, the pre-trained small model (i.e. the translation recognition model) is used to determine whether the translation is truncated at the current moment to correspond to the truncated speech.

[0201] First, input the three data points—Prob (correct token) - Prob (EOS), the length ratio of the truncated text to the complete translation at the current moment, and the length ratio of the truncated speech to the complete speech—into the small model.

[0202] Secondly, the prior rules for the small model are as follows:

[0203] If Prob(correct token) - Prob(EOS) < the threshold that automatically decreases as the sentence length increases, and the length ratio of the truncated text to the complete translation at the current moment is within a certain preset range above and below the length ratio of the truncated speech to the complete speech, then truncation is required; otherwise, translation inference needs to continue.

[0204] The threshold that automatically adjusts and decreases as the sentence (i.e., speech segment) length increases is a linearly increasing threshold, which is obtained by scaling the sentence length proportionally according to a preset scaling factor.

[0205] During the training of the small model, the input data is: P(correct token) - P(EOS), where the length ratio of the truncated text to the complete translation at the current time step and the length ratio of the truncated speech to the complete speech are used as input features; the output of the small model is a binary classification: whether or not it is truncated.

[0206] 2. If the output of the small model (i.e. the translation recognition result) is truncated, the token currently inferred by the end-to-end speech translation large model is used as the predicted translation of the speech segment; otherwise, the large model continues to perform the translation operation.

[0207] 3. Calculate the loss function using the predicted translation and the translated bilingual translation, and adjust the model parameters of the initialized end-to-end speech translation model based on the loss function to obtain the trained end-to-end speech translation model.

[0208] As can be seen from the above embodiments, the translation recognition model is used to perform translation recognition processing on the translation probability parameters and the translation end probability parameters, thereby detecting whether the translation probability parameters and the translation end probability parameters are accurate and obtaining accurate predicted speech segment translations.

[0209] It should be noted that the processing steps of the speech processing model for training data provided in this manual are the same as the processing steps of the initial speech processing model for training data of multiple speech segments. The processing steps of the speech processing model for training data can be referred to the processing steps of the initial speech processing model for training data of multiple speech segments, and will not be elaborated further here.

[0210] This specification provides one or more embodiments of a speech processing model training method. Considering the inaccuracy of speech processing results output by neural network models, this method, during the training of the speech processing model, can determine a large amount of complex speech training data and multiple speech segment training data. The speech training data includes speech training samples and corresponding speech sample labels. Each speech segment training data includes speech segment training samples and corresponding speech segment sample labels, with the speech segment training samples extracted from the speech training samples. By training the model using the speech training data and multiple speech segment training data, a target speech processing model capable of processing complex speech data and outputting accurate speech processing results is obtained. This improves the accuracy of the speech processing results from the neural network model and avoids the problem of inaccurate speech processing results due to the complexity of the speech data.

[0211] The following is in conjunction with the appendix Figure 3 Taking the application of the speech processing model training method provided in this specification in a speech translation scenario as an example, the speech processing model training method will be further explained. Among them, Figure 3 The diagram illustrates the processing steps of a speech processing model training method according to an embodiment of this specification, specifically including steps one through three.

[0212] Step 1: Build training data for the end-to-end speech translation large model.

[0213] Specifically, the training data is constructed as follows:

[0214] 1. Collect a batch of high-quality speech translation data, which includes the speech and the corresponding bilingual translation.

[0215] 2. Determine the interval threshold Xs for the speech, and truncate the speech at intervals of Xs.

[0216] Taking a 5-second speech as an example with an interval threshold Xs of 0.5 seconds;

[0217] First, divide 5s by 0.5s to determine the number of speech segments.

[0218] Secondly, based on the number of speech segments and the interval threshold Xs0.5s, the segmentation time range of multiple speech segments is calculated; the segmentation time range can be from 0s to 0.5s, 1s, 1.5s, 2s, 2.5s, 3s, 3.5s, 4s, 4.5s, or 5s.

[0219] Finally, the segments are truncated according to the time range, resulting in speech segments of different lengths: 0.5s, 1s, 1.5s, 2s, 2.5s, 3s, 3.5s, 4s, 4.5s, and 5s, along with corresponding bilingual translation segments.

[0220] 3. Retain audio segments that are 1 second or longer.

[0221] This method takes into account that short speech segments (such as 0.5s, 1s, 1.5s, etc.) often cannot express clear semantics. Therefore, this method first filters out speech segments shorter than 1s, retaining speech segments longer than or equal to 1s.

[0222] 4. Filter out speech segments that do not contain clear semantics.

[0223] Specifically, speech segments longer than 1 second are input into the empty speech discrimination model, which then determines whether shorter speech segments contain clear semantics.

[0224] If a speech segment is classified as an empty sound segment that does not contain clear semantics, these segments will be further filtered; if a speech segment is classified as a speech segment that contains clear semantics, these speech segments and their corresponding translations will be retained.

[0225] 5. Retain speech segments that are less than or equal to the time threshold.

[0226] This method takes into account the insufficient difference between speech segments and the original complete speech data when the speech segments are relatively complete (e.g., the speech segment duration is 4s, 4.5s, 5s).

[0227] Therefore, in order to provide complexity for the training data, this method can filter out speech segments that are longer than the time threshold and retain speech segments that are shorter than or equal to the time threshold; where the time threshold can be the speech duration of the complete speech time minus 1 second.

[0228] 6. Input speech segments less than or equal to the time threshold into the ASR model. The ASR model then recognizes longer speech segments to obtain the corresponding speech segment recognition text, which records the language content of the speech segment.

[0229] 7. Retain speech segments with a similarity less than or equal to the threshold theta.

[0230] This method calculates the similarity between the recognized text of a speech segment and the corresponding real speech text. If the similarity is greater than a threshold theta, these speech segments will be further filtered. If the similarity is less than or equal to the threshold theta, then the speech segment contains clear semantics and differs from the original complete speech after segmentation. Therefore, these speech segments and their corresponding real speech text will be retained.

[0231] Step 2: Use complete speech translation data to train and obtain an initial end-to-end speech translation model.

[0232] Specifically, this method can use high-quality speech translation data to train the original end-to-end speech translation model, and then use the original end-to-end speech translation model to translate the complete speech to obtain the predicted translation.

[0233] Based on the predicted translation and the translated bilingual text, a loss function is calculated to train an initialized end-to-end speech translation model. At this point, the model has the ability to translate complete audio, but it is prone to producing illusions in streaming audio with incomplete audio.

[0234] Step 3: Use the segmented speech segment translation data to train the initialized end-to-end speech translation model to obtain the trained end-to-end speech translation model.

[0235] Specifically, the method for training the initial end-to-end speech translation model is as follows:

[0236] 1. The filtered speech segments, the correct translation of the prefix, and the prompt text are concatenated to obtain the input data, which is then fed into the initialized end-to-end speech translation model.

[0237] For example, taking time t as an example, a given truncated speech segment and the correct translation of the prefix from time 0 to t-1 can be input into the model for translation.

[0238] It should be noted that if the speech segment at time t does not have a correct prefix translation, then the speech segment will be discarded and no translation operation will be performed.

[0239] 2. The input data is translated using the audio encoder (aud io encoder), input projector, and large language model (LLM) in the model, and the probability of the correct token and the probability of the token representing the end of the sentence are output.

[0240] Specifically, this method obtains a series of features from the large model decoding process:

[0241] Taking time t as an example, for the correct translation of the prefix of the given truncated speech segment from time +0 to time t-1, calculate the probability of the token at time t as Prob (correct token) and the probability of the token representing the end of the sentence as Prob (EOS).

[0242] 3. Use a pre-trained small model to determine whether the translation is truncated at the current moment to correspond to the truncated speech.

[0243] First, input the three data points—Prob (correct token) - Prob (EOS), the length ratio of the truncated text to the complete translation at the current moment, and the length ratio of the truncated speech to the complete speech—into the small model.

[0244] Secondly, the prior rules for the small model are as follows:

[0245] If Prob(correct token) - Prob(EOS) < the threshold that automatically decreases as the sentence length increases, and the length ratio of the truncated text to the complete translation at the current moment is within a certain preset range above and below the length ratio of the truncated speech to the complete speech, then truncation is required; otherwise, translation inference needs to continue.

[0246] The threshold that automatically adjusts and decreases as the sentence (i.e., speech segment) length increases is a linearly increasing threshold, which is obtained by scaling the sentence length proportionally according to a preset scaling factor.

[0247] This preset range can be set according to the actual application scenario.

[0248] It should be noted that this small model was trained using a certain amount of labeled data obtained through the aforementioned prior rules for pre-truncation and manual verification.

[0249] During the training of the small model, the model input data is: P(correct token) - P(EOS), where the length ratio of the truncated text to the complete translation at the current time step and the length ratio of the truncated speech to the complete speech are used as input features; the output of the small model is a binary classification: whether to truncate.

[0250] The trained small model can automatically determine the truncated translation corresponding to the truncated speech.

[0251] 4. If the output of the small model is truncated, use the token currently inferred by the end-to-end speech translation large model as the predicted translation of the speech segment.

[0252] 5. Calculate the loss function using the predicted translation and the translated bilingual translation, and adjust the model parameters of the initialized end-to-end speech translation model based on the loss function to obtain the trained end-to-end speech translation model.

[0253] At this point, the large model possesses the ability to translate complete audio, as well as the ability to accurately translate in streaming cases with incomplete audio and avoid hallucination problems.

[0254] Based on the above steps, the speech processing model training method in this specification provides a method for constructing streaming data for a large-scale end-to-end speech translation model. This method can obtain a large amount of streaming translations from complete speech translation data on a large scale. This data is then mixed with the original high-quality speech translation data to continue training the large-scale end-to-end speech translation model, enabling the large model to adaptively achieve the effect of waiting for audio / predicting translations. This significantly improves the translation capability of streaming input and output under incomplete audio input. Figure 4 This is a schematic diagram of the model prediction result in a speech processing model training method provided in one embodiment of this specification, based on... Figure 4 As can be seen, this method can automatically construct large batches of streaming data, enabling large models to learn to automatically wait and predict translations even when audio is incomplete. This largely solves the illusion problem of models and improves translation performance in streaming translation scenarios.

[0255] Based on this, this method proposes a way to construct streaming data for a large-scale end-to-end speech translation model. By automatically constructing streaming data, the model learns to automatically wait and predict translations even when the audio is incomplete. This largely solves the model's illusion problem and improves translation performance in streaming translation scenarios.

[0256] Compared to speech translation pipelines built using cascaded ASR+ translation systems, the end-to-end speech translation model trained using this method can achieve cross-modal audio translation, significantly reducing overall translation latency and increasing translation performance. Furthermore, it enables accurate streaming translation even when the audio is incomplete, avoiding the model's illusion problem; and it achieves accurate translation between languages ​​with different expression sequences, such as English, Chinese, and Japanese.

[0257] See Figure 5 , Figure 5 A flowchart of a speech translation model training method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0258] Step 502: Determine the speech training data and multiple speech segment training data, wherein the speech training data includes speech training samples and speech sample labels corresponding to the speech training samples, and each speech segment training data includes speech segment training samples and speech segment sample labels corresponding to the speech segment training samples, and the speech segment training samples are extracted from the speech training samples.

[0259] Step 504: Train the speech translation model based on the speech training data to obtain an initial speech translation model.

[0260] Step 506: Train the initial speech translation model based on the training data of the multiple speech segments to obtain the target speech translation model.

[0261] This specification provides one or more embodiments of a speech translation model training method. Considering the inaccuracy of speech translation results output by neural network models, this method, during the training of the speech translation model, can determine a large amount of complex speech training data and multiple speech segment training data. The speech training data includes speech training samples and corresponding speech sample labels. Each speech segment training data includes speech segment training samples and corresponding speech segment sample labels, with the speech segment training samples extracted from the speech training samples. By training the model using the speech training data and multiple speech segment training data, a target speech translation model capable of translating complex speech data and outputting accurate speech translation results is obtained. This improves the accuracy of the speech translation results of the neural network model and avoids the problem of inaccurate speech translation results due to the complexity of the speech data.

[0262] The above is an illustrative scheme of a speech translation model training method according to this embodiment. It should be noted that the technical solution of this speech translation model training method belongs to the same concept as the technical solution of the speech processing model training method described above. Details not described in detail in the technical solution of the speech translation model training method can be found in the description of the technical solution of the speech processing model training method described above.

[0263] See Figure 6 , Figure 6 A flowchart of a speech processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0264] Step 602: Determine the speech to be translated, wherein the speech to be translated is speech data or speech segment data determined in real time.

[0265] The speech to be translated can be streaming data, meaning it can be a real-time, incomplete stream of audio data. For example, the speech to be translated could be audio data collected during meetings, presentations, video playback, or live broadcasts.

[0266] The audio data can be understood as complete audio data, such as a complete dialogue or a complete explanatory audio recording; audio segment data can be understood as a segment of audio data rather than a complete audio data, such as a segment of audio from a complete explanatory audio recording or a segment of audio from a complete dialogue.

[0267] Step 604: Input the speech to be translated into the target speech processing model to obtain the speech translation result corresponding to the speech to be translated. The target speech processing model is obtained by training an initial speech processing model based on training data of multiple speech segments. The initial speech processing model is obtained by training a speech processing model based on speech training data.

[0268] The target speech processing model can be understood as the target speech processing model based on the speech processing model training method described above; the training method for the target speech processing model can be found in the corresponding or relevant content in the speech processing model training method described above; it will not be elaborated on here.

[0269] The result of speech translation can be understood as the translated text of the speech to be translated.

[0270] In one or more embodiments provided in this specification, the step of inputting the speech to be translated into a target speech processing model to obtain the speech translation result corresponding to the speech to be translated includes:

[0271] The speech to be translated is input into a target speech processing model, wherein the target speech processing model includes a speech recognition network layer, a speech translation network layer, and a text processing network layer;

[0272] The speech recognition network layer is used to perform text recognition on the speech to be translated to obtain the predicted speech text;

[0273] The predicted speech text is converted into text using a text processing network layer to obtain the speech text features corresponding to the predicted speech text.

[0274] Using a speech translation network layer, speech translation is performed based on the speech-text features to obtain the speech translation result corresponding to the speech to be translated.

[0275] The steps for processing the speech to be translated using the speech translation network layer, text processing network layer, and speech recognition network layer in this embodiment can be found in the corresponding steps of processing the target speech segment training samples and / or speech training samples using the speech translation network layer, text processing network layer, and speech recognition network layer in the speech processing model training method described above; these will not be elaborated upon here.

[0276] In one or more embodiments provided in this specification, the step of using a speech translation network layer to perform speech translation based on the speech-text features to obtain a speech translation result corresponding to the speech to be translated includes:

[0277] Using the speech translation network layer, speech translation is performed based on the speech-text features to obtain the translation probability parameters and the translation termination probability parameters corresponding to the target translation of the speech to be translated.

[0278] The translation probability parameter and the translation termination probability parameter are input into the translation recognition model to obtain the translation recognition result, wherein the translation recognition result is the translation execution result or the translation termination result;

[0279] If the translation recognition result is the translation execution result, the speech translation network layer is used to continue to perform speech translation based on the speech text features, and to obtain the translation probability parameters and the translation termination probability parameters corresponding to the target translation of the speech to be translated, until the translation recognition result is the translation termination result;

[0280] If the translation recognition result is the translation completion result, the target translation is taken as the speech translation result corresponding to the speech to be translated.

[0281] Taking the application of the speech processing method provided in this specification in a speech translation scenario as an example, the speech processing method is explained; wherein, the speech translation network layer is LLM; based on this, the speech translation process of this method is as follows:

[0282] 1. After inputting the input data (speech to be translated) into the trained end-to-end speech translation big model to obtain the audio representation, as well as the correct token probability and the token probability representing the end of the sentence, the pre-trained small model (i.e. the translation recognition model) is used to determine whether the translation is truncated at the current moment to correspond to the truncated speech.

[0283] Specifically, the Prob (correct token) - Prob (EOS) is input into the small model to obtain the output result of the small model (i.e., the translation recognition result);

[0284] 2. If the output of the small model is truncated, the token currently inferred by the end-to-end speech translation large model is used as the predicted translation of the speech segment. Otherwise, the large model continues to perform translation operations until the output is truncated.

[0285] As can be seen from the above embodiments, the translation recognition model is used to perform translation recognition processing on the translation probability parameters and the translation termination probability parameters, thereby detecting whether the translation probability parameters and the translation termination probability parameters are accurate and whether the translation can be terminated, and obtaining accurate speech translation.

[0286] In one or more embodiments provided in this specification, determining the speech to be translated includes:

[0287] Receive speech to be translated sent by the client, wherein the speech to be translated is speech data or speech segment data collected by the client in real time;

[0288] The step of inputting the speech to be translated into the target speech processing model to obtain the speech translation result corresponding to the speech to be translated includes:

[0289] The voice translation result is sent to the client for display.

[0290] Following the previous example, when the server in this speech processing method receives the speech data (i.e., the data to be translated) sent by the client in real time, it can input the speech data into the speech translation model for translation, obtain the translated text (i.e., the speech translation result), and send the translated text to the client for display, thereby meeting the user's speech translation needs.

[0291] This specification provides one or more embodiments of a speech processing method. Considering the inaccuracy of speech processing results output by neural network models, this method, during the training of the speech processing model, can determine a large number of complex speech training data and multiple speech segment training data. The model is trained using these speech training data and multiple speech segment training data to obtain a target speech processing model capable of processing complex speech data and outputting accurate speech processing results. The target speech processing model is then used to perform speech translation on the speech to be translated, obtaining accurate speech translation results. This improves the accuracy of the speech processing results from the neural network model and avoids the problem of inaccurate speech translation results due to the complexity of the speech data.

[0292] The above is an illustrative scheme of a speech processing method according to this embodiment. It should be noted that the technical solution of this speech processing method belongs to the same concept as the technical solution of the speech processing model training method described above. For details not described in detail in the technical solution of the speech processing method, please refer to the description of the technical solution of the speech processing model training method described above.

[0293] See Figure 7 , Figure 7 A flowchart of a speech translation method according to an embodiment of this specification is shown. The speech translation method is applied to a cloud-side device and specifically includes the following steps.

[0294] Step 702: The receiving device sends the voice message to be translated;

[0295] Step 704: Input the speech to be translated into the target speech translation model to obtain the speech translation corresponding to the speech to be translated. The target speech translation model is obtained by training an initial speech translation model based on training data of multiple speech segments. The initial speech translation model is obtained by training a speech translation model based on speech training data. The speech training data includes speech training samples and speech sample labels corresponding to the speech training samples. Each speech segment training data includes speech segment training samples and speech segment sample labels corresponding to the speech segment training samples. The speech segment training samples are extracted from the speech training samples.

[0296] Step 706: Send the translated speech to the terminal device.

[0297] In one or more embodiments provided in this specification, the cloud-side device can be a central cloud device in a distributed architecture or an edge cloud device in a distributed architecture. The cloud-side device can be a cloud-side device with a cloud desktop system or cloud desktop software installed and deployed, such as a cloud server or cloud host. The endpoint device can be understood as any terminal that interacts with the cloud-side device. This terminal can be a laptop, desktop computer, tablet, smart device, server, etc.

[0298] This specification provides one or more embodiments of a speech translation method. Considering the inaccuracy of speech translation results output by neural network models, this method, during the training of the speech translation model, can determine a large amount of complex speech training data and multiple speech segment training data. The speech training data includes speech training samples and corresponding speech sample labels. Each speech segment training data includes speech segment training samples and corresponding speech segment sample labels, with the speech segment training samples extracted from the speech training samples. By training the model using the speech training data and multiple speech segment training data, a target speech translation model is obtained. This target speech translation model can translate complex speech data and output accurate speech translation results, thereby improving the accuracy of the speech translation results of the neural network model and avoiding the problem of inaccurate speech translation results due to the complexity of the speech data.

[0299] The above is an illustrative scheme of a speech translation method according to this embodiment. It should be noted that the technical solution of this speech translation method belongs to the same concept as the technical solution of the speech processing model training method described above. For details not described in detail in the technical solution of the speech translation method, please refer to the description of the technical solution of the speech processing model training method described above.

[0300] Corresponding to the above method embodiments, this specification also provides embodiments of a speech processing model training device. Figure 8 A schematic diagram of a speech processing model training device according to one embodiment of this specification is shown. Figure 8 As shown, the device includes:

[0301] The data determination module 802 is configured to determine speech training data and multiple speech segment training data, wherein the speech training data includes speech training samples and speech sample labels corresponding to the speech training samples, and each speech segment training data includes speech segment training samples and speech segment sample labels corresponding to the speech segment training samples, and the speech segment training samples are extracted from the speech training samples.

[0302] The first model training module 804 is configured to train the speech processing model based on the speech training data to obtain an initial speech processing model.

[0303] The second model training module 806 is configured to train the initial speech processing model based on the training data of the multiple speech segments to obtain the target speech processing model.

[0304] Optionally, the data determination module 802 is further configured to:

[0305] Determine the speech training data for the speech processing model, and perform speech extraction on the speech training data to obtain multiple candidate speech segment training data. Each candidate speech segment training data includes a candidate speech segment training sample and a candidate speech segment sample label corresponding to the candidate speech segment training sample, and the candidate speech segment training sample is extracted from the speech training sample.

[0306] Data filtering is performed on the training data of the multiple candidate speech segments to obtain the training data of the multiple speech segments.

[0307] Optionally, the data determination module 802 is further configured to:

[0308] Based on the attribute information of each candidate speech segment training sample, the training data of each candidate speech segment is filtered in the first stage to obtain the first stage training data.

[0309] Based on the correlation between the candidate speech segment training samples included in the first-stage training data and the corresponding candidate speech segment sample labels, the first-stage training data is subjected to a second-stage data filtering to obtain the multiple speech segment training data.

[0310] Optionally, the attribute information is the duration of a speech segment;

[0311] The data determination module 802 is further configured to:

[0312] From the candidate speech segment training samples included in each candidate speech segment training data, determine the target candidate speech segment training sample whose speech segment duration is greater than or equal to the first time threshold, and determine the candidate speech segment training data corresponding to the target candidate speech segment training sample as the target speech segment training data;

[0313] Using a semantic recognition model, semantic recognition is performed on the target candidate speech segment training samples in the target speech segment training data to obtain the semantic recognition result of the target speech segment training data, wherein the semantic recognition result is a semantically clear result or a semantically unclear result;

[0314] The target speech segment training data corresponding to the semantically clear result is determined as the first stage training data.

[0315] Optionally, the correlation is the similarity between the speech segment recognition text of the candidate speech segment training sample and the candidate speech segment recognition text, the speech sample label includes speech text, the speech segment sample label is speech segment text, and the speech segment text is extracted from the speech text.

[0316] Optionally, the data determination module 802 is further configured to:

[0317] From the candidate speech segment training samples included in the first stage training data, target candidate speech segment training samples with a speech segment duration less than or equal to a second time threshold are determined, and the first stage training data corresponding to the target candidate speech segment training samples is determined as the target first stage training data.

[0318] Using a speech recognition model, speech recognition is performed on the target candidate speech segment training samples in the first stage training data of the target to obtain the candidate speech segment recognition text of the target candidate speech segment training samples;

[0319] Determine the similarity between the recognized text of the speech segment and the candidate speech segment text, and determine a target similarity less than or equal to a preset similarity threshold from the similarity;

[0320] The first-stage training data corresponding to the target similarity is used as the training data for the multiple speech segments.

[0321] Optionally, the data determination module 802 is further configured to:

[0322] Determine the speech segmentation time threshold for the speech training samples;

[0323] Based on the speech segmentation time threshold, the speech training data is segmented to obtain multiple candidate speech segment training data.

[0324] Optionally, the data determination module 802 is further configured to:

[0325] The number of speech segments corresponding to the speech training sample is calculated based on the speech length of the speech training sample and the speech segmentation time threshold.

[0326] Based on the number of speech segments and the speech segmentation time threshold, multiple speech segmentation ranges are calculated, wherein the number of multiple speech segmentation ranges is the same as the number of speech segments, each speech segment segmentation range is different, and the speech segmentation ranges overlap.

[0327] Based on the segmentation range of the multiple speech segments, the speech training samples and the speech sample labels corresponding to the speech training samples are segmented to obtain the multiple speech segment training samples and the speech segment sample labels corresponding to each speech segment training sample.

[0328] The training samples of the multiple speech segments, and the speech segment sample labels corresponding to each training sample of the speech segments, are used as the training data of the multiple candidate speech segments.

[0329] Optionally, the speech sample labels include speech translations, and the speech processing model is a speech translation model;

[0330] The first model training module 804 is also configured as follows:

[0331] The speech training samples are input into the speech translation model to perform speech translation, thereby obtaining the predicted speech translation and predicted speech text of the speech training samples;

[0332] Based on the predicted speech translation, the speech translation, the predicted speech text, and the speech text, a first loss function is determined, and the model parameters of the speech translation model are adjusted based on the first loss function to obtain an initial speech translation model.

[0333] Optionally, the speech sample labels are speech translations and speech texts, the speech segment sample labels are speech segment translations and speech segment texts, the speech segment translations are extracted from the speech translations, the speech segment texts are extracted from the speech texts, and the speech processing model is a speech translation model.

[0334] The second model training module 806 is also configured as follows:

[0335] The target speech segment training sample in the target speech segment training data is input into the initial speech translation model for speech translation to obtain the predicted speech segment translation and predicted speech segment text of the target speech segment training sample, wherein the target speech segment training data is any one of the plurality of speech segment training data;

[0336] A second loss function is determined based on the speech segment translation and speech segment text in the target speech segment training data, as well as the predicted speech segment translation and predicted speech segment text.

[0337] Based on the second loss function, the model parameters of the initial speech translation model are adjusted to obtain the target speech translation model.

[0338] Optionally, the second model training module 806 is further configured to:

[0339] Input the target speech segment training samples from the target speech segment training data into the initial speech translation model;

[0340] The initial speech translation model is used to translate the training sample of the target speech segment to obtain the predicted speech segment text of the training sample of the target speech segment, as well as the translation probability parameters and the translation termination probability parameters corresponding to the target translation of the training sample of the target speech segment.

[0341] The translation recognition model is used to perform translation recognition processing on the translation probability parameters and the translation termination probability parameters to obtain translation recognition results. Based on the translation recognition results, the predicted speech segment translation of the target speech segment training sample is determined.

[0342] Optionally, the second model training module 806 is further configured to:

[0343] In the initial speech translation model, a speech recognition network layer is used to perform text recognition on the training samples of the target speech segment to obtain the predicted speech segment text;

[0344] By using a text processing network layer, the predicted speech segment text is converted into text to obtain the speech segment text features corresponding to the predicted speech segment text;

[0345] Using a speech translation network layer, speech translation is performed based on the text features of the speech segment to obtain the translation probability parameters and the translation termination probability parameters corresponding to the target translation of the training sample of the target speech segment.

[0346] Optionally, the second model training module 806 is further configured to:

[0347] Determine the ratio of the length of the translated speech to the length of the translated speech segment of the target speech segment, and determine the ratio of the duration of the speech segment of the training sample of the target speech segment to the length of the speech segment among the training samples.

[0348] The translation probability parameter, the translation end probability parameter, the translation length ratio, and the speech length ratio are input into the translation recognition model to obtain the translation recognition result, wherein the translation recognition result is the translation execution result or the translation end result;

[0349] If the translation recognition result is the translation execution result, the speech translation network layer is used to continue to perform speech translation based on the text features of the speech segment, and to obtain the translation probability parameters and the translation termination probability parameters corresponding to the target translation of the target speech segment training sample, until the translation recognition result is the translation termination result.

[0350] If the translation recognition result is the translation completion result, the target translation is used as the predicted speech segment translation.

[0351] This specification provides one or more embodiments of a speech processing model training device. Considering the inaccuracy of speech processing results output by neural network models, this method, during the training of the speech processing model, can determine a large amount of complex speech training data and multiple speech segment training data. The speech training data includes speech training samples and corresponding speech sample labels. Each speech segment training data includes speech segment training samples and corresponding speech segment sample labels, with the speech segment training samples extracted from the speech training samples. By training the model using the speech training data and multiple speech segment training data, a target speech processing model capable of processing complex speech data and outputting accurate speech processing results is obtained. This improves the accuracy of the speech processing results from the neural network model and avoids the problem of inaccurate speech processing results due to the complexity of the speech data.

[0352] The above is an illustrative scheme of a speech processing model training device according to this embodiment. It should be noted that the technical solution of this speech processing model training device and the technical solution of the speech processing model training method described above belong to the same concept. For details not described in detail in the technical solution of the speech processing model training device, please refer to the description of the technical solution of the speech processing model training method described above.

[0353] Corresponding to the above method embodiments, this specification also provides embodiments of a speech translation model training device. Figure 9 A schematic diagram of a speech translation model training device according to one embodiment of this specification is shown. Figure 9 As shown, the device includes:

[0354] The data determination module 902 is configured to determine speech training data and multiple speech segment training data, wherein the speech training data includes speech training samples and speech sample labels corresponding to the speech training samples, and each speech segment training data includes speech segment training samples and speech segment sample labels corresponding to the speech segment training samples, and the speech segment training samples are extracted from the speech training samples.

[0355] The first model training module 904 is configured to train the speech translation model based on the speech training data to obtain an initial speech translation model.

[0356] The second model training module 906 is configured to train the initial speech translation model based on the training data of the multiple speech segments to obtain the target speech translation model.

[0357] This specification provides one or more embodiments of a speech translation model training device. Considering the inaccuracy of speech translation results output by neural network models, this method, during the training of the speech translation model, can determine a large amount of complex speech training data and multiple speech segment training data. The speech training data includes speech training samples and corresponding speech sample labels. Each speech segment training data includes speech segment training samples and corresponding speech segment sample labels, with the speech segment training samples extracted from the speech training samples. By training the model using the speech training data and multiple speech segment training data, a target speech translation model capable of translating complex speech data and outputting accurate speech translation results is obtained. This improves the accuracy of the speech translation results of the neural network model and avoids the problem of inaccurate speech translation results due to the complexity of the speech data.

[0358] The above is a schematic scheme of a speech translation model training device according to this embodiment. It should be noted that the technical solution of this speech translation model training device and the technical solution of the speech translation model training method described above belong to the same concept. For details not described in detail in the technical solution of the speech translation model training device, please refer to the description of the technical solution of the speech translation model training method described above.

[0359] Corresponding to the above method embodiments, this specification also provides embodiments of a voice processing device. Figure 10 A schematic diagram of the structure of a voice processing device according to one embodiment of this specification is shown. Figure 10 As shown, the device includes:

[0360] The speech determination module 1002 determines the speech to be translated, wherein the speech to be translated is real-time determined speech data or speech segment data;

[0361] The speech processing module 1004 is configured to input the speech to be translated into a target speech processing model to obtain the speech translation result corresponding to the speech to be translated. The target speech processing model is obtained by training an initial speech processing model based on training data of multiple speech segments. The initial speech processing model is obtained by training a speech processing model based on speech training data.

[0362] Optionally, the voice processing module 1004 is further configured to:

[0363] The speech to be translated is input into a target speech processing model, wherein the target speech processing model includes a speech recognition network layer, a speech translation network layer, and a text processing network layer;

[0364] The speech recognition network layer is used to perform text recognition on the speech to be translated to obtain the predicted speech text;

[0365] The predicted speech text is converted into text using a text processing network layer to obtain the speech text features corresponding to the predicted speech text.

[0366] Using a speech translation network layer, speech translation is performed based on the speech-text features to obtain the speech translation result corresponding to the speech to be translated.

[0367] Optionally, the voice processing module 1004 is further configured to:

[0368] Using the speech translation network layer, speech translation is performed based on the speech-text features to obtain the translation probability parameters and the translation termination probability parameters corresponding to the target translation of the speech to be translated.

[0369] The translation probability parameter and the translation termination probability parameter are input into the translation recognition model to obtain the translation recognition result, wherein the translation recognition result is the translation execution result or the translation termination result;

[0370] If the translation recognition result is the translation execution result, the speech translation network layer is used to continue to perform speech translation based on the speech text features, and to obtain the translation probability parameters and the translation termination probability parameters corresponding to the target translation of the speech to be translated, until the translation recognition result is the translation termination result;

[0371] If the translation recognition result is the translation completion result, the target translation is taken as the speech translation result corresponding to the speech to be translated.

[0372] Optionally, the voice determination module 1002 is further configured to:

[0373] Receive speech to be translated sent by the client, wherein the speech to be translated is speech data or speech segment data collected by the client in real time;

[0374] The voice processing device further includes a result sending module, configured as follows:

[0375] The voice translation result is sent to the client for display.

[0376] This specification provides one or more embodiments of a speech processing device. Considering the inaccuracy of speech processing results output by neural network models, this method, during the training of the speech processing model, can determine a large number of complex speech training data and multiple speech segment training data. The model is trained using these speech training data and multiple speech segment training data to obtain a target speech processing model capable of processing complex speech data and outputting accurate speech processing results. The target speech processing model is then used to perform speech translation on the speech to be translated, obtaining accurate speech translation results, improving the accuracy of the speech processing results of the neural network model, and avoiding the problem of inaccurate speech translation results due to the complexity of the speech data.

[0377] The above is an illustrative scheme of a voice processing device according to this embodiment. It should be noted that the technical solution of this voice processing device and the technical solution of the above-described voice processing method belong to the same concept. For details not described in detail in the technical solution of the voice processing device, please refer to the description of the technical solution of the above-described voice processing method.

[0378] Corresponding to the above method embodiments, this specification also provides embodiments of a speech translation device. Figure 11 A schematic diagram of a speech translation device according to one embodiment of this specification is shown. Figure 11 As shown, this device is applied to cloud-side equipment and includes:

[0379] The voice receiving module 1102 is configured to receive the voice to be translated sent by the end-side device;

[0380] The speech translation module 1104 is configured to input the speech to be translated into a target speech translation model to obtain the speech translation corresponding to the speech to be translated. The target speech translation model is obtained by training an initial speech translation model based on training data of multiple speech segments. The initial speech translation model is obtained by training a speech translation model based on speech training data. The speech training data includes speech training samples and speech sample labels corresponding to the speech training samples. Each speech segment training data includes speech segment training samples and speech segment sample labels corresponding to the speech segment training samples. The speech segment training samples are extracted from the speech training samples.

[0381] The voice translation sending module 1106 is configured to send the voice translation to the end-side device.

[0382] This specification provides one or more embodiments of a speech translation device. Considering the inaccuracy of speech translation results output by neural network models, this method, during the training of the speech translation model, can determine a large amount of complex speech training data and multiple speech segment training data. The speech training data includes speech training samples and corresponding speech sample labels. Each speech segment training data includes speech segment training samples and corresponding speech segment sample labels, with the speech segment training samples extracted from the speech training samples. By training the model using the speech training data and multiple speech segment training data, a target speech translation model is obtained. This target speech translation model can translate complex speech data and output accurate speech translation results, thereby improving the accuracy of the speech translation results of the neural network model and avoiding the problem of inaccurate speech translation results due to the complexity of the speech data.

[0383] The above is an illustrative scheme of a voice translation device according to this embodiment. It should be noted that the technical solution of this voice translation device and the technical solution of the above-described voice translation method belong to the same concept. For details not described in detail in the technical solution of the voice translation device, please refer to the description of the technical solution of the above-described voice translation method.

[0384] Figure 12 A structural block diagram of a computing device 1200 provided according to one embodiment of this specification is shown.

[0385] The computing device 1200 includes:

[0386] Memory 1210 and processor 1220;

[0387] The memory 1210 is used to store computer programs / instructions, and the processor 1220 is used to execute the computer programs / instructions, which, when executed by the processor 1220, implement the steps of any of the above methods.

[0388] In one or more embodiments of this specification, the computing device can be understood as an integrated smart terminal, including but not limited to a server, desktop computer, PC (Personal Computer), all-in-one model machine, mobile phone, tablet computer or other portable smart terminal, etc., and the computing device may have the model described in the above embodiments of this application pre-installed.

[0389] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other types of models), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API (Application Programming Interface) calling capabilities. Models can be called into created applications through the API interface, and application management tools are provided for application management and monitoring.

[0390] Furthermore, the computing device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI (Artificial Intelligence) technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0391] Figure 13 A structural block diagram of an electronic device 1300 provided according to one embodiment of this specification is shown.

[0392] The memory 1310 and the processor 1320 are connected via a bus 1330;

[0393] The memory 1310 is used to store computer programs / instructions, and the processor 1320 is used to execute the computer programs / instructions, which, when executed by the processor 1320, implement the steps of the method.

[0394] Specifically, the components of the electronic device 1300 include, but are not limited to, a memory 1310 and a processor 1320. The processor 1320 is connected to the memory 1310 via a bus 1330, and the database 1350 is used to store data.

[0395] Electronic device 1300 also includes access device 1340, which enables electronic device 1300 to communicate via one or more networks 1360. Examples of such networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. Access device 1340 may include one or more of any type of wired or wireless network interface (e.g., network interface card (NIC)), such as IEEE 802.11 Wireless Local Area Network (WLAN) interface, Wi-MAX (Worldwide Interoperability for Microwave Access) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, Near Field Communication (NFC) interface, and so on.

[0396] In one embodiment of this specification, the above-described components of the electronic device 1300 and Figure 13 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 13 The block diagram of the electronic device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0397] Electronic device 1300 can be any type of stationary or mobile electronic device, including mobile computers or mobile electronic devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable electronic devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary electronic devices such as desktop computers or personal computers (PCs). Electronic device 1300 can also be a mobile or stationary server.

[0398] The above is an illustrative scheme of an electronic device according to this embodiment. It should be noted that the technical solution of this electronic device belongs to the same concept as the technical solution of any of the above methods, and any details not described in detail in the technical solution of the electronic device can be referred to the description of the technical solution of any of the methods.

[0399] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to any of the above-described method embodiments, so the description is relatively simple; relevant parts can be referred to in the description of any of the above-described method embodiments.

[0400] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0401] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to any of the above-described method embodiments, so the description is relatively simple; relevant parts can be referred to in the description of any of the above-described method embodiments.

[0402] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0403] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solution of any of the above methods, and any details not described in detail in the technical solution of the computer program product can be referred to the description of the technical solution of any of the above methods.

[0404] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0405] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0406] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0407] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0408] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A speech processing method, comprising: The speech to be translated is determined, wherein the speech to be translated is real-time determined speech data or speech segment data; The speech to be translated is input into the target speech processing model to obtain the speech translation result corresponding to the speech to be translated. The target speech processing model is obtained by training an initial speech processing model based on training data of multiple speech segments. The initial speech processing model is obtained by training a speech processing model based on speech training data.

2. The speech processing method according to claim 1, wherein inputting the speech to be translated into a target speech processing model to obtain the speech translation result corresponding to the speech to be translated includes: The speech to be translated is input into a target speech processing model, wherein the target speech processing model includes a speech recognition network layer, a speech translation network layer, and a text processing network layer; The speech recognition network layer is used to perform text recognition on the speech to be translated to obtain the predicted speech text; The predicted speech text is converted into text using a text processing network layer to obtain the speech text features corresponding to the predicted speech text. Using a speech translation network layer, speech translation is performed based on the speech-text features to obtain the speech translation result corresponding to the speech to be translated.

3. The speech processing method according to claim 2, wherein the step of using a speech translation network layer to perform speech translation based on the speech-text features to obtain a speech translation result corresponding to the speech to be translated includes: Using the speech translation network layer, speech translation is performed based on the speech-text features to obtain the translation probability parameters and the translation termination probability parameters corresponding to the target translation of the speech to be translated. The translation probability parameter and the translation termination probability parameter are input into the translation recognition model to obtain the translation recognition result, wherein the translation recognition result is the translation execution result or the translation termination result; If the translation recognition result is the translation execution result, the speech translation network layer is used to continue to perform speech translation based on the speech text features, and to obtain the translation probability parameters and the translation termination probability parameters corresponding to the target translation of the speech to be translated, until the translation recognition result is the translation termination result; If the translation recognition result is the translation completion result, the target translation is taken as the speech translation result corresponding to the speech to be translated.

4. The speech processing method according to any one of claims 1 to 3, wherein determining the speech to be translated includes: Receive speech to be translated sent by the client, wherein the speech to be translated is speech data or speech segment data collected by the client in real time; The step of inputting the speech to be translated into the target speech processing model to obtain the speech translation result corresponding to the speech to be translated includes: The voice translation result is sent to the client for display.

5. A method for training a speech processing model, comprising: Determine speech training data and multiple speech segment training data, wherein the speech training data includes speech training samples and speech sample labels corresponding to the speech training samples, and each speech segment training data includes speech segment training samples and speech segment sample labels corresponding to the speech segment training samples, and the speech segment training samples are extracted from the speech training samples. The speech processing model is trained based on the speech training data to obtain an initial speech processing model. The initial speech processing model is trained using the training data of the multiple speech segments to obtain the target speech processing model.

6. The speech processing model training method according to claim 5, wherein determining the speech training data and multiple speech segment training data includes: Determine the speech training data for the speech processing model, and perform speech extraction on the speech training data to obtain multiple candidate speech segment training data. Each candidate speech segment training data includes a candidate speech segment training sample and a candidate speech segment sample label corresponding to the candidate speech segment training sample, and the candidate speech segment training sample is extracted from the speech training sample. The training data of the multiple candidate speech segments is filtered to obtain the training data of the multiple speech segments.

7. The speech processing model training method according to claim 6, wherein the step of filtering the training data of the plurality of candidate speech segments to obtain the training data of the speech segments includes: Based on the attribute information of each candidate speech segment training sample, the training data of each candidate speech segment is filtered in the first stage to obtain the first stage training data. Based on the correlation between the candidate speech segment training samples included in the first-stage training data and the corresponding candidate speech segment sample labels, the first-stage training data is subjected to a second-stage data filtering to obtain the multiple speech segment training data.

8. The speech processing model training method according to claim 7, wherein the attribute information is the duration of a speech segment; The first stage of data filtering, based on the attribute information of each candidate speech segment training sample, to obtain the first stage training data includes: From the candidate speech segment training samples included in each candidate speech segment training data, determine the target candidate speech segment training sample whose speech segment duration is greater than or equal to the first time threshold, and determine the candidate speech segment training data corresponding to the target candidate speech segment training sample as the target speech segment training data; Using a semantic recognition model, semantic recognition is performed on the target candidate speech segment training samples in the target speech segment training data to obtain the semantic recognition result of the target speech segment training data, wherein the semantic recognition result is a semantically clear result or a semantically unclear result; The target speech segment training data corresponding to the semantically clear result is determined as the first stage training data.

9. The speech processing model training method according to claim 7, wherein the correlation degree is the similarity between the speech segment recognition text of the candidate speech segment training sample and the candidate speech segment recognition text, the speech sample label includes speech text, the speech segment sample label is speech segment text, and the speech segment text is extracted from the speech text.

10. The speech processing model training method according to claim 9, wherein the step of performing a second-stage data filtering on the first-stage training data based on the correlation between the candidate speech segment training samples included in the first-stage training data and the corresponding candidate speech segment sample labels to obtain the plurality of speech segment training data includes: From the candidate speech segment training samples included in the first stage training data, target candidate speech segment training samples with a speech segment duration less than or equal to a second time threshold are determined, and the first stage training data corresponding to the target candidate speech segment training samples is determined as the target first stage training data. Using a speech recognition model, speech recognition is performed on the target candidate speech segment training samples in the first stage training data of the target to obtain the candidate speech segment recognition text of the target candidate speech segment training samples; Determine the similarity between the recognized text of the speech segment and the candidate speech segment text, and determine a target similarity less than or equal to a preset similarity threshold from the similarity; The first-stage training data corresponding to the target similarity is used as the training data for the multiple speech segments.

11. The speech processing model training method according to any one of claims 6 to 10, wherein the step of extracting speech from the speech training data to obtain multiple candidate speech segment training data includes: Determine the speech segmentation time threshold for the speech training samples; Based on the speech segmentation time threshold, the speech training data is segmented to obtain multiple candidate speech segment training data.

12. The speech processing model training method according to claim 11, wherein the step of performing speech segmentation on the speech training data according to the speech segmentation time threshold to obtain multiple candidate speech segment training data includes: The number of speech segments corresponding to the speech training sample is calculated based on the speech length of the speech training sample and the speech segmentation time threshold. Based on the number of speech segments and the speech segmentation time threshold, multiple speech segmentation ranges are calculated, wherein the number of multiple speech segmentation ranges is the same as the number of speech segments, each speech segment segmentation range is different, and the speech segmentation ranges overlap. Based on the segmentation range of the multiple speech segments, the speech training samples and the speech sample labels corresponding to the speech training samples are segmented to obtain the multiple speech segment training samples and the speech segment sample labels corresponding to each speech segment training sample. The training samples of the multiple speech segments, and the speech segment sample labels corresponding to each training sample of the speech segments, are used as the training data of the multiple candidate speech segments.

13. The speech processing model training method according to any one of claims 5 to 10, wherein the speech sample labels include speech translation and speech text, and the speech processing model is a speech translation model; The step of training the speech processing model based on the speech training data to obtain an initial speech processing model includes: The speech training samples are input into the speech translation model to perform speech translation, thereby obtaining the predicted speech translation and predicted speech text of the speech training samples; Based on the predicted speech translation, the speech translation, the predicted speech text, and the speech text, a first loss function is determined, and the model parameters of the speech translation model are adjusted based on the first loss function to obtain an initial speech translation model.

14. The speech processing model training method according to any one of claims 5 to 10, wherein the speech sample labels are speech translation and speech text, the speech segment sample labels are speech segment translation and speech segment text, the speech segment translation is extracted from the speech translation, the speech segment text is extracted from the speech text, and the speech processing model is a speech translation model.

15. The speech processing model training method according to claim 14, wherein training the initial speech processing model based on the plurality of speech segment training data to obtain a target speech processing model includes: The target speech segment training sample in the target speech segment training data is input into the initial speech translation model for speech translation to obtain the predicted speech segment translation and predicted speech segment text of the target speech segment training sample, wherein the target speech segment training data is any one of the plurality of speech segment training data; A second loss function is determined based on the speech segment translation and speech segment text in the target speech segment training data, as well as the predicted speech segment translation and predicted speech segment text. Based on the second loss function, the model parameters of the initial speech translation model are adjusted to obtain the target speech translation model.

16. The speech processing model training method according to claim 15, wherein inputting the target speech segment training sample from the target speech segment training data into the initial speech translation model for speech translation to obtain the predicted speech segment translation and predicted speech segment text of the target speech segment training sample, comprises: Input the target speech segment training samples from the target speech segment training data into the initial speech translation model; The initial speech translation model is used to translate the training sample of the target speech segment to obtain the predicted speech segment text of the training sample of the target speech segment, as well as the translation probability parameters and the translation termination probability parameters corresponding to the target translation of the training sample of the target speech segment. The translation recognition model is used to perform translation recognition processing on the translation probability parameters and the translation termination probability parameters to obtain translation recognition results. Based on the translation recognition results, the predicted speech segment translation of the target speech segment training sample is determined.

17. The speech processing model training method according to claim 16, wherein the step of using the initial speech translation model to perform speech translation on the target speech segment training sample to obtain the predicted speech segment text of the target speech segment training sample, and the translation probability parameters and translation termination probability parameters corresponding to the target translation of the target speech segment training sample, includes: In the initial speech translation model, a speech recognition network layer is used to perform text recognition on the training samples of the target speech segment to obtain the predicted speech segment text; By using a text processing network layer, the predicted speech segment text is converted into text to obtain the speech segment text features corresponding to the predicted speech segment text; Using a speech translation network layer, speech translation is performed based on the text features of the speech segment to obtain the translation probability parameters and the translation termination probability parameters corresponding to the target translation of the training sample of the target speech segment.

18. The speech processing model training method according to claim 17, wherein the step of using a translation recognition model to perform translation recognition processing on the translation probability parameter and the end-of-translation probability parameter to obtain a translation recognition result, and determining the predicted speech segment translation of the target speech segment training sample based on the translation recognition result, comprises: Determine the ratio of the length of the translated speech to the length of the translated speech segment of the target speech segment, and determine the ratio of the duration of the speech segment of the training sample of the target speech segment to the length of the speech segment among the training samples. The translation probability parameter, the translation end probability parameter, the translation length ratio, and the speech length ratio are input into the translation recognition model to obtain the translation recognition result, wherein the translation recognition result is the translation execution result or the translation end result; If the translation recognition result is the translation execution result, the speech translation network layer is used to continue to perform speech translation based on the text features of the speech segment, and to obtain the translation probability parameters and the translation termination probability parameters corresponding to the target translation of the target speech segment training sample, until the translation recognition result is the translation termination result. If the translation recognition result is the translation completion result, the target translation is used as the predicted speech segment translation.

19. A method for training a speech translation model, comprising: Determine speech training data and multiple speech segment training data, wherein the speech training data includes speech training samples and speech sample labels corresponding to the speech training samples, and each speech segment training data includes speech segment training samples and speech segment sample labels corresponding to the speech segment training samples, and the speech segment training samples are extracted from the speech training samples. The speech translation model is trained based on the speech training data to obtain an initial speech translation model. The initial speech translation model is trained using the training data from the multiple speech segments to obtain the target speech translation model.

20. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 19.

21. An electronic device, comprising: A memory and a processor, the memory and the processor being connected via a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 19.

22. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 19.

23. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 19.

Images