Cross-language speech recognition method, system and device and storage medium

Abstract

The invention discloses a cross-language speech recognition method, system and device and a storage medium, and the method comprises the steps: carrying out the preprocessing of training speech, and obtaining a training speech frame sequence; extracting content characterization, speaker characterization and pitch characterization from the training voice frame sequence; performing voice reconstruction according to the content representation, the speaker representation and the pitch representation to obtain a target language voice; based on the training voice, constructing a target language voice recognition model according to the target language voice; in response to the target language recognition instruction, obtaining a target voice; and inputting the target voice into the target language voice recognition model to obtain a recognition result output by the target language voice recognition model, so that accurate modeling of dialect acoustic characteristics can be realized under the condition of extremely low annotation data through cross-language feature decoupling and a self-supervised migration mechanism. The recognition robustness of the complex tones and the characteristic vocabularies of the Guiwili is remarkably improved, and efficient generalization application in a dialect scene is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a cross-lingual speech recognition method, system, device and storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, speech recognition technology has been widely applied in intelligent assistants, education, medical treatment and other fields, and has made remarkable progress in general languages such as English and Mandarin and has been widely applied to various intelligent devices and services. The existing speech recognition system is based on a deep neural network model, and its performance is highly dependent on large-scale, high-quality, accurately labeled speech-text paired datasets. For resource-rich general languages, by widely collecting public speech resources (such as news broadcasts, audio books, social media audio) and supplementing standardized manual labeling processes, a data foundation sufficient to support high-performance model training can be constructed.

[0003] However, in the speech recognition of low-resource dialects (such as Guilin-Liuzhou dialect), the existing technology faces multiple challenges. The systematic shortage of dialect data results in a lack of high-quality, structured corpus basis for model training: unstructured data sources (such as local radio, short videos) generally have noise interference, dialects mixed with Mandarin, and other problems, and the cost of data collection and copyright compliance review is high. Secondly, the high difficulty of dialect annotation further exacerbates the technical bottleneck: the six-tone system, characteristic vocabulary (such as "no" and "bleaching") and regional accent differences of Guilin-Liuzhou dialect require annotators to have dialect native ability and professional linguistic knowledge, and manual annotation is inefficient and costly. In addition, the existing model lacks the ability to model complex acoustic characteristics (such as subtle changes in secondary entry sound), making it difficult to accurately capture the phonological rules and semantic associations specific to dialects, resulting in recognition accuracy that is difficult to meet the needs of practical applications. SUMMARY

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] The main purpose of the embodiments of the present disclosure is to propose a cross-lingual speech recognition method, system, device and storage medium, which can realize accurate modeling of dialect acoustic characteristics under extremely low labeled data conditions through cross-lingual feature decoupling and self-supervised migration mechanism, significantly improve the recognition robustness of complex tones and characteristic vocabulary of Guilin-Liuzhou dialect, and realize efficient generalization application in dialect scenarios.

[0006] The first aspect of the embodiments of the present application provides a cross-lingual speech recognition method, for a central controller, the method comprising:

[0007] The training speech is preprocessed to obtain a training speech frame sequence;

[0008] extracting a content representation, a speaker representation and a pitch representation from the training speech frame sequence;

[0009] performing speech reconstruction according to the content representation, the speaker representation and the pitch representation to obtain target language speech;

[0010] constructing a target language speech recognition model according to the target language speech based on the training speech;

[0011] obtaining target speech in response to a target language recognition instruction;

[0012] inputting the target speech into the target language speech recognition model to obtain a recognition result output by the target language speech recognition model.

[0013] In some embodiments of the present application, the extracting a content representation, a speaker representation and a pitch representation from the training speech frame sequence comprises:

[0014] extracting prosodic features and spectral features from the training speech frame sequence;

[0015] constructing joint features of the training speech frame sequence according to the prosodic features and the spectral features;

[0016] extracting the content representation, the speaker representation and the pitch representation from the joint features.

[0017] In some embodiments of the present application, the extracting the content representation, the speaker representation and the pitch representation from the joint features comprises:

[0018] converting the joint features into a dense feature sequence;

[0019] generating the content representation according to the dense feature sequence;

[0020] extracting the speaker representation from the joint features;

[0021] extracting the pitch representation from the joint features.

[0022] In some embodiments of the present application, the performing speech reconstruction according to the content representation, the speaker representation and the pitch representation to obtain target language speech comprises:

[0023] reconstructing the content representation, the speaker representation and the pitch representation to obtain reconstructed features;

[0024] constructing a loss function among the content representation, the speaker representation and the pitch representation;

[0025] optimizing the reconstructed feature according to the loss function to obtain an optimized reconstructed feature;

[0026] decoding the reconstructed feature to obtain the target language speech.

[0027] In some embodiments of the present application, the target language speech recognition model is constructed according to the target language speech based on the training speech, including:

[0028] constructing a training language recognition model according to the training speech;

[0029] presetting an initial target language speech recognition model;

[0030] obtaining a target language training speech;

[0031] comparing and learning the training language recognition model and the initial target language speech recognition model based on the target language training speech to obtain a training target language speech recognition model;

[0032] optimizing the training target language speech recognition model according to the target language speech to obtain a target language speech recognition model.

[0033] In some embodiments of the present application, the training speech is preprocessed to obtain a training speech frame sequence, including:

[0034] pre-emphasizing the training speech to obtain a first speech;

[0035] segmenting the first speech frame by frame to obtain an overlapping frame sequence;

[0036] extracting the training speech frame sequence from the overlapping frame sequence.

[0037] In some embodiments of the present application, the training target language speech recognition model is optimized according to the target language speech to obtain a target language speech recognition model, including:

[0038] constructing an optimization model; wherein the optimization model includes an encoder and a decoder; the encoder is the training target language speech recognition model; and the decoder is a preset deep learning network;

[0039] inputting the target language speech into the encoder to extract a corresponding acoustic feature sequence;

[0040] based on the acoustic feature sequence, the optimization model is optimized through a two-stage parameter regulation strategy to obtain the target language speech recognition model; wherein the two-stage parameter regulation strategy is to divide the optimization process of the optimization model into a freezing stage and a joint stage for optimization; the freezing stage is to keep the encoder parameters unchanged and only optimize the decoder; the joint stage is to optimize the encoder and the decoder synchronously.

[0041] To achieve the above-mentioned purpose, a second aspect of the embodiment of the present application provides a cross-language speech recognition system, the system comprises:

[0042] The processing module is configured to preprocess the training speech to obtain a training speech frame sequence.

[0043] The extraction module is configured to extract content representation, speaker representation and pitch representation from the training speech frame sequence.

[0044] The reconstruction module is configured to reconstruct speech according to the content representation, the speaker representation and the pitch representation to obtain a target language speech.

[0045] The construction module is configured to construct a target language speech recognition model based on the training speech according to the target language speech.

[0046] The response module is configured to obtain a target speech in response to a target language recognition instruction.

[0047] The recognition module is configured to input the target speech into the target language speech recognition model to obtain a recognition result output by the target language speech recognition model.

[0048] To achieve the above-mentioned purpose, a third aspect of the embodiment of the present application provides an electronic device, comprising: at least one control processor and a memory connected in communication with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform the above-mentioned cross-language speech recognition method.

[0049] To achieve the above-mentioned purpose, a fourth aspect of the embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores computer executable instructions, and the computer executable instructions are used to make the computer execute the above-mentioned cross-language speech recognition method.

[0050] The embodiment of the present application provides a cross-language speech recognition method, which comprises the following steps: preprocessing training speech to obtain a training speech frame sequence; extracting content representation, speaker representation and pitch representation from the training speech frame sequence; reconstructing speech according to the content representation, the speaker representation and the pitch representation to obtain target language speech; constructing a target language speech recognition model based on the training speech according to the target language speech; obtaining target speech in response to a target language recognition instruction; inputting the target speech into the target language speech recognition model to obtain a recognition result output by the target language speech recognition model, which can realize accurate modeling of dialect acoustic characteristics under extremely low labeled data conditions through cross-language feature decoupling and self-supervised migration mechanism, significantly improve the recognition robustness of complex tones and characteristic words of Guilin and Liuzhou dialects, and realize efficient generalization application in a dialect scene.

[0051] It can be understood that the beneficial effects of the second aspect to the fourth aspect and the related technology compared with the beneficial effects of the first aspect and the related technology compared with the first aspect are the same. Please refer to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0052] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

[0053] Figure 1 is a flowchart of a cross-language speech recognition method provided by the embodiment of the present application;

[0054] Figure 2 is a schematic diagram of an embodiment of a cross-language speech recognition method provided by the embodiment of the present application;

[0055] Figure 3 is another schematic diagram of an embodiment of a cross-language speech recognition method provided by the embodiment of the present application;

[0056] Figure 4 is a structural schematic diagram of a cross-language speech recognition training system provided by the embodiment of the present application;

[0057] Figure 5 is a hardware structural schematic diagram of an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0058] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application.

[0059] In the description of the present application, if the first, second, etc. are described, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features or the order of the indicated technical features.

[0060] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, etc. is based on the orientation or position relationship shown in the drawings, only for the purpose of facilitating the description of the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the indicated device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present application.

[0061] In the description of the present application, it should be noted that, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0062] With the rapid development of artificial intelligence technology, speech recognition technology has been widely applied in intelligent assistants, education, medical treatment and many other fields, especially in general languages such as English and Mandarin, which has made significant progress and is widely used in various intelligent devices and services. The existing speech recognition system is based on a deep neural network model, and its performance is highly dependent on large-scale, high-quality, accurately labeled speech-text paired datasets. For resource-rich general languages, by widely collecting public speech resources (such as news broadcasts, audio books, social media audio) and supplemented by standardized manual labeling processes, a data foundation sufficient to support high-performance model training can be constructed.

[0063] However, in the speech recognition of low-resource dialects (such as Guilin-Liuzhou dialect), the existing technology faces multiple challenges. The systematic shortage of dialect data leads to a lack of high-quality, structured corpus basis for model training: unstructured data sources (such as local radio, short videos) are generally noisy, mixed with dialects and Mandarin, and the cost of data collection and copyright compliance review is high. Secondly, the high difficulty of dialect annotation further exacerbates the technical bottleneck: the six-tone system, characteristic vocabulary (such as “no” “bleaching”) and regional accent differences of Guilin-Liuzhou dialect require annotators to have dialect native ability and professional linguistic knowledge, and manual annotation is inefficient and costly. In addition, the existing model lacks the ability to model complex acoustic features (such as subtle changes in secondary entry sound), making it difficult to accurately capture the phonological rules and semantic associations unique to dialects, resulting in recognition accuracy that cannot meet the needs of practical applications.

[0064] Based on this, the embodiment of the application provides a cross-language speech recognition method, system, electronic device and storage medium, aiming to realize accurate modeling of dialect acoustic characteristics under extremely low labeled data conditions through cross-language feature decoupling and self-supervised migration mechanism, significantly improve the recognition robustness of complex tones and characteristic words of Guilin and Liuzhou dialects, and realize efficient generalization application in the dialect scene.

[0065] The cross-language speech recognition method, system, electronic device and storage medium provided by the embodiment of the application are specifically described through the following embodiments. First, the cross-language speech recognition method in the embodiment of the application is described.

[0066] The embodiment of the application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0067] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0068] The cross-language speech recognition method provided by the embodiment of the application relates to the field of artificial intelligence technology. The cross-language speech recognition method provided by the embodiment of the application can be applied in a terminal, can also be applied in a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms; the software can be an application that implements the cross-language speech recognition method, etc., but is not limited to the above forms.

[0069] The application is operable in a multitude of generic or specific computer system environments or configurations. Examples of well known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0070] It should be noted that in each specific embodiment of the present application, when it is necessary to perform relevant processing according to user information, user behavior data, user history data, and user location information, and other data related to the identity or characteristics of the user, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to function normally will be obtained.

[0071] To this end, with reference to Figure 1 The embodiments of the present application provide a cross-language speech recognition method, which is applied to a central controller. The controller can be a server, an electronic device, a mobile terminal, or the like, which is not specifically limited here. The method comprises the following steps S110 to S160.

[0072] Step S110: Preprocessing training speech to obtain a training speech frame sequence.

[0073] Step S120: Extracting content representation, speaker representation, and pitch representation from the training speech frame sequence.

[0074] Step S130: Reconstructing speech according to the content representation, the speaker representation, and the pitch representation to obtain target language speech.

[0075] Step S140: Constructing a target language speech recognition model according to the target language speech based on the training speech.

[0076] Step S150, in response to the target language recognition instruction, obtaining the target voice;

[0077] Step S160, inputting the target voice into the target language voice recognition model to obtain the recognition result output by the target language voice recognition model.

[0078] In this step, first, the training voice is preprocessed to obtain a training voice frame sequence. The training voice frame sequence refers to dividing continuous voice signals into short time periods of fixed length. The preprocessing process includes operations such as frame division and windowing on the original voice signal to extract time-frequency features. Specifically, pre-emphasis, frame division and windowing techniques can be used to achieve this. By eliminating high-frequency attenuation and spectral leakage, the time-frequency local characteristics of the voice signal are preserved, providing stable input for subsequent feature extraction.

[0079] Further, content representation, speaker representation, and pitch representation are extracted from the training voice frame sequence. The content representation refers to abstract features reflecting semantic information of the voice. Specifically, mel-frequency cepstral coefficients or deep neural network encoders can be used to achieve this. By capturing phoneme and syllable level language structure differences, the semantic mapping problem between dialects and general languages is solved. The speaker representation refers to a feature vector representing the individual pronunciation characteristics. Specifically, i-vector or x-vector models can be used to achieve this. By separating speaker identity related voiceprint information, the interference of accent differences on content recognition is reduced. The pitch representation refers to a quantitative parameter describing the fundamental frequency variation of the voice. Specifically, fundamental frequency trajectory extraction or tone encoding techniques can be used to achieve this. By modeling the tone patterns specific to dialects, the recognition robustness of the six-tone system is enhanced. Thus, multi-dimensional feature extraction helps to decouple voice information for subsequent processing.

[0080] Further, according to the extracted content representation, speaker representation, and pitch representation, the voice is reconstructed to obtain the target language voice. Voice reconstruction refers to generating a target language voice using separated features. Specifically, adversarial generative networks or sequence-to-sequence models can be used to achieve this. By converting cross-lingual acoustic features, the model generalization ability deficiency caused by insufficient low-resource dialect data is alleviated.

[0081] Further, based on the original training voice and the reconstructed target language voice, a target language voice recognition model is constructed. The target language voice recognition model refers to a recognition model optimized for a specific dialect. Specifically, transfer learning or contrastive learning frameworks can be used to achieve this. By reusing general language model parameters and fine-tuning dialect-related modules, the training efficiency under low-resource conditions is improved.

[0082] Further, in response to the target language recognition instruction, the target voice to be recognized is obtained and input into the constructed target language voice recognition model to obtain a recognition result, thereby realizing cross-language voice recognition from a source language to a target language.

[0083] In some embodiments, the content representation, the speaker representation, and the pitch representation are extracted from the training voice frame sequence in step S120, including the following steps S210 to S230:

[0084] Step S210, extracting prosodic features and spectral features from the training voice frame sequence;

[0085] Step S220, constructing joint features of the training voice frame sequence according to the prosodic features and the spectral features;

[0086] Step S230, extracting the content representation, the speaker representation, and the pitch representation from the joint features.

[0087] In this step, first, prosodic features and spectral features are extracted from the training voice frame sequence, wherein the prosodic features are obtained by analyzing the mode of the fundamental frequency and energy of the voice signal changing over time, and are used to capture intonation, stress, and rhythm information; the spectral features include mel-frequency cepstral coefficients (MFCC) and linear prediction coefficients (LPC), etc., and can be extracted by short-time Fourier transform or mel-frequency cepstral coefficients, and are used to represent the formant distribution and timbre characteristics of the voice. Then, the prosodic features and the spectral features are aligned and fused in the time dimension by using feature splicing or attention mechanism, forming a feature matrix containing multi-dimensional acoustic information, and finally obtaining the joint features of the training voice frame sequence, thereby effectively fusing prosodic and spectral information and improving the comprehensiveness of the feature representation.

[0088] Further, when extracting the content representation from the joint features, local pattern analysis of the joint features is performed by a convolutional neural network to filter speaker-related features and retain phoneme and semantic information; when extracting the speaker representation, the joint features are aggregated by using a global average pooling layer to generate a fixed-dimensional vector related to the speaker identity; when extracting the pitch representation, based on the fundamental frequency trajectory in the joint features, a continuous pitch contour is generated by linear interpolation and smoothing processing, thereby finally realizing multi-dimensional feature extraction of the training voice and laying a foundation for subsequent cross-language voice recognition, and further improving the accuracy and robustness of cross-language voice recognition.

[0089] In some embodiments, the content representation, the speaker representation, and the pitch representation are extracted from the joint features in step S230, including the following steps S310 to S340:

[0090] Step S310, converting the joint features into a dense feature sequence;

[0091] Step S320, generating the content representation according to the dense feature sequence;

[0092] Step S330, extracting the speaker representation from the joint feature;

[0093] Step S340, extracting the pitch representation from the joint feature.

[0094] In this step, the joint feature is first converted into a dense feature sequence, preferably by processing the joint feature through a fully connected layer and a normalization layer to obtain the dense feature sequence, wherein the fully connected layer maps the joint feature to a high-dimensional space, and the normalization layer standardizes the feature.

[0095] Further, the content representation is generated according to the dense feature sequence, preferably by extracting the time sequence information of the dense feature sequence through a recurrent neural network to obtain the content representation, wherein the recurrent neural network adopts a long short-term memory unit to capture long-term dependencies. Further, the speaker representation is obtained by processing the joint feature through a global average pooling layer, which averages the feature map in the spatial dimension to obtain a fixed-dimensional vector representation.

[0096] Specifically, the content representation retains the semantic information of the speech, the speaker representation captures the speaker feature, and the pitch representation extracts the pitch change, and further separation of the three representations provides a basis for subsequent speech reconstruction and cross-lingual recognition.

[0097] Further, the pitch representation is extracted from the joint feature through a one-dimensional convolutional neural network, wherein the one-dimensional convolutional neural network includes multiple convolutional layers and pooling layers to extract local features related to the pitch, and finally realizes the separation and extraction of content, speaker and pitch information from the joint feature.

[0098] In some embodiments, speech reconstruction is performed according to the content representation, the speaker representation and the pitch representation in step S130 to obtain target language speech, including steps S410 to S440 as follows:

[0099] Step S410, reconstructing the content representation, the speaker representation and the pitch representation to obtain a reconstructed feature;

[0100] Step S420, constructing a loss function between the content representation, the speaker representation and the pitch representation;

[0101] Step S430, optimizing the reconstructed feature according to the loss function to obtain an optimized reconstructed feature;

[0102] Step S440, decoding the reconstructed feature to obtain the target language speech.

[0103] In this step, reconstruction is performed through content representation, speaker representation and pitch representation to obtain reconstructed features. Preferably, the content representation, speaker representation and pitch representation are fused through a fully connected layer to generate initial reconstructed features. Further, a loss function is constructed among the content representation, speaker representation and pitch representation, wherein the loss function includes three parts of reconstruction loss, adversarial loss and consistency loss, the reconstruction loss is used to measure the difference between the reconstructed features and the original features, the adversarial loss is used to ensure the independence between different representations, and the consistency loss is used to maintain the consistency of each representation before and after reconstruction.

[0104] Further, the reconstructed features are optimized according to the loss function to obtain optimized reconstructed features. Preferably, the network parameters are iteratively updated through a back propagation algorithm to minimize the loss function, thereby obtaining the optimized reconstructed features. Then, the reconstructed features are decoded to obtain the target language speech, and finally the effective reconstruction and optimization of the content representation, speaker representation and pitch representation are realized, which provides high-quality training data for subsequent speech recognition tasks.

[0105] In some embodiments, in step S140, a target language speech recognition model is constructed according to the target language speech based on the training speech, including the following steps S510 to S550:

[0106] Step S510, constructing a training language recognition model according to the training speech;

[0107] Step S520, presetting an initial target language speech recognition model;

[0108] Step S530, obtaining target language training speech;

[0109] Step S540, based on the target language training speech, comparing and learning the training language recognition model and the initial target language speech recognition model to obtain a training target language speech recognition model;

[0110] Step S550, optimizing the training target language speech recognition model according to the target language speech to obtain a target language speech recognition model.

[0111] In this step, a training language recognition model is constructed according to the training speech, wherein the training language recognition model adopts a deep neural network structure including multiple convolution layers and recurrent layers, the convolution layers are used to extract local features of the speech, and the recurrent layers are used to capture long-time sequence dependence.

[0112] Further, a preset initial target language speech recognition model is adopted, and the initial model adopts the same network structure as the training language recognition model, but the parameters are randomly initialized. Then, the target language training speech is obtained. A certain number of speech samples are extracted from the speech corpus of the target language as training data.

[0113] Further, based on the target language training speech, the training language recognition model and the initial target language speech recognition model are compared and learned to obtain the training target language speech recognition model. Specifically, the target language training speech is input into the two models at the same time, the output features of the two models are compared, and the parameters of the initial target language model are adjusted by minimizing the feature difference, so that the initial target language model gradually approaches the training language recognition model. Therefore, through comparative learning and fine-tuning optimization, the target language speech recognition model can inherit the speech recognition ability of the universal language model, adapt to the characteristics of the target language, and improve the recognition accuracy of the low-resource language.

[0114] Further, according to the target language speech, the training target language speech recognition model is optimized to obtain the target language speech recognition model. Specifically, more target language speech data can be input into the training target language model for fine-tuning optimization to improve the adaptability of the model to the target language. Finally, the knowledge transfer from the resource-rich language to the low-resource language is realized, and the problem of insufficient data of the low-resource language is effectively solved.

[0115] In some embodiments, the training speech is preprocessed in step S110 to obtain a training speech frame sequence, including steps S610 to S630 as follows:

[0116] Step S610, pre-emphasis processing is performed on the training speech to obtain a first speech;

[0117] Step S620, the first speech is segmented frame by frame to obtain an overlapping frame sequence;

[0118] Step S630, the training speech frame sequence is extracted from the overlapping frame sequence.

[0119] In this step, the training speech is preprocessed to obtain a training speech frame sequence. The pre-emphasis processing can be realized by a high-pass filter. Further, the first speech is preferably segmented frame by frame by using a Hamming window function to obtain an overlapping frame sequence, so as to extract the training speech frame sequence from the overlapping frame sequence by applying a short-time Fourier transform. Therefore, the training speech is effectively preprocessed, and the accuracy of subsequent feature extraction is improved.

[0120] In some embodiments, the training target language speech recognition model is optimized according to the target language speech in step S550 to obtain a target language speech recognition model, including steps S710 to S730 as follows:

[0121] Step S710, constructing an optimization model; wherein the optimization model comprises an encoder and a decoder; the encoder is the training target language speech recognition model; and the decoder is a preset deep learning network;

[0122] Step S720, inputting the target language speech into the encoder to extract a corresponding acoustic feature sequence;

[0123] Step S730, optimizing the optimization model based on the acoustic feature sequence through a two-stage parameter regulation strategy to obtain the target language speech recognition model; wherein the two-stage parameter regulation strategy divides the optimization process of the optimization model into a freezing stage and a joint stage for optimization; the freezing stage keeps the encoder parameters unchanged and only optimizes the decoder; and the joint stage synchronously optimizes the encoder and the decoder.

[0124] In this step, an optimization model is constructed, which comprises an encoder and a decoder. The encoder is a training target language speech recognition model, and the decoder is a preset deep learning network. Specifically, the encoder can adopt a multi-layer convolutional neural network structure, including multiple convolutional layers, pooling layers, and fully connected layers. The decoder can adopt a recurrent neural network structure, such as a long short-term memory network (LSTM) or a gated recurrent unit (GRU).

[0125] Further, the target language speech is input into the encoder to extract a corresponding acoustic feature sequence, and then the optimization model is optimized based on the acoustic feature sequence through a two-stage parameter regulation strategy to obtain the target language speech recognition model. Wherein the two-stage parameter regulation strategy divides the optimization process of the optimization model into a freezing stage and a joint stage for optimization, so as to synchronously optimize the encoder and the decoder through the joint stage, effectively avoiding overfitting and improving the generalization ability of the model.

[0126] For example, in the freezing stage, the encoder acts as a fixed feature extractor, and its parameters remain unchanged. At this time, the decoder is independently trained by minimizing the reconstruction error of the acoustic feature sequence. For example, in the Guiyue dialect speech recognition task, the freezing stage lasts for fifteen training cycles, and the decoder uses a mean square error loss function to optimize the spectral reconstruction accuracy. After entering the joint stage, the encoder and decoder parameters participate in gradient update at the same time. At this time, the encoder adjusts the weights of the feature extraction layer based on the error signal feedback from the decoder, and the learning rate is 0.0001. This staged optimization mechanism limits the initial parameter adjustment range under low data conditions, avoids the encoder falling into a local optimal solution too early, and ensures that the decoder has a preliminary feature mapping capability before joint fine-tuning. By controlling the parameter update range in stages, the modeling stability of the model for complex acoustic features such as Guiyue dialect is improved, and the overfitting phenomenon caused by insufficient data is reduced.

[0127] In some embodiments, as shown in Figure 2 and Figure 3 The cross-lingual speech recognition method is realized by the following steps:

[0128] Step one, construct the joint feature parameter Pitch-MFCC:

[0129] 1. Speech preprocessing based on process collaboration:

[0130] Speech preprocessing refers to a series of processing steps before analyzing, processing or transmitting the speech signal to improve the quality of the signal and improve the recognition accuracy. In this embodiment, through the process collaboration of pre-emphasis, framing, windowing and endpoint detection technology, a closed-loop optimized speech processing system is constructed.

[0131] Further, a first-order difference filter is used to implement pre-emphasis processing on the original audio signal, and the specific formula is as follows:

[0132] y[n] = x[n] - a x[n-1];

[0133] Where a is the pre-emphasis coefficient, x[n] is the nth point of the original audio signal, x[n-1] is the n-1th point of the original audio signal, and y[n] is the pre-emphasized signal.

[0134] Further, different short-time signal overlap regions are set in the sliding window to make the adjacent signals have a certain continuity, which not only improves the retention effect of signal time domain characteristics, but also reduces the information break risk caused by segmentation.

[0135] It is worth noting that when the signal is cut into several short-time segments, discontinuous points will be introduced at the edges. These discontinuities will cause spectral leakage when performing Fourier transform, that is, the energy originally concentrated on a certain frequency is "leaked" to other frequencies, causing additional non-zero components in the frequency spectrum. To reduce the impact of spectral leakage, this embodiment introduces a window function, which is represented as:

[0136]

[0137] Where ω[i] is the value of the window function at the ith sampling point, i is the sampling point number, and I is the total number of sampling points.

[0138] After the above steps, the speech signal has been converted into a series of short-time frames with good spectral characteristics and stability, and then the endpoint detection technology is introduced to identify the speech activity interval, which is represented as:

[0139]

[0140] Among them, x i (n) is the speech signal of the nth point of the original audio signal after windowing at the i-th frame, x i (n-1) is the speech signal of the n-1th point of the original audio signal after windowing in the i-th frame, N is the frame length, and sgn[.] is the sign function;

[0141]

[0142] The zero-crossing rate Z of each frame is judged. If the Z of a frame i >Z 阈值 , then the frame is considered to belong to the speech activity segment and is retained, otherwise it is discarded, thereby realizing speech start and end point recognition and extracting effective speech frame sequences, providing high-quality speech input for subsequent steps.

[0143] 2. Joint feature extraction based on multi-level speech transformation:

[0144] To address the need for collaborative representation of pitch and Mel-Frequency Cepstral Coefficients (MFCC) in speech data, a joint feature extraction system based on multi-level speech transformation is constructed. Taking the MFCC feature extraction process as an example, its core technical modules include:

[0145] In the frequency domain analysis phase, the preprocessed time-domain sound signal is first converted to the frequency domain using a Fast Fourier Transform (FFT). This results in a spectrum that reflects the energy distribution of the sound signal at different frequencies. Specifically, the spectrum is represented based on a linear frequency scale, which treats the distance between each Hertz (Hz) as equal. However, the human ear's sensitivity to different frequencies is not linear, but rather follows the Mel scale. To simulate how the human auditory system processes sound, the following methods can be used:

[0146]

[0147] Convert the actual frequency (Hz) to Mel (f) Indicates Mel frequency, and f is the actual frequency it corresponds to. In order to extract more representative auditory features based on the difference in human ear’s sensitivity at different frequencies, a set of triangular filters is introduced, whose frequency response H n (k) can be expressed as:

[0148]

[0149] Among them, f(n) is the center frequency of the nth filter, f(n-1) is the center frequency of the n-1th filter, and f(n+1) is the center frequency of the n+1th filter, and then the energy spectrum E of the frequency response is calculated. n ;

[0150]

[0151] where |X(k)| is the fast Fourier transform amplitude at frequency k, H n (k) is the frequency response.

[0152] The cepstrum domain generation stage, since the human auditory system has a nonlinear characteristic of the perception of sound intensity, and shows a logarithmic relationship, so the data processed by logarithmic operation can be more close to the actual perception way of human ear, realize the effective compression of signal in dynamic range, the logarithmic energy spectrum E n can be expressed as:

[0153]

[0154] Further, through discrete cosine transform (DCT) to convert the above results to obtain MFCC, this process can be expressed as:

[0155]

[0156] where C j represents the jth cepstrum coefficient, l is the order of MFCC, and M is the number of logE n . In particular, after DCT conversion, the energy distribution of the signal is expressed in a compact form, which contains the results of all mel filter calculations, and usually only low-order cepstrum coefficients can capture the key characteristics of the speech signal, and higher-order coefficients contribute less to the overall characteristics. Therefore, this embodiment only retains the first 12 MFCCs, and the rest is discarded.

[0157] After the same process of MFCC, Pitch is obtained, and the first 12 are also retained. In order to make full use of the complementarity between the two, so as to more comprehensively and accurately capture the acoustic characteristics of the speech signal, the extracted Pitch and MFCC parameters are combined. It is assumed that the MFCC parameters form an M×N matrix A MFCC , where M represents the number of frames, and N represents the feature dimension. Similarly, the Pitch parameters also form an M×N matrix A Pitch . A MFCC and A Pitch are spliced by column to form a new M×2N matrix A MFCC-Pitch =[A MFCC A Pitch ] °

[0158] In particular, in order to ensure that the source speech and the target speech are accurately aligned on the time axis, a dynamic time warping (DTW) is used to establish a cross-speech timing alignment model, and the best match of the Pitch-MFCC feature parameters of the source speech and the target speech in time is calculated, which can be described as:

[0159] D(A sourceA target )=min P ∑ (i,j)∈P d(a source [i],a target [j]);

[0160] where D(A source ,A target ) denotes the total accumulated distance between the source and target speech, P is the optimal path consisting of a series of index pairs (i,j), and d(a source [i],a target [j]) is the distance measure between the i-th frame of the source speech and the j-th frame of the target speech.

[0161] Step two, SE-MI based low-resource data augmentation:

[0162] The data augmentation strategy of the embodiment mainly involves four modules: content encoder, speaker encoder, pitch extractor and decoder. The content encoder includes an h-net network, a quantizer VQ and a g-net network, and the dashed arrow represents the mutual information between the representations. Let the constructed Pitch-MFCC joint feature be X k , which is input into the content encoder, the speaker encoder and the pitch extractor to extract respective representations.

[0163] 1. Content representation extraction:

[0164] First, the convolutional module of the h-net network converts the joint feature X k into a dense feature sequence Z k,c , and the sequence length is changed from T to T / 2. In order to add global context information coding and improve the ability of content representation, the dense feature sequence Z k,c is input into the SE attention mechanism.

[0165] In order to solve the problem that the feature units cannot use the context information outside the region, the global average pooling operation can be used to compress the global spatial information into a channel descriptor. This step can be represented as:

[0166]

[0167] where z c′ ∈ Z = {z1, z2, …, z C′}, z c′ is the channel descriptor of the c ′ th channel, u c′ represents the feature of the c ′ th channel of the input feature, W ′ and H ′respectively, i is the index of feature width, j is the index of feature height, (i, j) represents the coordinate position of the current feature.

[0168] In addition, in order to fully capture the correlation between channels, the SE network uses a Sigmoid activation function to operate on Z, as follows:

[0169] S = σ(W2δ(W1Z));

[0170] where W1 represents the reduction parameter of the dimension reduction layer, W2 is the increment parameter of the dimension increment layer, σ is the Sigmoid activation function, δ is the ReLU activation function, S = {s1, s2, …, s C′} is the set of modulation weights corresponding to each generated channel. By multiplying the weight set S with the corresponding feature elements in the new feature U, the weighted new feature sequence Z k,c′ is obtained. k,c′ At this time, the feature sequence Z k,q output by the SE network contains global context information.

[0171] Finally, the content representation Z k is obtained through the VQ (Vector Quantization) and g-net network.

[0172] 2. Speaker representation extraction: input the acoustic feature X k into the speaker encoder to extract the speaker representation S k .

[0173] 3. Pitch representation extraction:

[0174] The fundamental frequency F0 is extracted from the acoustic feature X k through the pitch extractor, and the normalized pitch representation P k is obtained.

[0175] Further, the content representation Z k,q , the speaker representation S k and the pitch representation P k are input into the decoder to obtain the reconstructed feature , which can be represented as:

[0176]

[0177] where E C (·) and E S (·) represent the content encoder and the speaker encoder respectively, D(·) represents the decoder, Z k,q represents the content representation, which is obtained through the quantizer VQ and the g-net network from the dense feature sequence Z k,c′ , S k represents the speaker representation, and Pk represents a pitch representation, represents a reconstructed feature.

[0178] To reduce the correlation between each representation and realize high-quality dialect speech conversion, the embodiment introduces mutual information between content representation, speaker representation and pitch representation to reduce the leakage of each other's representation. The loss function can be represented as:

[0179] L VC,SE = L VQ,SE + L CPC,SE + L REC,SE + λ MI L MI,SE ;

[0180] Wherein, λ MI represents the weight of mutual information loss, L VQ,SE , L CPC,SE , L REC,SE and L MI,SE represent the vector quantization loss of quantizer VQ, the information noise contrast estimation loss of contrastive predictive coding, the decoder reconstruction loss and the mutual information loss, respectively.

[0181] The vector quantization loss can be represented as:

[0182]

[0183] Wherein, K represents the number of training sentences, k is the index of K, T represents the number of mel-spectrum feature frames, sg(·) represents the stop gradient operator, represents the square of 2-norm, z k,c′,t and z k,q,t represent the t-th frame feature element in the dense feature sequence z k,c′ and z k,q , respectively.

[0184] The information noise contrast estimation loss of contrastive predictive coding can be represented as:

[0185]

[0186] Wherein, M represents the step of contrastive prediction, m is the index of M, T'=T / 2-M, W m represents a trainable projection matrix, z k,q,t+m represents the t+m-th frame feature element in the content representation sequence z k,q , Ω k,t,m represents a negative sample set composed of randomly selected samples from the current training sentence, is a random sample in the set Ω k,t,m , z k,c,t represents the joint feature X kThe dense feature sequence Z output by the hnet network k,c The characteristic element of the t-th frame, For the set Ω k,t,m The transpose of a random sample.

[0187] The decoder reconstruction loss can be expressed as:

[0188]

[0189] Among them, x t and x se,t They represent the t-th feature element of the input Mel-spectrum feature and the t-th feature element of the Mel-spectrum feature reconstructed by the decoder, respectively. ||·||1 and ||·||2 represent the 1-norm and 2-norm, respectively.

[0190] The formula for mutual information loss can be expressed as:

[0191]

[0192] Represent the content representation Z k,q and speaker representation S k , content representation Z k,q and pitch representation P k , pitch representation P k and speaker representation S k The unbiased estimate of the upper bound of the variational contrast logarithm between . The specific expansion can be expressed as:

[0193]

[0194]

[0195]

[0196] in, Represents the positive sample's representation of the speaker S k and content representation Z k,q The conditional log-likelihood of , Indicates that the negative sample represents the speaker S k and content representation Z k,q The conditional log-likelihood of , Represents a given speaker representation S k When, after Content representation Z after network parameterization k,q The corresponding variational approximation to the true posterior, Represents the positive sample pitch representation P k and content representation Z k,q The conditional log-likelihood of , Represents the negative sample pitch representation P kand content representation Z k,q the conditional log-likelihood of the positive samples given the speaker representation S and the pitch representation P k and the pitch representation P k the conditional log-likelihood of the positive samples given the speaker representation S and the pitch representation P k and the pitch representation P k the conditional log-likelihood of the positive samples given the speaker representation S

[0197] Finally, the reconstructed features are input into the vocoder at the conversion stage to obtain the speech after cross-lingual conversion.

[0198] Step three, cross-language self-supervised migration of Guilin Liu speech recognition:

[0199] 1. Pre-training of the self-supervised model:

[0200] In this embodiment, Chinese Mandarin is used as the high-resource source language, and more than 10,000 hours of unlabeled Chinese Mandarin speech is used as pre-training data to pre-train the Wav2vec2.0 and HuBERT self-supervised models. Wav2vec2.0 learns context-dependent acoustic representations through a contrastive learning task, while HuBERT predicts hidden units based on a mask reconstruction task. During the pre-training process, the model takes the original speech waveform as input, extracts local features through a multi-layer convolutional neural network, and then learns global temporal dependencies through a Transformer encoder. The goal of this stage is to enable the model to capture general speech features from large-scale unsupervised data and provide high-robustness initialization parameters for the low-resource Guilin Liu recognition task.

[0201] 2. Loss optimization based on hybrid CTC / Attention:

[0202] After the pre-training is completed, the model parameters are saved and used for parameter initialization during fine-tuning, where the encoder is the self-supervised model and the top layer of the self-supervised model is used for fine-tuning.

[0203] A linear mapping layer is added, and a hybrid CTC / Attention loss function is used for optimization.

[0204] Let L be a set of modeling units, and the modeling unit in this embodiment is a word. To model silence and inter-word pauses, an additional "blank" label is introduced in CTC and represented by "-". Therefore, L can be extended to L' = L U {"-"}. The "blank" label also needs to be removed in the last step of recognition. In particular, the "blank" label is removed in the last step of recognition.

[0205] The training data set S is input into the self-supervised model to obtain X = {x1, x2, …, x T} and the corresponding high-level feature sequence H = {h1, h2,..., h r}.

[0206] Each training sample contains an input sequence X = {x1, x2,..., x T} and the corresponding output sequence Y = {y1, y2,..., y U}, where T represents the input time length, U represents the output label sequence length, and each label element y U is taken from a pre-defined set of modeling units L. The training goal of CTC is to maximize the matching probability P(Y|X) of the input and output sequences by optimizing the model parameters, where the output sequence Y is generated by connecting the time classification path space A CTC (X, Y), which contains all the legal paths that allow the insertion of blank labels and the repetition of labels.

[0207] The generation probability of the output sequence Y is defined as the sum of the probability values of all corresponding CTC paths, which can be represented as:

[0208]

[0209] where, represents a certain alignment path of X and Y under CTC. After removing the repetition and blank labels of , Y is obtained.

[0210] The probability of the path Y appearing is the product of the output probability at each time, which can be represented as

[0211]

[0212] where, represents the output label of the path at time t, is its corresponding output probability.

[0213] The CTC loss function is the sum of the negative logarithmic probabilities of all samples in the training set S, which can be represented as:

[0214] L CTC = -∑ (X,Y)∈S ln P(Y|X) ;

[0215] ​To further improve the accuracy of the model in complex speech recognition tasks, Attention is combined with CTC to construct a hybrid CTC / Attention model, and a Multi-task learning (MTL) strategy is adopted to form an encoder-decoder architecture, which fully utilizes the alignment advantage of CTC and the context modeling advantage of attention mechanism. In the model, a self-supervised model is used as a shared encoder to convert the input sequence X = {x1, x2, x3, x4, x5, x6, x7, x8…, x r} into a feature sequence = H = {h2, h4, h6, h8, …, h r}, and then the feature sequence H is sent into the CTC decoder and the attention decoder to predict the output sequence c = {c1, c2, …}. Where z = {…z2, …, z4, …} is the intermediate state in the CTC decoding process, q = {q0, q1, q2, …, q L} and r = {r0, r1, r2, …, r L} are the states of the attention decoder, starting with the sos state and ending with the eos state. When predicting the output sequence, the CTC and Attention decoders perform joint decoding. In particular, the encoder is shared by CTC and attention, so the loss function is the weighted sum of the CTC loss function L CTC and the attention loss function L ATT , which can be expressed as:

[0216] L MTL = ηL CTC + (1-η)L ATT ;

[0217] Where the weight η represents the weight of the CTC model.

[0218] 3. Self-supervised model two-stage freezing fine-tuning strategy:

[0219] To improve the generalization ability of the speech recognition model, the training state of the encoder and decoder is adjusted to achieve the collaborative optimization of the model's generalization ability and knowledge retention ability in the low-resource scenario of Gui-Li dialect. To address the problem of insufficient dialect feature learning caused by the traditional full freezing strategy and the problem of pre-training knowledge forgetting caused by full parameter fine-tuning, a two-stage training strategy of "freezing first and then combining" is adopted, and the specific steps are as follows:

[0220] The total training steps are divided into a freezing period and a joint period in a ratio of 2:8. In the freezing period, the core module parameters of the encoder, such as the Transformer layer and the feature projection layer, are completely frozen, and only the decoder is allowed to learn the context dependency of the dialect phoneme sequence through the masked self-attention mechanism. In the joint period, the encoder parameters are updated synchronously to enable the model to gradually adapt to the characteristics of the target dialect on the basis of inheriting general acoustic knowledge.

[0221] As shown in Figure 4 Some embodiments of the present application provide a cross-lingual speech recognition system, which includes a processing module 410, an extraction module 420, a reconstruction module 430, a construction module 440, a response module 450, and an identification module 460. Specifically:

[0222] The processing module 410 is configured to pre-process the training speech to obtain a training speech frame sequence.

[0223] The extraction module 420 is configured to extract content representation, speaker representation, and pitch representation from the training speech frame sequence.

[0224] The reconstruction module 430 is configured to reconstruct speech according to the content representation, speaker representation, and pitch representation to obtain target language speech.

[0225] The construction module 440 is configured to construct a target language speech recognition model based on the training speech according to the target language speech.

[0226] The response module 450 is configured to respond to a target language recognition instruction to obtain target speech.

[0227] The identification module 460 is configured to input the target speech into the target language speech recognition model to obtain an identification result output by the target language speech recognition model.

[0228] It should be noted that the cross-lingual speech recognition system provided by the present embodiment is based on the same inventive concept as the cross-lingual speech recognition method described above, and therefore the related content of the cross-lingual speech recognition method described above is also applicable to the content of the cross-lingual speech recognition system. Therefore, this will not be described again.

[0229] To this end, the system pre-processes the training speech to obtain a training speech frame sequence; extracts content representation, speaker representation and pitch representation from the training speech frame sequence; reconstructs the speech according to the content representation, the speaker representation and the pitch representation to obtain target language speech; constructs a target language speech recognition model based on the training speech and the target language speech; in response to a target language recognition instruction, acquires target speech; inputs the target speech into the target language speech recognition model to obtain a recognition result output by the target language speech recognition model. In this way, cross-language feature decoupling and self-supervised migration mechanism can be used to accurately model the acoustic characteristics of dialects under extremely low labeled data conditions, significantly improve the recognition robustness of complex tones and characteristic words of Guilin and Liuzhou dialects, and realize efficient generalization application in dialect scenarios.

[0230] The embodiment of the present application also provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the cross-language speech recognition method described above when executing the computer program.

[0231] As Figure 5 , Figure 5 The electronic device provided by the embodiment of the present application has a hardware structure diagram, which comprises:

[0232] At least one battery;

[0233] At least one memory;

[0234] At least one processor;

[0235] At least one program;

[0236] The program is stored in the memory, and the processor executes the at least one program to implement the cross-language speech recognition method described above.

[0237] The electronic device can be any intelligent terminal, including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.

[0238] The electronic device of the embodiment of the present application will be described in detail below.

[0239] The processor 1600 can be implemented in the form of a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present application;

[0240] Memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). Memory 1700 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in memory 1700 and is called by processor 1600 to execute a cross-language speech recognition method according to the embodiments of this disclosure.

[0241] Input / output interface 1800, used for information input and output;

[0242] Communication interface 1900, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0243] bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0244] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0245] An embodiment of the present disclosure further provides a storage medium, which is a computer-readable storage medium and stores computer-executable instructions. The computer-executable instructions are used to enable a computer to execute the above-mentioned cross-language speech recognition method.

[0246] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0247] The embodiments described in the present disclosure are to more clearly illustrate the technical solutions of the present disclosure, and do not constitute a limitation on the technical solutions provided by the present disclosure. Those skilled in the art can know that, as technology evolves and new application scenarios appear, the technical solutions provided by the present disclosure are also applicable to similar technical problems.

[0248] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the present disclosure, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0249] The device embodiments described above are merely illustrative, and units described as separate components can or can not be physically separated, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0250] Those skilled in the art can understand that all or some steps in the above disclosed method, functional modules / units in the system and device can be implemented as software, firmware, hardware and their appropriate combinations.

[0251] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0252] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b and c can be single or multiple.

[0253] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0254] The units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0255] In addition, the functional units in each embodiment of the application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0256] If the integrated unit is implemented in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various program storage media.

[0257] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned implementation. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the embodiments of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the embodiments of the present application.

[0258] The embodiments of the present application have been described in detail above in combination with the drawings, but the present application is not limited to the above-mentioned embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the purpose of the present application.

Claims

1. A cross-language speech recognition method, characterized in that: The method comprises: Preprocessing the training speech to obtain a training speech frame sequence; Extracting content representation, speaker representation and pitch representation from the training speech frame sequence; Reconstructing the speech according to the content representation, speaker representation, and pitch representation to obtain the speech of the target language; Based on the training speech, constructing a target language speech recognition model according to the target language speech; Respond to the target language recognition command and obtain the target speech; The target speech is input into the target language speech recognition model to obtain a recognition result output by the target language speech recognition model.

2. The cross-language speech recognition method according to claim 1, characterized in that The extracting content representation, speaker representation and pitch representation from the training speech frame sequence includes: Extracting prosodic features and spectral features from the training speech frame sequence; constructing a joint feature of the training speech frame sequence according to the prosodic feature and the spectral feature; The content representation, the speaker representation, and the pitch representation are extracted from the joint feature.

3. The cross-language speech recognition method according to claim 2, characterized in that: The extracting the content representation, the speaker representation, and the pitch representation from the joint features includes: converting the joint features into a dense feature sequence; generating the content representation according to the dense feature sequence; Extracting the speaker representation from the joint features; The pitch representation is extracted from the joint features.

4. The cross-language speech recognition method according to claim 1, wherein: The speech reconstruction based on the content representation, speaker representation, and pitch representation to obtain the target language speech includes: reconstructing the content representation, the speaker representation, and the pitch representation to obtain a reconstructed feature; Constructing a loss function among the content representation, the speaker representation, and the pitch representation; Optimizing the reconstruction feature according to the loss function to obtain an optimized reconstruction feature; The reconstructed features are decoded to obtain the target language speech.

5. The cross-language speech recognition method according to claim 1, wherein: The step of constructing a target language speech recognition model based on the training speech and the target language speech includes: Constructing a training language recognition model based on the training speech; Preset the initial target language speech recognition model; Obtain training speech in the target language; Based on the target language training speech, the training speech recognition model is compared with the initial target language speech recognition model to obtain a training target language speech recognition model; According to the target language speech, the training target language speech recognition model is optimized to obtain the target language speech recognition model.

6. The cross-language speech recognition method according to claim 1, characterized in that: The preprocessing of the training speech to obtain a training speech frame sequence includes: Performing pre-emphasis processing on the training speech to obtain a first speech; Segmenting the first speech frame by frame to obtain an overlapping frame sequence; The training speech frame sequence is extracted from the overlapping frame sequence.

7. The cross-language speech recognition method according to claim 5, characterized in that: The step of optimizing the trained target language speech recognition model according to the target language speech to obtain the target language speech recognition model includes: Constructing an optimization model; wherein the optimization model includes an encoder and a decoder; the encoder is the training target language speech recognition model; the decoder is a preset deep learning network; Inputting the target language speech into the encoder to extract the corresponding acoustic feature sequence; Based on the acoustic feature sequence, the optimization model is optimized through a two-stage parameter control strategy to obtain the target language speech recognition model; wherein, the two-stage parameter control strategy is to divide the optimization process of the optimization model into a freezing stage and a joint stage for optimization; the freezing stage is to keep the encoder parameters unchanged and only optimize the decoder; the joint stage is to synchronously optimize the encoder and the decoder.

8. A cross-language speech recognition system, characterized in that: The system comprises: A processing module, configured to pre-process the training speech to obtain a training speech frame sequence; An extraction module, configured to extract content representation, speaker representation, and pitch representation from the training speech frame sequence; A reconstruction module, configured to reconstruct speech based on the content representation, speaker representation, and pitch representation to obtain speech in the target language; A construction module, configured to construct a target language speech recognition model based on the training speech and the target language speech; The response module is used to respond to the target language recognition command and obtain the target speech; The recognition module is used to input the target speech into the target language speech recognition model to obtain the recognition result output by the target language speech recognition model.

9. An electronic device, characterized in that: The invention comprises at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute a cross-language speech recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the cross-language speech recognition method according to any one of claims 1 to 7.

Images