Audio language conversion method, system and related device

Abstract

The invention discloses an audio language conversion method and system and a related device, and the method comprises the steps: obtaining an initial audio stream of a target object, and determining an initial audio feature corresponding to the initial audio stream, and a current language corresponding to the initial audio stream; based on the initial audio feature and the current language, obtaining a target feature fragment corresponding to a current conversion round; wherein the target feature segments corresponding to different conversion rounds are segmented based on semantics of the initial audio features; and determining at least one conversion language, and based on the current language and the target feature segment, generating a conversion audio matched with the conversion language. In this way, the accuracy of audio cross-language conversion can be improved.

Description

[0001] This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office on October 23, 2024, with application number 2024114869196 and application name “Audio language conversion method, system and related device”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of audio processing technology, and in particular to an audio language conversion method, system and related devices. Background Art

[0003] With the continuous development of artificial intelligence, real-time audio language conversion is being applied in more and more scenarios. For example, in large-scale conferences or lectures, there is a language barrier between speakers and participants, which makes it impossible for participants to accurately understand what the speakers are saying. The current language conversion methods mainly rely on translators to perform manual translation or use neural network models to translate word by word based on the input audio stream. For the former, due to the short translation time, the quality requirements of translators are extremely high, and high labor costs are required. For the latter, due to the differences in expressions in different languages, the neural network model is prone to incoherent sentences during translation, resulting in poor conversion effect.

[0004] In view of this, how to improve the accuracy of audio cross-language conversion has become an urgent problem to be solved. Summary of the invention

[0005] The main technical problem solved by the present application is to provide an audio language conversion method, system and related devices, which can improve the accuracy of audio cross-language conversion.

[0006] To solve the above technical problems, a technical solution adopted in the present application is: to provide an audio language conversion method, including: obtaining an initial audio stream of a target object, determining an initial audio feature corresponding to the initial audio stream, and a current language corresponding to the initial audio stream; based on the initial audio feature and the current language, obtaining a target feature segment corresponding to the current conversion round; wherein the target feature segments corresponding to different conversion rounds are segmented based on the semantics of the initial audio feature; determining at least one conversion language, and generating a conversion audio matching the conversion language based on the current language and the target feature segment.

[0007] To solve the above technical problems, another technical solution adopted in the present application is: to provide an audio language conversion system, including: an acquisition module, used to acquire the initial audio stream of the target object, determine the initial audio features corresponding to the initial audio stream, and the current language corresponding to the initial audio stream; a processing module, used to acquire the target feature segments corresponding to the current conversion round based on the initial audio features and the current language; wherein the target feature segments corresponding to different conversion rounds are segmented based on the semantics of the initial audio features; a conversion module, used to determine at least one conversion language, generate a conversion audio matching the conversion language based on the current language and the target feature segments, and broadcast the conversion audio.

[0008] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising: a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the audio language conversion method mentioned in the above technical solution.

[0009] In order to solve the above technical problems, another technical solution adopted in the present application is: providing a computer-readable storage medium on which program instructions are stored, and when the program instructions are executed by a processor, the audio language conversion method mentioned in the above technical solution is implemented.

[0010] The beneficial effects of the present application are as follows: Different from the prior art, the audio language conversion method proposed in the present application obtains the initial audio stream of the target object in the target scene in real time, and extracts features from the initial audio stream to obtain the corresponding initial audio features and the current language. According to the semantics of the current language and the initial audio features, the initial audio features are semantically segmented to obtain target feature segments. Moreover, when translating and converting the target feature segments, there is no need to refer to the context semantics, thereby improving the accuracy of real-time conversion of audio languages. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them: Figure 1 It is a flowchart of an implementation method of the audio language conversion method of the present application; Figure 2 yes Figure 1 Step S101 corresponds to a flow chart of another implementation method; Figure 3 yes Figure 2Step S203 corresponds to a flow chart of another implementation method; Figure 4 yes Figure 1 Step S102 corresponds to a flow chart of another implementation method; Figure 5 yes Figure 1 Step S103 corresponds to a flow chart of another implementation method; Figure 6 yes Figure 5 Step S502 corresponds to a flowchart of another implementation method; Figure 7 It is a structural diagram of an implementation method of the audio language conversion system of the present application; Figure 8 It is a structural schematic diagram of an embodiment of the electronic device of the present application; Fig. 9 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments, and different embodiments can be adaptively combined. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0013] See also Figure 1 , Figure 1 1 is a flowchart of an embodiment of an audio language conversion method of the present application, the method comprising: S101: Acquire an initial audio stream of a target object, and determine initial audio features corresponding to the initial audio stream and a current language corresponding to the initial audio stream.

[0014] In one embodiment, a target object is determined from a target scene, and an initial audio stream generated when the target object is expressed in real time is extracted. The initial audio stream is subjected to feature extraction to obtain corresponding initial audio features. And, based on the obtained initial audio features, the current language currently expressed by the target object is determined. The current language is one of a plurality of language types, such as one of Chinese, English, Japanese, and Korean.

[0015] In another embodiment, after the initial audio stream of the target object is extracted, the initial audio stream is subjected to real-time noise reduction processing to remove environmental noise. Further, feature extraction is performed on the initial audio stream after noise removal to obtain corresponding initial audio features, and the current language is determined based on the initial audio features.

[0016] S102: Based on the initial audio features and the current language, a target feature segment corresponding to the current conversion round is obtained; wherein the target feature segments corresponding to different conversion rounds are segmented based on the semantics of the initial audio features.

[0017] In one implementation, semantic analysis and segmentation are performed on the initial audio features based on the obtained initial audio features and the current language to determine the target feature segment corresponding to the current conversion round.

[0018] Specifically, the above-mentioned target feature segment includes an initial audio feature segment whose semantic relevance is greater than a relevance threshold, that is, the target feature segment corresponding to the current conversion round has a higher degree of semantic completeness, so that when translating according to the target feature segment, there is no need to rely on contextual semantics, thereby helping to improve the accuracy of real-time conversion of audio languages.

[0019] S103: Determine at least one conversion language, and generate conversion audio matching the conversion language based on the current language and the target feature segment.

[0020] In one embodiment, at least one conversion language specified by a user is determined, and conversion audio matching each conversion language is generated according to the current language and the target feature segment.

[0021] In one implementation scenario, after determining the current language corresponding to the above-mentioned initial audio stream, multiple languages ​​to be selected are displayed, and the user is prompted to make a selection. In response to obtaining the user's selection instruction, the language to be selected corresponding to the selection instruction is used as the conversion language. The current language and the target feature segment are analyzed using the fine-tuned intelligent analysis model, and the intelligent analysis model is prompted to generate conversion audio matching each conversion language. Among them, the above-mentioned intelligent analysis model is a large language model with better data analysis capabilities. By inputting the current language and the target feature segment into the intelligent analysis model, the intelligent analysis model generates a translation text corresponding to the target feature segment, and generates a conversion audio matching the conversion language based on the translation text.

[0022] In a specific application scenario, the above-mentioned large language model may include but is not limited to deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTM), and generative pre-trained Transformer models, etc. No specific restrictions are made on the specific structure and specific deployment of the large language model. In addition, it should be noted that the specific structure and specific deployment of the intelligent analysis model mentioned in other implementations of this application can refer to this implementation.

[0023] In another embodiment, the conversion language is determined according to the first languages ​​of different objects in the target scene.

[0024] Specifically, the first language mastered by each object in the target scene is determined. For the target object in the target scene, the first language of other objects is used as the conversion language, so that after the audio language conversion, the other objects in the target scene can understand the expression content of the target object.

[0025] The audio language conversion method proposed in the present application obtains the initial audio stream of the target object in the target scene in real time, and extracts features from the initial audio stream to obtain the corresponding initial audio features and the current language. According to the semantics of the current language and the initial audio features, the initial audio features are semantically segmented to obtain target feature segments. Moreover, when translating and converting the target feature segments, there is no need to refer to the context semantics, thereby improving the accuracy of real-time conversion of audio languages.

[0026] See also Figure 2 , Figure 2 yes Figure 1 Step S101 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S101 includes: S201: Acquire an initial audio stream of a target object, and determine initial audio features corresponding to the initial audio stream.

[0027] In one embodiment, in response to the audio language conversion method proposed in the present application being applied to a target scene including at least one target object, scene audio data in the target scene is acquired in real time. The scene audio data is input into a trained speech detection model so that the speech detection model extracts an initial audio stream matching each target object from the scene audio data based on timbre differences. The initial audio is input into a trained speech coding model to obtain initial audio features.

[0028] In another embodiment, scene information matching the target scene is obtained, wherein the scene information includes scene video data and scene audio data.

[0029] Specifically, in order to improve the accuracy of detecting the initial audio streams of different target objects, scene video data and scene audio data corresponding to the target scene are obtained. The scene video data is used to determine the number of target objects in the target scene and the facial action information of the target objects, and the facial action information is used to determine whether the corresponding target object is expressing. By combining the scene video data and the scene audio data, it helps to improve the accuracy of audio separation of different target objects.

[0030] Furthermore, based on the scene information, at least one target object in the target scene and an initial audio stream corresponding to the target object are determined, and the initial audio is input into the trained speech coding model to obtain initial audio features.

[0031] Specifically, different timbre information contained in the scene audio data is determined. And the target object in the target scene is determined according to the scene video data. The timbre information is matched with the target object, and the scene audio data is audio-separated according to the facial action information of different target objects in the scene video data, so as to obtain an initial audio stream corresponding to each target object. The obtained initial audio is audio-encoded using the trained speech coding model to obtain initial audio features.

[0032] In another embodiment, the scene information further includes microphone information, and the scene audio data includes audio collected by different microphones in the target scene. Based on the audio collected by each microphone, an initial audio stream corresponding to the target object in the target scene and its corresponding initial audio features are determined.

[0033] Specifically, each target object in the target scene is matched with a corresponding microphone. For the audio collected by each microphone, the initial audio stream of the corresponding target object is extracted and audio encoding is performed on it to obtain the initial audio features.

[0034] S202: Based on the current time and the initial audio features, obtain current window features corresponding to a preset duration.

[0035] In one implementation, a preset duration is obtained, and the current window characteristics are determined according to the current time and the preset duration.

[0036] Specifically, the initial audio features corresponding to the initial audio stream of the preset time length before the current moment are used as the current window features. The preset time length can be set according to actual needs, such as 1 second, 2 seconds or 3 seconds.

[0037] S203: Determine the current language from a plurality of candidate languages ​​based on the current window feature.

[0038] In one embodiment, the current window feature is input into a trained language discrimination model, so that the language discrimination model outputs current discrimination information corresponding to the current window feature, and uses the candidate language corresponding to the maximum reference score as the current language. The current discrimination information includes reference scores matching each candidate language; and the higher the reference score, the higher the probability that the corresponding candidate language is the current language.

[0039] In another embodiment, after obtaining the reference scores of the current window features corresponding to each candidate language using the language discrimination model, it is determined whether the maximum reference score is greater than or equal to a preset score threshold. If so, the candidate language corresponding to the maximum reference score is used as the current language. If not, the preset time length is increased to obtain the current window features that match the adjusted preset time length, and return to step S203 to re-obtain the reference scores of the current window features corresponding to each candidate language until the maximum reference score is greater than or equal to the score threshold, and the candidate language corresponding to the maximum reference score is used as the current language.

[0040] In a specific application scenario, the preset duration before adjustment is 1 second, and the initial audio features corresponding to the initial audio stream from 1 second before to the current moment are used as the current window features. In response to the fact that the maximum reference score is less than the preset score threshold after the reference scores of the current window features corresponding to each candidate language are obtained using the language discrimination model, the preset duration is increased to 2 seconds, and the initial audio features corresponding to the initial audio stream within 2 seconds before the current moment are used as the current window features, and the current language is determined using the language discrimination model.

[0041] Furthermore, after determining the current language according to the current window feature, the above implementation method updates the current window feature to the historical window feature, updates the current discrimination information obtained according to the current window feature to the historical discrimination information, and continues to obtain the next window feature.

[0042] The above scheme determines the current window features by sliding the window, and identifies the current language expressed by the target object based on the current window features, so that the language conversion process can be performed in combination with the identified current language to improve the effect of language conversion.

[0043] See also Figure 3 , Figure 3 yes Figure 2 Step S203 in the flowchart corresponds to another implementation. Specifically, the implementation process of step S203 includes: S301: Acquire historical window features adjacent to the current window features and historical discrimination information corresponding to the historical window features; wherein the historical discrimination information includes historical scores matching each candidate language.

[0044] In one implementation, historical window features adjacent to the current window features and historical identification information corresponding to the historical window features are determined according to a preset duration.

[0045] S302: Based on the current window features, current discrimination information is obtained; wherein the current discrimination information includes current scores matching each candidate language.

[0046] In one embodiment, the obtained current window features are input into a trained language discrimination model so that the language discrimination model outputs a current score matching each candidate language, and each language and its matching current score are used as current discrimination information.

[0047] S303: Determine a first weight that matches the current discrimination information and a second weight that matches the historical discrimination information, obtain a target score for matching the current window feature with each candidate language based on the first weight, the second weight, the historical discrimination information and the current discrimination information, and determine the current language based on the target score.

[0048] In one embodiment, a predetermined first weight and a second weight are obtained. For each candidate language, a first product of a current score and the first weight, and a second product of a historical score and the second weight are obtained, and the sum of the first product and the second product is used as a target score for matching the candidate language.

[0049] Specifically, for any candidate language, the specific calculation formula for its corresponding target score is as follows:

[0050] in, Indicates candidate languages The corresponding target score, represents the first weight, Indicates candidate languages The corresponding current score, represents the second weight, Indicates candidate languages The corresponding historical score.

[0051] The above scheme improves the accuracy of current language determination by combining current discrimination information with historical discrimination information.

[0052] See also Figure 4 , Figure 4 yes Figure 1Step S102 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S102 includes: S401: Inputting the initial audio features and the current language into the trained semantic segmentation model to obtain semantic information corresponding to the initial audio features.

[0053] In one embodiment, a trained semantic segmentation model is obtained, and the initial audio features and the determined current language are input into the semantic segmentation model, so that the semantic segmentation model is used to perform real-time semantic analysis on the initial audio features, obtain corresponding semantic information in real time, and determine the correlation between the corresponding semantic information at different times.

[0054] S402: In response to the semantic information satisfying a preset segmentation condition, assigning a current segmentation flag to the initial audio feature; wherein the preset segmentation condition is related to the completeness of the semantic information.

[0055] In one embodiment, it is determined whether the correlation between the semantic information obtained at the current moment and the semantic information corresponding to the previous moment is greater than a correlation threshold. If it is less than the correlation threshold, it indicates that the correlation between the semantic information before the current moment and the voice information at and after the current moment is small, and it is determined that the preset segmentation condition is met, and the initial audio feature corresponding to the current moment is assigned to the current segmentation identifier.

[0056] S403: Determine a target feature segment corresponding to the current conversion round based on the historical segmentation identifier corresponding to the previous conversion round and the current segmentation identifier.

[0057] In one embodiment, the target feature segment corresponding to the current conversion round is determined based on the historical segmentation identifier corresponding to the previous conversion round and the current segmentation identifier; that is, the initial audio feature between the historical segmentation identifier corresponding to the previous conversion round and the current segmentation identifier is used as the target feature segment.

[0058] The above scheme obtains target feature fragments with complete semantic information through a semantic segmentation model to improve the accuracy of subsequent translation based on the target feature fragments.

[0059] See also Figure 5 , Figure 5 yes Figure 1 Step S103 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S103 includes: S501: Based on the target feature segment, a corresponding speech-level representation is obtained using an intelligent analysis model; wherein the speech-level representation includes at least one of language information corresponding to the current language, timbre information matching the target object, rhythm information, and speech speed information.

[0060] In one embodiment, the obtained target feature segment is input into the intelligent analysis model to obtain the corresponding speech-level representation using the intelligent analysis model. The speech-level representation includes language information corresponding to the current language, timbre information, prosody information and speech rate information matching the target object. Alternatively, the speech-level representation may also include at least one of the above-mentioned language information, timbre information, prosody information and speech rate information.

[0061] In one implementation scenario, in response to the target feature segment being an audio feature, before the target feature segment is input into the intelligent analysis model, the target feature segment is processed using a feature adapter so that the intelligent analysis model has the ability to process the target feature segment.

[0062] In another embodiment, the initial audio features and target feature segments are input into an intelligent analysis model, so that the intelligent analysis model is used to combine the initial audio features and the target feature segments to obtain corresponding speech-level representations, thereby avoiding the poor effect of speech-level representation extraction caused by the target feature segments being too short.

[0063] In another embodiment, reference features are obtained based on target feature segments corresponding to at least one historical conversion round and the current conversion round, respectively, and the reference features are input into an intelligent analysis model to obtain a speech-level representation.

[0064] Specifically, the target feature segment corresponding to the current conversion round is fused with the target feature segment corresponding to at least one historical conversion round to obtain a reference feature. The reference feature is processed by a feature adapter and then input into the intelligent analysis model to obtain a speech-level representation.

[0065] In a specific application scenario, in order to improve the efficiency of acquiring speech-level representations, the target feature fragments corresponding to the current conversion round and the historical conversion rounds adjacent to the current conversion round are fused to obtain reference features, and an intelligent analysis model is used to obtain speech-level representations based on the reference features.

[0066] S502: Based on the speech-level representation and the converted language, a corresponding converted audio is generated using an intelligent analysis model.

[0067] In one embodiment, a predetermined task template is obtained, and a corresponding task text is generated according to the speech-level representation and the conversion language. The task text is input into the intelligent analysis model so that the intelligent analysis model generates the corresponding conversion audio. The task text is used to prompt the intelligent analysis model to convert the target feature segment into text, and translate it into a translation text corresponding to the conversion language, and generate the corresponding conversion audio in combination with the speech-level representation.

[0068] Specifically, the predetermined task template is "You are an interpreter, and you have the ability to translate in real time. After you input <speech-level representation> and <target feature segment>, please output the translation text and conversion audio corresponding to each converted language of the translation, in the following format <conversion language 1><translation text corresponding to conversion language 1><conversion audio corresponding to conversion language 1>, <conversion language 2><translation text corresponding to conversion language 2><conversion audio corresponding to conversion language 2>...". Fill in the determined speech-level representation, target feature segment, and specific types of conversion languages ​​into the corresponding positions in the above task template to obtain the task text. The task text is input into the intelligent analysis model so that the intelligent analysis model outputs the conversion audio corresponding to each conversion text, and the conversion audio has the same timbre as the initial audio stream.

[0069] S503: broadcast the converted audio.

[0070] In one embodiment, in response to generating a conversion audio matching each conversion language, the conversion audio is broadcasted.

[0071] See also Figure 6 , Figure 6 yes Figure 5 Step S502 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S502 includes: S601: Acquire a conversion language representation that matches the conversion language; wherein the conversion language representation at least includes pronunciation information corresponding to the conversion language.

[0072] In one embodiment, a conversion language representation matching the determined conversion language is obtained, and the conversion language representation at least includes pronunciation information corresponding to the conversion language, and the pronunciation information at least includes pronunciation habits.

[0073] In one implementation scenario, the above-mentioned conversion language representation can be predetermined by using an intelligent analysis model.

[0074] S602: Generate a task text based on the speech-level representation, the target feature segment, the conversion language and the conversion language representation; wherein the task text is used to prompt the intelligent analysis model to generate a conversion audio that matches the conversion language.

[0075] In one embodiment, a predetermined task template is obtained, and a corresponding task text is generated according to the speech-level representation, the target feature segment, the conversion language, and the conversion language representation.

[0076] Specifically, the predetermined task template is "You are an interpreter, and you have the ability to perform real-time translation using the same intonation as the input speech. After you input <speech-level representation>, <target feature segment>, <conversion language>, and <conversion language representation>, please output the translation text and conversion audio corresponding to each conversion language of the translation, in the following format <conversion language 1><translation text corresponding to conversion language 1><conversion audio corresponding to conversion language 1>, <conversion language 2><translation text corresponding to conversion language 2><conversion audio corresponding to conversion language 2>...". The determined speech-level representation, target feature segment, conversion language, and conversion language representation are filled into the corresponding positions in the above task template to obtain the task text.

[0077] S603: Input the task text into the intelligent analysis model to obtain the converted audio generated by the intelligent analysis model.

[0078] In one embodiment, the task text is input into the intelligent analysis model, so that the intelligent analysis model generates corresponding converted audio after analyzing the task text.

[0079] The above scheme combines the conversion language representation corresponding to the conversion language when generating the conversion audio using the intelligent analysis model, so that the final conversion audio meets the pronunciation habits of the corresponding conversion language, thereby improving the user experience of the relevant objects.

[0080] The present application proposes an audio language conversion device, which is applied to a target scene to implement the audio language conversion method mentioned in any of the above-mentioned embodiments. The audio language conversion device stores the speech detection model, speech coding model, language discrimination model, semantic segmentation model and intelligent analysis model mentioned in the above-mentioned corresponding embodiments, and the specific structures of the above-mentioned speech detection model, speech coding model, language discrimination model and semantic segmentation model can refer to the existing neural network model structure. In addition, before using the audio language conversion device to perform audio language conversion, multiple training samples are obtained, and the training samples are used to jointly train the above-mentioned models until the preset stop conditions are met. Among them, the preset stop conditions are also related to the rounds of training or the degree of model convergence.

[0081] See also Figure 7 , Figure 7 1 is a schematic diagram of the structure of an audio language conversion system according to an embodiment of the present invention. The audio language conversion system comprises an acquisition module 10, a processing module 20 and a conversion module 30 which are coupled to each other.

[0082] The acquisition module 10 is used to acquire the initial audio stream of the target object, determine the initial audio features corresponding to the initial audio stream, and the current language corresponding to the initial audio stream.

[0083] The processing module 20 is used to obtain the target feature segments corresponding to the current conversion round based on the initial audio features and the current language; wherein the target feature segments corresponding to different conversion rounds are segmented based on the semantics of the initial audio features.

[0084] The conversion module 30 is used to determine at least one conversion language, generate conversion audio matching the conversion language based on the current language and the target feature segment, and broadcast the conversion audio.

[0085] In one embodiment, the acquisition module 10 acquires the initial audio stream of the target object, determines the initial audio features corresponding to the initial audio stream, and the current language corresponding to the initial audio stream, including: acquiring the initial audio stream of the target object, determining the initial audio features corresponding to the initial audio stream; acquiring the current window features corresponding to a preset duration based on the current time and the initial audio features; and determining the current language from multiple candidate languages ​​based on the current window features.

[0086] In one embodiment, the acquisition module 10 determines the current language from multiple candidate languages ​​based on the current window feature, including: acquiring historical window features adjacent to the current window feature and historical judgment information corresponding to the historical window features; wherein the historical judgment information includes historical scores matching each of the candidate languages; acquiring current judgment information based on the current window feature; wherein the current judgment information includes current scores matching each of the candidate languages; determining a first weight matching the current judgment information and a second weight matching the historical judgment information, acquiring a target score for matching the current window feature with each of the candidate languages ​​based on the first weight, the second weight, the historical judgment information and the current judgment information, and determining the current language based on the target score.

[0087] In one embodiment, the processing module 20 obtains the target feature segment corresponding to the current conversion round based on the initial audio feature and the current language, including: inputting the initial audio feature and the current language into a trained semantic segmentation model to obtain semantic information corresponding to the initial audio feature; in response to the semantic information satisfying a preset segmentation condition, assigning a current segmentation identifier to the initial audio feature; wherein the preset segmentation condition is related to the completeness of the semantic information; and determining the target feature segment corresponding to the current conversion round based on the historical segmentation identifier corresponding to the previous conversion round and the current segmentation identifier.

[0088] In one embodiment, the conversion module 30 generates a conversion audio matching the conversion language based on the current language and the target feature segment, including: based on the target feature segment, using an intelligent analysis model to obtain a corresponding speech-level representation; wherein the speech-level representation includes at least one of the language information corresponding to the current language, timbre information, prosody information and speech speed information matching the target object; based on the speech-level representation and the conversion language, using the intelligent analysis model to generate the corresponding conversion audio; and broadcasting the conversion audio.

[0089] In one embodiment, the conversion module 30 obtains the corresponding speech-level representation based on the target feature segment using an intelligent analysis model, including: obtaining reference features based on target feature segments corresponding to at least one historical conversion round and a current conversion round respectively; inputting the reference features into the intelligent analysis model to obtain the speech-level representation.

[0090] In one embodiment, the conversion module 30 generates the corresponding conversion audio based on the speech-level representation and the conversion language using the intelligent analysis model, including: obtaining a conversion language representation that matches the conversion language; wherein the conversion language representation at least includes pronunciation information corresponding to the conversion language; generating a task text based on the speech-level representation, the target feature segment, the conversion language and the conversion language representation; wherein the task text is used to prompt the intelligent analysis model to generate the conversion audio that matches the conversion language; and inputting the task text into the intelligent analysis model to obtain the conversion audio generated by the intelligent analysis model.

[0091] In one embodiment, the acquisition module 10 acquires the initial audio stream of the target object and determines the initial audio features corresponding to the initial audio stream, including: acquiring scene information matching the target scene; wherein the scene information includes scene video data and scene audio data; based on the scene information, determining at least one of the target objects in the target scene and the initial audio stream corresponding to the target object; and inputting the initial audio stream into a trained speech coding model to obtain the initial audio features.

[0092] See also Figure 8 , Figure 8: is a structural diagram of an embodiment of an electronic device of the present application. The electronic device comprises: a memory 40 and a processor 50 coupled to each other. The memory 40 stores program instructions, and the processor 50 is used to execute the program instructions to implement the method mentioned in any of the above embodiments. Specifically, the electronic device includes but is not limited to: a desktop computer, a laptop computer, a tablet computer, a server, etc., which are not limited here. In addition, the processor 50 can also be called a CPU (Center Processing Unit). The processor 50 may be an integrated circuit chip with signal processing capabilities. The processor 50 may also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 50 may be implemented by an integrated circuit chip.

[0093] See also Fig. 9 , Fig. 9 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 60 stores program instructions 70 that can be executed by a processor. When the program instructions 70 are executed by the processor, the method mentioned in any of the above embodiments is implemented.

[0094] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0095] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0096] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0097] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0098] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for converting audio language, characterized in that: include: Acquire an initial audio stream of a target object, and determine initial audio features corresponding to the initial audio stream and a current language corresponding to the initial audio stream; Based on the initial audio features and the current language, obtaining a target feature segment corresponding to the current conversion round; wherein the target feature segments corresponding to different conversion rounds are segmented based on the semantics of the initial audio features; At least one conversion language is determined, and based on the current language and the target feature segment, a conversion audio matching the conversion language is generated.

2. The method for converting audio language according to claim 1, characterized in that: The acquiring the initial audio stream of the target object and determining the initial audio features corresponding to the initial audio stream and the current language corresponding to the initial audio stream include: Acquire the initial audio stream of the target object, and determine the initial audio feature corresponding to the initial audio stream; Based on the current time and the initial audio feature, obtaining a current window feature corresponding to a preset duration; The current language is determined from a plurality of candidate languages ​​based on the current window feature.

3. The method for converting audio language according to claim 2, characterized in that: The determining the current language from a plurality of candidate languages ​​based on the current window feature includes: Acquire historical window features adjacent to the current window features and historical discrimination information corresponding to the historical window features; wherein the historical discrimination information includes historical scores matching each of the candidate languages; Based on the current window features, current discrimination information is obtained; wherein the current discrimination information includes a current score matching each of the candidate languages; Determine a first weight that matches the current discrimination information and a second weight that matches the historical discrimination information, obtain a target score for matching the current window feature with each of the candidate languages ​​based on the first weight, the second weight, the historical discrimination information and the current discrimination information, and determine the current language based on the target score.

4. The method for converting audio language according to claim 1, characterized in that: The acquiring, based on the initial audio features and the current language, a target feature segment corresponding to the current conversion round includes: Inputting the initial audio features and the current language into a trained semantic segmentation model to obtain semantic information corresponding to the initial audio features; In response to the semantic information satisfying a preset segmentation condition, assigning a current segmentation flag to the initial audio feature; wherein the preset segmentation condition is related to the completeness of the semantic information; The target feature segment corresponding to the current conversion round is determined based on the historical segmentation identifier corresponding to the previous conversion round and the current segmentation identifier.

5. The method for converting audio language according to claim 1, characterized in that: The step of generating a conversion audio that matches the conversion language based on the current language and the target feature segment includes: Based on the target feature segment, a corresponding speech-level representation is obtained using an intelligent analysis model; wherein the speech-level representation includes at least one of language information corresponding to the current language, timbre information, prosody information, and speech rate information matching the target object; Based on the speech-level representation and the conversion language, using the intelligent analysis model to generate the corresponding conversion audio; The converted audio is broadcasted.

6. The method for audio language conversion according to claim 5, characterized in that: The step of obtaining a corresponding speech-level representation based on the target feature segment by using an intelligent analysis model includes: Acquire reference features based on target feature fragments corresponding to at least one historical conversion round and the current conversion round respectively; The reference features are input into the intelligent analysis model to obtain the speech-level representation.

7. The method for converting audio language according to claim 5, characterized in that: The step of generating the corresponding converted audio based on the speech-level representation and the converted language using the intelligent analysis model includes: Acquire a conversion language representation that matches the conversion language; wherein the conversion language representation at least includes pronunciation information corresponding to the conversion language; Generate a task text based on the speech-level representation, the target feature segment, the conversion language and the conversion language representation; wherein the task text is used to prompt the intelligent analysis model to generate the conversion audio matching the conversion language; The task text is input into the intelligent analysis model to obtain the converted audio generated by the intelligent analysis model.

8. The method for converting audio language according to claim 2, characterized in that: The acquiring the initial audio stream of the target object and determining the initial audio feature corresponding to the initial audio stream includes: Acquire scene information matching the target scene; wherein the scene information includes scene video data and scene audio data; Based on the scene information, determining at least one target object in the target scene and the initial audio stream corresponding to the target object; The initial audio stream is input into the trained speech coding model to obtain the initial audio features.

9. An audio language conversion system, characterized in that: include: An acquisition module, used to acquire an initial audio stream of a target object, determine an initial audio feature corresponding to the initial audio stream, and a current language corresponding to the initial audio stream; A processing module, configured to obtain a target feature segment corresponding to a current conversion round based on the initial audio feature and the current language; wherein the target feature segments corresponding to different conversion rounds are segmented based on the semantics of the initial audio feature; The conversion module is used to determine at least one conversion language, generate conversion audio matching the conversion language based on the current language and the target feature segment, and broadcast the conversion audio.

10. An electronic device, characterized in that: include: A memory and a processor are coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the audio language conversion method as described in any one of claims 1-8.

11. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the audio language conversion method according to any one of claims 1 to 8 is implemented.

Images