Audio broadcast adjusting method, system and related device

Abstract

The invention discloses an audio broadcast adjusting method and system and a related device, and the method comprises the steps: obtaining an initial audio of a target object, and determining a translated text corresponding to the initial audio; obtaining a first reference duration corresponding to the initial audio and a second reference duration corresponding to the translated text; and generating an adjusted converted audio based on the translated text, the first reference duration and the second reference duration. By means of the mode, the flexibility of audio broadcasting can be improved, and the user experience can be improved.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio broadcast adjustment method, system and related devices. Background Art

[0002] With the continuous development of artificial intelligence, real-time audio language conversion is being applied in more and more scenarios. For example, in large-scale conferences or lectures, there is a language barrier between speakers and participants, which makes it difficult for participants to accurately understand what the speakers are saying. The current language conversion method mainly relies on neural network models to translate word by word based on the input audio stream. Due to the differences in expressions in different languages, neural network models are prone to asynchronous sentences during translation, resulting in a poor user experience.

[0003] In view of this, how to improve the flexibility of audio broadcasting has become an issue that needs to be addressed urgently. Summary of the invention

[0004] The main technical problem solved by the present application is to provide an audio broadcast adjustment method, system and related devices, which can improve the flexibility of audio broadcast and enhance the user experience.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: to provide an audio broadcast adjustment method, including: obtaining the initial audio of the target object, determining the translated text corresponding to the initial audio; obtaining a first reference duration corresponding to the initial audio and a second reference duration corresponding to the translated text; based on the translated text, the first reference duration and the second reference duration, generating an adjusted converted audio.

[0006] To solve the above technical problems, another technical solution adopted in the present application is: to provide an audio broadcast adjustment system, including: a first acquisition module, used to acquire the initial audio of the target object and determine the translated text corresponding to the initial audio; a second acquisition module, used to acquire a first reference duration corresponding to the initial audio and a second reference duration corresponding to the translated text; a processing module, used to generate an adjusted converted audio based on the translated text, the first reference duration and the second reference duration.

[0007] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising: a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the audio broadcast adjustment method mentioned in the above technical solution.

[0008] In order to solve the above technical problems, another technical solution adopted in the present application is: providing a computer-readable storage medium on which program instructions are stored, and when the program instructions are executed by a processor, the method for adjusting the audio broadcast mentioned in the above technical solution is implemented.

[0009] The beneficial effects of the present application are as follows: Different from the prior art, the method for adjusting audio broadcasts proposed in the present application determines the corresponding translation text according to the initial audio of the target object, and predicts the second reference duration required for broadcasting the corresponding conversion audio, thereby determining the corresponding target adjustment strategy according to the second reference duration. The conversion audio is generated according to the target adjustment strategy and the translation text, so that the final generated conversion audio is highly synchronized with the expression of the target object during broadcasting, thereby avoiding a backlog of broadcast content during real-time translation and improving the user experience of different target objects in the target scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0011] Figure 1 It is a flowchart of an implementation method of an audio broadcast adjustment method of the present application;

[0012] Figure 2 yes Figure 1 Step S101 corresponds to a flow chart of another implementation method;

[0013] Figure 3 yes Figure 1 Step S102 corresponds to a flow chart of another implementation method;

[0014] Figure 4 yes Figure 1 Step S103 corresponds to a flow chart of another implementation method;

[0015] Figure 5 yes Figure 1 Step S103 corresponds to a flow chart of another implementation method;

[0016] Figure 6 It is a structural schematic diagram of an implementation method of an adjustment system for audio broadcasting of the present application;

[0017] Figure 7 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0018] Figure 8It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments, and different embodiments can be adaptively combined. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of a method for adjusting audio broadcast of the present application, the method comprising:

[0021] S101: Acquire the initial audio of the target object and determine the translation text corresponding to the initial audio.

[0022] In one embodiment, the target object and the initial audio generated when the target object expresses are determined, and the initial audio is recognized and processed to obtain a corresponding transcribed text.

[0023] Furthermore, at least one conversion language is determined, and the transcribed text is translated into a translation text matching the conversion language.

[0024] In another embodiment, in order to improve the accuracy of obtaining the transcribed text, after obtaining the initial audio of the target object, the initial audio is subjected to noise reduction processing to remove environmental noise. The initial audio after noise removal is recognized to obtain the corresponding transcribed text.

[0025] S102: Obtain a first reference duration corresponding to the initial audio and a second reference duration corresponding to the translated text.

[0026] In one implementation, a first reference duration corresponding to the original audio is obtained, and a second reference duration corresponding to the translated text is determined based on text elements included in the translated text.

[0027] Specifically, the first reference duration is the duration corresponding to the initial audio. By obtaining the first reference duration and the second reference duration, it is helpful to subsequently generate the converted audio according to the comparison relationship between the first reference duration and the second reference duration.

[0028] S103: Generate adjusted converted audio based on the translated text, the first reference duration, and the second reference duration.

[0029] In one implementation, a target adjustment strategy is determined based on the first reference duration and the second reference duration.

[0030] Specifically, a comparison relationship between the first reference duration and the second reference duration is obtained, and a target adjustment strategy is determined according to the comparison relationship. The target adjustment strategy is used to adjust at least one of the speech rate of the generated conversion audio and the text length of the translated text.

[0031] Furthermore, based on the target adjustment strategy and the generated translation text, an adjusted conversion audio matching each conversion language is generated, and the conversion audio finally generated has a high synchronization with the initial audio of the expression of the target object.

[0032] Specifically, the fine-tuned intelligent analysis model is used to analyze the translation text and the target adjustment strategy, and prompts the intelligent analysis model to generate a conversion audio that matches each conversion language. The above-mentioned intelligent analysis model is a large language model with better data analysis capabilities. The translation text and the target adjustment strategy are input into the intelligent analysis model so that the intelligent analysis model generates a conversion audio that matches the conversion language.

[0033] In a specific application scenario, the above-mentioned large language model may include but is not limited to deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTM), and generative pre-trained Transformer models, etc., and no specific restrictions are made on the specific structure and specific deployment of the large language model. In addition, it should be noted that the specific structure and specific deployment of the intelligent analysis model mentioned in other implementations of this application can refer to this implementation.

[0034] In another embodiment, in order to improve the effect of the intelligent analysis model in generating converted audio, the intelligent analysis model is fine-tuned in advance using training samples.

[0035] The method for adjusting audio broadcast proposed in the present application determines the corresponding translation text according to the initial audio of the target object, and predicts the second reference duration required for broadcasting the corresponding conversion audio, thereby determining the corresponding target adjustment strategy according to the second reference duration. The conversion audio is generated according to the target adjustment strategy and the translation text, so that the final generated conversion audio is highly synchronized with the expression of the target object during broadcasting, thereby avoiding a backlog of broadcast content during real-time translation and improving the user experience of different target objects in the target scene.

[0036] See also Figure 2 , Figure 2 yes Figure 1 Step S101 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S101 includes:

[0037] S201: Acquire the initial audio of the target object, and determine the initial audio features corresponding to the initial audio and the current language corresponding to the initial audio.

[0038] In one embodiment, in response to the audio language conversion method proposed in the present application being applied to a target scene including at least one target object, scene audio data in the target scene is acquired in real time. The scene audio data is input into a trained speech detection model so that the speech detection model extracts initial audio matching each target object from the scene audio data based on timbre differences. The initial audio is input into a trained speech coding model to obtain initial audio features.

[0039] Furthermore, the initial audio features are input into the trained language discrimination model so that the language discrimination model outputs reference scores of the initial audio features matching each candidate language. The candidate language corresponding to the maximum reference score is used as the current language. The specific structures of the above-mentioned speech detection model and language discrimination model can refer to the existing neural network model structure.

[0040] S202: Based on the initial audio features and the current language, a target feature segment corresponding to the current conversion round is obtained; wherein the target feature segments corresponding to different conversion rounds are segmented based on the semantics of the initial audio features.

[0041] In one embodiment, the initial audio features are semantically analyzed and segmented based on the obtained initial audio features and the current language, and the target feature segments corresponding to the current conversion round are determined. The target feature segments include initial audio feature segments whose semantic relevance is greater than the relevance threshold, that is, the semantic integrity of the target feature segments corresponding to the current conversion round is high, so that the translation based on the target feature segments does not need to rely on contextual semantics, which helps to improve the accuracy of real-time conversion of audio languages.

[0042] In one implementation scenario, a trained semantic segmentation model is obtained, and the initial audio features and the determined current language are input into the semantic segmentation model. The semantic segmentation model is used to perform real-time semantic analysis on the initial audio features, and the corresponding semantic information and the correlation between the semantic information corresponding to different times are obtained in real time. It is determined whether the correlation between the semantic information obtained at the current moment and the semantic information corresponding to the previous moment is greater than the correlation threshold. If it is less than the correlation threshold, it indicates that the correlation between the semantic information before the current moment and the voice information at the current moment and thereafter is small, then it is determined that the preset segmentation conditions are met, and the initial audio features corresponding to the current moment are assigned to the current segmentation identifier. According to the historical segmentation identifier corresponding to the previous conversion round and the current segmentation identifier, the target feature segment corresponding to the current conversion round is determined, that is, the initial audio features between the historical segmentation identifier corresponding to the previous conversion round and the current segmentation identifier are used as the target feature segment. Among them, the specific structure of the above-mentioned semantic segmentation model can refer to the existing neural network model structure.

[0043] The above scheme obtains target feature fragments with complete semantic information through a semantic segmentation model, so as to improve the subsequent translation based on the target feature fragments without relying on contextual semantics, thereby improving the accuracy of translation.

[0044] S203: Determine at least one conversion language, and obtain a transcribed text and a translation text matching the conversion language based on the current language and the target feature segment.

[0045] In one embodiment, at least one conversion language specified by a user is determined, and the current language, the target feature segment, and the conversion language are input into an intelligent analysis model so that the intelligent analysis model preferentially converts the target feature segment into a conversion text, and then translates the conversion text into a translation text corresponding to the conversion language.

[0046] Specifically, a first task template is determined in advance, for example: "You are a translator, and you have the ability to translate in real time. When you input <current language> and <target feature fragment>, please output the corresponding <conversion text>, and the translation text corresponding to each conversion language, in the following format <conversion language 1><translation text corresponding to conversion language 1>, <conversion language 2><translation text corresponding to conversion language 2>...". Fill the determined current language, target feature fragment and specific type of conversion language into the corresponding positions in the above-mentioned first task template to obtain the first task text. The first task text is input into the intelligent analysis model so that the intelligent analysis model outputs the transcribed text corresponding to the target feature fragment, and the translation text matching each conversion language.

[0047] In another embodiment, the conversion language is determined according to the first languages ​​of different objects in the target scene.

[0048] Specifically, the first language mastered by each object in the target scene is determined. For the target object in the target scene, the first language of other objects is used as the conversion language, so that after the audio language conversion, the other objects in the target scene can understand the expression content of the target object.

[0049] See also Figure 3 , Figure 3 yes Figure 1 Step S102 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S102 includes:

[0050] S301: Obtain a reference sub-duration corresponding to each first text element in the transcribed text, and obtain a first reference duration based on all reference sub-durations corresponding to the transcribed text.

[0051] In one embodiment, after obtaining the transcribed text, all first text elements contained in the transcribed text and the reference sub-duration corresponding to each first text element are determined. The sum of the reference sub-durations corresponding to all first text elements in the transcribed text is taken as the first reference duration. The first text element is the smallest constituent unit of the transcribed text. For example, when the transcribed text is Chinese text, each character is taken as the first text element; the reference sub-duration corresponding to the first text element represents the average duration required for different objects to express the first text element.

[0052] In one implementation scenario, a predetermined target database is obtained, which includes different text elements corresponding to multiple language types and reference sub-durations corresponding to each text element. After the transcribed text is obtained, a search is performed in the target database to determine the reference sub-duration corresponding to each first text element in the transcribed text.

[0053] Among them, the above-mentioned target database is determined based on a large amount of reference data, which includes the expression audios of multiple reference objects; for example, for the same text element in different expression audios, the duration of the corresponding reference object expressing the text element is obtained, and the reference sub-duration corresponding to the text element is obtained by taking the average value.

[0054] S302: Obtain a reference sub-duration corresponding to each second text element in the translated text, and obtain a second reference duration based on all reference sub-durations corresponding to the translated text.

[0055] In one embodiment, after obtaining the translated text, all second text elements included in the translated text are determined, and the reference sub-duration corresponding to each second text element is determined using the target database. The sum of the reference sub-durations corresponding to all second text elements in the translated text is used as the second reference duration.

[0056] It should be noted that the execution order of the above-mentioned step S301 and step S302 may also be other, for example, step S301 and step S302 may be executed at the same time; or, step S302 may be executed first and then step S301.

[0057] In another embodiment, before step S103, the method includes: inputting the initial audio into the trained representation extraction model, and using the representation extraction model to obtain a speech-level representation matching the initial audio. The speech-level representation includes at least one of timbre information, speech rate information, and prosody information matching the target object. In addition, the specific structure of the above-mentioned representation extraction model can refer to the existing neural network model structure.

[0058] Specifically, the initial audio features corresponding to the initial audio are obtained, and the initial audio features are input into the representation extraction model to obtain a speech-level representation, which includes timbre information, speech rate information and prosody information that match the target object. By obtaining the speech-level representation, the converted audio subsequently generated in combination with the speech-level representation has the same timbre, speech rate and prosody as the target object, that is, the function of simultaneous interpretation is realized. Of course, in other embodiments, the speech-level representation may also include at least one of the above-mentioned timbre information, speech rate information and prosody information.

[0059] In one implementation scenario, the above-mentioned representation extraction model may be an intelligent analysis model. Specifically, after obtaining the initial audio features corresponding to the initial audio, the initial audio features are converted and processed by a feature adapter and then input into the intelligent analysis model, and the intelligent analysis model is prompted to obtain the corresponding speech-level representation according to the initial audio features. Among them, the initial audio features in this implementation scenario are the initial audio features obtained in step S201.

[0060] Optionally, in other embodiments, in order to improve the model processing efficiency and make the speech-level representation more consistent with the current language expression state of the target object, the target feature segment is input into the trained representation extraction model to obtain a speech-level representation that is more consistent with the current context of the target object.

[0061] In another embodiment, the above-mentioned voice-level representation can also be a default or user-instructed one. For example, in response to the target object receiving the converted audio in the target scene, a target virtual image is selected from multiple candidate virtual images in advance, and the voice-level representation is obtained according to the timbre information, speech rate information and prosody information matching the target virtual image; or, in response to the target object receiving the converted audio not selecting multiple candidate virtual images, any candidate virtual image is used as the target virtual image, and the voice-level representation is obtained according to the timbre information, speech rate information and prosody information matching the target virtual image.

[0062] See also Figure 4 , Figure 4 yes Figure 1 Step S103 in the flowchart corresponds to another implementation. Specifically, in response to the first reference duration being less than the second reference duration, and the speech-level representation matching the target object is obtained through the above corresponding implementation, the implementation process of step S103 includes:

[0063] S401: Obtain a first time difference between a second reference duration and a first reference duration.

[0064] In one implementation, a difference between the second reference duration and the first reference duration is obtained, and the difference is used as the first time difference.

[0065] Further, a predetermined first interval, a second interval and a third interval are obtained, and the interval to which the obtained first time difference belongs is determined, wherein any value in the first interval is smaller than any value in the second interval, and any value in the second interval is smaller than any value in the third interval.

[0066] In a specific application scenario, the first interval is (0, 2], the second interval is (2, 5], and the third interval is (5, ∞], and the unit of each value in the above intervals is seconds.

[0067] S402: In response to the first time difference being in the first interval, generating a speech speed increase instruction, and generating a converted audio based on the speech speed increase instruction, the translation text, and the speech-level representation.

[0068] In one embodiment, when the first time difference is within the first interval, it indicates that the predicted converted audio consumes slightly more time when broadcasting than the time of the initial audio corresponding to the target object's expression, thereby causing the expression of the target object to be out of sync with the translated broadcast, and easily causing a backlog of broadcast content after the target object's expression is completed. In order to improve the synchronization rate between the expression of the target object and the translated broadcast, and to improve the regulation efficiency, a speech speed-up instruction is generated, and the corresponding converted audio is generated according to the speech speed-up instruction, the translated text, and the voice-level representation.

[0069] In one implementation scenario, the generated speed-up instruction includes a speed-up multiple, and the value of the speed-up multiple is proportional to the first time difference, that is, the larger the first time difference, the larger the speed-up multiple. A predetermined second task template is obtained, and a corresponding second task text is generated according to the speed-up instruction, the translation text, the speech-level representation, and the conversion language. The second task text is input into the intelligent analysis model so that the intelligent analysis model generates the corresponding conversion audio.

[0070] In a specific application scenario, the predetermined second task template is "You are an interpreter and you have the ability to translate in real time. After you input <speech-level representation> and <translation text>, please output the conversion audio corresponding to each converted language of the translation according to <speech speed increase multiple>, in the following format <conversion language 1><conversion audio corresponding to conversion language 1>, <conversion language 2><conversion audio corresponding to conversion language 2>...". Fill the determined speech-level representation, translation text and specific types of conversion languages ​​into the corresponding positions in the above-mentioned second task template to obtain the second task text. The second task text is input into the intelligent analysis model so that the intelligent analysis model outputs the conversion audio corresponding to each conversion text.

[0071] Furthermore, the generated converted audio is broadcasted, wherein the timbre information of the converted audio is consistent with the timbre information of the target object, and the broadcasting speed of the converted audio is the product of the speed information of the target object when expressing the corresponding initial audio and the speed acceleration multiple.

[0072] S403: In response to the first time difference being in the second interval, determining redundant text elements in the translated text, and generating converted audio based on the translated text and the speech-level representation after deleting the redundant text elements.

[0073] In one embodiment, when the first time difference is within the second interval, in order to avoid a backlog of broadcast content during real-time translation, the translated text is input into a trained text simplification model, which is used to identify the part of speech of each text element in the input text, thereby determining redundant text elements in the translated text according to the part of speech; for example, repeated words, modal particles and decorative words in the text are regarded as redundant text elements. The redundant elements in the translated text are deleted to simplify the original translated text without changing the semantics of the original translated text.

[0074] Furthermore, a corresponding task text is generated according to the simplified translation text, the speech-level representation and the conversion language, and the task text is input into the intelligent analysis model so that the intelligent analysis model generates the corresponding conversion audio.

[0075] In another embodiment, after the redundant text elements in the translated text are determined using the text simplification model, at least some of the redundant text elements are deleted according to the first time difference determined above and the reference sub-durations corresponding to each redundant text element, so that the absolute value of the difference between the second reference duration corresponding to the translated text after the redundant text elements are deleted and the first reference duration is less than the duration threshold. This method deletes at least some of the redundant text elements according to the first time difference, thereby avoiding the situation where the final generated converted audio is too short after all the redundant text elements are deleted.

[0076] S404: In response to the first time difference being in the third interval, obtaining a summary text corresponding to the translated text, and generating a converted audio based on the summary text and the speech-level representation.

[0077] In one embodiment, when the first time difference is within the third interval, in order to avoid a large backlog of broadcast content during real-time translation, the translated text is input into a trained summary generation model, so that the summary generation model is used to summarize the translated text and output a corresponding summary text.

[0078] Furthermore, a corresponding task text is generated according to the generated summary text, the speech-level representation and the conversion language, and the task text is input into the intelligent analysis model so that the intelligent analysis model generates the corresponding conversion audio.

[0079] In another embodiment, the summary generation model may also be an intelligent analysis model, by inputting the translation text into the intelligent analysis model and prompting the intelligent analysis model to summarize the translation text, thereby obtaining a summary text with text elements within a target number range. The intelligent analysis model is used to generate a corresponding task text based on the summary text, voice-level representation and conversion language. The target number range is estimated based on the first reference duration.

[0080] The above scheme, by setting different intervals, determines the corresponding adjustment method according to the specific value of the first time difference between the first reference duration and the second reference duration, thereby improving the adjustment flexibility.

[0081] See also Figure 5 , Figure 5 yes Figure 1 Step S103 in the flowchart corresponds to another implementation. Specifically, in response to the first reference duration being greater than the second reference duration, and the speech-level representation matching the target object is obtained through the above corresponding implementation, the implementation process of step S103 includes:

[0082] S501: Obtain a second time difference between a first reference duration and a second reference duration, and determine a target pause duration based on the second time difference.

[0083] In one implementation, a second time difference between the first reference duration and the second reference duration is obtained, and the second time difference is used as the target pause duration.

[0084] S502: Generate a converted audio based on the target pause duration, the translated text and the speech-level representation; wherein the converted audio includes a silent gap of the target pause duration, and the position of the silent gap in the converted audio is determined based on the semantics of the corresponding translated text.

[0085] In one embodiment, an intelligent analysis model is used to generate corresponding conversion audio according to the target pause duration, the translation text and the speech-level representation. In addition, in the process of generating the conversion audio, the translation text is semantically analyzed to determine the semantic pause position in the translation text, and a silent gap matching the semantic pause position is added to the conversion audio. The semantic pause position is determined based on at least one of the component relationship between adjacent second text elements in the translation text and the degree of sentence completeness.

[0086] In a specific application scenario, in response to the translated text being a complete sentence, a blank audio segment is added at the end of the generated converted audio as a silent gap.

[0087] In another embodiment, in response to the translation text including multiple pause positions, when generating the converted audio, a silent gap is added at each pause position, and the total duration of all silent gaps in the converted audio corresponding to the translation text is made to be the target pause duration.

[0088] In the above scheme, when the first reference duration is greater than the second reference duration, it indicates that the time consumed by the predicted converted audio during broadcasting is less than the duration of the initial audio corresponding to the target object expression. Silent gaps are added to the generated converted audio according to semantics, thereby improving the fluency of the converted audio and enhancing the expression effect.

[0089] In another embodiment, in response to the first reference duration being greater than the second reference duration, and a speech-level representation matching the target object is obtained through the above corresponding embodiments, the implementation process of step S103 may further include: obtaining a second time difference between the first reference duration and the second reference duration.

[0090] In response to the second time difference being within the first interval, a speech rate reduction instruction is generated, and a converted audio is generated based on the speech rate reduction instruction, the translated text, and the speech-level representation.

[0091] Specifically, the generated speech speed slowing instruction includes a speech speed slowing multiple, such as 0.9, 0.8 or 0.75, etc. And the numerical value of the speech speed slowing multiple is inversely proportional to the second time difference, that is, the larger the second time difference, the smaller the speech speed slowing multiple. According to the speech speed slowing instruction, the translation text, the voice-level representation and the conversion language, the corresponding conversion audio is generated using the intelligent analysis model. Among them, the specific implementation process can refer to the corresponding implementation method mentioned above.

[0092] Alternatively, in response to the second time difference being in the second interval, a text modification element matching the translated text is determined, and the text modification element is added to a corresponding position in the translated text. Based on the modified translated text and the speech-level representation, a converted audio is generated.

[0093] Specifically, the translation text is input into the intelligent analysis model, at least part of the text modification elements are generated by the intelligent analysis model, and the text modification elements are added to the corresponding positions in the translation text. According to the modified translation text and the speech-level representation, the conversion audio is generated by the intelligent analysis model. Among them, the above-mentioned text modification elements include adjectives and modal particles, etc.

[0094] Alternatively, in response to the second time difference being in the third interval, a polished text corresponding to the translated text is obtained, and a converted audio is generated based on the polished text and the speech-level representation.

[0095] Specifically, the translated text is input into the intelligent analysis model, and the translated text is polished by the intelligent analysis model, so as to expand the number of second text elements in the original translated text. According to the polished text and speech-level representation obtained, the converted audio is generated by the intelligent analysis model.

[0096] See also Figure 6 , Figure 6 1 is a schematic diagram of the structure of an embodiment of the audio broadcast adjustment system of the present application. Specifically, the audio broadcast adjustment system includes a first acquisition module 10, a second acquisition module 20 and a processing module 30 coupled to each other.

[0097] The first acquisition module 10 is used to acquire the initial audio of the target object and determine the translation text corresponding to the initial audio.

[0098] The second acquisition module 20 is used to acquire a first reference duration corresponding to the initial audio and a second reference duration corresponding to the translated text.

[0099] The processing module 30 is used to generate an adjusted converted audio based on the translated text, the first reference duration and the second reference duration.

[0100] In one embodiment, the first acquisition module 10 acquires the initial audio of the target object and determines the translation text corresponding to the initial audio, including: acquiring the initial audio of the target object, determining the initial audio features corresponding to the initial audio and the current language corresponding to the initial audio; based on the initial audio features and the current language, acquiring the target feature segments corresponding to the current conversion round; wherein the target feature segments corresponding to different conversion rounds are segmented based on the semantics of the initial audio features; determining at least one conversion language, and based on the current language and the target feature segments, acquiring the transcribed text and the translation text matching the conversion language.

[0101] In one embodiment, the second acquisition module 20 acquires a first reference duration corresponding to the initial audio and a second reference duration corresponding to the translated text, including: acquiring a reference sub-duration corresponding to each first text element in the transcribed text, and obtaining a first reference duration based on all reference sub-durations corresponding to the transcribed text; and acquiring a reference sub-duration corresponding to each second text element in the translated text, and obtaining a second reference duration based on all reference sub-durations corresponding to the translated text.

[0102] In one embodiment, the processing module 30 generates an adjusted converted audio based on the translated text, the first reference duration, and the second reference duration, including: determining a target adjustment strategy based on the first reference duration and the second reference duration; wherein the target adjustment strategy is used to adjust at least one of the broadcast speed of the converted audio and the text length of the translated text; generating the converted audio based on the target adjustment strategy and the translated text.

[0103] In one embodiment, please continue to refer to Figure 6 The audio broadcast adjustment system proposed in the present application also includes a speech-level representation extraction module 40 coupled to the processing module 30. The speech-level representation extraction module 40 is used to input the initial audio into the trained representation extraction model, and use the representation extraction model to obtain a speech-level representation that matches the initial audio; wherein the speech-level representation includes at least one of timbre information, speech rate information, and prosody information that matches the target object. Based on this, the processing module 30 is used to generate the converted audio based on the target adjustment strategy, the translated text, and the speech-level representation.

[0104] In one embodiment, in response to the first reference duration being less than the second reference duration, the processing module 30 generates a conversion audio based on the target adjustment strategy, the translation text and the speech-level representation, including: obtaining a first time difference between the second reference duration and the first reference duration; in response to the first time difference being in the first interval, generating a speed-up instruction, and generating a conversion audio based on the speed-up instruction, the translation text and the speech-level representation; in response to the first time difference being in the second interval, determining redundant text elements in the translation text, and generating a conversion audio based on the translation text and the speech-level representation after deleting the redundant text elements; in response to the first time difference being in the third interval, obtaining a summary text corresponding to the translation text, and generating a conversion audio based on the summary text and the speech-level representation; wherein any numerical value in the first interval is less than any numerical value in the second interval, and any numerical value in the second interval is less than any numerical value in the third interval.

[0105] In one embodiment, in response to the first reference duration being greater than the second reference duration, the processing module 30 generates a converted audio based on the target adjustment strategy, the translated text, and the speech-level representation, including: obtaining a second time difference between the first reference duration and the second reference duration, and determining a target pause duration based on the second time difference; generating a converted audio based on the target pause duration, the translated text, and the speech-level representation; wherein the converted audio includes a silent gap of the target pause duration, and the position of the silent gap in the converted audio is determined based on the semantics of the corresponding translated text.

[0106] See also Figure 7 , Figure 7 : It is a structural diagram of an embodiment of an electronic device of the present application. The electronic device includes: a memory 50 and a processor 60 coupled to each other. The memory 50 stores program instructions, and the processor 60 is used to execute the program instructions to implement the method mentioned in any of the above embodiments. Specifically, the electronic device includes but is not limited to: a desktop computer, a laptop computer, a tablet computer, a server, etc., which are not limited here. In addition, the processor 60 can also be called a CPU (Center Processing Unit). The processor 60 may be an integrated circuit chip with signal processing capabilities. The processor 60 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 60 can be implemented by an integrated circuit chip.

[0107] See also Figure 8 , Figure 8 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 70 stores program instructions 80 that can be executed by a processor. When the program instructions 80 are executed by the processor, the method mentioned in any of the above embodiments is implemented.

[0108] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0109] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0110] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0111] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0112] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for adjusting audio broadcast, characterized in that: include: Acquire the original audio of the target object, and determine the translation text corresponding to the original audio; Obtaining a first reference duration corresponding to the initial audio and a second reference duration corresponding to the translated text; Based on the translated text, the first reference duration, and the second reference duration, an adjusted converted audio is generated.

2. The method for adjusting audio broadcast according to claim 1, characterized in that: The obtaining of the initial audio of the target object and determining the translation text corresponding to the initial audio includes: Acquire the initial audio of the target object, and determine the initial audio features corresponding to the initial audio and the current language corresponding to the initial audio; Based on the initial audio features and the current language, obtaining a target feature segment corresponding to the current conversion round; wherein the target feature segments corresponding to different conversion rounds are segmented based on the semantics of the initial audio features; At least one conversion language is determined, and based on the current language and the target feature segment, a transcribed text and the translated text matching the conversion language are acquired.

3. The method for adjusting audio broadcast according to claim 2, characterized in that: The obtaining of a first reference duration corresponding to the initial audio and a second reference duration corresponding to the translated text includes: Obtaining a reference sub-duration corresponding to each first text element in the transcribed text, and obtaining the first reference duration based on all reference sub-durations corresponding to the transcribed text; and A reference sub-duration corresponding to each second text element in the translated text is obtained, and the second reference duration is obtained based on all reference sub-durations corresponding to the translated text.

4. The method for adjusting audio broadcast according to claim 1, characterized in that: The step of generating an adjusted converted audio based on the translated text, the first reference duration, and the second reference duration includes: Determining a target adjustment strategy based on the first reference duration and the second reference duration; wherein the target adjustment strategy is used to adjust at least one of the broadcast speech rate of the converted audio and the text length of the translated text; The converted audio is generated based on the target adjustment strategy and the translated text.

5. The method for adjusting audio broadcast according to claim 4, characterized in that: Before generating the adjusted converted audio based on the translated text, the first reference duration, and the second reference duration, the method includes: Inputting the initial audio into a trained representation extraction model, and using the representation extraction model to obtain a speech-level representation matching the initial audio; wherein the speech-level representation includes at least one of timbre information, speech rate information, and prosody information matching the target object; The step of generating the converted audio based on the target adjustment strategy and the translated text includes: The converted audio is generated based on the target adjustment strategy, the translated text, and the speech-level representation.

6. The method for adjusting audio broadcast according to claim 5, characterized in that: In response to the first reference duration being less than the second reference duration, generating the converted audio based on the target adjustment strategy, the translated text, and the speech-level representation includes: Acquire a first time difference between the second reference duration and the first reference duration; In response to the first time difference being in a first interval, generating a speaking speed increase instruction, and generating the converted audio based on the speaking speed increase instruction, the translated text, and the speech-level representation; In response to the first time difference being in a second interval, determining redundant text elements in the translated text, and generating the converted audio based on the translated text after deleting the redundant text elements and the speech-level representation; In response to the first time difference being in a third interval, obtaining a summary text corresponding to the translated text, and generating the converted audio based on the summary text and the speech-level representation; Among them, any numerical value in the first interval is smaller than any numerical value in the second interval, and any numerical value in the second interval is smaller than any numerical value in the third interval.

7. The method for adjusting audio broadcast according to claim 5, characterized in that: In response to the first reference duration being greater than the second reference duration, generating the converted audio based on the target adjustment strategy, the translated text, and the speech-level representation includes: Acquire a second time difference between the first reference duration and the second reference duration, and determine a target pause duration based on the second time difference; The converted audio is generated based on the target pause duration, the translated text and the speech-level representation; wherein the converted audio includes a silent gap of the target pause duration, and the position of the silent gap in the converted audio is determined based on the semantics of the corresponding translated text.

8. An audio broadcast adjustment system, characterized in that: include: A first acquisition module, used to acquire the initial audio of the target object and determine the translation text corresponding to the initial audio; A second acquisition module, used to acquire a first reference duration corresponding to the initial audio and a second reference duration corresponding to the translated text; A processing module is used to generate an adjusted converted audio based on the translated text, the first reference duration and the second reference duration.

9. An electronic device, characterized in that: include: A memory and a processor are coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the audio broadcast adjustment method as described in any one of claims 1-7.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by the processor, the method for adjusting the audio broadcast as described in any one of claims 1 to 7 is implemented.

Images