Sound conversion method and device in network live broadcast scene, and electronic equipment
By obtaining and processing audio data in online live broadcasts, target audio data is generated based on the tone selected by the user, which solves the problem that the anchor's voice does not meet the needs and improves the live broadcast effect and retention rate.
Patent Information
- Application Number
- CN202310540529.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-12
AI Technical Summary
In the existing live broadcast scenarios, the anchor’s voice and tone may not meet the needs of the viewer, resulting in poor live broadcast effect and low retention rate.
By obtaining the audio data of the target live broadcast room, responding to the user's tone selection operation, determining the tone transformation model corresponding to the intended tone, processing the audio data, generating the target audio data that meets the user's needs, and sending it to the live broadcast data generation device. Generate live videos that meet user needs.
The live broadcast effect is improved, so that the voices of the anchor and the broadcast assist meet user needs, and the retention rate of the live broadcast room is improved.
Smart Images

Figure CN118945386A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of network live broadcast, and in particular to a sound conversion method and device, and electronic equipment in a network live broadcast scenario. Background Art
[0002] With the development of Internet technology, more and more people like to enrich their spare time by watching live broadcasts, and live broadcasts are becoming more and more popular among young people today.
[0003] However, in current live broadcast rooms, during live broadcasts, the timbre of some hosts' voices may not meet the needs of live viewers (for example, live viewers prefer female hosts, but the host is male), which will result in poor live broadcast effects and low retention rate of live viewers. Summary of the invention
[0004] In order to solve the above technical problems, the present application provides a sound conversion method and device, and an electronic device in a live broadcast scenario.
[0005] The technical solution of this application is as follows:
[0006] In a first aspect, the present application provides a sound transformation method in a network live broadcast scenario, which is applied to a content expression device in a live broadcast system. The live broadcast system also includes a live broadcast data generation device connected to the content expression device. In a possible implementation method of the first aspect, it includes: obtaining live audio data in a target live broadcast room; determining an intended sound transformation model corresponding to the intended timbre in response to a user's timbre selection operation; processing the live audio data based on the intended sound transformation model to obtain target audio data corresponding to the intended timbre; sending the target audio data to the live broadcast data generation device, so that the live broadcast data generation device generates live video data based on the target audio data for live broadcast.
[0007] In a possible implementation of the first aspect, after acquiring the live audio data in the target live broadcast room, the method also includes: determining an identifier of the intended timbre in response to the user's timbre selection operation; determining timbre feature data corresponding to the intended timbre based on the identifier of the intended timbre; acquiring semantic feature data and rhythmic feature data of the live audio data; fusing the semantic feature data, rhythmic feature data and timbre feature data to obtain target audio data; and sending the target audio data to a live broadcast data generating device so that the live broadcast data generating device generates live video data based on the target audio data for live broadcast.
[0008] In a possible implementation of the first aspect, determining the timbre feature data corresponding to the intended timbre includes: based on a timbre mapping network, determining the timbre feature data corresponding to the identification of the intended timbre; the timbre mapping network includes a correspondence between at least one identification of the timbre and at least one timbre feature data, and the identification of the at least one timbre includes the identification of the intended timbre.
[0009] In a possible implementation of the first aspect, after acquiring the live audio data in the target live broadcast room, the method also includes: determining the intended timbre variable in response to the user's timbre selection operation; processing the live audio data and the intended timbre variable based on the target sound transformation model to obtain the target audio data; and sending the target audio data to the live broadcast data generating device, so that the live broadcast data generating device generates live video data based on the target audio data for live broadcast.
[0010] In a possible implementation of the first aspect, after processing the live audio data based on the intentional sound transformation model to obtain the target audio data corresponding to the intended timbre, and before sending the target audio data to the live data generating device, the method also includes: adjusting the audio parameters in the target audio data according to preset rules to update the target audio data; wherein the audio parameters include any one or more of the following: pitch, loudness, bit rate.
[0011] In a possible implementation manner of the first aspect, the preset rule is derived from audio parameters of a reference audio corresponding to a specific live broadcast tag.
[0012] In a second aspect, the present application provides a sound conversion device, which is applied to a content expression device in a live broadcast system. The live broadcast system also includes a live broadcast data generation device connected to the content expression device. The device includes: an acquisition module, which is used to obtain live audio data in a target live broadcast room; a determination module, which is used to determine an intended sound conversion model corresponding to an intended timbre in response to a user's timbre selection operation; a processing module, which is used to process the live audio data obtained by the acquisition module based on the intended sound conversion model determined by the determination module to obtain target audio data corresponding to the intended timbre; a sending module, which is used to send the target audio data obtained by the processing module to the live broadcast data generation device, so that the live broadcast data generation device generates live video data based on the target audio data for live broadcast.
[0013] In a possible implementation of the second aspect, the determination module is also used to determine the identifier of the intended timbre in response to the user's timbre selection operation; the processing module is also used to determine the timbre feature data corresponding to the intended timbre based on the identifier of the intended timbre determined by the determination module; the processing module is also used to obtain the semantic feature data and rhythmic feature data of the live audio data obtained by the acquisition module; the processing module is also used to fuse the semantic feature data, rhythmic feature data and timbre feature data to obtain target audio data; the sending module is also used to send the target audio data to the live data generating device, so that the live data generating device generates live video data based on the target audio data for live broadcast.
[0014] In a possible implementation of the second aspect, the determination module is specifically used to: determine the timbre feature data corresponding to the identification of the intended timbre based on the timbre mapping network; the timbre mapping network includes a correspondence between at least one timbre identification and at least one timbre feature data, and the identification of at least one timbre includes the identification of the intended timbre.
[0015] In a possible implementation of the second aspect, the determination module is also used to determine the intended timbre variable in response to the user's timbre selection operation; the processing module is also used to process the live audio data acquired by the acquisition module and the intended timbre variable determined by the determination module based on the target sound transformation model to obtain the target audio data; the sending module is also used to send the target audio data to the live data generating device, so that the live data generating device generates live video data based on the target audio data for live broadcast.
[0016] In a possible implementation of the second aspect, the processing module processes the live audio data based on the intentional sound transformation model to obtain the target audio data corresponding to the intended timbre. Before the sending module sends the target audio data to the live data generating device, the processing module is also used to adjust the audio parameters in the target audio data according to preset rules to update the target audio data; wherein the audio parameters include any one or more of the following: pitch, loudness, bit rate.
[0017] In a possible implementation manner of the second aspect, the preset rule is derived from audio parameters of a reference audio corresponding to a specific live broadcast tag.
[0018] In a third aspect, the present application provides an electronic device, comprising a processor and a memory; the memory is used to store computer instructions, and when the electronic device is running, the processor executes the computer instructions stored in the memory to enable the electronic device to implement the live video generation method provided in the first aspect above.
[0019] In a fourth aspect, the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on an electronic device, the electronic device implements the sound conversion method in the live broadcast scenario as provided in the first aspect above.
[0020] In a fifth aspect, the present application also provides a computer program product comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device implements the sound conversion method in a live broadcast scenario as provided in the first aspect above.
[0021] In a sixth aspect, a device (for example, the device may be a chip system) is provided, the device including a processor for supporting an electronic device to implement the functions involved in the first aspect above. In one possible design, the device also includes a memory for storing program instructions and data necessary for the electronic device. When the device is a chip system, it may be composed of a chip, or may include a chip and other discrete devices.
[0022] It should be noted that the above computer instructions may be stored in whole or in part on a first computer-readable storage medium, wherein the first computer-readable storage medium may be packaged together with the processor of the data processing device, or may be packaged separately from the processor of the data processing device, which is not limited in the present disclosure.
[0023] In this application, the above names do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalent technologies.
[0024] The descriptions of the second to sixth aspects in the present disclosure can refer to the detailed description of the first aspect; and the beneficial effects of the descriptions of the second to sixth aspects can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.
[0025] These and other aspects of the present application will become more apparent from the following description.
[0026] In this application, the above names do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalent technologies.
[0027] These and other aspects of the present application will become more apparent from the following description.
[0028] In the technical solution provided by the present application, the live audio data in the target live broadcast room can be first obtained, and then, in response to the user's timbre selection operation, the intended sound transformation model corresponding to the intended timbre is determined. Afterwards, the live audio data can be converted into the target audio data corresponding to the intended timbre based on the intended sound transformation model. Finally, after sending the target audio data to the live broadcast data generation device, the live video data whose timbre meets the user's needs can be obtained and broadcast live. It can be seen that in this method, after receiving the user's timbre selection operation, the timbre in the live audio data can be converted into the intended timbre required by the user, thereby obtaining live video data that meets the user's timbre requirements, so that the voice of the host and even the voice of the assistant in the live broadcast room meet the user's needs, improve the live broadcast effect, and also improve the retention rate of the live broadcast viewers in the live broadcast room. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0031] Figure 1 A schematic diagram of a structure of an implementation environment provided for an embodiment of the present application;
[0032] Figure 2 A schematic diagram of the structure of a live broadcast system provided in an embodiment of the present application;
[0033] Figure 3 A flow chart of a method for converting sound in a live broadcast scenario provided in an embodiment of the present application Figure 1 ;
[0034] Figure 4 A scenario diagram of a tone selection operation provided in an embodiment of the present application Figure 1 ;
[0035] Figure 5 A scenario diagram of a tone selection operation provided in an embodiment of the present application Figure 2 ;
[0036] Figure 6 A scenario diagram of a tone selection operation provided in an embodiment of the present application Figure 3 ;
[0037] Figure 7A scenario diagram of a tone selection operation provided in an embodiment of the present application Figure 4 ;
[0038] Figure 8 A flow chart of a method for converting sound in a live broadcast scenario provided in an embodiment of the present application Figure 2 ;
[0039] Fig. 9 A flow chart of a method for converting sound in a live broadcast scenario provided in an embodiment of the present application Figure 3 ;
[0040] Fig.10 A flow chart of a method for converting sound in a live broadcast scenario provided in an embodiment of the present application Figure 4 ;
[0041] Fig.11 A schematic diagram of the structure of a sound conversion device provided in an embodiment of the present application;
[0042] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0043] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0044] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.
[0045] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0046] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.
[0047] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.
[0048] In addition, in the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.
[0049] In the existing online live broadcast scenario, during the live broadcast, the host's voice timbre may not meet the needs of the live viewers, resulting in poor live broadcast effects and low retention rate of live viewers.
[0050] In view of the above problems, the embodiment of the present application provides a sound conversion method in a network live broadcast scenario, which is applied to a content expression device in a live broadcast system, and the live broadcast system also includes a live broadcast data generation device connected to the content expression device. In this method, the live broadcast audio data in the target live broadcast room can be first obtained, and then, in response to the user's timbre selection operation, the intended sound conversion model corresponding to the intended timbre is determined. After that, based on the intended sound conversion model, the live broadcast audio data can be converted into the target audio data corresponding to the intended timbre. Finally, after the target audio data is sent to the live broadcast data generation device, the live broadcast video data whose timbre meets the user's needs can be obtained and broadcast live. It can be seen that in this method, after receiving the user's timbre selection operation, the timbre in the live broadcast audio data can be converted into the intended timbre of the user's needs, thereby obtaining live broadcast video data that meets the user's timbre needs, so that the voice of the anchor and even the voice of the assistant in the live broadcast room meet the user's needs, improve the live broadcast effect, and also improve the retention rate of the live broadcast viewers in the live broadcast room.
[0051] The executor of the sound conversion method in a live broadcast scenario provided by the embodiment of the present disclosure may be the content expression device including the sound conversion device provided by the embodiment of the present application, or may be an electronic device including the content expression device. The specific execution may be determined according to actual usage requirements, and the embodiment of the present disclosure is not limited thereto.
[0052] The technical solution provided in the embodiments of the present application can be applied to Figure 1 In the implementation environment shown, the implementation environment may include a live broadcast system 01, a live broadcast platform 02, and a user terminal 03. The live broadcast system 01 and the live broadcast platform 02 may communicate with each other via a wired or wireless communication method, and the live broadcast platform 02 and the user terminal 03 may also communicate with each other via a wired or wireless communication method.
[0053] The live broadcast system 01 is mainly used to generate live broadcast videos, and then push the live broadcast videos to the live broadcast platform 02 for live broadcast. In the embodiment of the present application, there can be one or more live broadcast platforms 02. Figure 1 Only one live broadcast platform 02 is used as an example, and this application does not make any specific restrictions on this.
[0054] The live broadcast platform 02 is mainly used to push the live broadcast data to the user terminal 03 according to specific rules after receiving the live broadcast video from the live broadcast system 01, so that the user terminal 03 plays the live broadcast data corresponding to the live broadcast video for the user to watch. In the embodiment of the present application, there can be one or more user terminals 03. The user terminal 03 can be an electronic device with a live broadcast application corresponding to the live broadcast platform 02 installed.
[0055] Reference Figure 2 As shown, in the present application, the live broadcast system 01 may include a decision-making device 011 , a process driving device 012 , a content expression device 013 and a live broadcast data generating device 014 .
[0056] Among them, the decision-making device 011 establishes a communication connection with the process driving device 012 and the content expression device 013, and the process driving device 012 and the content expression device 013 establish a communication connection with the live broadcast data generation device 014.
[0057] It should be noted that in the present application, the decision device 011, the process driving device 012, the content expression device 013 and the live broadcast data generation device 014 can be separate devices or different parts of the same device, or modules that implement different functions in the data center, depending on actual needs. This application does not impose specific restrictions on this. Among them, the process driving device 012 can obtain the user's process demand data, and generate live broadcast process data and live broadcast rule data based on the user's process demand data. The content expression device 013 can obtain the user's content demand data, and generate live broadcast content data based on the user's content demand data. Among them, the live broadcast content data can include live broadcast audio data. The live broadcast data generation device 014 can generate live video based on the live broadcast process data, live broadcast rule data and live broadcast content data.
[0058] The decision device 011 can generate live broadcast decision data according to the playback data of the live broadcast video, and adjust the live broadcast process data and live broadcast rule data generated by the process driving device 012 based on the live broadcast decision data, and / or adjust the live broadcast content data generated by the content expression device 013.
[0059] The following describes the method for generating live video provided by the embodiment of the present application. A person skilled in the art will know that with the development of technology and the emergence of new scenarios, the technical solution provided by the embodiment of the present application is also applicable to similar technical problems.
[0060] based on Figure 1 The implementation environment shown and Figure 2 The live broadcast system shown in the figure, the embodiment of the present application provides a method for changing the sound in a live broadcast scenario, which can be implemented by a sound changing device, which can be a content expression device in the above embodiment or a part thereof. Figure 3 As shown, the method may include S301-S304:
[0061] S301, obtaining live audio data in the target live broadcast room.
[0062] The target live broadcast room is any live broadcast room that is currently broadcasting or about to broadcast live, corresponding to the content expression device. If the target live broadcast room is a live broadcast room that is about to broadcast live, then the target live broadcast room should be a digital human live broadcast room, that is, the anchor is a digital human, and the live video and audio are pre-generated live broadcast room. Only when the target live broadcast room is a digital human live broadcast room can the live broadcast audio data be obtained when it is not broadcasting. If the target live broadcast room is a real-person live broadcast room, the live broadcast audio data needs to be obtained in real time after the live broadcast room starts broadcasting.
[0063] Taking the target live broadcast room as a digital human live broadcast as an example, in the embodiment of the present application, the live broadcast audio data may specifically be the speech content that the user requires the digital human anchor to output (or say) in the target live broadcast room. Taking live broadcast sales as an example, the live broadcast audio data may include the digital human anchor's introduction and promotion audio of the goods during the live broadcast sales process, as well as the interactive audio between the digital human anchor and the live broadcast viewers.
[0064] The live audio data can be converted from the live speech text uploaded by the user to the content expression device in the live broadcast system, or it can be the audio data directly uploaded by the user to the content expression device in the live broadcast system. In some special cases, if an emergency situation that the user did not expect occurs during the live broadcast of the target live broadcast room (for example, a large number of bullet screens displaying negative comments), and the digital human anchor is unable to respond to the emergency situation, the user can adjust the live audio in the target live broadcast room to the voice spoken by the user in real time through an audio acquisition device (such as a microphone). At this time, the live audio data can be the sound data of the user speaking in real time obtained by the audio acquisition device.
[0065] In addition, it should be noted that in most cases, the actual target live broadcast room may only have the host speaking, but in some cases, the host may not be able to interact with the live broadcast viewers. Based on this, the live broadcast audio data in the embodiment of the present application may include audio data corresponding to the assistant broadcast in addition to the audio data corresponding to the host.
[0066] S302: In response to the user's timbre selection operation, determine an intended sound transformation model corresponding to the intended timbre.
[0067] The timbre selection operation performed by the user may specifically be the selection (or a series of triggering operations) of a timbre option (i.e., a selection line corresponding to a timbre) required by the user from among the multiple timbre options available to the host displayed on the content expression device. Each timbre option corresponds to a sound change model, and the sound change model can convert the timbre of the input audio data into the timbre corresponding to the sound change model.
[0068] There are three scenarios in which users can select a tone:
[0069] Scene 1:
[0070] For example, refer to Figure 4 As shown, the content expression device displays a first interface 401 that may include multiple timbre options. The timbre options may include corresponding names, which may be pre-defined names similar to names or names that reflect the characteristics of the timbre. For example, Jiachuan, Chaodong, Huihui, Intellectual Senior Sister, Lolita, Royal Sister, Shota, etc. In order to bring users a better user experience, a corresponding image will also be displayed in each timbre option. If the user requires the host to present the timbre of the intellectual senior sister, the user can trigger the intellectual senior sister option 402 in the first interface 401 (for example, click). In response to the user's triggering operation on the intellectual senior sister option 402, the content expression device can determine the timbre transformation model corresponding to the intellectual senior sister option 402 as the intended timbre transformation model. In the first scenario, the user's triggering operation on the intellectual senior sister option 402 is a timbre selection operation.
[0071] In addition, in order to facilitate user selection, the content expression device can pre-classify all timbres into multiple categories, such as male voice, female voice, child voice, mine, etc. Among them, the timbres in the category of "mine" can be timbres customized by the user. Figure 4 As shown, a classification window 403 may also be displayed in the first interface 401. The classification window 403 includes category options corresponding to male voice, female voice, child voice, mine, etc. When the user selects the intended timbre, the real-time timbre selection operation may specifically be a trigger operation (e.g., a click operation) on a certain category option plus a trigger operation (e.g., a click operation) on a certain timbre option among all timbre options corresponding to the category option.
[0072] Second scene:
[0073] In addition, if the live broadcast audio data includes not only the host's audio data, but also the assistant's audio data. Figure 5 As shown in (a), the second interface 500 displayed by the content expression may include a host voice option 501 and an assistant voice option 502. If the user only needs to adjust the voice of the host's audio data in the live audio data, the user can click the host option 501. In response to the user's click operation on the host option 501, the content expression device can display the following Figure 5 The third interface 503 shown in (b) is for the user to perform a timbre selection operation on a timbre option. The contents other than the host option 501 and the assistant option 502 in the third interface 503 can be Figure 4 The tone selection interface shown is similar and will not be described again here.
[0074] The content presentation device can also respond to the user's click operation on the auxiliary broadcast option 502 by displaying a Figure 5 The fourth interface 504 shown in (c) is for the user to perform a timbre selection operation on a timbre option. The fourth interface 504 is different from the third interface 503 in that the assistant broadcast option 502 in the fourth interface 504 is displayed in bold and underlined, while the host option 501 in the third interface 503 is displayed in bold and underlined.
[0075] In the second scenario described above, the user's triggering operation on the host option or the assistant option and the subsequent triggering operation on a certain tone option on the third interface or the fourth interface constitute a tone selection operation.
[0076] Scene 3:
[0077] In some embodiments, if the target live broadcast room is a digital human live broadcast room, then in some emergency situations in the actual target live broadcast room, the inherent rhetoric of the digital human anchor may not be able to cope well. Based on this, the content expression device can provide the user with an avatar mode, in which the audio output by the digital human anchor can be the user's real audio obtained through an audio acquisition device. However, since there is a difference between the timbre of the audio previously output by the digital human anchor and the timbre of the user himself, in this mode, the content expression device can provide the user with a voice changing function. For example, refer to Figure 6 As shown, the setting menu 601 of the content expression device may include a host voice change option 602. After the user turns on the avatar mode, if the user needs to make the voice timbre of the live video seen by the live viewers the same as before, the host voice change option 602 needs to be triggered to turn on the voice change function, so that the timbre of the voice input by the user through the audio acquisition device (such as a microphone) is the same as the timbre of the audio output by the digital human host before. Of course, in order to improve the viewing experience of the live viewers, the host voice change option 602 can be triggered by default, that is, the voice change function is turned on by default.
[0078] In the third scenario, the user's triggering operation on the host's voice changing option is a timbre selection operation.
[0079] Of course, the above-mentioned scenario examples of the user performing the timbre selection operation are only illustrative, and in practice, any other feasible methods may be used, and this application does not impose any specific restrictions on this.
[0080] In the embodiment of the present application, the training process of the intended sound transformation model corresponding to the intended timbre is as follows:
[0081] S11. Obtain training corpus.
[0082] The training corpus includes multiple groups of sample data, each group of sample data includes multiple first sample audio data and second sample audio data. The text content corresponding to the first sample audio data and the second sample audio data in each group of sample data is the same, that is, the first sample audio data and the second sample audio data are obtained based on different people (including real people and virtual characters) reading various types of text content. For example, multiple first sample audio data can be obtained by multiple people of different ages and genders and different game characters reading (or outputting in voice form) the text content "I want to eat fried chicken today". The text content corresponding to different groups of sample data may be different.
[0083] The second sample audio data may be obtained by people with the intended timbre reading various text contents. For example, if the intended timbre is a child's voice, the second sample audio data may be audio data of a child reading various text contents. The multiple first sample audio data may be audio data obtained by people with different timbre reading various text contents.
[0084] In some embodiments, in order to make the target audio data obtained by the final trained sound transformation model after transforming the live audio data similar or even identical to the live audio data in features other than timbre, the rhythmic features of the first sample audio data and the second sample audio data may have a large degree of similarity or be identical.
[0085] S12. Using the first sample audio data as training data and the second sample audio data as supervisory information, train the initial sound transformation model to obtain an intended sound transformation model corresponding to the intended timbre.
[0086] Specifically, S12 may include S121-S123:
[0087] S121. Input the first sample audio data into an initial sound transformation model to obtain predicted audio data.
[0088] In the embodiment of the present application, the specific framework of the initial sound transformation model can be any feasible neural network architecture, such as generative adversarial nets (GAN), etc. Generally, the bias parameter in the initial sound transformation model is 0, and the weight parameter can be a random value.
[0089] S122. Determine a loss value based on the predicted audio data and the second sample audio data.
[0090] Specifically, the loss value is determined based on the predicted audio data and the second sample audio data, and specifically, the loss value can be determined based on the Mel spectrum (or Mel-Mel cepstral coefficients, linear prediction cepstral coefficients, perceptual linear prediction coefficients, etc.) of the predicted audio data and the second sample audio data. The loss value can be determined in any feasible way, such as using at least one loss function to calculate the loss value, etc., and this application does not make specific restrictions on this.
[0091] S123, iteratively updating the initial sound transformation model based on the loss value until the loss value is less than a preset threshold, and determining the latest updated initial sound transformation model as the intended sound transformation model corresponding to the intended timbre.
[0092] Among them, iteratively updating the initial sound transformation model based on the loss value means adjusting the internal parameters in the initial sound transformation model according to the loss value, re-inputting the new first sample audio data into the updated initial sound transformation model to obtain new predicted audio data, and obtaining the loss value; repeatedly executing this process until the loss value is less than a preset threshold and stopping.
[0093] Of course, in practice, the sound transformation model corresponding to the intended timbre may also be any other feasible training method, and this application does not impose any specific restrictions on this.
[0094] It should be noted that, taking the target live broadcast room as a digital human live broadcast room as an example, when the live broadcast audio data of the target live broadcast room is first generated, it can be the audio synthesized by the content expression device based on the text content provided or selected by the user (selected from multiple text contents provided by the content expression device). During synthesis, the content expression device can display Figure 4 or Figure 5 The interface shown in the figure allows the user to select the timbre of the audio synthesized from the text. Afterwards, the content expression device can synthesize the text into live audio data based on the TTS (text to speech) model corresponding to the timbre selected by the user. For example, taking the example that the text content only includes the audio content corresponding to the digital human anchor, the process of the content expression device displaying the synthesized audio timbre selection interface for the user to select the timbre can be as follows: Figure 7 See Figure 7 As shown in (a) of FIG. 1 , the synthesized audio timbre selection interface may include a timbre selection control 701. In response to the user's selection of the timbre selection control 701, the content presentation device displays the following Figure 7 Thereafter, in response to a user triggering operation (e.g., clicking) on a first timbre option 702 among the multiple timbre options, the content expression device may determine the TTS model corresponding to the first timbre option as the TTS model for subsequent audio synthesis of the text, and synthesize the text content provided or selected by the user into audio based on the TTS model.
[0095] In one possible implementation, the training process of the TTS model is as follows:
[0096] S21. Obtain training samples.
[0097] The training samples include multiple sections of sample audio data and sample texts corresponding to the sample audio data. The multiple sections of sample audio data are obtained by the same person (including real people and virtual people) reading the corresponding sample texts. For example, if the TTS model is capable of synthesizing text into audio data of a first tone, the multiple sections of sample audio data are all obtained by a person with the first tone reading the corresponding sample text.
[0098] In some embodiments, taking the example of a TTS model that synthesizes text into audio data of a first timbre, the method for obtaining training samples can be to let a person with a first timbre record a high-definition audio file for a long time (e.g., 20 minutes), and the recording content is a specified text. Afterwards, the high-definition audio file can be divided into multiple audio segments as multiple sample audio data, and the text content corresponding to each sample audio data segment is determined. In addition, in order to make the pronunciation quality of the audio data in the recorded high-definition audio file good enough to facilitate subsequent training. After the recording is completed, ASR (automatic speech recognition) technology can be used to identify the high-definition audio file to determine the accuracy of the pronunciation in the high-definition audio file. If the accuracy is high (e.g., higher than 90%), the subsequent segmentation can be continued, and if the accuracy is low, re-recording is required. Furthermore, in order to make the subsequent training effect better, before training, the words with incorrect pronunciation in the sample audio need to be adjusted and modified.
[0099] S22. Using the sample text as training data and the sample audio data as supervision information, train the initial TTS model to obtain the final TTS model.
[0100] In some embodiments, the training process S22 may last for more than 48 hours, so that the effect of the final TTS model is as good as possible.
[0101] Specifically, S22 may include S221-S223:
[0102] S221. Input the sample text into the initial TTS model to obtain predicted audio data.
[0103] In the embodiment of the present application, the specific framework of the initial TTS model can be any feasible neural network architecture, such as generative adversarial nets (GAN), feed forward neural network (FFNN), etc. Generally, the bias parameter in the initial TTS model is 0, and the weight parameter can be a random value.
[0104] In the embodiment of the present application, inputting the sample text into the initial TTS model may be converting the sample text into a word vector and a sentence vector and then inputting them into the initial TTS model.
[0105] S222. Determine a loss value based on the predicted audio data and the sample audio data.
[0106] Specifically, the loss value is determined based on the predicted audio data and the sample audio data, and specifically, the loss value can be determined based on the Mel spectrum (or Mel-Mel cepstral coefficients, linear prediction cepstral coefficients, perceptual linear prediction coefficients, etc.) of the predicted audio data and the sample audio data. The loss value can be determined in any feasible way, such as using at least one loss function to calculate the loss value, etc., and this application does not make specific restrictions on this.
[0107] S223. Iteratively update the initial TTS model based on the loss value until the loss value is less than a preset threshold, and determine the latest updated initial TTS model as the final TTS model.
[0108] Among them, iteratively updating the initial TTS model based on the loss value means adjusting the internal parameters in the initial TTS model according to the loss value, re-inputting the new first sample audio data into the updated initial TTS model to obtain new predicted audio data, and obtaining the loss value; repeatedly executing this process until the loss value is less than the preset threshold.
[0109] Of course, in practice, the TTS model corresponding to the intended timbre may also be trained in any other feasible manner, and this application does not impose any specific restrictions on this.
[0110] It should be noted that in the embodiment of the present application, the live broadcast system will have a corresponding client, and the client may be installed with an application corresponding to the live broadcast system. The user can interact with the content expression device in the live broadcast system based on the client. In the aforementioned embodiment, the content displayed by the content expression device may be specifically displayed through the client.
[0111] S303: Process the live audio data based on the intended sound transformation model to obtain target audio data corresponding to the intended timbre.
[0112] In the embodiment of the present application, S303 may be specifically completed by a sound conversion service in the content expression device.
[0113] In one possible implementation, the intended sound transformation model may directly process the live audio data to obtain the target audio data.
[0114] In another possible implementation, if the target live broadcast room is a digital human live broadcast room, a sound transformation model may be the TTS model corresponding to the intended timbre mentioned in the above embodiment. In this implementation, after acquiring the live audio data, the text content corresponding to the live audio data may be acquired. Based on this, S302 may specifically include: acquiring the live text content corresponding to the live audio data, processing the live text content based on the TTS model corresponding to the intended timbre, and obtaining the target audio data.
[0115] S304: Send target audio data to a live broadcast data generating device.
[0116] In some embodiments, after the content expression device obtains the target audio data, it can send the target audio data to the live data generation device, so that the live data generation device generates live video data based on the target audio data for live broadcast (or continues to live broadcast). In an embodiment of the present application, the generation of live video data can be specifically completed by a rendering service in the live data generation device.
[0117] For example, taking the target live broadcast room as a digital human live broadcast room, after receiving the target audio data, the live broadcast data generating device can convert the target audio data into multiple frames of digital human live broadcast images based on its own rendering service. Afterwards, the live broadcast data generating device synchronizes the target live broadcast audio data and multiple frames of digital human live broadcast images for live broadcast display.
[0118] Based on the technical solution provided in the embodiment of the present application, the live audio data in the target live broadcast room can be first obtained, and then, in response to the user's timbre selection operation, the intended sound transformation model corresponding to the intended timbre can be determined. Afterwards, the live audio data can be converted into the target audio data corresponding to the intended timbre based on the intended sound transformation model. Finally, after sending the target audio data to the live broadcast data generation device, the live video data whose timbre meets the user's needs can be obtained and broadcast live. It can be seen that in this method, after receiving the user's timbre selection operation, the timbre in the live audio data can be converted into the intended timbre required by the user, thereby obtaining live video data that meets the user's timbre requirements, so that the voice of the host and even the voice of the assistant in the live broadcast room meet the user's needs, improve the live broadcast effect, and also improve the retention rate of the live broadcast viewers in the live broadcast room.
[0119] In some embodiments, in response to the user's timbre selection operation, the content expression device can determine the timbre feature data corresponding to the intended timbre required by the user. Then, based on the feature data other than the timbre feature data in the live audio data, the target audio data can be obtained by combining the timbre feature data. Figure 3 , refer to Figure 8 As shown, after S301 and before S304, S302 and S303 in the method can be replaced by S801-S804.
[0120] S801. In response to a timbre selection operation by a user, an identifier of an intended timbre is determined.
[0121] The specific implementation of the user's timbre selection operation may refer to the relevant description after S302 in the aforementioned embodiment, which will not be repeated here.
[0122] The content expression device can determine the intended timbre selected by the user through the timbre selection operation. Afterwards, the content expression device can determine the identifier of the intended timbre from the preset timbre identifier table based on the intended timbre selected by the user. For example, the identifier corresponding to the child's voice is 00, the identifier corresponding to the mature voice is 01, the identifier corresponding to the uncle's voice is 02, and so on. The preset timbre identifier table can be pre-configured by the content expression device.
[0123] S802: Determine timbre feature data corresponding to the intended timbre according to the identifier of the intended timbre.
[0124] In the embodiment of the present application, the timbre characteristic data is specifically used to characterize the characteristics or properties that are different from other sounds in terms of waveform. The specific expression form or content can be any feasible way or content.
[0125] In a possible implementation, a timbre feature library may be pre-stored in the content expression device, and the timbre feature library may contain multiple timbre identifiers and associations between multiple timbre feature data. After obtaining the identifier of the intended timbre, the content expression device may search the timbre feature data corresponding to the identifier of the intended timbre from the timbre feature library.
[0126] In another possible implementation, a timbre mapping network may be pre-existing in the content expression device, and the timbre may convert a timbre identifier into a 256-dimensional feature, and the 256-dimensional feature may be used to represent the timbre of the speaker, and the timbre mapping network may output corresponding timbre feature data based on the identifier of the input intended timbre. The timbre mapping network may be generated by training with a variety of timbres and corresponding identifiers. The training process is not specifically limited in this embodiment of the present application. In this implementation, S802 may specifically be: based on the timbre mapping network, determine the timbre feature data corresponding to the identifier of the intended timbre. Among them, the timbre mapping network includes a correspondence between at least one timbre identifier and at least one timbre feature data, and the identifier of at least one timbre includes the identifier of the intended timbre.
[0127] S803: Acquire semantic feature data and rhythmic feature data of the live audio data.
[0128] For an audio data composed of human speech, the audio data should include text information, prosody information and timbre information. Among them, the text information can be represented by semantic feature data, the prosody information can be represented by prosody feature data, and the timbre information can be represented by timbre feature data. In some embodiments, the semantic feature data and the prosody feature data can be called bottleneck features, also known as BNF features.
[0129] In some possible implementations, S803 may be to use a pre-trained feature extraction model capable of extracting bottleneck features to extract bottleneck features, and extract the bottleneck features of the live audio data. The feature extraction model may be trained in any feasible manner, and this application does not impose any specific restrictions on this.
[0130] In some other possible implementations, S803 may also utilize a specific feature extraction method to extract bottleneck features of the live audio data.
[0131] It should be noted that S803 can be executed at any feasible time after S301 and before S804. Figure 8 The step sequence shown is only an example. In practice, the execution timing of S803 can be determined according to actual needs, and this application does not impose any specific restrictions on this.
[0132] S804: fusing the semantic feature data, the prosodic feature data and the timbre feature data to obtain target audio data.
[0133] In an embodiment of the present application, the fusion of semantic feature data, prosodic feature data and timbre feature data can be achieved through a pre-trained acoustic model network.
[0134] Exemplarily, the training process of the acoustic model network may be as follows:
[0135] S31. Obtain training samples.
[0136] The training sample includes multiple sample audio features and sample audio data corresponding to each audio feature, wherein each sample audio feature may include sample semantic feature data, sample rhythm feature data and sample timbre feature data, and the sample audio feature is an audio feature of the corresponding sample audio data. The multiple sample audio data may be recorded by different people (including real people and virtual people) reading different texts.
[0137] S32. Using the sample audio features as training data and the sample audio data as supervision information, train the initial acoustic model network to obtain the final acoustic model network.
[0138] Specifically, S32 may include S321-S323:
[0139] S321. Input the sample audio features into the initial acoustic model network to obtain predicted audio data.
[0140] In the embodiment of the present application, the specific framework of the initial acoustic model network can be any feasible neural network architecture, such as generative adversarial nets (GAN), feedforward neural network (FFNN), etc. Generally, the bias parameter in the initial acoustic model network is 0, and the weight parameter can be a random value.
[0141] S322. Determine a loss value based on the predicted audio data and the sample audio data.
[0142] Specifically, the loss value is determined based on the predicted audio data and the sample audio data, and specifically, the loss value can be determined based on the Mel spectrum (or Mel-Mel cepstral coefficients, linear prediction cepstral coefficients, perceptual linear prediction coefficients, etc.) of the predicted audio data and the sample audio data. The loss value can be determined in any feasible way, such as using at least one loss function to calculate the loss value, etc., and this application does not make specific restrictions on this.
[0143] S323. Iteratively update the initial acoustic model network based on the loss value until the loss value is less than a preset threshold, and determine the latest updated initial acoustic model network as the final acoustic model network.
[0144] Among them, iteratively updating the initial acoustic model network based on the loss value means adjusting the internal parameters in the initial acoustic model network according to the loss value, re-inputting the new first sample audio data into the updated initial acoustic model network to obtain new predicted audio data, and obtaining the loss value; repeatedly executing this process until the loss value is less than a preset threshold.
[0145] Of course, in practice, the acoustic model network can also be trained in any other feasible way, and this application does not impose any specific restrictions on this.
[0146] In the embodiment of the present application, the fusion of semantic feature data, prosodic feature data and timbre feature data can be achieved through any feasible fusion algorithm, and the present application does not impose any specific limitation on this.
[0147] Based on the above technical solution, after obtaining the live audio data in the target live broadcast room, the identifier of the intended timbre can be determined based on the user's timbre selection operation, and the timbre feature data of the intended timbre can be obtained based on the identifier. Afterwards, the timbre feature data and the bottleneck features of the live audio data are fused and processed to obtain the target audio data corresponding to the intended timbre. Finally, after sending the target audio data to the live broadcast data generation device, the live video data whose timbre meets the user's needs can be obtained and broadcast live. It can be seen that in this method, after receiving the user's timbre selection operation, the timbre in the live audio data can be converted into the intended timbre required by the user, thereby obtaining live video data that meets the user's timbre requirements, so that the voice of the host and even the voice of the assistant in the live broadcast room meet the user's needs, improve the live broadcast effect, and also improve the retention rate of the live broadcast viewers in the live broadcast room.
[0148] In some embodiments, in addition to the technical solution shown in S301-S304 above, where each timbre corresponds to a sound transformation model, a large sound transformation model can also exist in the content expression device, which has variables corresponding to the intended timbre selected by the user and the live audio data, so as to obtain the target audio data. Figure 3 , refer to Fig. 9 As shown, between S301 and S304, S302 and S303 in the method can also be replaced by 901 and S902.
[0149] S901. In response to a user's timbre selection operation, determine an intended timbre variable.
[0150] The intended timbre variable may be any possible information for indicating the intended timbre selected by the user.
[0151] S902. Based on a target sound transformation model, the live audio data and the intended timbre variable are processed to obtain the target audio data.
[0152] In the embodiment of the present application, the training process of the target sound conversion model is as follows:
[0153] S41. Obtain training corpus.
[0154] The training corpus includes multiple groups of sample data, each group of sample data includes multiple first sample audio data, multiple second sample audio data, and timbre variables corresponding to the second sample audio data. The text content corresponding to the first sample audio data and the second sample audio data in each group of sample data is the same. In the same group of sample data, multiple first sample audio data can be obtained by multiple people of different age groups and genders and different game characters reading a certain text content (or outputting it in the form of voice); multiple second sample audio data can be obtained by people with different timbres reading the certain text content (or outputting it in the form of voice). For example, multiple first sample audio data may include audio data with a child's voice version, audio data with a queen's voice version, audio data with an uncle's voice version, etc.
[0155] The sample contents corresponding to different groups of sample data may be different.
[0156] In some embodiments, in order to make the target audio data obtained by the final trained sound transformation model after transforming the live audio data similar or even identical to the live audio data in features other than timbre, in the same group of sample data, the rhythmic features of the first sample audio data and the second sample audio data can have a large degree of similarity or be identical.
[0157] S42. Using the first sample audio data and timbre variables as training data and the second sample audio data corresponding to the timbre variables as supervisory information, train the initial sound transformation model to obtain an intended sound transformation model corresponding to the intended timbre.
[0158] The first sample audio data and timbre variables of the same group in S42 are the contents of the same group of sample data.
[0159] Specifically, S42 may include S421-S423:
[0160] S421. Input the first sample audio data into the initial sound transformation model to obtain predicted audio data.
[0161] In the embodiment of the present application, the specific framework of the initial sound transformation model can be any feasible neural network architecture, such as generative adversarial nets (GAN), etc. Generally, the bias parameter in the initial sound transformation model is 0, and the weight parameter can be a random value.
[0162] S422. Determine a loss value based on the predicted audio data and the second sample audio data corresponding to the timbre variable.
[0163] Specifically, the loss value is determined based on the predicted audio data and the second sample audio data corresponding to the timbre variable, and specifically, the loss value can be determined based on the Mel spectrum (or Mel-Mel cepstral coefficients, linear prediction cepstral coefficients, perceptual linear prediction coefficients, etc.) of the second sample audio data corresponding to the predicted audio data and the timbre variable. The loss value can be determined in any feasible way, such as using at least one loss function to calculate the loss value, etc., and this application does not make specific restrictions on this.
[0164] S423, iteratively updating the initial sound transformation model based on the loss value until the loss value is less than a preset threshold, and determining the latest updated initial sound transformation model as the target sound transformation model.
[0165] Among them, iteratively updating the initial sound transformation model based on the loss value means adjusting the internal parameters in the initial sound transformation model according to the loss value, and then re-inputting new first sample audio data and new timbre variables into the updated initial sound transformation model to obtain new predicted audio data and obtain the loss value; repeatedly executing this process until the loss value is less than a preset threshold.
[0166] Of course, in practice, the target sound transformation model may also be trained in any other feasible manner, and this application does not impose any specific limitation on this.
[0167] Based on the above technical solution, when the user needs it, one model can be used to achieve the effects of multiple tone changes, so that the voice of the host and even the assistant in the live broadcast room meets the needs of the user, improving the live broadcast effect and the retention rate of the live broadcast viewers in the live broadcast room.
[0168] In some embodiments, if the target live broadcast room is a digital human live broadcast room, and the live broadcast audio data is a fixed audio set in advance, in order to make the audio data after the timbre change conform to the vocal characteristics of the real person (emotions are not fixed, intonation is not fixed, etc.), combined with Figure 3 , refer to Fig.10 As shown, between S303 and S304, the method may further include S1001:
[0169] S1001. Adjust audio parameters in the target audio data according to preset rules to update the target audio data.
[0170] The audio parameters include any one or more of the following: pitch, loudness, and bit rate. The bit rate is mainly used to characterize the speech speed in the audio data.
[0171] In a possible implementation manner, the preset rule may be to randomly adjust audio parameters.
[0172] In another possible implementation, the preset rule may be derived from audio parameters of reference audio corresponding to a specific live broadcast tag. For example, the preset rule may be derived from audio parameter transformation of reference audio with a live broadcast tag of "selling goods".
[0173] In this way, the transformation of audio parameters in the finally updated target audio data will be more in line with reality, and the target audio data can also show richer situations, thereby improving the viewing experience of live viewers, improving the live broadcast effect, and also improving the retention rate of live viewers in the live broadcast room.
[0174] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.
[0175] The embodiment of the present application can divide the functional modules of the sound conversion device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.
[0176] Corresponding to the method in the above embodiment, the present application embodiment further provides a sound conversion device, which may be the content expression device in the above embodiment or a part thereof. Fig.11 As shown, the sound conversion device includes an acquisition module 111 , a determination module 112 , a processing module 113 and a sending module 114 .
[0177] Among them, the acquisition module 111 is used to acquire the live audio data in the target live broadcast room; the determination module 112 is used to determine the intended sound transformation model corresponding to the intended timbre in response to the user's timbre selection operation; the processing module 113 is used to process the live audio data acquired by the acquisition module 111 based on the intended sound transformation model determined by the determination module 112, and obtain the target audio data corresponding to the intended timbre; the sending module 114 is used to send the target audio data obtained by the processing module 113 to the live broadcast data generation device, so that the live broadcast data generation device generates live video data based on the target audio data for live broadcast.
[0178] Optionally, the determination module 112 is also used to determine the identification of the intended timbre in response to the user's timbre selection operation; the processing module 113 is also used to determine the timbre feature data corresponding to the intended timbre according to the identification of the intended timbre determined by the determination module 112; the processing module 113 is also used to obtain the semantic feature data and rhythmic feature data of the live audio data obtained by the acquisition module 111; the processing module 113 is also used to fuse the semantic feature data, rhythmic feature data and timbre feature data to obtain target audio data; the sending module 114 is also used to send the target audio data to the live data generating device, so that the live data generating device generates live video data based on the target audio data for live broadcast.
[0179] Optionally, the determination module 112 is specifically used to: determine the timbre feature data corresponding to the identification of the intended timbre based on the timbre mapping network; the timbre mapping network includes a correspondence between at least one timbre identification and at least one timbre feature data, and the identification of at least one timbre includes the identification of the intended timbre.
[0180] Optionally, the determination module 112 is also used to determine the intended timbre variable in response to the user's timbre selection operation; the processing module 113 is also used to process the live audio data acquired by the acquisition module 111 and the intended timbre variable determined by the determination module 112 based on the target sound transformation model to obtain the target audio data; the sending module 114 is also used to send the target audio data to the live data generating device, so that the live data generating device generates live video data based on the target audio data for live broadcast.
[0181] Optionally, after the processing module 113 processes the live audio data based on the intended sound transformation model to obtain the target audio data corresponding to the intended timbre, before the sending module 114 sends the target audio data to the live data generating device, the processing module 113 is also used to adjust the audio parameters in the target audio data according to preset rules to update the target audio data; wherein the audio parameters include any one or more of the following: pitch, loudness, bit rate.
[0182] Optionally, the preset rule is derived from audio parameters of a reference audio corresponding to a specific live broadcast tag.
[0183] The effects that can be achieved by the above-mentioned sound conversion device can refer to the effects of the relevant content of the sound conversion method in the live broadcast scenario in the aforementioned embodiment, and will not be repeated here.
[0184] Fig.12 A structural diagram of an electronic device provided in an embodiment of the present disclosure, such as Fig.12 As shown, the electronic device includes one or more processors 1201 and a memory 1202 .
[0185] The processor 1201 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.
[0186] The memory 1202 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1201 may run the program instructions to implement the sound conversion method or other desired functions in the network live broadcast scenario of each embodiment of the present disclosure above.
[0187] In one example, the electronic device may further include: an input device 1203 and an output device 1204 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0188] Of course, to simplify, Fig.12 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.
[0189] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the sound conversion method in the live broadcast scenario in the aforementioned embodiments of this specification.
[0190] The computer program product may be written in any combination of one or more programming languages to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0191] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the sound conversion method in the live broadcast scenario in the aforementioned embodiment of this specification.
[0192] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0193] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.
[0194] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.
[0195] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.
[0196] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
[0197] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.
Claims
1. A sound conversion method in a network live broadcast scenario, applied to a content expression device in a live broadcast system, wherein the live broadcast system also includes a live broadcast data generation device connected to the content expression device, characterized in that: include: Get the live audio data in the target live room; In response to a timbre selection operation by a user, determining an intended sound transformation model corresponding to an intended timbre; Processing the live audio data based on the intended sound transformation model to obtain target audio data corresponding to the intended timbre; The target audio data is sent to the live broadcast data generating device, so that the live broadcast data generating device generates live broadcast video data based on the target audio data for live broadcast.
2. The method according to claim 1, characterized in that After acquiring the live audio data in the target live broadcast room, the method further includes: In response to a timbre selection operation by a user, determining an identification of an intended timbre; Determining timbre characteristic data corresponding to the intended timbre according to the identifier of the intended timbre; Acquire semantic feature data and rhythm feature data of the live audio data; fusing the semantic feature data, the prosodic feature data and the timbre feature data to obtain the target audio data; The target audio data is sent to the live broadcast data generating device, so that the live broadcast data generating device generates live broadcast video data based on the target audio data for live broadcast.
3. The method according to claim 2, characterized in that The step of determining the timbre characteristic data corresponding to the intended timbre includes: Based on a timbre mapping network, the timbre feature data corresponding to the identification of the intended timbre is determined; the timbre mapping network includes a correspondence between at least one timbre identification and at least one timbre feature data, and the identification of the at least one timbre includes the identification of the intended timbre.
4. The method according to claim 1, characterized in that: After acquiring the live audio data in the target live broadcast room, the method further includes: In response to a user's timbre selection operation, determining an intended timbre variable; Based on the target sound transformation model, the live audio data and the intended timbre variable are processed to obtain the target audio data; The target audio data is sent to the live broadcast data generating device, so that the live broadcast data generating device generates live broadcast video data based on the target audio data for live broadcast.
5. The method according to any one of claims 1 to 4, characterized in that: After the live audio data is processed based on the intended sound transformation model to obtain target audio data corresponding to the intended timbre, and before the target audio data is sent to the live data generating device, the method further includes: The audio parameters in the target audio data are adjusted according to preset rules to update the target audio data; wherein the audio parameters include any one or more of the following: pitch, loudness, bit rate.
6. The method according to claim 5, characterized in that The preset rule is derived from audio parameters of the reference audio corresponding to the specific live broadcast tag.
7. A sound conversion device, applied to a content expression device in a live broadcast system, wherein the live broadcast system further comprises a live broadcast data generation device connected to the content expression device, characterized in that: include: The acquisition module is used to obtain the live audio data in the target live broadcast room; A determination module, for determining an intended sound transformation model corresponding to an intended timbre in response to a timbre selection operation by a user; A processing module, configured to process the live audio data acquired by the acquisition module based on the intended sound transformation model determined by the determination module, to obtain target audio data corresponding to the intended timbre; The sending module is used to send the target audio data obtained by the processing module to the live data generating device, so that the live data generating device generates live video data based on the target audio data for live broadcast.
8. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program; and the processor is used to enable the electronic device to implement the sound conversion method in a live network broadcast scenario as described in any one of claims 1 to 6 when executing the computer program.
9. A computer-readable storage medium, characterized in that: It includes computer instructions; when the computer instructions are executed on an electronic device, the electronic device executes the sound conversion method in a live broadcast scenario as described in any one of claims 1-6.
10. A live broadcast system, characterized in that: It includes the sound conversion device as claimed in claim 7 or the electronic device as claimed in claim 8.
Citation Information
Patent Citations
Live broadcast data processing method and device, electronic device and readable storage medium
CN110062267A
Speech playing method and device, and electronic equipment
CN111627417A
Microphone connection live broadcast method and related equipment
CN112019874A
Audio processing method and device
CN112261470A
Video playing method and device
CN114302153A
