Display device and tone template generation method

Abstract

The invention relates to a display device and a tone template generation method. The method comprises the steps of determining voice input prompt information in response to a trigger operation of a tone creation entry displayed in a content generation platform by a display device; receiving a user voice segment obtained by collecting the voice of the user by the terminal; generating a user tone template corresponding to the user according to the user voice segment; wherein the user tone template is used for generating the multimedia content in the content generation platform, and the tone of the audio data in the multimedia content is matched with the tone of the user. The timbre of the multimedia content can be matched with the timbre of the user as much as possible, so that the multimedia content meeting the user requirement can be quickly generated, and the generation efficiency of the multimedia content is improved.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a method for generating a timbre template. Background Art

[0002] Display devices are generally used to display multimedia contents. Currently, in order to improve the flexibility of content display of display devices, the function of customizing multimedia contents through artificial intelligence has emerged.

[0003] However, in the traditional technology, during the process of multimedia customization, there is a lack of the function of customizing the user's voice in the display device, resulting in the inability to read and present the customized multimedia content on the display device in a personalized voice, making the customized multimedia function of the display device inflexible and individuated.

[0004] Therefore, how to make the timbre of multimedia content match the timbre of the user as close as possible, quickly generate multimedia content that meets the user's needs, and improve the efficiency of multimedia content generation are issues that need to be addressed urgently. Summary of the invention

[0005] The present application provides a display device and a method for generating a tone template, which can improve the efficiency of generating multimedia content in the display device.

[0006] In a first aspect, some embodiments provide a display device, including: a display and a controller. The display is configured to display a sound creation portal in a content generation platform;

[0007] The controller is configured as:

[0008] In response to a trigger operation for a timbre creation entry, determining voice entry prompt information;

[0009] The display is further configured to display voice input prompt information;

[0010] Among them, the voice input prompt information is used to instruct the terminal to collect the user's voice to obtain a user voice segment, the user voice segment is used to generate a user timbre template corresponding to the user, the user timbre template is used to generate multimedia content in the content generation platform, and the timbre of the audio data in the multimedia content matches the timbre of the user.

[0011] In one of the embodiments, when the controller determines the voice entry prompt information, it is also configured to: determine the voice entry prompt information including a graphic code; wherein the graphic code is used for the terminal to display a voice entry entrance, and the voice entry entrance is used for the terminal to collect voice to obtain a user voice segment.

[0012] The voice input entrance is displayed through the terminal. Since the voice input entrance is used by the terminal to collect voice to obtain user voice segments, the method of obtaining user voice segments can be made more efficient and flexible.

[0013] In one of the embodiments, when the controller determines the voice entry prompt information including the graphic code, it is also configured to: when the specified video application in the display device is in a logged-in state, determine the voice entry prompt information including the graphic code; the controller is also configured to: when the specified video application in the display device is in a non-logged-in state, generate a login page for the specified video application; the display is also configured to: display the login page for the specified video application, and the login page is used to log in to the specified video application.

[0014] The controller determines the voice input prompt information including the graphic code when the designated video application is in a logged-in state in the display device. When the designated video application is in a non-logged-in state in the display device, the controller generates a login page for the designated video application so that the generated user voice template matches the user logged in by the designated video application.

[0015] In one of the embodiments, the controller is further configured to: receive a user voice segment sent by the terminal; and send the user voice segment to the server, so that the server generates a user timbre template corresponding to the user according to the user voice segment.

[0016] The controller can obtain the user voice segment in multiple ways, which improves the flexibility of obtaining the user voice segment. Further, the controller sends the user voice segment to the server corresponding to the display device, so that the server can generate the user voice template corresponding to the user according to the user voice segment.

[0017] In one of the embodiments, the controller is further configured to: when a trigger operation for a tone creation entry is detected, output a voice recording instruction, where the voice recording instruction is used to instruct the collection of user voice clips through the terminal.

[0018] When the controller detects a trigger operation for the timbre creation entry, it outputs a voice input instruction, thereby instructing the control device to obtain the user voice segment, which can make the method of obtaining the user voice segment more flexible and efficient.

[0019] In one of the embodiments, the controller is further configured to: obtain a template card corresponding to the user timbre template; the display is further configured to: display the template card corresponding to the user timbre template in the content generation platform; the controller is further configured to: when the template card corresponding to the user timbre template is in a selected state, if a content generation instruction is detected, then send a content generation instruction indicating the user timbre template to the server, so that the server responds to the content generation instruction and generates multimedia content according to the user timbre template; the display is further configured to: display the playback entrance of the multimedia content in the content generation platform.

[0020] The template card corresponding to the user's timbre template is displayed on the display, and a content generation instruction can be created by selecting the template card on the display, so that the server generates multimedia content according to the user's timbre template, which can improve the operability and efficiency of the multimedia content generation method. Furthermore, the playback entrance of the multimedia content is displayed on the display, which is convenient for users to watch the multimedia content.

[0021] In one of the embodiments, the user timbre template is bound to a target account, and the target account is the account used to log in to a specified video application in a display device; when the controller obtains the template card corresponding to the user timbre template, it is also configured to: when the specified video application in the display device is in a logged-in state, obtain the template card corresponding to the user timbre template and the template card corresponding to at least one preset timbre template respectively; the controller is also configured to: when the specified video application in the display device is in a logged-in state, obtain the template card corresponding to at least one preset timbre template respectively; when the display displays the template card corresponding to the user timbre template in the content generation platform, it is also configured to: when the specified video application in the display device is in a logged-in state, display the template card corresponding to the user timbre template in the content generation platform, and display the template card corresponding to at least one preset timbre template respectively; the display is also configured to: when the specified video application in the display device is in a logged-in state, display the template card corresponding to the user timbre template in the content generation platform.

[0022] When the specified video application is not logged in in the display device, only the template card corresponding to the preset timbre template is displayed. When the specified video application is logged in in the display device, the template card corresponding to the user timbre template is displayed. Ensuring that the template card corresponding to the user timbre template customized by the user is displayed only when the user is logged in can improve the security of the user timbre template and avoid leakage of the user's timbre privacy.

[0023] In one of the embodiments, the display is further configured to: display the input text in the content generation platform; when the controller sends a content generation instruction indicating the user's voice template to the server, it is also configured to: send a content generation instruction carrying the input text and indicating the user's voice template to the server; when the server generates multimedia content according to the user's voice template, it is also configured to: generate audio data whose timbre matches the user's timbre according to the user's voice template and the input text; and generate multimedia content based on the audio data.

[0024] By processing the input text, a story picture book of multimedia content is generated, and according to the user's voice template, audio data with a voice matching the user's voice is generated, and then the final multimedia content is generated according to the story picture book and audio data. The picture book content can be intelligently customized, and combined with the audio data matching the user's voice, it can improve the personalized customization of multimedia content.

[0025] In one of the embodiments, the user timbre template is a target audio generation model corresponding to the user, the target audio generation model includes a phoneme encoder and a spectrum generator, the phoneme encoder and the spectrum generator each include at least one network layer after parameter adjustment by the user voice segment; when the server generates audio data whose timbre matches the user's timbre according to the user timbre template and the input text, it is also configured to: obtain a phoneme vector generated based on the input text; input the phoneme vector into the phoneme encoder, encode each phoneme in the phoneme vector to obtain a corresponding phoneme coding feature, and obtain a phoneme coding feature vector composed of each phoneme coding feature; input the phoneme coding feature vector into the spectrum generator to generate a speech spectrum corresponding to the input text; input the speech spectrum into the vocoder to generate audio data whose timbre matches the user's timbre.

[0026] The user timbre template is a target audio generation model including a phoneme encoder and a spectrum generator, wherein the phoneme encoder and the spectrum generator each include at least one network layer after parameter adjustment by the user's voice segment. Since the network layer after parameter adjustment by the user's voice segment is based on the parameter adjustment by the user's voice segment, adding the network layer can make the generated audio data more compatible with the user's timbre.

[0027] In one of the embodiments, the spectrum generator includes a speech feature prediction layer and a spectrum decoder; when the server inputs the phoneme encoding feature vector into the spectrum generator to generate a speech spectrum corresponding to the input text, it is also configured to: input the phoneme encoding feature vector into the speech feature prediction layer, and predict the speech features based on the attention mechanism layer and the phoneme encoding feature vector in the speech feature prediction layer; input the speech features into the spectrum decoder to generate a speech spectrum corresponding to the input text.

[0028] Using an attention mechanism layer to replace the variance adapter built based on a convolutional neural network can make the positional encoding information of each phoneme encoding feature more accurate, thereby making the audio data generated by decoding based on the spectrum decoder more coherent and smooth.

[0029] In a second aspect, some embodiments provide a method for generating a timbre template, the method being applied to a display device, comprising:

[0030] Displaying a sound creation portal in a content generation platform;

[0031] In response to a trigger operation on a timbre creation entry, displaying a voice input prompt message;

[0032] Among them, the voice input prompt information is used to instruct the terminal to collect the user's voice to obtain a user voice segment, the user voice segment is used to generate a user timbre template corresponding to the user, the user timbre template is used to generate multimedia content in the content generation platform, and the timbre of the audio data in the multimedia content matches the timbre of the user.

[0033] It can be seen from the above technical scheme that some embodiments of the present application provide a display device and a method for generating a timbre template, wherein the display displays a timbre creation entrance in a content generation platform, and correspondingly, the controller determines voice entry prompt information in response to a trigger operation for the timbre creation entrance, and displays the voice entry prompt information in the display. Among them, the voice entry prompt information is used to indicate that the user's voice is collected to obtain a user voice segment. The user voice segment is used to generate a user timbre template corresponding to the user. The user timbre template is used to generate multimedia content in the content generation platform, and the timbre of the audio data in the multimedia content matches the timbre of the user. The above-mentioned display device and timbre template generation method, in the process of generating multimedia content using the display device, can obtain a user timbre template that matches the timbre of the user by collecting the user voice segment, and by using the user timbre template, the timbre of the multimedia content can be matched with the timbre of the user, that is, as close as possible, so that multimedia content that meets the needs of the user can be quickly generated, thereby improving the generation efficiency of multimedia content. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0036] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0037] Figure 3 A schematic diagram of the hardware configuration of a control device provided in some embodiments of the present application;

[0038] Figure 4 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0039] Figure 5A A schematic diagram of a flow chart of a method for generating a timbre template provided in some embodiments of the present application;

[0040] Figure 5B A schematic diagram of a content generation platform provided in some embodiments of the present application;

[0041] Figure 5C A schematic diagram of a timbre creation entry display interface provided in some embodiments of the present application;

[0042] Fig. 6A A schematic diagram of a voice input prompt information display interface provided in some embodiments of the present application;

[0043] Figure 6B A schematic diagram showing a voice input entry provided in some embodiments of the present application;

[0044] Figure 6C A schematic diagram of a voice input mode selection interface provided in some embodiments of the present application;

[0045] Figure 7 A schematic diagram of a login page provided for some embodiments of the present application;

[0046] Fig. 8A A schematic diagram of a voice input interface provided in some embodiments of the present application;

[0047] Figure 8B A schematic diagram of timbre naming provided for some embodiments of the present application;

[0048] Figure 8C A schematic diagram of a voice input interface provided in some embodiments of the present application;

[0049] Fig. 9 A schematic diagram of displaying voice input prompt information provided in some embodiments of the present application;

[0050] Fig.10 A schematic diagram of displaying template cards provided in some embodiments of the present application;

[0051] Fig.11 A flowchart of the steps of generating audio data provided in some embodiments of the present application;

[0052] Fig.12 A flowchart of speech spectrum generation steps provided in some embodiments of the present application;

[0053] Fig.13 A timing diagram of a timbre template generation method provided in some embodiments of the present application. DETAILED DESCRIPTION

[0054] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0055] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and common meanings.

[0056] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0057] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0058] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0059] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0060] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1As shown in FIG. 1 , the user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0061] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0062] like Figure 1 It is also shown that the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN) and other networks.

[0063] The display device 200 can provide a broadcast receiving television function and a multimedia content creation function; wherein the multimedia content can be a picture book, and can also provide an intelligent network TV function that provides computer support functions, including but not limited to network TV, smart TV, Internet Protocol TV (IPTV), etc.

[0064] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0065] In some embodiments, the display device 200 may include at least one of a tuner 210 , a communication device 220 , a detector 230 , a device interface 240 , a controller 250 , a display 260 , an audio output device 270 , a memory, a power supply, and a user input interface 280 .

[0066] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0067] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0068] In some embodiments, the communication device 220 is a component for communicating with an external device or a server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0069] The communication device 220 can enable the display device 200 to communicate with the external device or the server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0070] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0071] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0072] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0073] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0074] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0075] Figure 3 Some embodiments of the present application provide Figure 1 The hardware configuration diagram of the control device in the figure. Figure 3 As shown, the control device 100 may include: a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0076] The control device 100 is configured to control the display device 200 , and can receive user input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200 .

[0077] In some embodiments, the control device 100 may be a smart device, for example, the control device 100 may be installed with various applications for controlling the display device 200 according to user needs.

[0078] In some embodiments, Figure 1 As shown, the mobile terminal 300 or other intelligent electronic devices can play a similar function as the control device 100 after installing the application for controlling the display device 200 .

[0079] The controller 110 includes a processor 112, a RAM 113, a ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication and cooperation between the internal components and the external and internal data processing functions.

[0080] The communication interface 130 implements communication of control signals and data signals with the display device 200 under the control of the controller 110. The communication interface 130 may include at least one of other near field communication modules such as a WiFi chip 131, a Bluetooth module 132, and an NFC module 133.

[0081] The user input / output interface 140 , wherein the input interface includes at least one of other input interfaces such as a microphone 141 , a touch panel 142 , a sensor 143 , and a button 144 .

[0082] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with a communication interface 130, such as a WiFi, Bluetooth, NFC or other module, and can encode the user input command through the WiFi protocol, Bluetooth protocol, or NFC protocol and send it to the display device 200.

[0083] The memory 190 is used to store various operating programs, data and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0084] The power supply 180 is used to provide operating power support for each component of the control device 100 under the control of the controller.

[0085] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system may provide a user interface (control the display device), allow the user to interact with the display device 200, and support the running of various application programs.

[0086] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0087] The operating system can be divided into different modules or layers according to the functions implemented, for example Figure 4 As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications layer) (referred to as "application layer"), the application framework layer (Application Framework layer) (referred to as "framework layer"), the system library layer and the kernel layer.

[0088] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0089] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions of applications in the application layer. Applications can access system resources and obtain system services during execution through the API interface.

[0090] like Figure 4 As shown, the application layer in the embodiment of the present application includes various applications. For example, the various applications include content generation applications to support users in generating multimedia content in the content generation applications.

[0091] In the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application package currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0092] In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes, for example, reducing the display window, shaking the display, distorting the display, etc.

[0093] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.

[0094] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 4 As shown, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB (Universal Serial Bus) driver, HDMI (High-Definition Multimedia Interface) driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0095] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0096] The above embodiments illustrate the hardware / software architecture and functional implementation of a display device 200. Based on this, in some embodiments, the present application provides a display device 200, including a display 260 and a controller 250. Among them, the display 260 is configured to display a timbre creation entrance in the content generation platform; the controller 250 is configured to: determine the voice entry prompt information in response to the trigger operation for the timbre creation entrance; the display 260 is also configured to display the voice entry prompt information; wherein the voice entry prompt information is used to instruct the terminal to collect the user's voice to obtain a user voice segment, the user voice segment is used to generate a user timbre template corresponding to the user, and the user timbre template is used to generate multimedia content in the content generation platform, and the timbre of the audio data in the multimedia content matches the timbre of the user.

[0097] It is worth noting that, when generating multimedia content, the prior art only provides preset timbre templates in the content generation platform for users to select timbre, and the timbre selection is relatively limited. Based on this, the present application provides a display device 200, which can generate a user timbre template corresponding to the user by inputting a user voice segment, so that the timbre of the audio data of the generated multimedia content can match the timbre of the user. In other words, the timbre can be customized for the multimedia content, thereby improving the user experience.

[0098] For details, see Figure 5A The method for generating a timbre template provided in this application comprises the following steps:

[0099] Step 502: Display a tone creation portal in the content generation platform.

[0100] The content generation platform may be a platform for personalized customization of multimedia content, and the content generation platform may also play the multimedia content stored therein and customized therein. The multimedia content may include video, audio, and text. For example, the multimedia content in this embodiment may be a picture book.

[0101] Exemplarily, the content generation platform may be a picture book application, which has multiple picture books preset and supports users to create new picture books. Each picture book includes multiple picture book pictures, and the character images in the multiple picture book pictures are unified; each picture book picture is generated based on a story text in the picture book story, and each picture book picture corresponds to audio data, which is obtained based on the story text corresponding to the picture book picture.

[0102] For example, the content generation platform may be an artificial intelligence (AI) picture book generation tool, and a custom picture book may be customized in the content generation platform. For example, the content generation platform may analyze and process the user's input text and generate a customized picture book that matches the user's input text.

[0103] Among them, the timbre creation entrance can be an entrance for users to customize timbre. Exemplarily, the timbre creation interface can be entered by triggering the timbre creation entrance. In this embodiment, the display position of the timbre creation entrance in the content generation platform can be set as needed, without any limitation. Exemplarily, it can be set in the upper right corner of the content generation platform, or in the upper left corner of the content generation platform, or in the middle of the content generation platform. In this embodiment, the triggering operation for the timbre creation entrance can be a click operation or a double-click operation, without any limitation.

[0104] Optionally, when the display device 200 is turned on, the display 260 displays a preset display homepage, in which a platform entrance of a content generation platform is displayed or hidden. When there is a need to generate multimedia content, the user can manipulate the control device 100 to interact with the display device 200, for example, manipulating the control device 100 to touch the platform entrance corresponding to the content generation platform. The controller 250 displays the content generation platform in response to a trigger operation on the platform entrance. Further, in response to a picture book customization operation in the content generation platform, a timbre creation entrance is displayed.

[0105] See also Figure 5B The schematic diagram of the content generation platform shown in FIG. 1 shows, in which a picture book customization entrance is provided in the content generation platform. When a picture book customization entrance (such as Figure 5BIn the case of a touch operation such as a click operation, it is determined that there is a customized operation for the picture book in the content generation platform, and the sound color creation entrance is displayed. Figure 5C The timbre creation entry display interface shown in the figure displays a template card corresponding to a preset timbre template (such as Figure 5C "Voice 1", "Voice 2" and "Voice 3" in the Figure 5C "Clone My Voice" in .

[0106] Step 504: In response to the triggering operation on the timbre creation entry, display the voice input prompt information.

[0107] Among them, the voice input prompt information is used to instruct the terminal to collect the user's voice to obtain a user voice segment, the user voice segment is used to generate a user timbre template corresponding to the user, the user timbre template is used to generate multimedia content in the content generation platform, and the timbre of the audio data in the multimedia content matches the timbre of the user.

[0108] The voice input prompt information may be information for prompting the user to input voice. For example, the voice input prompt information may be a pre-set text, a pre-set voice, or a pre-set graphic information. The present application does not impose any limitation on the display content and display form of the voice input prompt information.

[0109] The user voice segment may be a recording information input by the user. For example, the user voice segment may be a recording information randomly input by the user; or may be a recording information input by the user reading a preset text. This application does not impose any limitation on the voice content of the user voice segment.

[0110] In an optional embodiment, the controller 250 responds to a trigger operation for a tone creation entry and determines voice entry prompt information, including: the controller 250 responds to a trigger operation for a tone creation entry, displays a first option corresponding to a first entry method through a display, and displays a second option corresponding to a second entry method, the first entry method refers to a method of recording voice using a mobile terminal 300, and the second entry method refers to a method of recording voice using a control device 100, when a selection operation for the first option is received, the voice entry prompt information corresponding to the first entry method is determined; when a selection operation for the second option is received, the voice entry prompt information corresponding to the second entry method is determined.

[0111] The user's selection operation for the first option may be triggered by voice or by a touch operation in the content generation platform, and this application does not impose any limitation on this.

[0112] In an optional implementation, the controller 250 randomly selects an entry method from the first entry method and the second entry method in response to a trigger operation for the timbre creation entry, and displays voice entry prompt information corresponding to the selected entry method.

[0113] In an optional embodiment, the controller 250 determines the voice entry prompt information in response to a trigger operation for a timbre creation entry, including: the controller 250 responds to the trigger operation for a timbre creation entry and uses preset voice entry prompt information as the voice entry prompt information.

[0114] Furthermore, when the controller 250 determines the voice input prompt information, the display 260 may display the voice input prompt information.

[0115] It should be noted that the terminal can collect the user's voice according to the voice input prompt information, and output the collected user voice segment to the controller 250. Exemplarily, the terminal can be the control device 100, and the voice input prompt information can be a text prompt information, such as "Please press and hold the voice input button for voice collection". Correspondingly, the user collects the user's voice by pressing the voice input button set on the control device 100.

[0116] Exemplarily, the terminal may be a mobile terminal 300, and the voice entry prompt information may include a graphic code, such as a QR code. Accordingly, the mobile terminal 300 scans the QR code, enters the voice entry interface, and collects user voice in the voice entry interface.

[0117] Furthermore, the user voice segment can be used to generate a user timbre template corresponding to the user, wherein the user timbre template is used to generate multimedia content in the content generation platform, and the timbre of the audio data in the multimedia content matches the timbre of the user. Optionally, the user timbre template can be a voice package. Accordingly, based on the multimedia content and the voice package, multimedia content in which the timbre of the audio data matches the timbre of the user can be generated.

[0118] Optionally, the user timbre template may also be a model constructed based on a common neural network. Accordingly, the user timbre template may also be generated in the following manner: obtaining the user voice features in the user voice segment, adjusting the model parameters of the preset audio generation model according to the user voice features, obtaining the target audio generation model corresponding to the user, and using the target audio generation model as the user timbre template.

[0119] Correspondingly, when the user timbre template is a target audio generation model, the multimedia content can be input into the target audio generation model to obtain multimedia content whose timbre matches the timbre of the user.

[0120] In the above-mentioned display device and timbre template generation method, the display 260 displays the timbre creation entrance in the content generation platform, and accordingly, the controller 250 determines the voice entry prompt information in response to the trigger operation for the timbre creation entrance, and displays the voice entry prompt information in the display 260. Among them, the voice entry prompt information is used to indicate that the user's voice is collected to obtain a user voice segment. The user voice segment is used to generate a user timbre template corresponding to the user. The user timbre template is used to generate multimedia content in the content generation platform, and the timbre of the audio data in the multimedia content matches the timbre of the user. In the above-mentioned display device and timbre template generation method, in the process of generating multimedia content using the display device 200, a user timbre template that matches the timbre of the user can be obtained by collecting the user voice segment, and the user timbre template can be used to make the timbre of the multimedia content match the timbre of the user as close as possible, so that multimedia content that meets the user's needs can be quickly generated, and the generation efficiency of multimedia content is improved. And the above-mentioned display device 200 can improve the personalization of the timbre presented in the multimedia content.

[0121] In some exemplary embodiments, when the terminal is a mobile terminal 300, the controller 250 is further configured to: determine the voice input prompt information including a graphic code when determining the voice input prompt information. The graphic code is used for the terminal to display a voice input entrance, and the voice input entrance is used for the terminal to collect voice to obtain a user voice segment.

[0122] Exemplarily, the graphic code may be a QR code or a barcode, and the present application does not impose any limitation on the type of the graphic code.

[0123] See also Fig. 6A The voice input prompt information display interface schematic diagram shown, the voice input prompt information can include a graphic code and voice input process information.

[0124] In an optional implementation, the user can display the voice input entrance by scanning the graphic code through the terminal. Figure 6B The voice input entry display schematic diagram shown in FIG. 1 may include a voice input entry (such as Figure 6B Start Recording in the Figure 6B "Recording Preparation" and the preparation prompts below) and rule agreement prompts (such as Figure 6B "I have read and agreed to the Voice Customization Rules").

[0125] Specifically, when the rule agreement prompt information is selected, the touch operation on the voice input entry can be performed. Optionally, when the rule agreement prompt information is not selected, the voice input entry cannot be touched. Exemplarily, in the two cases of being touchable and not touchable, the display form of the voice input entry can be different.

[0126] In an optional implementation, the user can display the voice recording mode selection interface by scanning the graphic code. The voice recording mode selection interface displays mode cards corresponding to at least two voice recording modes, and different voice recording modes correspond to different voice recording contents. Figure 6C The voice recording mode selection interface shown in FIG. 1 shows a first recording mode (such as Figure 6C in the "standard mode") and the second entry mode (such as Figure 6C The mode cards corresponding to the two voice input modes ("children's customization") are shown in the figure. The mode cards also include description information of the corresponding voice input modes. It should be noted that Figure 6C The two voice input modes shown are merely examples provided in this application and are not intended to limit the number of voice input modes.

[0127] Furthermore, when the terminal detects a touch operation on a mode card corresponding to a voice entry mode, the terminal displays a voice entry entry corresponding to the corresponding voice entry mode.

[0128] In the above embodiment, the voice input portal can be displayed by the terminal. Since the voice input portal is used by the terminal to collect voice to obtain user voice segments, the method of obtaining user voice segments can be made more efficient and flexible.

[0129] In some exemplary embodiments, when the controller 250 determines the voice entry prompt information including the graphic code, it is further configured to: determine the voice entry prompt information including the graphic code when the designated video application in the display device 200 is in a logged-in state.

[0130] The designated video application refers to an application program installed in the display device 200 and can be used to provide a video viewing service.

[0131] It should be noted that, in order to improve the flexibility of using user timbre templates, in this embodiment, the user timbre template customized by the user corresponds to the user account. That is to say, no matter which display device 200 the user logs in to his account, the display device 200 can display the user timbre template customized by the user in history.

[0132] Based on this, when the controller 250 determines the voice entry prompt information containing the graphic code, it is necessary to ensure that the specified video application in the display device 200 is in a logged-in state. Exemplarily, the specified video application in the display device 200 has different corresponding login status identifiers in the logged-in state and the non-logged-in state. For example, when the specified video application is in the logged-in state, the login status identifier is 1, and when it is in the non-logged-in state, the login status identifier is 0. In this embodiment, the controller 250 can obtain the login status identifier of the specified video application, and determine whether the specified video application is in the logged-in state according to the login status identifier.

[0133] It is understandable that the controller 250 is further configured to generate a login page for the designated video application when the designated video application in the display device 200 is in a non-logged-in state.

[0134] The login page of the designated video application may be a pre-set page for logging in to the designated video application. For example, the login page of the designated video application may be a login page including a graphic code, or a login page including an account input entry and a password input entry, and this application does not make any limitation on this.

[0135] Specifically, in this embodiment, when it is determined that the specified video in the display device 200 is in a non-logged-in state, the controller 250 generates a login page for the specified video application. Accordingly, the display device 200 is also configured to: display a login page for the specified video application, and the login page is used to log in to the specified video application. Figure 7 The login page schematic diagram shown in FIG. 1 shows a graphical code (such as Figure 7 ), and other login options (such as Figure 7 You can scan the graphic code through the terminal to log in to the specified video application. You can also select the touch operation of the entrance for other login methods and choose other login methods to log in.

[0136] In the above embodiment, the controller 250 determines the voice input prompt information including the graphic code when the designated video application is in the logged-in state in the display device 200. When the designated video application is in the unlogged-in state in the display device 200, the controller 250 generates a login page for the designated video application, so that the generated user voice template can match the user logged in by the designated video application.

[0137] In some exemplary embodiments, the controller 250 is further configured to: receive a user voice segment sent by the terminal; and send the user voice segment to the server, so that the server generates a user timbre template corresponding to the user according to the user voice segment.

[0138] In an optional implementation, when the terminal is a mobile terminal 300, upon detecting a touch operation on a voice entry entry, the mobile terminal 300 displays a voice entry interface and uses the voice information entered by the user in the voice entry interface as a user voice segment.

[0139] See also Fig. 8A As shown in the voice input interface schematic diagram, the voice input interface of the mobile terminal 300 displays preset text and a voice input control, and prompts the user to long press the voice input control and read out the preset text for voice input in a quiet environment.

[0140] Further, see Figure 8B The timbre naming diagram shown in the figure displays the timbre naming interface when it is detected that the user has completed the entry of the preset text and the voice information entered by the user completely matches the preset text. The timbre can be named in the timbre naming interface, or the picture corresponding to the timbre can be changed in the timbre naming interface.

[0141] In an optional implementation, when the terminal is the control device 100, the controller 250 may use the voice information entered by the user in the voice entry prompt information interface as the user voice segment.

[0142] See also Figure 8C The voice input interface diagram shown in FIG. 1 shows a preset text on the display 260, and is used to prompt the user to perform voice input in a quiet environment by pressing a voice input button set on the control device 100 and reading the preset text (such as Figure 8C Please keep the surrounding environment quiet, press the voice input button on the control device 100, and read the text on the screen).

[0143] It should be noted that, during the process of voice recording by the user, whether the terminal is a mobile terminal 300 or a control device 100, noise detection can be performed on the user's environment, and when the environmental noise is greater than a preset noise threshold, a prompt message indicating that the environment does not meet the voice recording requirements is output. When the environment meets the voice recording requirements, voice recording continues. The preset noise threshold can be determined based on manual experience or through a large number of experiments, and this application does not impose any limitation on this.

[0144] At the same time, during the user's voice input process, the voice information input by the user can be matched with the preset text, and the text segments that failed to match, the text segments that successfully matched, and the unread text segments in the preset text can be displayed separately.

[0145] It is worth noting that when the voice input mode includes "standard mode" and "child customized", the preset text displayed to the user will be different. For example, the preset text in the standard mode is a complete sentence, and the preset text in the child customized mode is a simple word with pinyin annotated.

[0146] It should be noted that, when the terminal is the control device 100 , when the user performs voice input, the display 260 may also display a voice waveform.

[0147] Exemplarily, the server 400 generates a user timbre template corresponding to the user according to the user voice segment by: inputting the user voice segment into a pre-trained timbre template generation model to generate a user timbre template corresponding to the user.

[0148] Among them, the timbre template generation model can be constructed based on a common neural network, which will not be described in detail here. Further, in the process of training the timbre template generation model, a sample voice segment can be input into the timbre template generation model to obtain a sample timbre template, and the timbre template generation model is trained according to the difference between the sample timbre template and the timbre template label corresponding to the sample voice segment, so as to improve the model accuracy of the timbre template generation model.

[0149] Exemplarily, the server 400 can generate a user timbre template corresponding to the user based on the user voice segment by: determining the user voice feature vector based on the user voice segment, adjusting the model parameters of a pre-trained speech synthesis model based on the user voice feature vector, and using the adjusted speech synthesis model as the user timbre template corresponding to the user.

[0150] In the above embodiment, the controller 250 can obtain the user voice segment in a variety of ways, which improves the flexibility of obtaining the user voice segment. Further, the controller 250 sends the user voice segment to the server 400 corresponding to the display device 200, so that the server 400 can generate a user timbre template corresponding to the user according to the user voice segment.

[0151] In some exemplary embodiments, when the terminal is the control device 100, the controller 250 may also output a preset prompt text as voice input prompt information when a trigger operation for a timbre creation entry is detected, and display it on the display 260. Fig. 9 The voice input prompt information display schematic diagram shown in FIG. 2 displays “Please press and hold the voice input button to collect voice” on the display 260 , and uses the voice information randomly input by the user through the control device 100 as the user voice segment.

[0152] Optionally, the controller 250 is further configured to: when a trigger operation for a timbre creation entry is detected, output a voice recording instruction, where the voice recording instruction is used to instruct the user voice segment to be collected through the terminal.

[0153] Specifically, in this embodiment, when the controller 250 detects a trigger operation for a tone creation entry, it sends a voice entry instruction to the control device 100 corresponding to the display device 200 to instruct the control device 100 to use the voice information entered by the user as a user voice segment.

[0154] In the above embodiment, when the controller 250 detects a trigger operation for a timbre creation entry, it outputs a voice input instruction, thereby instructing the control device 100 to obtain a user voice segment, which can make the method of obtaining the user voice segment more flexible and efficient.

[0155] In some exemplary embodiments, the controller 250 is further configured to: obtain a template card corresponding to the user timbre template. The display 260 is further configured to: display the template card corresponding to the user timbre template in the content generation platform. Among them, the template card can be used to indicate different user timbre templates, that is, different user timbre templates correspond to different template cards. Exemplarily, the template card can be a picture, and for the user timbre template of user 1, the corresponding template card can be picture 1 set by user 1. The template card can also be a preset card, and the card description information of the template cards corresponding to different user timbre templates is different. For example, for the user timbre template of user 1, the card description information of the template card corresponding to it can be "User 1 timbre".

[0156] It should be noted that, when the template card is a preset card, the card attribute information such as the shape and background color of the preset card can be determined based on manual experience, and this application does not impose any limitation. For example, the preset card can be a blue rectangular card.

[0157] Furthermore, the controller 250 is also configured to: when the template card corresponding to the user voice template is in a selected state, if a content generation instruction is detected, send a content generation instruction indicating the user voice template to the server, so that the server responds to the content generation instruction and generates multimedia content according to the user voice template; accordingly, the display 260 is also configured to: display the playback entrance of the multimedia content in the content generation platform.

[0158] It should be noted that when the specified multimedia content is selected and confirmed to be played in the content generation platform, the selected multimedia content can be played in video mode. For example, when the multimedia content is a picture book video, during the picture book video playback, the picture book picture can be displayed statically or with sliding special effects. The story text corresponding to the picture page is also displayed on the picture book picture, and the display duration of each picture book picture corresponds to the broadcast audio duration of the picture book picture.

[0159] Optionally, during the picture book video playback, the switching display of two adjacent picture book pictures can be continuous, but the broadcast audio of the two adjacent picture book pictures is played at a preset time interval (for example, 1 second). In order to increase the fun of the picture book video playback, in this embodiment, there can be background music during the picture book playback, and the background music matches the story content of the picture book.

[0160] It is understandable that before multimedia generation, the display device 200 can be customized in multiple dimensions in the content generation platform according to multimedia generation requirements, for example, including timbre customization, character customization, background sound customization, story content customization, and other aspects.

[0161] Specifically, when the template card corresponding to the user timbre template is in a selected state, the controller 250 determines that the user selects the user timbre template corresponding to the template card as the timbre template for generating multimedia content. Further, after the user customizes each dimension, a content generation instruction can be generated by triggering the content generation entry in the content generation platform. Accordingly, when the controller 250 detects a trigger operation for the content generation entry, it generates a content generation instruction, which indicates that the timbre of the audio data in the generated multimedia content is the target timbre, wherein the target timbre is the timbre generated by the user timbre template corresponding to the template card in the selected state.

[0162] In the above embodiment, the template card corresponding to the user's timbre template is displayed on the display, and a content generation instruction can be created by selecting the template card on the display, so that the server generates multimedia content according to the user's timbre template, which can improve the operability and efficiency of the multimedia content generation method. Furthermore, the playback entrance of the multimedia content is displayed on the display, which is convenient for the user to watch the multimedia content.

[0163] In some exemplary embodiments, since the user voice template is bound to a target account, the target account is the account used to log in to the specified video application in the display device 200. Therefore, when the controller 250 obtains the template card corresponding to the user voice template, it is also configured to: when the specified video application in the display device 200 is in a logged-in state, obtain the template card corresponding to the user voice template and the template cards corresponding to at least one preset voice template.

[0164] Among them, the user timbre template is a user timbre template generated by the timbre customization performed by the user corresponding to the target account before the current moment.

[0165] The controller 250 is further configured to: when the designated video application in the display device 200 is not logged in, obtain a template card corresponding to at least one preset tone template.

[0166] Accordingly, when the display 260 displays the template card corresponding to the user's timbre template in the content generation platform, it is also configured to: when the designated video application in the display device 200 is in a logged-in state, display the template card corresponding to the user's timbre template in the content generation platform, and display the template card corresponding to at least one preset timbre template. Fig.10 As shown in the schematic diagram of template card display, when the specified video application is in the logged-in state in the display device 200, the display 260 displays the template card corresponding to the user's timbre template (such as Fig.10 1 sound"), and display the template card corresponding to at least one preset sound template (such as Fig.10 "Tone 1" and "Tone 2" in the Fig.10 "Clone My Voice" in .

[0167] The display 260 is further configured to: when the designated video application in the display device 200 is not logged in, display a template card corresponding to at least one preset tone template. Figure 5C As shown in the schematic diagram of template card display, when the specified video application is in the unlogged-in state in the display device 200, the display 260 displays at least one template card corresponding to the preset tone template (such as Figure 5C 1, 2, and 3 in the Figure 5C "Clone My Voice" in .

[0168] In the above embodiment, when the designated video application is not logged in in the display device 200, only the template card corresponding to the preset timbre template is displayed. When the designated video application is logged in in the display device 200, the template card corresponding to the user timbre template is displayed. Ensuring that the template card corresponding to the user timbre template customized by the user is displayed only when the user is logged in can improve the security of the user timbre template and avoid leakage of the user's timbre privacy.

[0169] In some exemplary embodiments, the display 260 is further configured to: display input text in the content generation platform; wherein the input text may be multimedia generation requirement text input by the user. For example, the input text may be "I want to generate a story, the protagonist of the story is a rabbit".

[0170] When the controller 250 sends a content generation instruction indicating a user's voice template to the server 400, it is also configured to: send a content generation instruction carrying input text and indicating the user's voice template to the server 400; when the server 400 generates multimedia content based on the user's voice template, it is also configured to: generate audio data whose timbre matches the user's timbre based on the user's voice template and the input text; and generate multimedia content based on the audio data.

[0171] For example, when the server 400 generates multimedia content according to the user's timbre template, it can first generate a story picture book according to the input text, then input the text information in the story picture book into the user's timbre template, generate audio data whose timbre matches the user's timbre, and combine the audio data with the story picture book to generate multimedia content. The multimedia content is a story picture book in which the protagonist of the story is a rabbit, which can be a story video or an audio picture book.

[0172] In the above embodiment, a story picture book of multimedia content is generated by processing the input text, and audio data whose timbre matches the user's timbre is generated according to the user's timbre template, and then the final multimedia content is generated according to the story picture book and the audio data. The picture book content can be intelligently customized, and combined with the audio data that matches the user's timbre, the personalized customization of multimedia content can be improved.

[0173] In some exemplary embodiments, the user timbre template is a target audio generation model corresponding to the user, and the target audio generation model includes a phoneme encoder and a spectrum generator, and the phoneme encoder and the spectrum generator respectively include at least one network layer after parameter adjustment by the user voice segment.

[0174] The audio encoder is used to process the phoneme to obtain the phoneme coding features corresponding to the phoneme. The spectrum generator can also be called a decoder, which is used to process the phoneme coding features to generate a speech spectrum, for example, the speech spectrum can be a Mel spectrum.

[0175] See also Fig.11 As shown, when the server 400 generates audio data whose timbre matches the timbre of the user according to the user timbre template and the input text, it is also configured as follows:

[0176] Step 1102, obtaining a phoneme vector generated based on the input text.

[0177] Among them, the phoneme vector refers to the vector composed of phonemes corresponding to the text.

[0178] Exemplarily, in this embodiment, text information of a picture story book generated based on an input text may be obtained, and a phoneme vector generated based on the text information of the picture story book may be obtained.

[0179] Step 1104, input the phoneme vector into the phoneme encoder, encode each phoneme in the phoneme vector to obtain the corresponding phoneme coding feature, and obtain the phoneme coding feature vector composed of the phoneme coding features.

[0180] Exemplarily, in this embodiment, the phoneme vector is input into the phoneme encoder, and the phoneme encoder encodes each phoneme in the phoneme vector to obtain a phoneme coding feature corresponding to each phoneme. Further, each phoneme coding feature is combined to form a phoneme coding feature vector.

[0181] Among them, the phoneme encoding features corresponding to each phoneme include the fundamental frequency (F0) and energy information (energy) corresponding to each phoneme. The fundamental frequency affects the pitch of the synthesized target speech to a certain extent, and thus also has a certain impact on the tone of the user's voice segment. The energy information affects the volume of the target speech and can also reflect the logical stress of the user's voice segment.

[0182] It should be noted that the phoneme encoder includes a phoneme embedding layer and a phoneme encoding layer. In order to make the audio data generated by the vocoder more closely match the user's timbre, in this embodiment, at least one of the phoneme embedding layer and the phoneme encoding layer has a network layer added with parameters adjusted with the user's voice fragment (also called conditional layer normalization).

[0183] Among them, the phoneme embedding layer can process the input phonemes and convert them into continuous features in a high-dimensional space. The phoneme encoding layer can further process and transform the phoneme features to generate phoneme encoding features suitable for subsequent tasks.

[0184] Among them, conditional layer normalization can normalize the input data. Since the network parameters of conditional layer normalization are adjusted based on the user voice features corresponding to the user voice segment, the data after conditional layer normalization can better match the user voice features.

[0185] Exemplarily, in this embodiment, in order to improve the timbre matching effect, conditional layer normalization can be added after the phoneme embedding layer and the phoneme coding layer. Specifically, the phoneme vector is input into the phoneme embedding layer, and the result output by the phoneme embedding layer is input into the conditional layer normalization; the result output by the conditional layer normalization is input into the phoneme coding layer, and the result output by the phoneme coding layer is input into the conditional layer normalization, and each phoneme in the phoneme vector is encoded to obtain the corresponding phoneme coding feature.

[0186] Step 1106: input the phoneme encoding feature vector into a spectrum generator to generate a speech spectrum corresponding to the input text.

[0187] Exemplarily, in this embodiment, the phoneme coding feature vector is input into the spectrum generator, and the spectrum generator decodes each phoneme coding feature in the phoneme coding feature vector to generate a speech spectrum corresponding to the text information of the story picture book.

[0188] It should be noted that the spectrum generator includes a speech feature prediction layer and a spectrum decoder, wherein the speech feature prediction layer includes a global style tag module (Global Style Tokens, GST) and a variance adapter. Correspondingly, the spectrum generator includes a global style tag module (Global Style Tokens, GST), a variance adapter and a spectrum decoder.

[0189] The speech feature prediction layer is used to perform speech feature prediction on the phoneme coding features to obtain predicted speech features.

[0190] The model parameters of the GST module are adjusted based on the user voice features corresponding to the user voice segment, and the GST module is used to process the phoneme coding features according to the user voice features so that the processed phoneme coding features include the user voice features. In other words, the processed phoneme coding features include the user voice features.

[0191] Among them, the variance adapter is used to optimize the features of the processed phoneme coding features so that the subsequently generated speech spectrum can be better integrated with the user's speech features.

[0192] Among them, the spectrum decoder is used to restore the input data to the original speech spectrum to ensure the accurate reproduction of the audio information.

[0193] In order to make the audio data generated by the vocoder more closely match the user's timbre, in this embodiment, at least one of the GST module, variance adapter, and spectrum decoder has a network layer added with parameters adjusted with the user's voice fragment (also referred to as conditional layer normalization).

[0194] Exemplarily, in this embodiment, in order to improve the timbre matching effect, conditional layer normalization can be added after the GST module, the variance adapter and the spectrum decoder. Specifically, the phoneme encoding feature vector is input into the GST module, and the result output by the GST module is input into the conditional layer normalization; the result output by the conditional layer normalization is input into the variance adapter, and the result output by the variance adapter is input into the conditional layer normalization; and the result output by the conditional layer normalization is input into the spectrum decoder to obtain the speech spectrum corresponding to the text information of the story picture book.

[0195] Step 1108: Input the speech spectrum into a vocoder to generate audio data whose timbre matches the timbre of the user.

[0196] The vocoder is used to process the speech spectrum and generate audio data corresponding to the text information of the story picture book, and the timbre of the audio data matches the timbre of the user.

[0197] In the above embodiment, the user timbre template is a target audio generation model including a phoneme encoder and a spectrum generator, and the phoneme encoder and the spectrum generator each include at least one network layer after parameter adjustment by the user's voice segment. Since the network layer after parameter adjustment by the user's voice segment is based on the user's voice segment parameter adjustment, adding the network layer can make the generated audio data more compatible with the user's timbre.

[0198] In some exemplary embodiments, the spectrum generator includes a speech feature prediction layer and a spectrum decoder; see Fig.12 As shown, the server inputs the phoneme encoding feature vector into the spectrum generator to generate the speech spectrum corresponding to the input text, and is also configured as follows:

[0199] Step 1202, input the phoneme coding feature vector into the speech feature prediction layer, and predict the speech feature based on the attention mechanism layer and the phoneme coding feature vector in the speech feature prediction layer.

[0200] Step 1204: input the speech features into a spectrum decoder to generate a speech spectrum corresponding to the input text.

[0201] The speech feature prediction layer is used to perform speech feature prediction on the phoneme coding features to obtain predicted speech features.

[0202] In the traditional technology, the speech feature prediction layer includes a GET module and a variance adapter. Different from the traditional technology, in this embodiment, the speech feature prediction layer includes a GST module and an attention mechanism layer. Since the attention mechanism layer is a network layer in the Transformer model, and the attention mechanism layer in the Transformer model can encode the position information of the phoneme coding features, the attention mechanism layer is used instead of the variance adapter. Therefore, compared with the traditional technology, this embodiment can make the position coding information of each phoneme coding feature more accurate, thereby making the speech features output by the speech feature prediction layer more accurate.

[0203] In the above embodiment, the variance adapter constructed based on the convolutional neural network is replaced by the attention mechanism layer, which can make the position encoding information of each phoneme encoding feature more accurate, thereby making the audio data generated by decoding based on the spectrum decoder more coherent and smooth.

[0204] In some exemplary embodiments, an application timing diagram of the method for generating a timbre template of the present application is provided. As shown in FIG. 13 , the method includes the following steps:

[0205] 1. The display 260 displays a sound creation entrance in the content generation platform. When the content generation platform detects an opening operation for the content generation platform, the content generation platform responds to the instruction to open the content generation platform and loads the platform homepage of the content generation platform; illustratively, the platform homepage of the content generation platform displays a multimedia content creation entrance;

[0206] 2. The content generation platform responds to the instruction to open multimedia content creation, displays a timbre creation portal, and responds to the triggering operation of the timbre creation portal to generate a timbre creation instruction;

[0207] 3. The content generation platform sends the tone creation instruction to the controller 250;

[0208] 4. The designated video application monitors the login status of its own application in real time and synchronizes the login status to the controller;

[0209] 5. When the controller 250 detects that the designated video application is in a logged-in state, it responds to the tone creation instruction, determines the voice input prompt information including the graphic code, and displays the voice input prompt information on the display 260;

[0210] 6. The content generation platform responds to the voice input operation, receives the user voice segment, and sends the user voice segment to the controller 250;

[0211] 7. The controller 250 sends the user voice segment to the server 400, so that the server 400 generates a user voice template corresponding to the user according to the user voice segment;

[0212] 8. The content generation platform monitors the generation of the user's timbre template, and when the user's timbre template is generated, displays the template card corresponding to the user's timbre template;

[0213] 9. The content generation platform responds to the template card selection operation and the input text, and generates a content generation instruction that picks up the input text and indicates the user's timbre template;

[0214] 10. The content generation platform sends the content to the controller 250;

[0215] 11. The controller 250 sends a content generation instruction to the server 400, so that the server 400 generates multimedia content, and the timbre of the audio data in the multimedia content matches the timbre of the user;

[0216] 12. The content generation platform monitors the multimedia content generation status and displays the multimedia playback entrance when the multimedia content is generated.

[0217] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0218] Based on the same inventive concept, the embodiment of the present application further provides a device for realizing the above-mentioned timbre template generation, comprising:

[0219] The entry display module is used to display the tone creation entry in the content generation platform.

[0220] The information display module is used to display voice input prompt information in response to a trigger operation on a timbre creation entry.

[0221] Among them, the voice input prompt information is used to instruct the terminal to collect the user's voice to obtain a user voice segment, the user voice segment is used to generate a user timbre template corresponding to the user, the user timbre template is used to generate multimedia content in the content generation platform, and the timbre of the audio data in the multimedia content matches the timbre of the user.

[0222] In one embodiment, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0223] Displaying a sound creation portal in a content generation platform;

[0224] In response to a trigger operation on a timbre creation entry, displaying a voice input prompt message;

[0225] Among them, the voice input prompt information is used to instruct the terminal to collect the user's voice to obtain a user voice segment, the user voice segment is used to generate a user timbre template corresponding to the user, the user timbre template is used to generate multimedia content in the content generation platform, and the timbre of the audio data in the multimedia content matches the timbre of the user.

[0226] In one embodiment, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0227] Displaying a sound creation portal in a content generation platform;

[0228] In response to a trigger operation on a timbre creation entry, displaying a voice input prompt message;

[0229] Among them, the voice input prompt information is used to instruct the terminal to collect the user's voice to obtain a user voice segment, the user voice segment is used to generate a user timbre template corresponding to the user, the user timbre template is used to generate multimedia content in the content generation platform, and the timbre of the audio data in the multimedia content matches the timbre of the user.

[0230] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0231] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0232] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A display device, characterized in that: include: A display configured to display a sound creation portal in a content generation platform; The controller is configured as: In response to a triggering operation for the timbre creation entry, determining voice entry prompt information; The display is further configured to display the voice input prompt information; Among them, the voice input prompt information is used to instruct the terminal to collect the user's voice to obtain a user voice segment, the user voice segment is used to generate a user timbre template corresponding to the user, the user timbre template is used to generate multimedia content in the content generation platform, and the timbre of the audio data in the multimedia content matches the timbre of the user.

2. The display device according to claim 1, characterized in that When the controller determines the voice input prompt information, it is further configured to: Determine the voice input prompt information including the graphic code; wherein the graphic code is used for the terminal to display a voice input entrance, and the voice input entrance is used for the terminal to collect voice to obtain a user voice segment.

3. The display device according to claim 2, characterized in that When the controller determines the voice input prompt information including the graphic code, the controller is further configured to: When the designated video application is in a logged-in state in the display device, determining voice input prompt information including a graphic code; The controller is further configured to: generate a login page for the designated video application when the designated video application is in a non-logged-in state in the display device; The display is further configured to: display a login page for the specified video application, where the login page is used to log in to the specified video application.

4. The display device according to claim 1, characterized in that The controller is also configured to: Receiving the user voice segment sent by the terminal; The user voice segment is sent to a server, so that the server generates a user timbre template corresponding to the user according to the user voice segment.

5. The display device according to claim 1, characterized in that The controller is further configured to: When a trigger operation for the timbre creation entry is detected, a voice recording instruction is output, where the voice recording instruction is used to instruct the terminal to collect a user voice segment.

6. The display device according to claim 1, characterized in that The controller is further configured to: obtain a template card corresponding to the user timbre template; The display is further configured to: display a template card corresponding to the user timbre template in the content generation platform; The controller is further configured to: when the template card corresponding to the user timbre template is in a selected state, if a content generation instruction is detected, send a content generation instruction indicating the user timbre template to the server, so that the server generates multimedia content according to the user timbre template in response to the content generation instruction; The display is further configured to: display a playback entrance of the multimedia content in the content generation platform.

7. The display device according to claim 5, characterized in that The user voice template is bound to a target account, and the target account is an account used to log in to a specified video application in the display device; When the controller obtains the template card corresponding to the user timbre template, the controller is further configured to: When the designated video application is in a logged-in state in the display device, obtaining a template card corresponding to the user timbre template and a template card corresponding to at least one preset timbre template respectively; The controller is further configured to: When the designated video application in the display device is in a non-logged-in state, obtaining a template card corresponding to at least one preset timbre template; When the display displays the template card corresponding to the user timbre template in the content generation platform, the display is further configured to: When the designated video application is in a logged-in state in the display device, a template card corresponding to the user timbre template is displayed in the content generation platform, and a template card corresponding to at least one preset timbre template is displayed respectively; The display is further configured as: When the designated video application is in a non-logged-in state in the display device, a template card corresponding to at least one preset timbre template is displayed.

8. The display device according to claim 6, characterized in that The display is further configured to: display input text in the content generation platform; When the controller sends a content generation instruction for instructing the user timbre template to the server, the controller is further configured to: Sending a content generation instruction carrying the input text and indicating the user voice template to the server; When the server generates multimedia content according to the user timbre template, the server is further configured to: Generating audio data whose timbre matches the timbre of the user according to the user timbre template and the input text; Multimedia content is generated based on the audio data.

9. The display device according to claim 8, characterized in that The user timbre template is a target audio generation model corresponding to the user, the target audio generation model includes a phoneme encoder and a spectrum generator, the phoneme encoder and the spectrum generator each include at least one network layer after parameter adjustment of the user voice segment; When the server generates audio data whose timbre matches the timbre of the user according to the user timbre template and the input text, the server is further configured to: Obtaining a phoneme vector generated based on the input text; Inputting the phoneme vector into the phoneme encoder, encoding each phoneme in the phoneme vector to obtain a corresponding phoneme coding feature, and obtaining a phoneme coding feature vector composed of the phoneme coding features; Inputting the phoneme encoding feature vector into the spectrum generator to generate a speech spectrum corresponding to the input text; The speech spectrum is input into a vocoder to generate audio data having a timbre matching that of the user.

10. The display device according to claim 9, characterized in that The spectrum generator includes a speech feature prediction layer and a spectrum decoder; When the server inputs the phoneme encoding feature vector into the spectrum generator to generate a speech spectrum corresponding to the input text, the server is further configured to: Inputting the phoneme encoding feature vector into the speech feature prediction layer, and predicting speech features based on the attention mechanism layer in the speech feature prediction layer and the phoneme encoding feature vector; The speech feature is input into the spectrum decoder to generate a speech spectrum corresponding to the input text.

11. A method for generating a timbre template, characterized in that: The method is applied to a display device, and the method comprises: Displaying a sound creation portal in a content generation platform; In response to a triggering operation for the timbre creation entry, displaying the voice input prompt information; Among them, the voice input prompt information is used to instruct the terminal to collect the user's voice to obtain a user voice segment, the user voice segment is used to generate a user timbre template corresponding to the user, the user timbre template is used to generate multimedia content in the content generation platform, and the timbre of the audio data in the multimedia content matches the timbre of the user.

Images