Audio generation method, display device, storage medium and program product

Abstract

The embodiment of the invention belongs to the display technology, and provides an audio generation method, a display device, a storage medium and a program product, and the audio generation method comprises the steps: obtaining a generation instruction used for generating audio, and the generation instruction comprises a text instruction or a voice instruction; based on the generation instruction, rule parameters used for representing audio generation requirements are extracted; searching in an audio database according to the rule parameter to obtain a reference audio matched with the rule parameter; extraction is carried out based on a beginning segment of the reference audio, prompt words used for representing audio elements in the beginning segment are obtained, and the audio elements comprise at least one of timbre, rhythm or speed; and through a trained audio generation model, reasoning is performed based on the cue word so as to perform audio customization, and a correspondingly generated audio file is obtained. According to the invention, the requirement of a user for self-expression can be met, and personalized audio customization is realized on the household electrical appliance.

Description

Technical Field

[0001] The embodiments of the present application relate to display technology, and more specifically, to an audio generation method, a display device, a storage medium, and a program product. Background Art

[0002] With the development of intelligent home appliances, users' demand for personalized customization of home appliances is increasing. Personalized customization of home appliances is mainly reflected in the customization of the audio played by home appliances.

[0003] Traditional home appliances are unable to achieve personalized audio customization based on the user's customization needs, resulting in poor use of home appliances.

[0004] Therefore, how to meet users' needs for self-expression and realize personalized customized audio on home appliances is still an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide a display device that can meet the user's needs for self-expression and realize personalized audio customization on home appliances.

[0006] In a first aspect, an embodiment of the present application provides an audio generation method, which is applied to a display device as described in the second aspect below;

[0007] The method comprises:

[0008] Obtaining a generation instruction for generating audio, wherein the generation instruction includes a text instruction or a voice instruction;

[0009] Based on the generation instruction, extracting rule parameters for characterizing audio generation requirements;

[0010] Searching in an audio database according to the rule parameters to obtain reference audio that matches the rule parameters;

[0011] Extracting based on the beginning segment of the reference audio, obtaining prompt words for characterizing audio elements in the beginning segment, wherein the audio elements include at least one of timbre, rhythm or speed;

[0012] The trained audio generation model is used to perform reasoning based on the prompt words to customize the audio and obtain a corresponding generated audio file.

[0013] The audio generation method provided in this embodiment, after obtaining the generation instruction for generating audio, extracts the rule parameters for characterizing the audio generation requirements based on the generation instruction. Then, according to the rule parameters, the audio database is searched to obtain the reference audio that matches the rule parameters. Based on the beginning segment of the reference audio, the prompt words for characterizing the audio elements in the beginning segment are extracted. Through the trained audio generation model, the prompt words are inferred to perform audio customization and obtain the corresponding generated audio file.

[0014] That is to say, the audio file is associated with the generation instruction, and when the generation instruction is changed, the audio file will also be changed. The user can customize the desired audio file through the generation instruction to meet the need for self-expression. Compared with the disadvantage that traditional household appliances cannot realize personalized customization of audio according to the customization needs of users, the method provided in this embodiment can realize personalized customization of audio according to the customization needs of users, meet the needs of users for self-expression, and improve the use effect of household appliances.

[0015] In one embodiment, after obtaining the corresponding generated audio file, the method further includes:

[0016] Audio playback is performed based on the audio file, and a corresponding visual music score is displayed.

[0017] The method provided in this embodiment displays the corresponding visual music score when playing audio based on the audio file. The purpose of this is to enable the user to know the audio playing process, check whether the visual music score is wrong, etc., so as to improve the user's autonomy. The user can also select the audio clips that he does not like or have errors through the visual music score, so as to improve the effect of personalized audio customization.

[0018] In one embodiment, after obtaining the corresponding generated audio file, the method further includes:

[0019] Based on the audio segment to be adjusted in the audio file, the audio file is updated to obtain an updated audio file.

[0020] The method provided in this embodiment enables the user to independently select the audio segment to be adjusted, and can also update the audio file of the audio segment to be adjusted, and finally play the audio that the user is satisfied with. By selecting the audio segment to be adjusted, the user can further improve the playing effect of the personalized customized audio.

[0021] In one embodiment, the updating of the audio file based on the audio segment to be adjusted in the audio file to obtain the updated audio file includes:

[0022] In response to a selection operation on the visual music score, a corresponding selected audio segment is cut from the audio file; and extraction is performed based on the cut audio file to obtain a corresponding prompt word;

[0023] Through the trained audio generation model, reasoning is performed based on the corresponding prompt words to obtain multiple candidate audio clips;

[0024] A target audio segment is determined from the multiple candidate audio segments, and the target audio segment is filled into the trimmed audio file to obtain an updated audio file.

[0025] In the method provided in this embodiment, the user can perform a selection operation on the visual music score. Correspondingly, in response to the selection operation on the visual music score, the corresponding selected audio segment is trimmed from the audio file, and then multiple candidate audio segments are obtained based on the trimmed audio file. After determining the target audio segment from the multiple candidate audio segments, the target audio segment is filled into the trimmed audio file to obtain an updated audio file.

[0026] Thus, the user can adjust the generated audio file again, so that the user can independently select the audio clip that needs to be adjusted, and can also update the audio file for the audio clip that needs to be adjusted, and finally play the audio that satisfies the user. The user does not need to have music theory knowledge to edit and modify the audio, that is, to achieve interactivity. By selecting the audio clip that needs to be adjusted, the user can further improve the playback effect of the personalized audio. Moreover, the method of updating the audio file by operating the visualized music score provided in this embodiment is not only intuitive, but also simple and efficient to operate, which can improve the customization speed and user experience of the personalized audio.

[0027] In one embodiment, the method further comprises:

[0028] Extracting based on the sample audio file to obtain musical note information, and using the musical note information and the sample audio file as training samples;

[0029] Based on the training samples, the audio generation model is trained to obtain a trained audio generation model.

[0030] The method provided in this embodiment describes how to train an audio generation model. Specifically, firstly extract based on a sample audio file to obtain musical note information, and use the musical note information and the sample audio file as training samples. Then, the audio generation model is trained based on the training samples. Thus, the trained audio generation model can be used to generate audio files that meet the user's personalized customization needs, thereby improving the user's experience of personalized audio customization.

[0031] In one embodiment, the extracting based on the sample audio file to obtain the musical note information includes:

[0032] Extract based on the sample audio file to obtain sample rule parameters;

[0033] The sample rule parameters are converted into computer codes, and the computer codes are converted into binary information as musical note information.

[0034] The method provided in this embodiment describes how to train an audio generation model when the audio generation model includes a model that can only recognize binary information (such as GPT). The trained audio generation model can be used to generate audio files that meet the user's personalized customization needs, thereby improving the user's experience of personalized audio customization.

[0035] In one embodiment, the audio file is an audio interface file for describing audio composition; the method further comprises:

[0036] According to the specified audio format, audio synthesis is performed based on the audio interface file to obtain a playable file corresponding to the specified audio format.

[0037] The method provided in this embodiment allows the user to specify the audio format of the playable file. When the specified audio format is the Midi format, the storage size of the playable file can be reduced, saving resource waste.

[0038] In a second aspect, the present application provides a display device, including:

[0039] a display configured to display content from a broadcast system or network and / or a user interface;

[0040] One or more external device interfaces, for communicating with one or more external devices according to a communication protocol, the communication protocol including at least a Bluetooth protocol;

[0041] and at least one processor interfaced with the display and the one or more external devices;

[0042] The processor is configured to:

[0043] Obtaining a generation instruction for generating audio, wherein the generation instruction includes a text instruction or a voice instruction;

[0044] Based on the generation instruction, extracting rule parameters for characterizing audio generation requirements;

[0045] Searching in an audio database according to the rule parameters to obtain reference audio that matches the rule parameters;

[0046] Extracting based on the beginning segment of the reference audio, obtaining prompt words for characterizing audio elements in the beginning segment, wherein the audio elements include at least one of timbre, rhythm or speed;

[0047] The trained audio generation model is used to perform reasoning based on the prompt words to customize the audio and obtain a corresponding generated audio file.

[0048] In one embodiment, the processor is further configured to:

[0049] Audio playback is performed based on the audio file, and a corresponding visual music score is displayed.

[0050] In one embodiment, the processor is further configured to:

[0051] Based on the audio segment to be adjusted in the audio file, the audio file is updated to obtain an updated audio file.

[0052] In one embodiment, the processor executes updating the audio file based on the audio segment to be adjusted in the audio file to obtain the updated audio file, and is configured to:

[0053] In response to a selection operation on the visual music score, a corresponding selected audio segment is cut from the audio file; and extraction is performed based on the cut audio file to obtain a corresponding prompt word;

[0054] Through the trained audio generation model, reasoning is performed based on the corresponding prompt words to obtain multiple candidate audio clips;

[0055] A target audio segment is determined from the multiple candidate audio segments, and the target audio segment is filled into the trimmed audio file to obtain an updated audio file.

[0056] In one embodiment, the processor is further configured to:

[0057] Extracting based on the sample audio file to obtain musical note information, and using the musical note information and the sample audio file as training samples;

[0058] Based on the training samples, the audio generation model is trained to obtain a trained audio generation model.

[0059] In one embodiment, the processor performs extraction based on the sample audio file to obtain musical note information, and is configured to:

[0060] Extract based on the sample audio file to obtain sample rule parameters;

[0061] The sample rule parameters are converted into computer codes, and the computer codes are converted into binary information as musical note information.

[0062] In one embodiment, the audio file is an audio interface file for describing audio composition; the processor is further configured to:

[0063] According to the specified audio format, audio synthesis is performed based on the audio interface file to obtain a playable file corresponding to the specified audio format.

[0064] In a third aspect, the present application provides a computer-readable storage medium having computer-executable instructions stored thereon. When the computer-executable instructions are executed by a processor, the method described in the first aspect is implemented.

[0065] In a fourth aspect, the present application provides a computer program product, comprising a computer program, which implements the method described in the first aspect when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the implementation methods in the embodiments of the present application or the related technologies, the following is a brief introduction to the drawings required for use in the embodiments or the related technology descriptions. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

[0067] Figure 1 A schematic diagram of a display device provided by an embodiment of the present application;

[0068] Figure 2 Another schematic diagram of a display device provided by an embodiment of the present application;

[0069] Figure 3 Another schematic diagram of a display device provided by an embodiment of the present application;

[0070] Figure 4 Another schematic diagram of a display device provided by an embodiment of the present application;

[0071] Figure 5 A schematic diagram of a flow chart of an audio generation method provided by an embodiment of the present application;

[0072] Figure 6 Another schematic diagram of a flow chart of an audio generation method provided by an embodiment of the present application;

[0073] Figure 7 Another flowchart of an audio generation method provided by an embodiment of the present application;

[0074] Figure 8 Another schematic diagram of a flow chart of an audio generation method provided by an embodiment of the present application;

[0075] Fig. 9 Another flowchart of an audio generation method provided by an embodiment of the present application;

[0076] Fig.10 Another schematic diagram of a flow chart of an audio generation method provided by an embodiment of the present application;

[0077] Fig.11 A schematic diagram of an audio generating device provided in accordance with an embodiment of the present application. DETAILED DESCRIPTION

[0078] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0079] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0080] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0081] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0082] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0083] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0084] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , the user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0085] The mobile terminal 300 can be used as a control device for performing human-computer interaction between a user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 for data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0086] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0087] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.

[0088] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0089] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0090] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0091] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0092] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0093] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0094] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0095] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0096] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0097] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0098] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0099] Figure 3 Some embodiments of the present application provide Figure 1 The hardware configuration diagram of the control device in the figure. Figure 3 As shown, the control device 100 may include: a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0100] The control device 100 is configured to control the display device 200 , and can receive user input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, thereby playing the role of an interaction medium between the user and the display device 200 .

[0101] In some embodiments, the control device 100 may be a smart device, for example, the control device 100 may be installed with various applications for controlling the display device 200 according to user needs.

[0102] In some embodiments, Figure 1 As shown, the mobile terminal 300 or other intelligent electronic devices can play a similar function as the control device 100 after installing the application for controlling the display device 200 .

[0103] The controller 110 includes a processor 112, a RAM 113, a ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication and cooperation between the internal components and the external and internal data processing functions.

[0104] The communication interface 130 implements communication of control signals and data signals with the display device 200 under the control of the controller 110. The communication interface 130 may include at least one of other near field communication modules such as a WiFi chip 131, a Bluetooth module 132, and an NFC module 133.

[0105] The user input / output interface 140 , wherein the input interface includes at least one of other input interfaces such as a microphone 141 , a touch panel 142 , a sensor 143 , and a button 144 .

[0106] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with a communication interface 130, such as a WiFi, Bluetooth, NFC or other module, and can encode the user input command through the WiFi protocol, Bluetooth protocol, or NFC protocol and send it to the display device 200.

[0107] The memory 190 is used to store various operating programs, data and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0108] The power supply 180 is used to provide operating power support for each component of the control device 100 under the control of the controller.

[0109] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system may provide a user interface (control the display device), allow the user to interact with the display device 200, and support the running of various application programs.

[0110] It should be noted that the operating system can be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0111] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 4 As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the system library layer and the kernel layer.

[0112] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on the applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0113] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0114] like Figure 4 As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application package currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0115] In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes, for example, reducing the display window, shaking the display, distorting the display, etc.

[0116] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.

[0117] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 4As shown, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0118] In some embodiments, the display device 200 includes a display 260 as described above, one or more external device interfaces, and at least one processor 112. The display 260 is configured to display content from a broadcast system or network and / or a user interface. The one or more external device interfaces are used to communicate with one or more external devices according to a communication protocol, and the communication protocol includes at least a Bluetooth protocol. The processor 112 is connected to the display 260 and the one or more external device interfaces, and the processor 112 is configured to perform the audio generation method as described above.

[0119] In some embodiments, the processor 112 is configured to: obtain generation instructions for generating audio, the generation instructions including text instructions or voice instructions; based on the generation instructions, extract rule parameters for characterizing audio generation requirements; search in an audio database according to the rule parameters to obtain reference audio that matches the rule parameters; extract based on the beginning segment of the reference audio to obtain prompt words for characterizing audio elements in the beginning segment, the audio elements including at least one of timbre, rhythm or speed; perform inference based on the prompt words through a trained audio generation model to perform audio customization and obtain a corresponding generated audio file.

[0120] In some embodiments, the processor 112 is further configured to: perform audio playback based on the audio file and display a corresponding visual music score.

[0121] In some embodiments, the processor 112 is further configured to: update the audio file based on the audio segment to be adjusted in the audio file to obtain an updated audio file.

[0122] In some embodiments, the processor 112 executes updating of the audio file based on the audio segment to be adjusted in the audio file to obtain an updated audio file, and is configured to: in response to a selection operation on the visual music score, trim the corresponding selected audio segment from the audio file; extract based on the trimmed audio file to obtain corresponding prompt words; perform reasoning based on the corresponding prompt words through a trained audio generation model to obtain multiple candidate audio segments; determine a target audio segment from the multiple candidate audio segments, and fill the target audio segment into the trimmed audio file to obtain an updated audio file.

[0123] In some embodiments, the processor 112 is further configured to: extract based on the sample audio file to obtain musical note information, and use the musical note information and the sample audio file as training samples; and train the audio generation model based on the training samples to obtain a trained audio generation model.

[0124] In some embodiments, the processor 112 performs extraction based on the sample audio file to obtain musical note information, and is configured to: extract based on the sample audio file to obtain sample rule parameters; convert the sample rule parameters into computer code, and convert the computer code into binary information as musical note information.

[0125] In some embodiments, the audio file is an audio interface file for describing the audio composition. The processor 112 is further configured to: perform audio synthesis based on the audio interface file according to a specified audio format to obtain a playable file corresponding to the specified audio format.

[0126] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0127] The following uses the processor of the display device as an example to illustrate how the display device implements the audio generation method.

[0128] The technical solution of the present application is described in detail below in conjunction with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0129] Figure 5 This is a flowchart of an audio generation method provided by this application. Figure 5 As shown, the method comprises the following steps:

[0130] S510, obtaining a generation instruction for generating audio, where the generation instruction includes a text instruction or a voice instruction.

[0131] The generation instruction is issued by the user, and the user informs the audio generation requirement through the generation instruction. The generation instruction can be a text instruction, for example, the user enters text on the interactive interface of the display device, and the display device obtains the generation instruction by recognizing the text entered by the user.

[0132] The generation instruction may be a voice instruction. For example, after the voice interaction function of the display device is enabled, the user starts to speak, and the display device determines the generation instruction based on the user's speech content. When the generation instruction is a voice instruction, the display device may convert the user's speech content into text information through a text transcription model, and then determine the generation instruction based on the text information.

[0133] The generation instruction may also be an instruction in other forms, such as a gesture instruction, a selection instruction, etc., which is not limited in this embodiment.

[0134] Among them, the user can set the description, atmosphere and usage scenario corresponding to the audio to be generated through the generation instruction. For example, the user's speech content is to generate a lyrical and moving violin piece, which sets the description corresponding to the audio to be generated. For example, the user's speech content is "a soothing song suitable for the waiting room of a dentist's clinic", which sets the usage scenario corresponding to the audio to be generated. The user's speech content is "a beach party in Qingdao", which sets the atmosphere corresponding to the audio to be generated.

[0135] S520: Based on the generation instruction, extract rule parameters for characterizing audio generation requirements.

[0136] The rule parameters may include one or more of the following: beat, instrument, speed, and pitch, etc. For example, if the generation instruction is "generate a lyrical and moving violin piece" as described above, the extracted rule parameters may include the instrument being violin, the beat and speed being the beat and speed corresponding to the lyrical music. For example, if the generation instruction is "a beach party in Qingdao" as described above, the extracted rule parameters may include the beat and speed being the beat and speed corresponding to the party audio.

[0137] Before extracting the rule parameter based on the generation instruction, an extraction rule of the rule parameter may be obtained. After obtaining the generation instruction, the rule parameter is extracted from the generation instruction based on the extraction rule. The extraction rule may be set according to actual needs and is not limited in this embodiment.

[0138] In one embodiment, when the user wants to customize the background audio through the voice command, a large language model (LLM) can be used to extract the rule parameters.

[0139] S530: Search in an audio database according to the rule parameters to obtain reference audio that matches the rule parameters.

[0140] The audio database stores audio parameters, which are used to describe the beat, instrument, speed, and pitch of the audio.

[0141] When searching for the reference audio in the audio database according to the rule parameters, the reference audio can be determined by matching the rule parameters with the parameters of each audio in the audio database. For example, when the rule parameters and the parameters of a certain audio are substantially the same, the certain audio is determined to be the reference audio. The substantially same can be understood as the same rate reaching above the preset probability, and the preset probability can be set according to actual needs, for example, the preset probability is 95%. The audio database used in this embodiment can be a COOKBOOK library, which includes two spaces, one of which is a one-dimensional space composed of tags, and the other is a 15-dimensional space composed of musical instruments. Each pre-generated song occupies a precise position in these two spaces. The rule parameters are converted into a list of instruments and a list of tags for querying a point occupied with different songs. After determining a point occupied, a Euclidean distance operation is performed to find the instance closest to the user query as the reference audio.

[0142] S540: extracting based on the beginning segment of the reference audio to obtain prompt words for representing audio elements in the beginning segment.

[0143] The opening segment may be the first 1 minute or the first 30 seconds of the reference audio. The start time and the end time of the opening segment may be set according to actual needs, and this embodiment does not limit this.

[0144] The audio element includes at least one of timbre, rhythm or speed, and may also include other factors, which are not limited in this embodiment. Correspondingly, when the prompt word is used to represent the timbre, it may be violin timbre. Correspondingly, when the prompt word is used to represent the rhythm, it may be six or eight beats. Correspondingly, when the prompt word is used to represent the speed, it may be 120 beats per minute.

[0145] When extracting the prompt word based on the opening segment, the opening segment may be first converted into element information, and then the prompt word may be extracted based on the element information. The element information may include all audio elements and specific information of each audio element (such as beat, speed value, etc.), and correspondingly, the prompt word includes beat, speed value, etc.

[0146] S550, using the trained audio generation model, inference is performed based on the prompt word to customize the audio and obtain a corresponding generated audio file.

[0147] The audio generation model may be a symbolic audio generation model. A symbolic audio generation model refers to a method of using computer algorithms and models to represent audio as a series of symbols (such as pitch, interval, time, etc.), and to generate new audio works by learning the rules and features of these symbols. The symbolic audio generation model is used to generate symbolic audio.

[0148] After the prompt word is input into the audio generation model, the output of the audio generation model is obtained as the audio file.

[0149] Since the generation instruction is issued by the user based on the audio generation requirement, the audio file output by the audio generation model is a user-customized audio file.

[0150] In one embodiment, taking a television-based picture book generation application as an example, the audio generation method can achieve highly personalized and educational audio picture book creation by combining AIGC (AI-Generated Content) technology with professional concepts of children's education.

[0151] In one embodiment, when a user uses a display device to browse electronic photo albums or Vlogs, a large model is generated through audio. The user only needs a simple sentence instruction, and the display device can automatically complete the operations of image semantic understanding, background audio generation, and audio fusion, providing the user with a novel and innovative experience.

[0152] In one embodiment, for users with professional audio knowledge, some functions may also be provided to allow the users to edit certain musical instruments.

[0153] In one embodiment, for users with keyboards, the generated song can be edited by the user by disabling or enabling the number of tracks.

[0154] In one embodiment, in terms of interactive functionality, an audio learner is allowed to play and record his performance based on the generated audio.

[0155] In one embodiment, after the audio file is obtained, the audio file can be delivered to the user in a preset manner. The preset manner is, for example, a sound clip + extended paradigm. Specifically, a batch of several shorter sound clips are generated and displayed based on the audio file, and then the user can decide whether to expand the sound clip based on the extended paradigm. In this way, the user can decide to expand high-quality sound clips to save computing resources and time, and improve the response speed. In addition, it is also possible to allow the change of musical instruments. Specifically, the user can be shown which instruments are used in the audio he is listening to, and he can be allowed to replace the piano with a xylophone, or the harmonica with a saxophone. In addition, percussion instruments can be turned off, the volume of the audio track can be changed, etc., which are not limited in this embodiment.

[0156] In summary, the audio generation method provided in this embodiment, after obtaining the generation instruction for generating audio, extracts the rule parameters for characterizing the audio generation requirements based on the generation instruction. Then, according to the rule parameters, search in the audio database to obtain the reference audio that matches the rule parameters. Extraction is performed based on the beginning segment of the reference audio to obtain the prompt words for characterizing the audio elements in the beginning segment. Through the trained audio generation model, reasoning is performed based on the prompt words to perform audio customization and obtain the corresponding generated audio file.

[0157] That is to say, the audio file is associated with the generation instruction. When the generation instruction is changed, the audio file will also change. Users can customize the desired audio file through the generation instruction to meet the needs of self-expression. In actual applications, when users use the display device, they only need a simple sentence instruction, and the display device can generate qualified audio. During the generation, the user does not need to have audio knowledge or provide audio parameters. The audio generation model supports any natural language description, providing users with a novel and innovative experience.

[0158] Compared with the disadvantage that traditional household appliances cannot realize personalized customized audio according to the customization needs of users, the method provided in this embodiment can realize personalized customized audio according to the customization needs of users, meet the user's needs for self-expression, and improve the use effect of household appliances.

[0159] In some embodiments, after step S550, the method further includes:

[0160] Step 1: Play the audio based on the audio file and display the corresponding visual music score.

[0161] After the audio file is converted into audio, the audio is played. Before or while the audio is played, the corresponding visual score can be displayed. When the corresponding visual score is displayed, the current playing position can also be marked on the visual score, so that the user can know the playing progress in real time, and it is also convenient for the user to modify the unsatisfactory audio segment.

[0162] When displaying the corresponding visual score, the display position, display size, and background color of the visual score on the interface can be selected according to actual needs. For example, the visual score is displayed in full screen on the interface with a white background color.

[0163] Furthermore, before displaying the visualized music score, a display rule may be obtained first, and then the visualized music score may be displayed based on the display rule, wherein the display rule may be set by the user according to actual needs.

[0164] The method provided in this embodiment displays the corresponding visual music score when playing audio based on the audio file. The purpose of this is to enable the user to know the audio playing process, check whether the visual music score is wrong, etc., so as to improve the user's autonomy. The user can also select the audio clips that he does not like or have errors through the visual music score, so as to improve the effect of personalized audio customization.

[0165] In some embodiments, after step S550, the audio generation method further includes:

[0166] Step 1: Based on the audio segment to be adjusted in the audio file, the audio file is updated to obtain an updated audio file.

[0167] The method in which the user selects the audio segment to be adjusted can be selected according to actual needs.

[0168] For example, the visual music score described above is displayed on the interface of the display device, and the user can select the audio segment to be adjusted by selecting the audio segment in the visual music score. The described selection can be double-click selection, sliding selection, single-click selection, etc., which is not limited in this embodiment.

[0169] For another example, the user selects the audio segment to be adjusted through voice interaction. For example, the user indicates through voice interaction that the audio segment from the 1st minute to the 2nd minute of the played audio needs to be adjusted, and the audio segment from the 1st minute to the 2nd minute is marked as the audio segment to be adjusted.

[0170] For another example, a playback progress bar of the played audio is displayed, and the user selects a playback duration in the playback progress bar to select an audio segment corresponding to the playback duration as the audio segment to be adjusted.

[0171] Wherein, based on the audio segment to be adjusted, the audio file is updated, and when the updated audio file is obtained, the audio segment to be adjusted is replaced with another audio segment, and then the updated audio file is obtained. The other audio segment can be a user segment edited by the user himself, or an audio segment regenerated based on the played audio segment, which is not limited in this embodiment.

[0172] The method provided in this embodiment enables the user to independently select the audio segment to be adjusted, and can also update the audio file of the audio segment to be adjusted, and finally play the audio that the user is satisfied with. By selecting the audio segment to be adjusted, the user can further improve the playing effect of the personalized customized audio.

[0173] See also Figure 6In some embodiments, the above step 1 updates the audio file based on the audio segment to be adjusted in the audio file to obtain the updated audio file, specifically including:

[0174] S610: In response to a selection operation on the visual music score, a corresponding selected audio segment is trimmed from the audio file.

[0175] The selection operation may be double-click selection, single-click selection, swipe selection, etc., or may be selection through voice interaction, which is not limited in this embodiment.

[0176] Here, trimming the corresponding selected audio segment from the audio file can be understood as removing the corresponding selected audio segment from the audio file.

[0177] S620: Extract based on the trimmed audio file to obtain corresponding prompt words.

[0178] The trimmed audio file may be understood as an audio file from which the corresponding selected audio segment has been removed.

[0179] The prompt word is used to characterize the features of the trimmed audio file, and the prompt word includes, for example, that the instrument is violin and the speed is 120 beats per minute.

[0180] S630, performing reasoning based on corresponding prompt words through the trained audio generation model to obtain multiple candidate audio segments.

[0181] After the prompt word obtained in step S620 is input into the audio generation model, the output of the audio generation model is obtained as multiple candidate audio segments. The multiple candidate audio segments can be displayed to the user so that the user can select the candidate audio segment for audio segment replacement.

[0182] S640: Determine a target audio segment from the multiple candidate audio segments, and fill the target audio segment into the trimmed audio file to obtain an updated audio file.

[0183] As described above, the multiple candidate audio segments can be displayed to the user, and when it is detected that the user selects a candidate audio segment, the candidate audio segment is determined to be the target audio segment. The user can select a candidate audio segment by single-clicking, double-clicking, swiping, etc., which is not limited in this embodiment.

[0184] Furthermore, if the user is not satisfied with the displayed multiple candidate audio segments, the user may instruct the display device to regenerate multiple candidate audio segments.

[0185] After determining the target audio segment, the target audio segment is filled into the trimmed audio file to obtain an updated audio file. Filling the target audio segment into the trimmed audio file can also be understood as overwriting the audio segment that the user wants to replace with the target audio segment.

[0186] Furthermore, the visual score corresponding to the updated audio file can be displayed again, and the user can replace the unsatisfactory audio segment based on the visual score. Furthermore, if the user is still not satisfied after replacing the audio segment multiple times, the audio generation instruction can be re-issued to instruct the display device to regenerate an audio file.

[0187] For a clearer description, see Figure 7 In the example shown, after the audio generation model outputs an audio file, the audio file is converted into a visual score through a plug-in. After the user selects an unsatisfactory audio segment based on the visual score, the generated audio file is trimmed, and the trimmed audio file is input into the audio generation model again as a prompt. The audio generation model generates 10 candidate audio segments for the user to preview, and the user selects a satisfactory candidate audio segment as the target audio segment, and the unsatisfactory audio segment is covered with the target audio segment. After the complete audio file is processed again, it is output as an updated audio file.

[0188] In the method provided in this embodiment, the user can perform a selection operation on the visual music score. Correspondingly, in response to the selection operation on the visual music score, the corresponding selected audio segment is trimmed from the audio file, and then multiple candidate audio segments are obtained based on the trimmed audio file. After determining the target audio segment from the multiple candidate audio segments, the target audio segment is filled into the trimmed audio file to obtain an updated audio file.

[0189] Thus, the user can adjust the generated audio file again, so that the user can independently select the audio clip that needs to be adjusted, and can also update the audio file for the audio clip that needs to be adjusted, and finally play the audio that satisfies the user. The user does not need to have music theory knowledge to edit and modify the audio, that is, to achieve interactivity. By selecting the audio clip that needs to be adjusted, the user can further improve the playback effect of the personalized audio. Moreover, the method of updating the audio file by operating the visualized music score provided in this embodiment is not only intuitive, but also simple and efficient to operate, which can improve the customization speed and user experience of the personalized audio.

[0190] See also Figure 8 The audio generation method also includes a step of training an audio generation model. Specifically,

[0191] S810, extracting based on the sample audio file to obtain musical note information, and using the musical note information and the sample audio file as training samples.

[0192] The sample audio file can be an audio interface file (such as a Musical Instrument Digital Interface file, referred to as a Midi file) or a musical score. A Midi file contains Midi data, which is a digital interface of audio information. It contains various audio information, such as the start and end time of the note, pitch, and intensity. Due to the non-audio characteristics of the Midi file itself and the small storage space, the training efficiency of the audio generation model can be improved.

[0193] The number of the sample audio files can be selected according to actual needs, for example, it can be 160,000 sample audio files.

[0194] The musical note information may include information such as musical instruments, beats, rhythm, speed, etc., and the musical note information may be binary information.

[0195] S820: Based on the training sample, train the audio generation model to obtain a trained audio generation model.

[0196] The audio generation model may be the symbolic audio generation model described above. The core of the symbolic audio generation model is to convert audio into processable symbolic data and model and generate these data through a machine learning algorithm.

[0197] The purpose of fully training the audio generation model based on the training samples is to enable the trained audio generation model to use natural language as input and to generate audio files that meet the user's personalized customization needs.

[0198] Among them, the number of training times and training stop conditions of the audio generation model can be selected according to actual needs and are not limited in this embodiment.

[0199] In one embodiment, before training the audio generation model, the parameters of the audio generation model may be set according to actual needs.

[0200] The method provided in this embodiment describes how to train an audio generation model. Specifically, firstly extract based on a sample audio file to obtain musical note information, and use the musical note information and the sample audio file as training samples. Then, the audio generation model is trained based on the training samples. Thus, the trained audio generation model can be used to generate audio files that meet the user's personalized customization needs, thereby improving the user's experience of personalized audio customization.

[0201] In some embodiments, step S810 includes:

[0202] Step 1: Extract based on the sample audio file to obtain sample rule parameters.

[0203] The sample rule parameters may include one or more of the following: beat, instrument, speed, pitch, etc.

[0204] Step 2: convert the sample rule parameters into computer codes, and convert the computer codes into binary information as musical note information.

[0205] The computer code may be the American Standard Code for Information Interchange (ASCII code for short).

[0206] It should be noted that the audio generation model is a symbolic audio generation model, which includes a generative pre-trained transformer (GPT) under the open source Transformer architecture. The sample rule parameters are converted into binary information and used as musical note information, so that GPT can recognize the musical note information.

[0207] See also Fig. 9 In the example shown, a Midi file is used as a sample audio file to train an audio generation model containing GPT. The sample audio file is converted into a binary data set (binary information) by a decoder and a binary encoder, and then GPT training is performed based on the binary data set. After the original song is inferred based on GPT, the original song is decoded by a binary decoder, and the Midi file is output by the decoder, thereby completing a training of the symbolic audio generation model. It should be noted that the Midi file output at this time is the Midi file output by the symbolic audio generation model after learning the input Midi file, which is a simulation file of the original song. It should also be noted that, among them, the binary encoder can customize the format and add a customized dictionary, and the dictionary can, for example, contain musical instruments, notes, audio scenes, and word sets. Among them, the binary decoder can customize the format and add a customized dictionary, and the dictionary can, for example, contain musical instruments, notes, audio scenes, and word sets. By adding customized dictionaries and customized formats, algorithms, etc., the problem of understanding the duration of notes in the traditional symbolic audio generation model can be improved, and the quality of the generated audio can be optimized.

[0208] The method provided in this embodiment describes how to train an audio generation model when the audio generation model includes a model that can only recognize binary information (such as GPT). The trained audio generation model can be used to generate audio files that meet the user's personalized customization needs, thereby improving the user's experience of personalized audio customization.

[0209] In some embodiments, the audio file is an audio interface file for describing the audio composition, such as a Midi file. The audio generation method also includes:

[0210] Step 1: Perform audio synthesis based on the audio interface file according to the specified audio format to obtain a playable file corresponding to the specified audio format.

[0211] The designated audio format may be a Midi format, or a Moving Picture Experts Group Audio Layer III (MP3) format.

[0212] When the audio interface file is used for audio synthesis, the audio interface file can be sent to an audio synthesizer for rendering, and a playable file corresponding to the specified audio format can be exported.

[0213] The user can choose to save the playable file locally. Since the storage size of the Midi format is at the Kilobyte (KB) level, in order to reduce the storage size of the play file, the specified audio format can be set to the Midi format. Correspondingly, during the saving process, the user can choose to save the play file in the Midi format. Without considering the storage size of the play file, the specified audio format can be set to the MP3 format. Correspondingly, during the saving process, the user can choose to save the play file in the MP3 format.

[0214] The method provided in this embodiment allows the user to specify the audio format of the playable file. When the specified audio format is the Midi format, the storage size of the playable file can be reduced, saving resource waste.

[0215] In some embodiments, after the audio file is obtained in step S550, the audio file may be combined with other creative scenes, such as electronic photo albums, wallpaper generation, home KTV (full name Karaoke TV), short videos, Vlog (video blog), etc.

[0216] In some embodiments, some steps in the audio generation method are executed on the processor of the display device, and the remaining steps are executed on the cloud device. For example, step S510 described above can be executed on the processor of the display device, and step S520, step S530, step S540 and step S550 can be executed on the cloud device.

[0217] See also Fig.10 In some embodiments, some steps in the audio generation method are executed on a processor of the display device, and the remaining steps are executed on a cloud device.

[0218] The audio generation method comprises:

[0219] S1010: The display device obtains a generation instruction for generating audio.

[0220] The generating instruction includes a text instruction or a voice instruction. When the generating instruction is a voice instruction, the voice instruction is obtained based on a voice interaction module in the display device.

[0221] For the description of step S1010, reference may be made to the above description of step S510, which will not be repeated here.

[0222] S1020: The display device sends the generated instruction to the cloud device.

[0223] This embodiment does not limit the interaction method and communication protocol between the display device and the cloud device.

[0224] S1030, the cloud device uses a large language model to extract rule parameters for characterizing audio generation requirements from the generation instruction.

[0225] The large language model may be the LLM described above.

[0226] For the description of step S1030, reference may be made to the above description of step S520, which will not be repeated here.

[0227] S1040, the cloud device searches in the audio database according to the rule parameters to obtain reference audio that matches the rule parameters, extracts based on the beginning segment of the reference audio, and obtains prompt words for characterizing the audio elements in the beginning segment.

[0228] The audio element includes at least one of timbre, rhythm or speed.

[0229] For the description of step S1040, reference may be made to the above description of step S530 and step S540, which will not be repeated here.

[0230] S1050, the cloud device uses the trained audio generation model to perform inference based on the prompt word to customize the audio and obtain a corresponding generated audio file.

[0231] For the description of step S1050, reference may be made to the above description of step S550, which will not be repeated here.

[0232] S1060: The cloud device sends the audio file to the display device.

[0233] S1070: The display device plays audio based on the audio file and displays corresponding visual music score.

[0234] The display device plays audio based on the audio playing module and displays the corresponding visual music score.

[0235] When the display device is playing audio, the audio file can be input into the audio synthesizer and the audio output by the audio synthesizer can be played.

[0236] For the description of step S1070, please refer to the above related description, which will not be repeated here.

[0237] S1080: In response to the selection operation on the visual music score, the display device trims the corresponding selected audio segment from the audio file, and sends the trimmed audio segment to the cloud device.

[0238] The display device responds to a selection operation on the visualized music score based on a page interaction module.

[0239] For the description of step S1080, please refer to the above related description, which will not be repeated here.

[0240] S1090, the cloud device uses the trained audio generation model to perform reasoning based on corresponding prompt words to obtain multiple candidate audio segments, and determines a target audio segment from the multiple candidate audio segments, fills the target audio segment into the trimmed audio file, and obtains an updated audio file.

[0241] For the relevant description of step S1090, please refer to the above related description, which will not be repeated here.

[0242] S1100: The cloud device sends the updated audio file to the display device.

[0243] S1110: The display device plays audio based on the updated audio file.

[0244] The display device performs audio playback based on the audio playback module.

[0245] When the display device is playing audio, the updated audio file can be input into the audio synthesizer to play the audio output by the audio synthesizer.

[0246] In the method provided in this embodiment, the user can perform a selection operation on the visual music score. Correspondingly, in response to the selection operation on the visual music score, the corresponding selected audio segment is trimmed from the audio file, and then multiple candidate audio segments are obtained based on the trimmed audio file. After determining the target audio segment from the multiple candidate audio segments, the target audio segment is filled into the trimmed audio file to obtain an updated audio file.

[0247] Thus, the user can adjust the generated audio file again, so that the user can independently select the audio clip that needs to be adjusted, and can also update the audio file for the audio clip that needs to be adjusted, and finally play the audio that satisfies the user. The user does not need to have music theory knowledge to edit and modify the audio, that is, to achieve interactivity. By selecting the audio clip that needs to be adjusted, the user can further improve the playback effect of the personalized audio. Moreover, the method of updating the audio file by operating the visualized music score provided in this embodiment is not only intuitive, but also simple and efficient to operate, which can improve the customization speed and user experience of the personalized audio.

[0248] Fig.11 Schematic diagram of the structure of an audio generating device 10 provided in this application. Fig.11 As shown, the device 10 comprises:

[0249] The acquisition module 11 is used to acquire a generation instruction for generating audio, where the generation instruction includes a text instruction or a voice instruction.

[0250] The processing module 12 is used to extract rule parameters for characterizing audio generation requirements based on the generation instruction.

[0251] The acquisition module 11 is further used to search in the audio database according to the rule parameters to obtain reference audio that matches the rule parameters.

[0252] The processing module 12 is further used to extract, based on the beginning segment of the reference audio, a prompt word for characterizing an audio element in the beginning segment, where the audio element includes at least one of timbre, rhythm or speed.

[0253] The processing module 12 is also used to perform inference based on the prompt word through the trained audio generation model to customize the audio and obtain a corresponding generated audio file.

[0254] In some embodiments, the processing module 12 is further used to play audio based on the audio file and display a corresponding visual music score.

[0255] In some embodiments, the processing module 12 is further configured to update the audio file based on the audio segment to be adjusted in the audio file to obtain an updated audio file.

[0256] In some embodiments, the processing module 12 is used to respond to a selection operation on the visual music score, crop a corresponding selected audio segment from the audio file; extract based on the cropped audio file to obtain corresponding prompt words; perform reasoning based on the corresponding prompt words through a trained audio generation model to obtain multiple candidate audio segments; determine a target audio segment from the multiple candidate audio segments, and fill the target audio segment into the cropped audio file to obtain an updated audio file.

[0257] In some embodiments, the acquisition module 11 is further used to extract based on the sample audio file to obtain musical note information, and use the musical note information and the sample audio file as training samples. The processing module 12 is further used to train the audio generation model based on the training samples to obtain the trained audio generation model.

[0258] In some embodiments, the acquisition module 11 is used to extract based on the sample audio file to obtain sample rule parameters; convert the sample rule parameters into computer codes, and convert the computer codes into binary information as musical note information.

[0259] In some embodiments, the audio file is an audio interface file for describing the audio composition. The processing module 12 is also used to perform audio synthesis based on the audio interface file according to a specified audio format to obtain a playable file corresponding to the specified audio format.

[0260] The audio generation device 10 provided in the embodiment of the present application can execute the audio generation method in the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here. Fig.10 The division of the modules shown is only a schematic diagram, and the present application does not limit the division of the modules and the naming of the modules.

[0261] The present application also provides a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes. Specifically, the computer-readable storage medium stores program instructions, and the program instructions are used for the methods in the above embodiments.

[0262] The present application also provides a program product, which includes an execution instruction, which is stored in a readable storage medium. At least one control module of the display device can read the execution instruction from the readable storage medium, and at least one control module executes the execution instruction so that the display device implements the handwriting erasing method provided by the various embodiments described above.

[0263] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0264] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that: include: a display configured to display content from a broadcast system or network and / or a user interface; One or more external device interfaces, for communicating with one or more external devices according to a communication protocol, wherein the communication protocol includes at least a Bluetooth protocol; and at least one processor interfaced with the display and the one or more external devices; The processor is configured to: Obtaining a generation instruction for generating audio, wherein the generation instruction includes a text instruction or a voice instruction; Based on the generation instruction, extracting rule parameters for characterizing audio generation requirements; Searching in an audio database according to the rule parameters to obtain reference audio that matches the rule parameters; Extracting based on the beginning segment of the reference audio, obtaining prompt words for characterizing audio elements in the beginning segment, wherein the audio elements include at least one of timbre, rhythm or speed; The trained audio generation model is used to perform reasoning based on the prompt words to customize the audio and obtain a corresponding generated audio file.

2. The display device according to claim 1, characterized in that The processor is further configured to: Audio playback is performed based on the audio file, and a corresponding visual music score is displayed.

3. The display device according to claim 2, characterized in that The processor is further configured to: Based on the audio segment to be adjusted in the audio file, the audio file is updated to obtain an updated audio file.

4. The display device according to claim 3, characterized in that The processor executes updating the audio file based on the audio segment to be adjusted in the audio file to obtain an updated audio file, and is configured to: In response to a selection operation on the visual music score, a corresponding selected audio segment is cut from the audio file; and extraction is performed based on the cut audio file to obtain a corresponding prompt word; Through the trained audio generation model, reasoning is performed based on the corresponding prompt words to obtain multiple candidate audio clips; A target audio segment is determined from the multiple candidate audio segments, and the target audio segment is filled into the trimmed audio file to obtain an updated audio file.

5. The display device according to claim 1, characterized in that The processor is further configured to: Extracting based on the sample audio file to obtain musical note information, and using the musical note information and the sample audio file as training samples; Based on the training samples, the audio generation model is trained to obtain a trained audio generation model.

6. The display device according to claim 5, characterized in that The processor performs extraction based on the sample audio file to obtain musical note information, and is configured to: Extract based on the sample audio file to obtain sample rule parameters; The sample rule parameters are converted into computer codes, and the computer codes are converted into binary information as musical note information.

7. The display device according to claim 1, characterized in that The audio file is an audio interface file for describing audio composition; the processor is further configured to: According to the specified audio format, audio synthesis is performed based on the audio interface file to obtain a playable file corresponding to the specified audio format.

8. An audio generation method, characterized in that: Applicable to a display device as claimed in any one of claims 1 to 7; The method comprises: Obtaining a generation instruction for generating audio, wherein the generation instruction includes a text instruction or a voice instruction; Based on the generation instruction, extracting rule parameters for characterizing audio generation requirements; Searching in an audio database according to the rule parameters to obtain reference audio that matches the rule parameters; Extracting based on the beginning segment of the reference audio, obtaining prompt words for characterizing audio elements in the beginning segment, wherein the audio elements include at least one of timbre, rhythm or speed; The trained audio generation model is used to perform reasoning based on the prompt words to customize the audio and obtain a corresponding generated audio file.

9. The method according to claim 8, characterized in that After obtaining the corresponding generated audio file, the method further includes: Audio playback is performed based on the audio file, and a corresponding visual music score is displayed.

10. The method according to claim 9, characterized in that After obtaining the corresponding generated audio file, the method further includes: Based on the audio segment to be adjusted in the audio file, the audio file is updated to obtain an updated audio file.

11. The method according to claim 10, characterized in that The updating of the audio file based on the audio segment to be adjusted in the audio file to obtain the updated audio file comprises: In response to a selection operation on the visual music score, a corresponding selected audio segment is cut from the audio file; and extraction is performed based on the cut audio file to obtain a corresponding prompt word; Through the trained audio generation model, reasoning is performed based on the corresponding prompt words to obtain multiple candidate audio clips; A target audio segment is determined from the multiple candidate audio segments, and the target audio segment is filled into the trimmed audio file to obtain an updated audio file.

12. The method according to claim 8, characterized in that The method further comprises: Extracting based on the sample audio file to obtain musical note information, and using the musical note information and the sample audio file as training samples; Based on the training samples, the audio generation model is trained to obtain a trained audio generation model.

13. The method according to claim 12, characterized in that The extracting based on the sample audio file to obtain the musical note information includes: Extract based on the sample audio file to obtain sample rule parameters; The sample rule parameters are converted into computer codes, and the computer codes are converted into binary information as musical note information.

14. The method according to claim 8, characterized in that The audio file is an audio interface file for describing audio composition; the method further comprises: According to the specified audio format, audio synthesis is performed based on the audio interface file to obtain a playable file corresponding to the specified audio format.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the method described in any one of claims 8 to 14 is implemented.

16. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 8 to 14 when being executed by a processor.

Images