Display device, audio generation method and content display method

Abstract

The embodiment of the invention discloses a display device, an audio generation method and a content display method, and relates to the field of audio generation, and the method comprises the steps: responding to an application starting instruction, and displaying a first application page; receiving first input information; controlling the first target application to determine a first audio beginning clip according to the first input information; controlling the first target application to input the first audio beginning fragment into a symbol audio generation model to obtain a first symbol audio file; and controlling the first target application to generate the first audio. Through the target application, the audio beginning fragment can be directly determined according to the user input information, the audio beginning fragment is input into the symbol audio generation model to generate the corresponding audio, the user does not need to use the symbol audio generation model after knowing related knowledge of the symbol file, the difficulty of using the symbol audio generation model by the user is reduced, and the user experience is improved. And the audio beginning segment is used as input, so that the audio generation accuracy can be improved, and audios which can better meet different user requirements can be generated.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device, an audio generation method, and a content display method. Background Art

[0002] Audio generation methods include symbol-based audio generation methods and audio-based audio generation methods. Audio-based audio generation methods often directly use audio data for training to obtain an audio generation model that can generate audio based on user input information. However, this method needs to process a large amount of continuous audio data, which takes up a lot of computing resources and storage space. In addition, due to the large amount of data, the training time of the audio generation model is usually long and the training difficulty is also relatively large.

[0003] In order to reduce the occupation of computing resources and storage controls, shorten the training time of the audio generation model, and reduce the difficulty of training the audio generation model, a symbol-based audio generation method can be used to generate audio. This method converts the audio into a Musical Instrument Digital Interface (MIDI) file, transcribes the notes therein based on rules, and then uses the transcribed data for training to obtain an audio generation model that can generate audio based on MIDI file information. However, if the user does not understand the MIDI file information, the audio generation model cannot be used, and thus audio that better meets the user's needs cannot be generated. Summary of the invention

[0004] Some embodiments of the present application provide a display device, an audio generation method, and a content display method, which can reduce the difficulty for users to use the symbolic audio generation model, and use the beginning segment of the audio as input to improve the accuracy of the generated audio and generate audio that can better meet the needs of different users.

[0005] In a first aspect, some embodiments of the present application provide a display device, including:

[0006] A display; configured to display a user interface;

[0007] A controller is coupled to the display and is configured to:

[0008] In response to an application startup instruction to run the first target application, controlling the display to display a first application page of the first target application;

[0009] Receiving first input information inputted on the first application page;

[0010] Controlling the first target application to determine a first audio beginning segment according to the first input information;

[0011] Controlling the first target application to input the first audio beginning segment into the symbolic audio generation model to obtain a first symbolic audio file;

[0012] The first target application is controlled to generate a first audio according to the first symbolic audio file.

[0013] The above technical solution has the following advantages or beneficial effects: after receiving the first input information, the first target application can determine the beginning segment of the audio according to the first input information, and generate corresponding audio according to the beginning segment of the audio. The user does not need to understand the relevant knowledge of the symbolic file before using the symbolic audio generation model, which reduces the difficulty of the user using the symbolic audio generation model. Moreover, using the beginning segment of the audio as input can improve the accuracy of the generated audio and generate audio that can better meet the needs of different users.

[0014] In some embodiments, the first input information includes voice data, and the controller executes and controls the first target application to determine the first audio start segment according to the first input information, and is further configured to:

[0015] Controlling the first target application to recognize voice data and obtain voice text;

[0016] Controlling the first target application to input the speech text into the text transcription model to obtain at least one audio parameter;

[0017] The first target application is controlled to determine a first audio start segment corresponding to at least one audio parameter.

[0018] The above technical solution has the following advantages or beneficial effects: the first target application can recognize the received voice data as voice text, convert the voice text into audio parameters, determine the starting segment of the audio according to the audio parameters, and thus generate the corresponding audio, thereby solving the problem that the symbolic audio generation model cannot generate audio files according to the voice data, and reducing the difficulty for users to use the symbolic audio model to generate models.

[0019] In some embodiments, the first input information includes an input image, and the controller executes control of the first target application to determine the first audio start segment according to the first input information, and is further configured to:

[0020] Controlling the first target application to input the input image into the image recognition model to obtain image description text;

[0021] Controlling the first target application to input the image description text into the text transcription model to obtain at least one audio parameter;

[0022] The first target application is controlled to determine a first audio start segment corresponding to at least one audio parameter.

[0023] The above technical solution has the following advantages or beneficial effects: the first target application can generate audio that matches the input image based on the input image, solve the problem that the symbolic audio generation model cannot generate audio files based on image data, and reduce the difficulty for users to use the symbolic audio model to generate models.

[0024] In some embodiments, the first input information includes voice data and an input image, and the controller executes control of the first target application to determine the first audio start segment according to the first input information, and is further configured to:

[0025] Controlling the first target application to recognize voice data to obtain voice text, and inputting the input image into the image recognition model to obtain image description text;

[0026] Controlling the first target application to input the speech text and the image description text into the text transcription model to obtain at least one audio parameter;

[0027] The first target application is controlled to determine a first audio start segment corresponding to at least one audio parameter.

[0028] The above technical solution has the following advantages or beneficial effects: the first target application can receive the voice data and input image input by the user, and generate corresponding audio according to the text corresponding to the voice data and the description text of the input image. The audio generated based on the above conditions can both meet the needs of the user's expression and adapt to the input image, thereby improving the accuracy of the generated audio.

[0029] In some embodiments, the controller controls the first target application to determine a first audio start segment corresponding to at least one audio parameter, and is further configured to:

[0030] determining in an audio database a symbolic audio information description that matches the at least one audio parameter;

[0031] An audio start segment corresponding to the symbolic audio information description in an audio database is determined as a first audio start segment, and the audio database includes a correspondence between the symbolic audio information description and the audio start segment.

[0032] The above technical solution has the following advantages or beneficial effects: the audio database stores the symbolic audio information description and the audio beginning segment corresponding to the audio with relatively good generation quality. The audio beginning segment corresponding to the symbolic audio information description matching the audio parameters is used as a prompt word, which helps to improve the accuracy of the audio and meet the needs of users.

[0033] In some embodiments, when controlling the first target application to generate the first audio according to the first symbolic audio file, the controller is further configured to:

[0034] Controlling the first target application to convert the first symbolic audio file into musical score information;

[0035] The display is controlled to display a score page drawn based on the score information.

[0036] The above technical solution has the following advantages or beneficial effects: displaying the generated audio in the form of music score helps users to visually understand the generated audio, achieves the purpose of visualizing the audio score, and provides operational convenience for subsequent users to replace unsatisfactory segments.

[0037] In some embodiments, after controlling the display to display the music score page drawn based on the music score information, the controller is further configured to:

[0038] Receive confirmation of the audio segment to be modified in the score page;

[0039] Controlling the first target application to determine the audio segment before the audio segment to be modified as the target audio segment;

[0040] Controlling the first target application to input the target audio segment into the symbolic audio generation model to obtain a second symbolic audio file;

[0041] The first target application is controlled to generate a second audio according to the second symbolic audio file.

[0042] The above technical solution has the following advantages or beneficial effects: the first target application can receive the audio segment that the user selects as unsatisfactory on the score page, and re-input the segment before the audio segment into the symbolic audio generation model to obtain new audio. This technical solution can perform secondary editing on some segments of the generated audio to make the generated audio more accurate, and avoid the situation where the entire audio is repeatedly regenerated due to dissatisfaction with some segments, and the regenerated complete audio may still not meet the user's needs.

[0043] In some embodiments, after controlling the display to display the music score page drawn based on the music score information, the controller is further configured to:

[0044] Receive modification operations on audio parameters in the score page;

[0045] Controlling the first target application to input the first symbolic audio file and the modified audio parameters into the symbolic audio generation model to obtain a third symbolic audio file;

[0046] The first target application is controlled to generate a third audio according to the third symbolic audio file.

[0047] The above technical solution has the following advantages or beneficial effects: the first target application can receive the user's modification of the audio parameters, and re-input the modified audio parameters and the generated audio file into the symbolic audio generation model to obtain new audio. The technical solution can perform secondary editing on the audio parameters of the generated audio to generate audio that better meets the user's needs, and provide the user with the option of modifying some details of the generated audio, thereby avoiding the situation where the entire audio is repeatedly regenerated and the regenerated complete audio still cannot meet the user's needs.

[0048] In a second aspect, some embodiments of the present application provide a display device, including:

[0049] A display; configured to display a user interface;

[0050] A controller is coupled to the display and is configured to:

[0051] In the case of displaying a first page of a first application, in response to an instruction to add audio, controlling the display to display a second application page of a second target application;

[0052] receiving second input information inputted on the second application page;

[0053] Controlling the second target application to determine a second audio start segment according to the second input information;

[0054] Controlling the second target application to input the second audio beginning segment into the symbolic audio generation model to obtain a fourth symbolic audio file;

[0055] Controlling the second target application to generate a fourth audio according to the fourth symbolic audio file;

[0056] Controlling the first application to generate first content based on the fourth audio;

[0057] The display is controlled to display a first page, and first content is displayed on the first page.

[0058] The above technical solution has the following advantages or beneficial effects: the technical solution provided by the present application can jump to the second target application when the first application is running and audio needs to be added, and jump back to the first application interface after generating the target audio using the second target application, and the first application can use the audio to generate the target content. An interface for calling the second target application is provided in the first application, so that the first application can use the audio generated by the second target application, which helps the first application generate content containing audio that satisfies the user.

[0059] In a third aspect, some embodiments of the present application provide a display device, including:

[0060] A display; configured to display a user interface;

[0061] The controller is configured as:

[0062] When displaying the second page of the second application, receiving third input information input on the second page;

[0063] In response to the instruction to add audio, calling the target service;

[0064] Controlling the target service to determine a third audio start segment according to the third input information;

[0065] Controlling the target service to input the third audio beginning segment into the symbolic audio generation model to obtain a fifth symbolic audio file;

[0066] Controlling the target service to generate a fifth audio according to the fifth symbolic audio file;

[0067] controlling the second application to generate second content based on the fifth audio;

[0068] The display is controlled to display a second page, and second content is displayed on the second page.

[0069] The above technical solution has the following advantages or beneficial effects: the technical solution provided by the present application can jump to the target service when the second application is running and needs to add audio. After the target audio is generated by the target service, the second application can use the audio to generate the target content. An interface for calling the target service is provided in the second application, so that the second application can use the audio generated by the target service, which helps the second application to generate content containing audio that satisfies the user.

[0070] In a fourth aspect, some embodiments of the present application provide an audio generation method, including:

[0071] In response to an application start instruction to run the first target application, displaying a first application page of the first target application;

[0072] Receiving first input information inputted on the first application page;

[0073] Controlling the first target application to determine a first audio beginning segment according to the first input information;

[0074] Controlling the first target application to input the first audio beginning segment into the symbolic audio generation model to obtain a first symbolic audio file;

[0075] The first target application is controlled to generate a first audio according to the first symbolic audio file.

[0076] The above technical solution has the following advantages or beneficial effects: after receiving the first input information, the first target application can determine the beginning segment of the audio according to the first input information, and generate corresponding audio according to the beginning segment of the audio. The user does not need to understand the relevant knowledge of the symbolic file before using the symbolic audio generation model, which reduces the difficulty of the user using the symbolic audio generation model. Moreover, using the beginning segment of the audio as input can improve the accuracy of the generated audio and generate audio that can better meet the needs of different users.

[0077] In a fifth aspect, some embodiments of the present application provide a content display method, including:

[0078] In the case of displaying the first page of the first application, in response to the instruction to add audio, displaying the second application page of the second target application;

[0079] receiving second input information inputted on the second application page;

[0080] Controlling the second target application to determine a second audio start segment according to the second input information;

[0081] Controlling the second target application to input the second audio beginning segment into the symbolic audio generation model to obtain a fourth symbolic audio file;

[0082] Controlling the second target application to generate a fourth audio according to the fourth symbolic audio file;

[0083] Controlling the first application to generate first content based on the fourth audio;

[0084] Display the first page and show the first content on the first page.

[0085] The above technical solution has the following advantages or beneficial effects: the technical solution provided by the present application can jump to the second target application when the first application is running and audio needs to be added, and jump back to the first application interface after generating the target audio using the second target application, and the first application can use the audio to generate the target content. An interface for calling the second target application is provided in the first application, so that the first application can use the audio generated by the second target application, which helps the first application generate content containing audio that satisfies the user.

[0086] In a sixth aspect, some embodiments of the present application provide a content display method, including:

[0087] When displaying the second page of the second application, receiving third input information input on the second page;

[0088] In response to the instruction to add audio, calling the target service;

[0089] Controlling the target service to determine a third audio start segment according to the third input information;

[0090] Controlling the target service to input the third audio beginning segment into the symbolic audio generation model to obtain a fifth symbolic audio file;

[0091] Controlling the target service to generate a fifth audio according to the fifth symbolic audio file;

[0092] controlling the second application to generate second content based on the fifth audio;

[0093] The second page is displayed, and the second content is presented on the second page.

[0094] The above technical solution has the following advantages or beneficial effects: the technical solution provided by the present application can jump to the target service when the second application is running and needs to add audio. After the target audio is generated by the target service, the second application can use the audio to generate the target content. An interface for calling the target service is provided in the second application, so that the second application can use the audio generated by the target, which helps the second application to generate content containing audio that satisfies the user.

[0095] The technical solution provided in the embodiment of the present application can use the target application to directly determine the beginning segment of the audio based on the user input information, and input the beginning segment of the audio into the symbolic audio generation model to generate the corresponding audio. The user does not need to understand the relevant knowledge of the symbolic file before using the symbolic audio generation model, which reduces the difficulty for the user to use the symbolic audio generation model. In addition, using the beginning segment of the audio as input can improve the accuracy of the generated audio and generate audio that can better meet the needs of different users. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0097] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0098] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0099] Figure 3 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0100] Figure 4 A flowchart of an audio generation method provided in some embodiments of the present application;

[0101] Figure 5 A schematic diagram of a first application page provided in some embodiments of the present application;

[0102] Figure 6 A schematic diagram of a parameter selection page provided for some embodiments of the present application;

[0103] Figure 7 A schematic diagram of an image selection page provided in some embodiments of the present application;

[0104] Figure 8 A flowchart of training and reasoning of a symbolic audio generation model provided for some embodiments of the present application;

[0105] Fig. 9 A technical pipeline diagram of an audio generation method provided for some embodiments of the present application;

[0106] Fig.10 A schematic diagram of a first music score page provided for some embodiments of the present application;

[0107] Fig.11 A schematic diagram of a fragment clipping page provided in some embodiments of the present application;

[0108] Fig.12 A schematic diagram of a second music score page provided in some embodiments of the present application;

[0109] Fig.13 A schematic diagram of a third type of music score page provided for some embodiments of the present application;

[0110] Fig.14 A schematic diagram of a fourth music score page provided for some embodiments of the present application;

[0111] Fig.15 A flowchart of an audio segment modification method provided in some embodiments of the present application;

[0112] Fig.16 A schematic diagram of an audio display page provided in some embodiments of the present application;

[0113] Fig.17 A timing diagram of an audio generation method provided in some embodiments of the present application;

[0114] Fig.18 A timing diagram of another audio generation method provided in some embodiments of the present application;

[0115] Fig.19 A flowchart of a content display method provided in some embodiments of the present application;

[0116] Fig. 20A schematic diagram of a publishing reminder page provided in some embodiments of the present application;

[0117] Fig.21 A schematic diagram of another publishing prompt page provided in some embodiments of the present application;

[0118] Fig. 22 A timing diagram of a content display method provided in some embodiments of the present application;

[0119] Fig.23 A flowchart of another content display method provided for some embodiments of the present application;

[0120] Fig.24 A schematic diagram of an electronic photo album setting page provided in some embodiments of the present application;

[0121] Fig.25 A timing diagram of another content display method provided in some embodiments of the present application;

[0122] Fig.26 A schematic diagram of an audio generation method and use provided in some embodiments of the present application. DETAILED DESCRIPTION

[0123] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0124] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0125] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0126] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0127] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0128] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0129] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , a user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device 100 is used to receive operation instructions input by the user and convert the operation instructions into control instructions that can be recognized and responded to by the display device 200. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0130] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0131] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application program for controlling the display device 200 .

[0132] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0133] After receiving the user input information, the display device 200 may directly generate the corresponding audio according to the input information, or send the input information to the server 400, and the server 400 may generate the corresponding audio according to the input information.

[0134] The specific implementation of the operation of generating corresponding audio according to the input information may include: converting the input information into text. If the input information is text, no processing is required. If the input information is voice data, the voice data is converted into text using voice recognition technology. If the input information is an image, the image is input into the multimodal large model to obtain the image description text.

[0135] After the input information is converted into text, the text is input into the text transcription model to obtain audio parameters. Then, the audio beginning segment that matches the audio parameters is found from the audio database. After that, the audio beginning segment is input into the symbolic audio generation model to obtain a symbolic audio file, and finally the symbolic audio file is converted into playable audio using an audio synthesizer. The display device 200 can directly generate audio or indirectly receive audio sent by the server 400, and use the generated audio to generate display content that meets user needs in scenarios such as short videos and electronic photo albums.

[0136] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.

[0137] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0138] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0139] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0140] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0141] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0142] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0143] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0144] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0145] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0146] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0147] In some embodiments, the user input interface 280 can be used to receive instructions from the user. Exemplarily, the user input interface 280 can receive instructions from the user to select an image. The user input interface 280 can receive text information input by the user in the user interface. Among them, the text information includes a text description related to the generated audio. The user input interface 280 can receive a user's confirmation instruction for a control or image in the user interface. The user input interface 280 can also receive a voice instruction input by the user.

[0148] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system can control the display device to provide a user interface. For example, the operating system can directly control the display device to provide a user interface, or provide a user interface by running an application program. The operating system also allows the user to interact with the display device 200.

[0149] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0150] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 3 As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (referred to as "application layer"), the application framework layer (referred to as "framework layer"), the system library layer and the kernel layer.

[0151] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on the applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0152] In an embodiment of the present application, the application layer may include an application for generating audio, such as an audio application, an audio playback application, and an audio generation application. The application may generate corresponding audio according to the text, voice, and image input by the user. After receiving the user input information, the application may convert the input information into text, and then input the voice text into the text transcription model to obtain audio parameters, and find the audio beginning segment that matches the audio parameters from the symbolic audio file database. Afterwards, the audio beginning segment is input into the symbolic audio generation model to obtain a symbolic audio file, and the symbolic audio file is converted into playable audio using an audio synthesizer.

[0153] In the embodiment of the present application, the application layer may include applications that need to use audio, such as Vlog (video log) applications, short video applications, wallpaper applications, electronic album applications, and home singing applications. The application can choose to add audio that has been saved locally, audio downloaded from the network, and audio generated according to user input information, thereby combining audio with other data to generate a new content. For example, audio can be combined with images to generate a video.

[0154] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0155] In an embodiment of the present application, the system service may include a target service, namely an audio generation service, and the target service may be used to generate a corresponding symbol audio file according to user input information.

[0156] like Figure 3As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application package currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0157] In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes, for example, reducing the display window, shaking the display, distorting the display, etc.

[0158] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.

[0159] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 3 As shown, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0160] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0161] Audio generation methods include symbol-based audio generation methods and audio-based audio generation methods. Audio-based audio generation methods often directly use audio data for training to obtain an audio generation model that can generate audio based on user input information. Symbol-based audio generation methods convert audio into MIDI files, transcribe the notes based on rules, and then use the transcribed data for training to obtain an audio generation model that can generate audio based on MIDI file information. There are some key technical differences between the above two audio generation methods:

[0162] 1. Data representation: Symbol-based audio usually uses symbolic representation, such as MIDI or Extensible Markup Language (XML). These representations describe the abstract structure of music, such as notes, rhythm, and dynamics. Audio-based audio generation methods directly process audio waveforms or spectrum data, which contain all the sound information of the music, including pitch, timbre, dynamics, echo, etc.

[0163] 2. Complexity: Since audio data contains more information, audio-based audio generation methods are usually more technically complex than symbol-based audio generation methods. For example, audio-based audio generation methods need to process a large amount of continuous audio data, while symbol-based audio generation methods only need to process discrete symbol data.

[0164] 3. Generation quality: The audio-based audio generation method can generate more realistic and rich audio because it directly generates audio signals, which contain all the sound information of music. However, the symbol-based audio generation method may need to be played through a sound source (such as a virtual instrument or a real instrument) to obtain audio.

[0165] 4. Application areas: Symbol-based audio generation methods are usually used in music composition, music score generation and other fields, while audio-based audio generation methods can be used in music synthesis, sound effect design, music restoration and other fields.

[0166] 5. Training data: The training data for symbol-based audio generation methods are usually musical scores or MIDI files, while the training data for audio-based audio generation methods are audio files or audio data.

[0167] 6. Model design: Due to the difference in training data, symbol-based audio generation methods and audio-based audio generation methods usually need to use different models. For example, symbol-based audio generation methods usually use recurrent neural networks (RNN) or variational autoencoders (VAE), while audio-based audio generation methods may use generative adversarial networks (GAN) or WaveNet models.

[0168] In audio-based audio generation methods, audio data usually exists in the form of waveforms or spectra, while the Transformer model (a deep learning model based on the self-attention mechanism) requires a structured data form. Therefore, audio data needs to go through some preprocessing steps before it can be input into the Transformer model.

[0169] Here are some common preprocessing steps:

[0170] 1) Feature extraction: Audio data usually needs to be converted into a form that is more suitable for machine learning. This process is called feature extraction. Common audio features include Mel Frequency Cepstrum Coefficient (MFCC), Spectrogram, Chromagram, etc. These features can capture key information of audio data, such as pitch, timbre, rhythm, etc.

[0171] 2) Standardization: After feature extraction, the data usually needs to be standardized to eliminate the scale differences of the data. Common standardization methods include Z-score standardization and minimum and maximum standardization.

[0172] 3) Serialization: Since audio data is continuous and the Transformer model processes discrete sequence data, the audio data needs to be converted into a sequence form. This can be achieved by dividing the audio data into a series of frames, each of which can be regarded as an element in the sequence.

[0173] 4) Encoding: If necessary, the serialized data can be encoded to facilitate the Transformer model processing. For example, one-hot encoding or embedding methods can be used.

[0174] After preprocessing, the audio data can be fed into the Transformer model for training. During the training process, the Transformer model will try to learn the patterns and structures in the audio data in order to generate new audio data.

[0175] In the symbol-based audio generation method, MIDI data is a digital interface of music information, which contains various information of the audio, such as the start and end time of the note, pitch, dynamics, etc. In order to input MIDI data into the Transformer model for training, some preprocessing steps are required:

[0176] 1) Parsing MIDI: Use a specialized library (such as Python's midi or pretty_midi library) to parse the MIDI file and convert it into a data structure that can be processed, that is, a human-readable and tokenizable format. After parsing, you usually get a series of events, such as note_on (note start), note_off (note end), etc.

[0177] 2) Event encoding: These events need to be converted into a form that the model can understand. A common approach is to use one-hot encoding, assigning a unique index to each possible event (such as each possible note and intensity), and then use this index to represent the event. Another approach is to use a more complex encoding scheme, such as the event-based encoding used by Music Transformer, which can capture more musical information.

[0178] 3) Serialization: Since MIDI events are arranged in time order, the encoded event sequence can be directly input into the Transformer model. If necessary, padding or truncation can also be used to adjust the length of the sequence.

[0179] After preprocessing, the MIDI data can be fed into the Transformer model for training. During the training process, the Transformer model will try to learn the patterns and structures in the MIDI data in order to generate new MIDI data.

[0180] Although audio-based audio generation methods can generate more realistic and rich music, they need to process a large amount of continuous audio data, which takes up a lot of computing resources and storage space. In addition, due to the large amount of data, the training time for audio music generation is usually long and the training difficulty is also greater.

[0181] In order to reduce the usage of computing resources and storage controls, shorten the training time of the audio generation model, and reduce the difficulty of audio generation model training, a symbol-based audio generation method can be used to generate audio. However, if the user does not understand the MIDI file information, the audio generation model cannot be used, and thus audio that can better meet the needs of different users cannot be generated.

[0182] In order to generate audio that can better meet the needs of different users, the embodiment of the present application provides a display device 200. The structure and functions of each part of the display device 200 can refer to the above embodiment. In addition, based on the display device 200 shown in the above embodiment, this embodiment further improves some functions of the display device 200. Figure 4 As shown, the controller 250 is configured to perform the following steps:

[0183] Step S401: receiving an application start instruction for running a first target application.

[0184] In some embodiments, the display device 200 displays the home page of the desktop application after startup, wherein the home page of the desktop application includes the first target application control. The application startup instruction to run the first target application is received by receiving a confirmation operation of the user on the first target application control.

[0185] In other embodiments, the display device 200 may receive voice data of "open the first target application" input by the user by pressing the voice key of the control device 100; or receive voice data of "open the first target application" input by the user through a sound collector. The application start instruction for running the first target application is received by recognizing the voice data and generating a voice instruction.

[0186] The first target application includes an application for generating audio. The first target application may include an audio application, an audio synthesis application, an audio playback application, and the like.

[0187] Step S402: In response to an application start instruction for running a first target application, controlling a display to display a first application page of the first target application.

[0188] In some embodiments, in response to the application start instruction for running the first target application, the first target application is started, and the display 260 is controlled to display the first application page of the first target application. The first application page may be a home page, which includes an information input area for receiving user input information, and the information input area includes a language input area and an image input area.

[0189] In other embodiments, in response to an application startup instruction for running the first target application, the first target application is started, and the display 260 is controlled to display the home page of the first target application. The home page includes an audio synthesis control. In response to a user's confirmation operation on the audio synthesis control, the display 260 is controlled to display the first application page, which is an audio synthesis page, and the audio synthesis page includes an information input area.

[0190] Step S403: receiving first input information input on the first application page.

[0191] The first input information includes at least one of language data, reference data and input image. Language data includes text data or voice data. Reference data includes multiple preset parameters set in terms of audio style, implicit emotions, applicable scenarios and professional audio parameters. Reference data can provide multiple reference options when the user cannot express the audio requirements well. Exemplarily, the implicit emotions include joy, sadness and indignation. The input image can be selected from local storage, downloaded from the network, shot in real time, uploaded from other devices and generated by AI.

[0192] For example, Figure 5 As shown, the first application page includes an information input area, a generate audio control 53, an audio control 54 corresponding to the user's work, and a focus 55. The information input area includes a language input area 51 and an image input area 52. The language input area 51 includes a text input box 511, a switch control 512, and a reference parameter control 513. The text input box 511 is used to receive text information input by the user. The switch control 512 is used to switch the text input mode to the voice input mode, so that the display device 200 can receive the voice data input by the user. Among them, the mode of receiving the user input voice data includes receiving the voice data input by the user by pressing the voice key of the control device 100, and starting the voice data collected by the sound collector of the display device 200. The reference parameter control 513 is used to trigger the setting page of related parameters such as the display audio style. The image input area 52 includes an add image control 521. After receiving the user's confirmation operation on the add image control 521 and selecting at least one image, the image selected by the user, that is, the input image, can be displayed in the image input area 52.

[0193] After receiving the user's confirmation operation on the reference parameter control 513, the display 260 can display a parameter selection interface, which includes at least one parameter control. After receiving the user's selection instruction for at least one parameter control, the parameter name corresponding to the parameter control can be used as the content in the first input information.

[0194] For example, in Figure 5 After receiving the user's confirmation operation on the reference parameter control 513, the following is displayed: Figure 6 The parameter selection page shown in FIG. 6 includes a parameter control 61, a confirmation control 62, and a cancel control 63. After receiving a selection operation of the parameter control 61 corresponding to rock and guitar by the user and then receiving a confirmation operation of the confirmation control 62 by the user, rock and guitar can be used as the content of the first input information.

[0195] After receiving the user's confirmation operation for adding the image control 521, the following is displayed: Figure 7 The image selection page shown in the figure includes at least one of a capture image control 71, a select image control 72, a generate image control 73 and an other device upload image control 74.

[0196] After receiving the user's confirmation operation on the capture image control 71, the display device 200 starts the built-in or external camera to use the camera to capture the target image. After receiving the user's confirmation operation on the select image control 72, the display device 200 displays the file dialog page. The file dialog page is used to display images under different storage paths and for the user to select the target image. After receiving the user's confirmation operation on the generate image control 73, the text input area or the voice input area is displayed. After receiving the text or voice information input by the user, the text or voice information is input into the image generation model to obtain the target image. After receiving the user's confirmation operation on the upload image control 74 of other devices, the prompt page is displayed, and the prompt page includes prompt information and an identification code. The prompt information is used to prompt the user to use the terminal to open the specified application and scan the identification code, and complete the upload of the target image on the terminal interface according to the prompt process. After receiving the confirmation operation on the target image, the target image can be reduced and displayed in the image input area 52, and the target image is determined as the content in the first input information.

[0197] After receiving the user's confirmation operation on generating the audio control 53, the language data input by the user in the language input area 51 and / or the input image in the image input area 52 are determined as the first input information.

[0198] Step S404: controlling the first target application to determine a first audio beginning segment according to the first input information.

[0199] Several possible specific implementations of step S404 provided in the embodiments of the present application are described below.

[0200] The first implementation method: when the first input information is voice data, the first target application can first recognize the voice data to obtain voice text, then input the voice text into the text transcription model to obtain at least one audio parameter, and finally determine the first audio beginning segment corresponding to the at least one audio parameter.

[0201] The specific implementation method of recognizing speech data and obtaining speech text may include: after obtaining the speech data, firstly subjecting the speech data to acoustic processing such as sound source localization, denoising and sound quality enhancement. Sound source localization is used to enhance or retain the signal of the target speaker when multiple speakers are speaking, suppress the signals of other speakers, track the speaker's signal and perform subsequent speech directional pickup. Denoising is used to remove environmental noise from speech data, etc. Sound quality enhancement is used to increase the speaker's voice intensity when the speaker's voice intensity is low. Then, the acoustically processed speech data is input into the hidden Markov model to obtain speech text.

[0202] In order to ensure the smooth implementation of the step of obtaining audio parameters using the text writing model, the text transcription model needs to be trained before the step is executed. Exemplarily, the training method of the text transcription model may include: using a large amount of training text and its corresponding audio parameters to train the initial model to obtain the corresponding text transcription model. The text transcription model can use a large language model (LLM). The text transcription model is used to convert the description text into audio parameters. Among them, the training text includes text for describing information such as audio emotions and musical instruments, for example, generating a lyrical and moving violin piece. The training text also includes text for describing information such as usage scenarios, for example, a soothing song suitable for the waiting room of a dental clinic. The training text also includes text for describing information such as music atmosphere, for example, Qingdao's beach party. The training text includes text for describing audio-related parameters, for example, I want the melody to jump, a piano minor with a 4 / 4 beat. Audio parameters include beat parameters, instrument parameters, speed parameters, and pitch parameters. The instrument parameter optional range is 128 instruments, and multiple instruments can be supported as instrument parameters at the same time.

[0203] A specific implementation of determining the first audio start segment corresponding to at least one audio parameter may include: determining a symbolic audio information description matching the at least one audio parameter in an audio database, and determining the audio start segment corresponding to the symbolic audio information description in the audio database as the first audio start segment. The audio database includes a correspondence between the symbolic audio information description and the audio start segment.

[0204] The opening segment is also called the starting melody, which includes information such as rhythm, speed, notes, timbre, etc. It can set the tone, style and mood of the entire audio and pave the way for subsequent audio generation. The opening segment includes the data before note_on (note start) in the MIDI file, that is, the opening segment does not include notes.

[0205] In order to facilitate the determination of the beginning segment of the audio based on the audio parameters, the display device needs to pre-establish and store an audio database. In some embodiments, a specific implementation method of establishing an audio database may include: after obtaining an initial symbolic audio generation model through training, randomly selecting a description text to input into the initial symbolic audio generation model to obtain a generated symbolic audio file. After obtaining a large number of symbolic audio files in this way, symbolic audio files with relatively good quality are screened out through manual filtering and screening methods and audios with common characteristics. The information description of the screened symbolic audio file, i.e., the MIDI file, is saved in the audio database corresponding to the beginning segment.

[0206] In other embodiments, another specific implementation of establishing an audio database may include: after obtaining the training data of the symbolic audio generation model, the information description of the symbolic audio file in the training data is saved in the audio database corresponding to the beginning segment through manual filtering and screening methods.

[0207] In some embodiments, a specific implementation of determining a symbolic audio information description that matches at least one audio parameter may include: vectorizing data such as audio parameters, and calculating the matching scores of the audio parameters and the symbolic audio information description through a weight matching method and prior information, and selecting the symbolic audio information description with the highest matching score with at least one audio parameter. Weight matching and prior information are effective means to improve the performance of information matching models. By reasonably allocating weights and utilizing prior information, it is possible to ensure that the model pays more attention to important features while improving the accuracy and robustness of the model.

[0208] In other embodiments, another specific implementation method of determining a symbolic audio information description that matches at least one audio parameter may include: creating a first space and a second space. The first space includes a one-dimensional space composed of labels, and the labels may include information such as beats, speed, and pitch. The second space includes a 15-dimensional space composed of musical instruments. Each symbolic audio information description in the audio database occupies a precise position in the first space and the second space. Based on the audio parameters, a list of instruments and a list of labels are determined, the positions of the list of instruments and the list of labels in the first space and the second space are determined, and then a Euclidean distance operation is performed to find the symbolic audio information description corresponding to the audio parameter that is closest.

[0209] Second implementation: When the first input information is input text, the first target application converts the input text into a text transcription model to obtain at least one audio parameter, and finally determines the first audio beginning segment corresponding to the at least one audio parameter. The input text includes the text entered by the user in the text input box 511 and / or the parameter name corresponding to the parameter control 61 selected by the user.

[0210] The third implementation method: when the first input information is an input image, the first target application inputs the input image to an image recognition model to obtain an image description text, and then inputs the image description text to a text transcription model to obtain at least one audio parameter, and finally determines a first audio beginning segment corresponding to the at least one audio parameter.

[0211] The image recognition model can use a multimodal large model. A multimodal large model is a machine learning model that can understand and process multiple types, and the types are also called modalities, including text, pictures, audio, video, etc. After obtaining the input image, the input image is preprocessed first. Preprocessing methods include scaling, cropping, color standardization, and image enhancement. The preprocessed input image is input into the multimodal large model, which can perform feature extraction and image recognition on the preprocessed input image and output the image description text corresponding to the input image.

[0212] Fourth implementation: when the first input information is voice data and an input image, the first target application recognizes the voice data to obtain voice text, and inputs the input image into an image recognition model to obtain image description text, and then inputs the voice text and the image description text into a text transcription model to obtain at least one audio parameter. Finally, a first audio start segment corresponding to the at least one audio parameter is determined.

[0213] Step S405: Control the first target application to input the first audio beginning segment into the symbolic audio generation model to obtain a first symbolic audio file.

[0214] The first audio beginning segment is input as a prompt word into the symbolic audio generation model, and the symbolic audio generation model can generate a first symbolic audio file based on the first audio beginning segment.

[0215] In order to ensure the smooth implementation of the step of generating the first symbolic audio file using the symbolic audio generation model, the symbolic audio generation model needs to be trained first, and then the symbolic audio generation model is inferred. Figure 8 As shown in Figure 1, the symbolic audio file used for training, i.e., the MIDI file, is encoded. The encoding process is divided into two levels: the conversion of the MIDI file to the American Standard Code for Information Interchange (ASCII), and the conversion of the ASCII code to binary. The MIDI file is first encoded by a rule-based encoder to obtain the ASCII code. The ASCII code is used to convert the MIDI file into a human-readable and tokenizable format.

[0216] For example, the ASCII code of a piece of audio is as follows:

[0217] …control_change / 0 / 38 / 0 / 1 / note_on / 0 / 76 / 69 / 1 / slip / 5 / note_off / 0 / 76 / 0 / 1 / slip / 1273+124 / note_on / 0 / 75 / 64 / 1 / slip / 31 / note_off / 0 / 75 / 0 / 1 / slip / 1273+96 / note_on / 0 / 73 / 65 / 1 / slip / 31 / note_off / 0 / 73 / 0 / 1 / slip / 1273+93 / note_on / 0 / 71 / 64 / 1 / slip / 31 / control_chang e / 0 / 7 / 102 / 1 / slip / 1273+52 / note_off / 0 / 71 / 0 / 1 / …

[0218] Since the Transformer architecture has a poor understanding of numbers, the symbolic audio generation model also has a poor understanding of the duration of notes. For example, if a note lasts for 2.75863s, the symbolic audio generation model cannot understand this duration well. After generating a preset number of notes, the duration of the notes is poorly constrained and may be very long. However, the complexity of the audio is inversely correlated with the duration of the note, resulting in low complexity of the generated audio and a certain impact on the audio quality.

[0219] In order to solve the problem of low audio quality due to the misunderstanding of duration, the embodiment of the present application simplifies the duration of the notes in the MIDI file in the scheme of rule-based encoding. One simplified processing method is rounding. For example: if the duration of a note is 2.75863s, the duration of the note after rounding is 3s. Another simplified processing method is to take a multiple of the nearest preset value. For example, the preset value is 3, and the duration of a note is 6.5633s, then the duration of the simplified note is 6s. In addition, after generating the symbolic audio file, the duration of the notes in the symbolic audio file can be measured, and once the duration of the note (in seconds) exceeds a certain threshold, it stops, which can ensure that the duration of the note does not exceed the threshold. Among them, the threshold can be set to 30 seconds. The solution provided in the embodiment of the present application can improve audio complexity and audio quality.

[0220] After getting the ASCII code, the ASCII code is converted into a list of numbers through a binary encoder and a dictionary, which is stored in binary format, i.e. a binary file. Binary data is a way to read and learn during Transformer training. Among them, the dictionary includes instruments, notes, and music events. The dictionary is stored in the format used by the NanoGPT library.

[0221] Based on multiple MIDI files, multiple binary files can be generated, and the binary files are collected into a binary data set. The binary data set is used to train a Generative Pre-Trained Transformer (GPT) model.

[0222] After the beginning of the audio segment is input into the trained symbolic audio generation model, a binary audio file can be generated. The binary audio file is converted into an ASCII code audio file through a binary decoder and a dictionary. The ASCII code audio file is decoded by a rule-based decoder to obtain a symbolic audio file, i.e., a MIDI file.

[0223] Step S406: Control the first target application to generate the first audio according to the first symbolic audio file.

[0224] The first target application inputs the first symbolic audio file into an audio synthesizer for rendering to obtain a first audio that can be played directly.

[0225] In order to facilitate the user to quickly use or play the generated first audio, the first audio needs to be saved in the display device 200. In some embodiments, after receiving the user input instruction to save the first audio, the first audio can be saved in a format that can be directly played, such as MP3, and the audio in this format can be directly played after the user selects the control corresponding to the audio, thereby reducing the user's waiting time.

[0226] In other embodiments, after receiving an instruction from the user to save the first audio, the first audio may be saved in a MIDI file format, thereby reducing storage space usage.

[0227] In the embodiment of the present application, combined with the content of the above embodiment, the technical route pipeline diagram of the audio generation method can be as follows Fig. 9 The audio generation method may include the following three steps:

[0228] The first step is to convert text to audio parameters. Get the text corresponding to the user input information, input the text into the text transcription model, and output the audio parameters. Among them, you can train the text writing model or call the QWEN2.5 interface to realize the conversion of user input information into audio parameters. Audio parameters include instruments, pitch, beat, and speed.

[0229] The second step is to convert audio parameters to MIDI files. An audio database is established, and the MIDI file description information that best matches the audio parameters is found in the audio database. The audio beginning segment corresponding to the MIDI file description information is obtained. The audio beginning segment is encoded into a binary input symbol audio generation model, and the output generates a MIDI file.

[0230] The third step is to convert MIDI files to audio. Use an audio synthesizer to convert MIDI files to audio.

[0231] In order to allow the user to view the music score corresponding to the first audio when playing the first audio, when the first target application generates the first audio according to the first symbolic audio file, the first symbolic audio file can also be converted into music score information, and the display can be controlled to display the music score page drawn based on the music score information. The music score page is synchronously displayed during the playing of the first audio.

[0232] The score interface includes a score, a clip trimming control, at least one audio parameter control, an audio control control, a save control, and a regenerate control. The score is used to display the note information of the audio. The score includes five-line notation and simplified notation. A progress mark is also displayed in the score, and the audio playback progress corresponds to the position of the progress mark. The selected clip trimming control is used to trigger the audio clip trimming function. The audio parameter control includes controls corresponding to the audio parameters corresponding to the first audio, and the audio parameter control may also include common audio parameter modification controls. The audio control control includes controls for controlling functions such as audio playback, specifically including a play / pause control, a fast forward control, a fast rewind control, a next song control, a previous song control, and a speed control. The save control can be used to trigger the save function of the first audio.

[0233] When the music score page is displayed, a modification instruction for the first audio input by the user on the music score page can be received, and the first audio can be modified according to the modification instruction.

[0234] In some embodiments, a specific implementation of receiving a modification instruction input by a user on a music score page for a first audio, and modifying the first audio according to the modification instruction may include: upon receiving a confirmation operation of a user on a segment trimming control, the display 260 may be controlled to display a segment trimming page. After receiving a segment to be modified selected by a user on the segment trimming page, the first target application determines the audio segment before the audio segment to be modified as a target audio segment, and then inputs the target audio segment into a symbolic audio generation model to obtain one or more second symbolic audio files. A second audio is generated according to the second symbolic audio file, and the second symbolic audio file is converted into music score information, and then the second audio is played, and a music score page drawn based on the music score information is displayed at the same time.

[0235] In other embodiments, another specific implementation of receiving a modification instruction of a first audio inputted by a user on a score page, and modifying the first audio according to the modification instruction may include: after receiving a confirmation operation on an audio parameter control, a plurality of audio parameter change controls may be displayed. After receiving a confirmation operation on an audio parameter change control from a user, the first target application may input the first symbolic audio file and the modified audio parameter into a symbolic audio generation model to obtain one or more third symbolic audio files, or may input the first symbolic audio file and the audio parameters corresponding to the audio parameter controls before and after the modification into the symbolic audio generation model to obtain one or more third symbolic audio files. Exemplarily, after receiving an instruction from a user to modify a piano to a saxophone, the first symbolic audio file + delete piano + add saxophone as prompt words are input into the symbolic audio generation model to obtain a third symbolic audio file. Among them, the audio parameter control is different from the audio parameter corresponding to the selected audio parameter change control. A third audio is generated according to the third symbolic audio file and the third symbolic audio file is converted into score information, and then the third audio is played, while displaying a score page drawn based on the score information.

[0236] In some embodiments, in addition to modifying the unsatisfactory audio clips and audio parameters individually, the unsatisfactory audio clips and audio parameters can also be modified at the same time. The specific implementation method may include: inputting the target audio clip and the modified audio parameters into the symbolic audio generation model to generate the corresponding symbolic audio file.

[0237] For example, Fig.10 As shown, the score page includes an input information area 101, a score area 102, an audio modification area 103, an audio control area 104, a save control 105, and a regenerate control 106. The input information area 101 includes a voice control 1011 and an image control 1012. After receiving a user confirmation operation on the voice control 1011, the voice data can be re-input; after receiving a user confirmation operation on the image control 1012, the input image can be reselected; and then the audio start segment is determined according to the updated input information.

[0238] The music score area 102 includes a music score 1021 and a progress mark 1022. The progress mark 1022 represents the playing progress of the audio at the position of the music score 1021.

[0239] The audio modification area 103 includes a segment trimming control 1031, a first instrument control 1032, a second instrument control 1033, and a track quantity control 1034. After receiving the user's operation on the segment trimming control 1031, the following is displayed: Fig.11The segment trimming page shown. The segment trimming page includes a start mark control 1101, an end mark control 1102, a confirmation control 1103 and a cancel control 1104. Among them, the starting position of the start mark control 1101 and the end mark control 1102 can be set at the current position of the progress mark 922. After receiving the user's movement of the start mark control 1101 and the end mark control 1102 in the music score 921 and receiving the user's confirmation operation on the confirmation control 1103, it can be determined that the segment to be modified is the segment from the position of the start mark control 1101 to the position of the end mark control 1102. New audio can be generated directly according to the segment to be modified, or it can be jumped to Fig.12 The sheet music page shown. Fig.12 The selected audio segment 121 can be displayed in the music score page, and the user's modification of the audio parameters in the music score page can continue to be received.

[0240] Upon receiving the user's Fig.10 After the confirmation operation of the first instrument control 1032, multiple audio parameter change controls 131 are displayed, such as Fig.13 After selecting one of the audio parameter change controls 131, the name corresponding to the first musical instrument control is changed from piano to xylophone, such as Fig.14 As shown. Among them, the instruments can also be modified uniformly according to the instrument classification. After receiving the user's confirmation operation on the second instrument control 1033, it can be controlled whether to delete the participation of all orchestral instruments in the audio.

[0241] The audio control area 104 includes a pause control 1041 , a fast forward control 1042 , a fast rewind control 1043 , and a multiple speed control 1044 .

[0242] After receiving the user's confirmation operation on the save control 105, the audio can be saved locally. The user can be provided with a format to select the audio to be saved, MIDI or MP3 format.

[0243] After receiving a confirmation operation of the user on the regeneration control 106, the audio may be regenerated based on the modified audio segment and audio parameters selected on the current page.

[0244] For example, Fig.15As shown, user input information (any natural language) is received, the input information is converted into audio parameters through a text transcription model, and the audio beginning segment corresponding to the audio parameters is determined. The audio beginning segment is then input into the symbolic audio generation model, a MIDI file preview is output, and the MIDI file is converted into a musical score and displayed through a plug-in. After receiving the audio segment that the user selects that he is not satisfied with, a new MIDI file is generated by cutting, that is, the audio segment before the unsatisfactory audio segment is retained, and the new MIDI file is input into the symbolic audio generation model as a prompt word, and 10 audio segments can be generated for the user to preview. After receiving the user's confirmation operation on one of the segments, the selected segment is overwritten with the original audio segment to obtain a complete MIDI file. The complete MIDI file is synthesized and output audio. After receiving the save instruction input by the user, the audio is saved in any format.

[0245] The solution provided by the embodiment of the present application allows users to change the instrument that generates the audio, such as replacing the piano with a xylophone, turning off the orchestral instruments, changing the volume of the track, etc., which is impossible for the audio generation model based on audio. For users with professional music knowledge, it allows users to edit certain instruments. For users with keyboards, the number of tracks can be disabled or enabled, allowing users to edit the generated audio, which can generate audio that satisfies users more.

[0246] In some embodiments, the first symbolic audio file generated by the symbolic audio generation model may be a symbolic audio file corresponding to a complete audio segment, which is played to the user after being rendered by an audio synthesizer.

[0247] In other embodiments, the first symbolic audio file generated by the symbolic audio generation model can be a symbolic audio file corresponding to multiple short audio clips, which is rendered by an audio synthesizer and played to the user. After receiving an instruction from the user to select an audio clip, the symbolic audio file corresponding to the audio clip is input into the symbolic audio generation model to obtain a symbolic audio file corresponding to the complete audio. In this solution, multiple shorter sound clips are generated, and then the user can decide whether to expand them. Allowing the user to decide to expand high-quality sound clips can save computing resources and time, and improve response speed.

[0248] Exemplarily, when the first audio beginning segment is input into the symbolic audio generation model to generate multiple short audio segments, the following may be displayed: Fig.16The audio display page shown. The audio display page includes a first segment control 161, a second segment control 162, a third segment control 163, a modification control 164, a generate complete audio control 165, and a change batch control 166. After receiving the user moving the focus to the first segment control 161, the audio corresponding to the first segment control 161 is automatically played. After the first segment control 161 is selected, if the user confirms the generate complete audio control 165, the symbolic audio file corresponding to the first segment control 161 is input into the symbolic audio generation model to generate the complete audio. After the first segment control 161 is selected, if the user confirms the modify control 164, you can jump to Fig.10 The music score page shown in FIG. The audio parameters can be modified in the music score interface to regenerate the audio clip. After receiving the user's confirmation operation for the newly generated audio clip, a complete audio can be generated based on the audio clip. After receiving the user's confirmation operation for changing a batch of controls 166, multiple short audio clips can be regenerated.

[0249] In order to make the execution flow and order of the steps in the audio generation method clearer, illustratively, the timing diagram of the audio generation method can be as follows: Fig.17 As shown. The application layer includes a first target application, and the first target application includes an image interaction module, a voice interaction module, an audio playback module, a page interaction module, a text transcription module, an image recognition module, an audio database, and an audio generation module. The image interaction module receives an input image selected by a user, and sends the input image to the image recognition module. Among them, the image interaction module can perform preprocessing such as image enhancement on the input image. The image recognition module uses a multimodal large model to recognize the input image, obtains an image description text, and sends the image description text to the text transcription module. The voice interaction module can receive voice data input by the user, recognize the voice data using voice recognition technology, obtain voice text, and send the voice text to the text transcription module. Among them, the voice interaction module can perform acoustic processing such as denoising on the voice data.

[0250] The text transcription module inputs the image description text and / or the speech text into the text transcription model to obtain multiple audio parameters. The text transcription module calls the data of the audio database, finds the MIDI file information description matching the multiple audio parameters from the audio database, obtains the audio beginning segment corresponding to the MIDI file information description, and sends the audio beginning segment to the audio generation module. The audio generation module inputs the audio beginning segment as a prompt word into the symbol audio generation model to obtain the symbol audio file, and sends the symbol audio file to the audio playback module. The audio playback module converts the symbol audio file into audio and music score information, then plays the audio and sends the music score information to the page interaction module. The page interaction module draws the music score page based on the music score information and displays the music score page.

[0251] After receiving the instruction that the user selects the re-edited audio clip and / or the audio parameter modification on the music score page, the page interaction module sends the generated symbol audio file, the re-edited audio clip and the modified audio parameters to the audio generation module. The audio generation module obtains the symbol audio file before the re-edited audio clip from the generated symbol audio file, inputs the symbol audio file and / or the modified audio parameters as prompt words into the symbol audio generation model, obtains a new symbol audio file, and sends the new symbol audio file to the audio playback module. The audio playback module performs steps such as converting the new symbol audio file into audio and music score information.

[0252] When the memory and computing resources of the display device 200 are insufficient to support the storage or computing of the text transcription model and other models, at least one of the text transcription module, the image recognition module, the audio database and the audio generation module may be set in the server 400. For example, the timing diagram of the audio generation method is as follows: Fig.18 The audio generation method has been Fig.17 The details are described in detail in the chapter and will not be repeated here.

[0253] The solution provided in the embodiment of the present application can solve the problem that the audio output by the audio-based audio generation model cannot be edited again, and at the same time solves the problem that the traditional symbol-based audio generation model cannot handle the note duration well. In addition, during the application process, due to the non-audio characteristics of the MIDI file itself and the small storage space, the processing efficiency is greatly improved, making interactive editing possible. After the MIDI corresponding music score is visualized, the accuracy is higher and the user experience is better. Interactive editing and music creation can meet the user's growing "self-expression" needs.

[0254] In some application scenarios, the user needs to add audio before the final work can be presented. In order to allow the user to add satisfactory audio, the embodiment of the present application also provides a display device 200, such as Fig.19 As shown, the controller 250 is configured to perform the following steps:

[0255] Step S1901: When displaying a first page of a first application, in response to an instruction to add audio, control the display to display a second application page of a second target application.

[0256] When the first page is displayed, in response to an instruction to add audio, a second target application is started, and a second application page of the second target application is displayed.

[0257] In some embodiments, when only one application can be run in the foreground, the second target application is started and runs in the foreground, and the first application retreats to the background to run.

[0258] In other embodiments, when multiple applications can be run in the foreground, for example, the display area is divided into multiple areas to run multiple applications. The first page of the first application is displayed in the first area, and after the second target application is started, the second application page can be displayed in the second area.

[0259] The first application includes an application that can add audio, such as a Vlog application, a short video application, a wallpaper application, an electronic album application, and a home singing application. The application can choose to add audio that has been saved locally, audio downloaded from the Internet, and audio generated according to user input information, thereby combining audio with other data to generate a new content. For example, a short video application can add background music (audio) to the video shot by the user to form a new video.

[0260] The second target application can be the same application as the first target application mentioned above, or different applications with the same function. The execution steps of the second target application can refer to the execution steps of the first target application, and the second application page can be the same page as the first application page, or different pages providing the same function controls, which will not be repeated here.

[0261] Exemplarily, the first application is a short video application, and after receiving the publishing instruction for the target video, the following is displayed: Fig. 20 The publishing prompt page shown in FIG. 1 is the first page. The publishing prompt page includes a video control 201 and a first prompt box 202. The first prompt box 202 includes a first prompt message 2021, an add audio control 2022, and a direct publish control 2023. After receiving the user's confirmation operation for adding the audio control 2022, the following may be displayed: Figure 5 The information input page shown.

[0262] Step S1902: receiving second input information input on the second application page.

[0263] Step S1903: Control the second target application to determine the second audio start segment according to the second input information.

[0264] Step S1904: Control the second target application to input the second audio beginning segment into the symbolic audio generation model to obtain a fourth symbolic audio file.

[0265] Step S1905: Control the second target application to generate a fourth audio according to the fourth symbolic audio file.

[0266] The second input information is essentially the same as the first input information. The method for determining the beginning segment of the second audio is essentially the same as the method for determining the beginning segment of the first audio. The method for generating the fourth symbolic audio file is essentially the same as the method for generating the first symbolic audio file. The method for generating the fourth audio is essentially the same as the method for generating the first audio. For the above, please refer to the above and will not be repeated here.

[0267] Step S1906: Control the first application to generate first content based on the fourth audio.

[0268] Step S1907: Control the display to display the first page, and display the first content on the first page.

[0269] After receiving the user's confirmation operation on the fourth audio, the second target application sends the fourth audio to the first application and exits the background operation or directly stops the operation. After receiving the fourth audio, the first application generates the first content based on the fourth audio and changes to the foreground operation, that is, controls the display to display the first page and displays the first content on the first page.

[0270] In some embodiments, the first content includes a fourth audio, and the fourth audio may be associated with one or more events in the first application, and the corresponding fourth audio may be played after the event is triggered. Exemplarily, an audio corresponding to a screen saver image may be generated in a screen saver application. When the display device 200 displays the screen saver image, the fourth audio may be played.

[0271] In other embodiments, the first content includes content formed by merging the fourth audio with other data. Exemplarily, in a short video application, the fourth audio can be combined with the video screen to generate a new video, that is, the fourth audio replaces the audio in the original video, thereby forming a new video.

[0272] After the first page is displayed, a control corresponding to the first content may be displayed on the first page, and after receiving a confirmation operation of the user on the control, the first content is displayed on the first page.

[0273] Exemplarily, after generating the fourth audio, the display device 200 jumps back to the release prompt page. Fig.21 As shown, the release prompt page includes an update video control 211 and a second prompt box 212. The second prompt box 212 includes a second prompt message 2121, a change audio control 2122, and a release control 2123. The corresponding video can be played directly in the update video control 211, or after receiving a user input to select the update video control 211, the corresponding video can be played.

[0274] In order to make the execution flow and sequence of the steps in the content display method more clear, for example, the timing diagram of the content display method can be as follows: Fig. 22 As shown. The application layer includes a first application and a second target application. The first application includes a first page interaction module and a content generation module. The second target application includes an image interaction module, a voice interaction module, an audio playback module, a second page interaction module, a text transcription module, an image recognition module, an audio database, and an audio generation module. The framework layer includes an application management module.

[0275] The first page interaction module displays the first page and sends a request to call the second target application to the application management module after receiving the user input of the instruction to add audio. The application management module sends a start instruction to the second target application. After the second target application is started, it can execute the same Fig.17 The same steps as in are not repeated here. After receiving the user's confirmation operation on the generated audio, the audio is sent to the content generation module of the first application, and a task completion notification is sent to the application management module. The application management module sends a foreground operation notification to the first page interaction module, and the first page interaction module displays the first page again. The content generation module generates target content based on the audio, and sends the target content to the first page interaction module, which displays the target content on the first page.

[0276] When the memory and computing resources of the display device 200 are insufficient to support the storage or calculation of models such as text transcription models, at least one of the text transcription module, image recognition module, audio database, audio generation module and content generation module may be set in the server 400.

[0277] In some application scenarios, the user needs to add audio before the final work can be presented. In order to allow the user to add satisfactory audio, the embodiment of the present application also provides a display device 200, such as Fig.23 As shown, the controller 250 is configured to perform the following steps:

[0278] Step S2301: When displaying the second page of the second application, receiving third input information input on the second page.

[0279] Step S2302: In response to the instruction to add audio, call the target service.

[0280] Step S2303: Control the target service to determine the third audio start segment according to the third input information.

[0281] Step S2304: Control the target service to input the third audio beginning segment into the symbolic audio generation model to obtain the fifth symbolic audio file.

[0282] Step S2305: Control the target service to generate the fifth audio according to the fifth symbolic audio file.

[0283] Step S2306: controlling the second application to generate second content based on the fifth audio;

[0284] Step S2307: Control the display to display the second page, and display the second content on the second page.

[0285] The second application can be the same application as the first application mentioned above, or it can be a different application with the same function. The third input information is essentially the same as the first input information. The second page can be the same page as the first application page, or it can be a different page that provides the same function controls. The steps executed by the target service can refer to the steps executed by the first target application. The method for determining the beginning segment of the second audio is essentially the same as the method for generating the beginning segment of the first audio. The fourth symbol audio file is essentially the same as the method for generating the first symbol audio file. The fourth audio is essentially the same as the method for generating the first audio. The second content is essentially the same as the method for generating the first content. Please refer to the above for all the above, and I will not repeat them here.

[0286] Exemplarily, the second application includes an electronic photo album application. Fig.24 As shown, the electronic album setting page includes a first image control 2401, a second image control 2402, a third image control 2403, an add image control 2404, an add audio control 2405, and a do not add audio control 2406. After receiving a user confirmation operation on the add image control 2404, the image selected by the user can be received as the electronic album image. After receiving a user confirmation operation on the add audio control 2405, the target service can be called to generate audio based on the selected electronic album image.

[0287] In order to make the execution flow and sequence of the steps in the content display method more clear, for example, the timing diagram of the content display method can be as follows: Fig.25 As shown. The application layer includes a second application. The second application includes a page interaction module and a content generation module. The framework layer includes a target service. The target service includes an image interaction module, a voice interaction module, an audio playback module, a text transcription module, an image recognition module, an audio database and an audio generation module. The framework layer includes an application management module.

[0288] The page interaction module displays the second page and after receiving the user input to add audio, calls the target service to send the input image received by the second page to the image interaction module, and / or sends the voice data received by the second page to the voice interaction module. Fig.17 After receiving the user's confirmation operation on the generated audio, the page interaction module sends the audio to the content generation module, and the content generation module generates target content based on the audio, and sends the target content to the first page interaction module, which displays the target content on the second page.

[0289] When the memory and computing resources of the display device 200 are insufficient to support the storage or calculation of models such as text transcription models, at least one of the text transcription module, image recognition module, audio database, audio generation module and content generation module may be set in the server 400.

[0290] Combined with the description in the above embodiments, the application process of the content display method provided by the present application can be as follows: Fig.26 As shown, after obtaining the text corresponding to the user input information, the display device can input the text into the text transcription model to obtain audio parameters. Among them, the text can be used to describe the audio, usage scenarios, and atmosphere. After determining the audio beginning segment corresponding to the audio parameters, the audio beginning segment can be input into the symbolic audio generation model to obtain a MIDI file. MIDI files can synthesize audio to play to users, and can also be used in electronic albums, wallpaper generation, home singing, and Vlog / short videos.

[0291] The solution provided in the embodiment of the present application can realize the interactive / editable audio generation function and enhance the user experience of the display device 200. It advocates allowing users to create short video background music by themselves, and can also be extended to support electronic photo album background music, wallpaper generation background music, etc., to meet people's growing "self-expression" needs in the current traffic era.

[0292] Some embodiments of the present application also provide a computer-readable storage medium, which may store a program. When the computer storage medium is configured in a display device or a server, the program may include the program steps involved in the audio generation and content display methods in the above embodiments when the program is executed. Among them, the computer storage medium may be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0293] The embodiment of the present application provides an electronic device, which includes: a processor and a memory for storing processor executable instructions. The processor is used to read the executable instructions from the memory and execute the instructions to implement the audio generation and content display method in the above embodiment.

[0294] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0295] For ease of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are intended to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that: include: monitor; configured to display a user interface; A controller is coupled to the display and is configured to: In response to an application startup instruction for running a first target application, controlling the display to display a first application page of the first target application; Receiving first input information inputted on the first application page; Controlling the first target application to determine a first audio beginning segment according to the first input information; Controlling the first target application to input the first audio beginning segment into a symbolic audio generation model to obtain a first symbolic audio file; The first target application is controlled to generate a first audio according to the first symbolic audio file.

2. The display device according to claim 1, characterized in that The first input information includes voice data, and the controller controls the first target application to determine a first audio start segment according to the first input information, and is further configured to: Controlling the first target application to recognize the voice data to obtain voice text; Controlling the first target application to input the speech text into a text transcription model to obtain at least one audio parameter; The first target application is controlled to determine a first audio start segment corresponding to the at least one audio parameter.

3. The display device according to claim 1, characterized in that The first input information includes an input image, and the controller controls the first target application to determine a first audio start segment according to the first input information, and is further configured to: Controlling the first target application to input the input image into an image recognition model to obtain image description text; Controlling the first target application to input the image description text into a text transcription model to obtain at least one audio parameter; The first target application is controlled to determine a first audio start segment corresponding to the at least one audio parameter.

4. The display device according to claim 1, characterized in that The first input information includes voice data and an input image, and the controller controls the first target application to determine a first audio start segment according to the first input information, and is further configured to: Controlling the first target application to recognize the voice data to obtain voice text, and inputting the input image into an image recognition model to obtain image description text; Controlling the first target application to input the speech text and the image description text into a text transcription model to obtain at least one audio parameter; The first target application is controlled to determine a first audio start segment corresponding to the at least one audio parameter.

5. The display device according to any one of claims 2 to 4, characterized in that: The controller controls the first target application to determine a first audio start segment corresponding to the at least one audio parameter, and is further configured to: determining in an audio database a symbolic audio information description that matches the at least one audio parameter; An audio start segment in the audio database corresponding to the symbolic audio information description is determined as a first audio start segment, wherein the audio database includes a correspondence between the symbolic audio information description and the audio start segment.

6. The display device according to claim 1, characterized in that When controlling the first target application to generate the first audio according to the first symbolic audio file, the controller is further configured to: controlling the first target application to convert the first symbolic audio file into music score information; The display is controlled to display a music score page drawn based on the music score information.

7. The display device according to claim 6, characterized in that After controlling the display to display the score page drawn based on the score information, the controller is further configured to: receiving a confirmation operation on the audio segment to be modified in the music score page; Controlling the first target application to determine the audio segment before the audio segment to be modified as the target audio segment; Controlling the first target application to input the target audio segment into the symbolic audio generation model to obtain a second symbolic audio file; The first target application is controlled to generate a second audio according to the second symbolic audio file.

8. The display device according to claim 6, characterized in that After controlling the display to display the score page drawn based on the score information, the controller is further configured to: receiving a modification operation on audio parameters in the music score page; Controlling the first target application to input the first symbolic audio file and the modified audio parameters into the symbolic audio generation model to obtain a third symbolic audio file; The first target application is controlled to generate a third audio according to the third symbolic audio file.

9. A display device, characterized in that: include: monitor; configured to display a user interface; A controller is coupled to the display and is configured to: In the case of controlling the display to display a first page of a first application, in response to an instruction to add audio, controlling the display to display a second application page of a second target application; receiving second input information inputted on the second application page; Controlling the second target application to determine a second audio start segment according to the second input information; Controlling the second target application to input the second audio beginning segment into a symbolic audio generation model to obtain a fourth symbolic audio file; Controlling the second target application to generate a fourth audio according to the fourth symbolic audio file; Controlling the first application to generate first content based on the fourth audio; The display is controlled to display the first page, and the first content is displayed on the first page.

10. A display device, characterized in that: include: monitor; configured to display a user interface; A controller is coupled to the display and is configured to: In a case where the display is controlled to display a second page of a second application, receiving third input information input on the second page; In response to the instruction to add audio, calling the target service to determine a third audio beginning segment according to the third input information; Controlling the target service to input the third audio beginning segment into a symbolic audio generation model to obtain a fifth symbolic audio file; Controlling the target service to generate a fifth audio according to the fifth symbolic audio file; Control the second application to generate second content based on the fifth audio; The display is controlled to display the second page, and the second content is displayed on the second page.

11. An audio generation method, characterized in that: include: In response to an application start instruction to run a first target application, displaying a first application page of the first target application; Receiving first input information inputted on the first application page; Controlling the first target application to determine a first audio beginning segment according to the first input information; Controlling the first target application to input the first audio beginning segment into a symbolic audio generation model to obtain a first symbolic audio file; The first target application is controlled to generate a first audio according to the first symbolic audio file.

12. A content display method, characterized in that: include: In the case of displaying the first page of the first application, in response to the instruction to add audio, displaying the second application page of the second target application; receiving second input information inputted on the second application page; Controlling the second target application to determine a second audio start segment according to the second input information; Controlling the second target application to input the second audio beginning segment into a symbolic audio generation model to obtain a fourth symbolic audio file; Controlling the second target application to generate a fourth audio according to the fourth symbolic audio file; Controlling the first application to generate first content based on the fourth audio; The first page is displayed, and the first content is presented on the first page.

13. A content display method, characterized in that: include: When displaying a second page of a second application, receiving third input information input on the second page; In response to the instruction to add audio, calling the target service to determine a third audio beginning segment according to the third input information; Controlling the target service to input the third audio beginning segment into a symbolic audio generation model to obtain a fifth symbolic audio file; Controlling the target service to generate a fifth audio according to the fifth symbolic audio file; Control the second application to generate second content based on the fifth audio; The second page is displayed, and the second content is presented on the second page.

Images