A method, apparatus and system for music generation based on a smart speaker

Abstract

Embodiments of the present application disclose a method, device and system for music generation based on a smart sound box. In the embodiments of the present application, user voice instruction information is obtained; the user voice instruction information is converted into user text instructions, and the voiceprint feature of the user is determined according to the user voice instruction information; the lyrics text is determined according to the user text instructions, and the lyrics vector feature is generated according to the lyrics text; the reference song information is determined according to the user text instructions, and the audio vector feature is generated according to the reference song information; the historical audio information of the user is determined according to the voiceprint feature of the user, the user voice instruction information and the historical audio information are subjected to feature extraction, and the target voiceprint feature of the user is determined; and the lyrics vector feature, the audio vector feature and the target voiceprint feature are input into a large language model to generate a real-time target audio stream. Through the above method, personalized music can be generated for the user simply and quickly, and the experience of the user is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, more particularly, to a method, device and system for music generation based on a smart speaker. BACKGROUND

[0002] In recent years, with the development of artificial intelligence technology, various smart devices have entered people's daily life, providing a lot of convenience for people's life. For example, a smart speaker, as a core component of smart home, has evolved from a simple voice assistant into a multifunctional platform integrating entertainment, education and life services. Users' expectations for the smart speaker are no longer limited to basic voice interaction and playback control, but hope to obtain more personalized and immersive multi-dimensional experience. For example, the traditional smart speaker relies on a preset music library and streaming media service, and lacks personalized music creation capability.

[0003] In the prior art, in order to generate personalized music, a sound-based music generation scheme and a text-based music generation scheme are adopted. In the sound-based music generation scheme, the user needs to record a long sound by himself, which can be personalized, but the long sound recording process is a burden for the user, which can easily lead to user boredom or loss of patience during the experience, resulting in user loss. In the text-based music generation scheme, the text is input into an agent or a plug-in to generate music, but the text-based music generation by the agent or the plug-in is limited by the system architecture. The user cannot interact with the user for at least 2 to 3 minutes after triggering the music generation task, and can only wait for the end of the music generation before playing the music, which seriously affects the user experience, and the scheme cannot provide personalized timbre matching, resulting in weak user participation.

[0004] In summary, how to simply and quickly generate personalized music for users and improve user experience is a problem to be solved at present. SUMMARY

[0005] Therefore, the embodiments of the present application provide a method, device and system for music generation based on a smart speaker, which can simply and quickly generate personalized music for users and improve user experience.

[0006] In a first aspect, the embodiments of the present application provide a method for music generation based on a smart speaker, which comprises:

[0007] obtaining user voice instruction information;

[0008] converting the user voice instruction information into user text instructions, and determining a voiceprint feature of the user according to the user voice instruction information;

[0009] determine lyrics text according to the user text instruction, and generate lyrics vector features according to the lyrics text;

[0010] determine reference track information according to the user text instruction, and generate audio vector features according to the reference track information;

[0011] determine historical audio information of the user according to the voiceprint features of the user, perform feature extraction on the user voice instruction information and the historical audio information, and determine target voiceprint features of the user;

[0012] input the lyrics vector features, the audio vector features, and the target voiceprint features into a large language model to generate a real-time target audio stream.

[0013] Optionally, the method further comprises:

[0014] output the real-time target audio stream to a smart speaker through a gateway.

[0015] Optionally, the method further comprises:

[0016] perform intent recognition on the user text instruction, determine at least one parameter of the text instruction in response to a result of the intent recognition being music generation, and

[0017] in response to the at least one parameter being all necessary parameters for the music generation, call a music generation service, wherein the music generation service represents generating a real-time target audio stream through a large language model.

[0018] Optionally, the method further comprises:

[0019] in response to the at least one parameter being non-all necessary parameters for the music generation, send a reacquisition instruction to a smart speaker, wherein the reacquisition instruction is used to control the smart speaker to reacquire user voice instruction information from a user.

[0020] Optionally, the determining lyrics text according to the user text instruction, and generating lyrics vector features according to the lyrics text specifically comprises:

[0021] output the user text instruction to a basic large model to generate the lyrics text;

[0022] input the lyrics text into a text encoder to generate lyrics vector features.

[0023] Optionally, the determining reference track information according to the user text instruction, and generating audio vector features according to the reference track information specifically comprises:

[0024] Identify the sentiment words in the user's text command;

[0025] The reference music library is queried based on the aforementioned emotional words to determine the reference track information;

[0026] The reference track information is input into the audio encoder to generate audio vector features.

[0027] Optionally, obtaining user voice command information specifically includes:

[0028] Receives the instruction audio stream sent by the smart speaker through the gateway;

[0029] Based on voice activity detection, it is determined that the user has completed issuing the command, and the user's voice command information is determined.

[0030] Optionally, outputting the real-time target audio stream to the smart speaker via a gateway specifically includes:

[0031] Send the real-time target audio stream to the content platform;

[0032] The content platform sends playback commands to the smart speaker through a gateway;

[0033] Upon receiving a read command from the smart speaker, the content platform sends the real-time target audio stream to the smart speaker.

[0034] Secondly, embodiments of the present invention provide a method for music generation based on a smart speaker, the method comprising:

[0035] Get the audio commands sent by the user and generate the command audio stream;

[0036] The instruction audio stream is sent to the cloud server through the gateway, and the user completes the instruction issuance based on the microphone mute detection, thus determining the user's voice instruction information.

[0037] Convert the user's voice command information into user text commands;

[0038] The user's text command is sent to the cloud server through the gateway;

[0039] The cloud server receives a real-time target audio stream sent through a gateway, wherein the real-time target audio stream is generated by the cloud server by inputting lyrics vector features, audio vector features, and target voiceprint features into a large language model.

[0040] Optionally, the method further includes:

[0041] Received a reacquisition command sent by the cloud server through the gateway;

[0042] sending follow-up information to the user to reacquire the user voice instruction information.

[0043] Optionally, the method further comprises:

[0044] receiving a wake-up word sent by the user;

[0045] switching from an idle state to a working state, wherein the idle state indicates that the smart speaker cannot receive the instruction audio stream, and the working state indicates that the smart speaker can receive the instruction audio stream.

[0046] Optionally, the method further comprises:

[0047] receiving a play instruction sent by a content platform of the cloud server through a gateway;

[0048] sending a read instruction to the content platform of the cloud server through the gateway, wherein the read instruction is used to acquire the real-time target audio stream from the content platform of the cloud server.

[0049] In a third aspect, an apparatus for music generation based on a smart speaker is provided, and the apparatus comprises:

[0050] a first acquisition unit configured to acquire user voice instruction information;

[0051] a first processing unit configured to convert the user voice instruction information into user text instruction and determine a voiceprint feature of the user according to the user voice instruction information;

[0052] a first generation unit configured to determine a lyric text according to the user text instruction and generate a lyric vector feature according to the lyric text;

[0053] a second generation unit configured to determine reference song information according to the user text instruction and generate an audio vector feature according to the reference song information;

[0054] a first determination unit configured to determine historical audio information of the user according to the voiceprint feature of the user, extract features from the user voice instruction information and the historical audio information, and determine a target voiceprint feature of the user;

[0055] a third generation unit configured to input the lyric vector feature, the audio vector feature, and the target voiceprint feature into a large language model to generate a real-time target audio stream.

[0056] Optionally, the apparatus further comprises:

[0057] an output unit configured to output the real-time target audio stream to the smart speaker through a gateway.

[0058] Optionally, the apparatus further comprises:

[0059] an intention recognition unit, configured to perform intention recognition on the user text instruction, determine at least one parameter of the text instruction in response to a result of the intention recognition being music generation, and invoke a music generation service in response to the at least one parameter being all necessary parameters for the music generation, wherein the music generation service represents generating a real-time target audio stream through a large language model.

[0060] Optionally, the first acquisition unit is further configured to:

[0061] in response to the at least one parameter being non-all necessary parameters for the music generation, send a re-acquisition instruction to the smart speaker, wherein the re-acquisition instruction is used to control the smart speaker to re-acquire user voice instruction information from the user.

[0062] Optionally, the first generation unit is specifically configured to:

[0063] output the user text instruction to a basic large model to generate the lyric text;

[0064] input the lyric text to a text encoder to generate a lyric vector feature.

[0065] Optionally, the second generation unit is specifically configured to:

[0066] determine an emotional word in the user text instruction;

[0067] query a reference audio source library according to the emotional word to determine reference track information;

[0068] input the reference track information to an audio encoder to generate an audio vector feature.

[0069] Optionally, the first acquisition unit is specifically configured to:

[0070] receive an instruction audio stream sent by the smart speaker through a gateway;

[0071] determine that the user voice instruction information is issued by the user according to voice activity detection that the user completes instruction issuance.

[0072] Optionally, the output unit is specifically configured to:

[0073] send the real-time target audio stream to a content platform;

[0074] the content platform sends a play instruction to the smart speaker through a gateway;

[0075] The content platform sends the real-time target audio stream to the smart speaker after receiving the reading instruction sent by the smart speaker.

[0076] In a fourth aspect, an apparatus for music generation based on a smart speaker is provided, and the apparatus comprises:

[0077] A second obtaining unit is configured to obtain an audio instruction sent by a user and generate an instruction audio stream.

[0078] A second determining unit is configured to send the instruction audio stream to a cloud server through a gateway and determine that the user completes the instruction issuing according to a microphone closing detection, and determine the user voice instruction information sent by the user.

[0079] A second processing unit is configured to convert the user voice instruction information into a user text instruction.

[0080] A first sending unit is configured to send the user text instruction to the cloud server through the gateway.

[0081] A first receiving unit is configured to receive a real-time target audio stream sent by the cloud server through the gateway, wherein the real-time target audio stream is generated by inputting a lyric vector feature, an audio vector feature and a target voiceprint feature into a large language model by the cloud server.

[0082] Optionally, the first sending unit is further configured to:

[0083] receive a re-obtaining instruction sent by the cloud server through the gateway;

[0084] The second obtaining unit is further configured to send a follow-up question to the user and re-obtain the user voice instruction information.

[0085] Optionally, the first receiving unit is further configured to:

[0086] receive a wake-up word sent by the user;

[0087] The second processing unit is further configured to switch an idle state to a working state, wherein the idle state indicates that the smart speaker cannot receive the instruction audio stream, and the working state indicates that the smart speaker can receive the instruction audio stream.

[0088] Optionally, the first receiving unit is further configured to:

[0089] receive a playing instruction sent by a content platform of the cloud server through the gateway;

[0090] The first sending unit is further configured to send a reading instruction to a content platform of the cloud server through the gateway, where the reading instruction is used to acquire the real-time target audio stream from the content platform of the cloud server.

[0091] In a fifth aspect, an embodiment of the present application provides a music generation system based on an intelligent sound box, comprising a cloud server and an intelligent sound box; wherein the cloud server is configured to execute the method in the first aspect or any possible one of the first aspect, and the intelligent sound box is configured to execute the method in the second aspect or any possible one of the second aspect.

[0092] In a sixth aspect, an embodiment of the present application provides an electronic device comprising a memory and a processor, the memory being configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method in the first aspect, any possible one of the first aspect, the method in the second aspect, or any possible one of the second aspect.

[0093] In a seventh aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer program instructions, the computer program instructions being executed by a processor to implement the method in the first aspect, any possible one of the first aspect, the method in the second aspect, or any possible one of the second aspect. In the embodiment of the present application, the user voice instruction information is acquired; the user voice instruction information is converted into user text instructions, and the voiceprint feature of the user is determined according to the user voice instruction information; the lyrics text is determined according to the user text instructions, and the lyrics vector feature is generated according to the lyrics text; the reference song information is determined according to the user text instructions, and the audio vector feature is generated according to the reference song information; the historical audio information of the user is determined according to the voiceprint feature of the user, the user voice instruction information and the historical audio information are feature extracted to determine the target voiceprint feature of the user; the lyrics vector feature, the audio vector feature and the target voiceprint feature are input into a large language model to generate a real-time target audio stream. Through the above method, the personalized music meeting the user's demand can be generated simply and quickly for the user, and the user's experience is improved. BRIEF DESCRIPTION OF DRAWINGS

[0094] The above and other objects, features and advantages of the present application will become more apparent from the following description of embodiments of the present application, taken in conjunction with the accompanying drawings, in which:

[0095] Figure 1 is a schematic diagram of a music generation system based on an intelligent sound box in an embodiment of the present application;

[0096] Figure 2is a method flowchart of music generation based on a smart sound box in an embodiment of the present application;

[0097] Figure 3 is another method flowchart of music generation based on a smart sound box in an embodiment of the present application;

[0098] Figure 4 is still another method flowchart of music generation based on a smart sound box in an embodiment of the present application;

[0099] Figure 5 is yet another method flowchart of music generation based on a smart sound box in an embodiment of the present application;

[0100] Figure 6 is another method flowchart of music generation based on a smart sound box in an embodiment of the present application;

[0101] Figure 7 is still another method flowchart of music generation based on a smart sound box in an embodiment of the present application;

[0102] Figure 8 is yet another method flowchart of music generation based on a smart sound box in an embodiment of the present application;

[0103] Figure 9 is a flowchart of interaction between a cloud server and a smart sound box in an embodiment of the present application;

[0104] Figure 10 is a device schematic diagram of music generation based on a smart sound box in an embodiment of the present application;

[0105] Figure 11 is another device schematic diagram of music generation based on a smart sound box in an embodiment of the present application;

[0106] Figure 12 is a schematic diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0107] The present application is described in the following based on embodiments, but the present application is not limited to these embodiments only. In the following detailed description of the present application, some specific details are described in detail. The present application can also be completely understood without the description of these details by those skilled in the art. In order to avoid confusion of the essence of the present application, the well-known methods, processes, flows, elements and circuits are not described in detail.

[0108] In addition, the drawings provided herein are for illustrative purposes only and the drawings are not necessarily drawn to scale.

[0109] Unless the context clearly requires otherwise, throughout the application the general term "comprise", "comprising" or the like is to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in the sense of "including, but not limited to".

[0110] In the description of the application, it should be understood that the terms "first", "second" and the like are used only for the purpose of description, and cannot be understood as indicating or implying relative importance. In addition, in the description of the application, unless otherwise stated, the meaning of "a plurality of" is two or more.

[0111] In the prior art, in order to generate personalized music and provide users with an immersive experience, the following two ways are adopted:

[0112] Method one, sound-based music generation scheme.

[0113] Specifically, some common music applications require users to record a long period of sound. The above music application generates music using the voiceprint features of the recorded sound. Although this can be personalized to create music that belongs to their own voiceprint and enhance the interactive experience, the above music application generally requires recording an audio sample of more than 60 seconds to ensure the similarity of the timbre of the generated music to the user's timbre. The timbre is determined by the voiceprint features. Different voiceprint features will generate different timbres. However, the long sound recording process is a burden for users, which can easily lead to user boredom or loss of patience during the experience, resulting in user loss.

[0114] Method two, text-based music generation scheme.

[0115] Specifically, some large language model providers open agents or plug-ins, input text into the agents or plug-ins to generate music, but the text-to-music generation through the agents or plug-ins is limited by the system architecture, and the user cannot interact with the user for at least 2-3 minutes after triggering the music generation task, and can only wait for the end of the generation before playing the music, which seriously affects the user experience, and the scheme cannot provide personalized timbre matching, so that the user's participation and experience are weak, and it is difficult to meet the user's demand for instant rewards and interaction. The large-scale language model represents the frontier of the deep integration of artificial intelligence and various fields, and has shown amazing effects in different applications. In addition to the music field, it has shown better ability in language understanding, knowledge memory, text understanding, text generation, image recognition, and even complex decision making. The above large-scale language model is a language model constructed by a deep neural network containing more than 100 billion parameters, which is trained by a large amount of unlabeled text using a self-supervised learning method, can process various natural language tasks, and is an important way to artificial intelligence. For example, the large-scale language model applied to ChatGPT can have a natural and smooth conversation with the user and answer various questions. In addition, the large-scale language model can accurately capture and simulate various expressions of humans, from text to sound, and even subtle emotional fluctuations. In the music field, with the application of large-scale language models, intelligent speakers are moving towards higher levels of intelligence, opening up new possibilities for future human-computer interaction.

[0116] Therefore, how to simply and quickly generate personalized music for users and improve user experience is a problem to be solved at present.

[0117] In the embodiment of the application, in order to solve the above problems, a system for music generation based on a smart speaker is provided, as shown in Figure 1 The system includes a cloud server 101 and a smart speaker 102. The execution steps of the cloud server 101 and the smart speaker 102 are described in detail below.

[0118] In a possible implementation, the smart speaker can also be other devices integrated with voice interaction functions, such as a smart TV, a smart tablet, a smart wearable device, etc.

[0119] In the embodiment of the application, the processing flow of the cloud server is as shown in Figure 2 The processing flow of the cloud server is as shown in Figure 2 A method flowchart for music generation based on a smart speaker, specifically including the following steps:

[0120] Step S201, obtaining user voice instruction information.

[0121] Specifically, the smart speaker receives the instruction audio stream sent by the gateway, determines that the user completes the instruction issuing according to voice activity detection (VAD), and determines the user voice instruction information issued by the user.

[0122] For example, the voice instruction information issued by the user is "play a lyrical music", the voice instruction information is real-time relayed by the smart speaker through the gateway, the smart speaker sends the voice of one word to the cloud server through the gateway every time it receives the voice of one word, the instruction audio stream received by the cloud server is real-time received, the voice activity detection VAD is used to determine that the user's instruction is completely issued, and it is determined that the user voice instruction information issued by the user is "play a lyrical music", wherein the real-time voice instruction information is audio information.

[0123] In a possible implementation, the voice instruction information issued by the user can also be other music generation instructions, such as "generate a light background music", etc., which is determined according to actual conditions.

[0124] In a possible implementation, the language instruction information can also have other functions in addition to generating music, such as "read a Tang poem", "tell a fairy tale suitable for a 3-year-old child", etc.

[0125] Step S202, convert the user voice instruction information into user text instruction, and determine the voiceprint feature of the user according to the user voice instruction information.

[0126] Specifically, the user voice instruction information is converted into user text instruction by automatic speech recognition (ASR), and the voiceprint feature of the user can be extracted from the user voice instruction by voiceprint recognition (VPR), wherein the voiceprint feature is used in the subsequent step S205, under the premise of user authorization, the historical audio information of the user can be obtained in a non-perceptual manner, the historical audio information is the historical dialogue record between the user and the smart speaker of the voiceprint feature, and the historical dialogue record is saved in the cloud server.

[0127] Step S203, determine the lyrics text according to the user text instruction, and generate the lyrics vector feature according to the lyrics text.

[0128] Specifically, the user text instruction is output to the base model to generate the lyrics text; and the lyrics text is input to the text encoder to generate the lyrics vector feature.

[0129] In a possible implementation, the user text instruction is input as a prompt word to the base large model without the user specifying the lyrics, the base large model expands the user text instruction to generate lyrics text; wherein the base large model is also a large language model.

[0130] In the embodiment of the application, the lyrics text is encoded according to the text editor to generate lyrics vector features.

[0131] In a possible implementation, in the case that the user specifies the lyrics, the lyrics text is encoded according to the text editor to generate lyrics vector features.

[0132] In step S204, reference song information is determined according to the user text instruction, and audio vector features are generated according to the reference song information.

[0133] Specifically, an emotional word in the user text instruction is determined, the reference song library is queried according to the emotional word to determine reference song information, and the reference song information is input to an audio encoder to generate audio vector features.

[0134] In the embodiment of the application, the reference song library includes reference songs of various styles, different emotional words correspond to reference songs of different styles, and the emotional word in the user text instruction is queried in the reference song library to determine a reference song that matches the style of the emotional word, for example, if the emotional word is 'happy', a reference song of a happy or cheerful style is searched in the reference song library, if the emotional word is'sad', a reference song of a sad style is searched in the reference library, and then the determined reference song information is input to the audio encoder for encoding to generate audio vector features.

[0135] In a possible implementation, the style and emotional word can also be modeled in advance, and after the emotional word is input to the established model, the style corresponding to the emotional word is directly generated, so as to avoid the situation that the style cannot be determined through the emotional word when the emotional word and the style do not completely correspond.

[0136] In step S205, the historical audio information of the user is determined according to the voiceprint feature of the user, the user voice instruction information and the historical audio information are feature extracted to determine the target voiceprint feature of the user.

[0137] Specifically, different user voiceprint features are completely different and are unique identifiers of users, and the users can match historical audio information belonging to themselves in a historical audio database through the unique identifiers, and the cloud server can directly extract voiceprint features through the user voice instruction information and the historical audio information, without the need for the user to deliberately record a long audio for voiceprint feature extraction, and directly uses voice information saved in a historical interaction process of the user and the smart speaker, thereby facilitating and quick extraction of the target voiceprint feature of the user.

[0138] Step S206, inputting the lyric vector feature, the audio vector feature and the target voiceprint feature into a large language model to generate a real-time target audio stream.

[0139] In the embodiment of the application, the lyric vector feature, the audio vector feature and the target voiceprint feature are input into a large language model (LLM) to generate a real-time target audio stream meeting the user demand, and the real-time target audio stream is generated based on the voiceprint feature of the user, thereby meeting the personalized and diversified needs of the user.

[0140] In a possible implementation, after the user text instruction is given, the determining of the user voiceprint feature based on the user voice instruction information, the generating of the lyric vector feature based on the lyric text, and the generating of the audio vector feature based on the reference song information can be simultaneously performed.

[0141] In a possible implementation, after the step S206, other steps are further included, specifically as Figure 3 As shown in the figure, the Figure 3 Another method flowchart for music generation based on a smart speaker, specifically including the following steps:

[0142] Step S207, outputting the real-time target audio stream to the smart speaker through a gateway.

[0143] Specifically, the real-time target audio stream is sent to a content platform; the content platform sends a playing instruction to the smart speaker through a gateway; after receiving a reading instruction sent by the smart speaker, the content platform sends the real-time target audio stream to the smart speaker.

[0144] In the embodiment of the present application, the content platform is part of the cloud server, the cloud server generates a target audio stream through the large language model and then puts it into the content platform, after the content platform receives the target audio stream, it sends a play instruction to the smart speaker through the gateway, the content platform sends the real-time target audio stream to the smart speaker, and after the target audio stream is completely output, the target audio that meets the user's demand and the user's voiceprint feature are generated, the target audio is pushed in full-link streaming, realizing real-time transmission of the target audio, ensuring low delay and high efficiency during target audio transmission, reducing the speed of generating target audio after the user initiates user voice instruction information, and improving the user experience.

[0145] In a possible implementation, after the step S202, other steps are further included, specifically as Figure 4 As shown in the figure, the Figure 4 is another method flow chart for music generation based on a smart speaker, specifically including the following steps:

[0146] Step S208, performing intent recognition on the user text instruction, and determining at least one parameter of the text instruction in response to a result of the intent recognition being music generation.

[0147] Specifically, the intent recognition on the user text instruction is performed through natural language understanding (NLU), the result of the intent recognition is determined, and in response to the result being music generation, the parameters need to be extracted in the text instruction, wherein the parameters are prompt words and / or styles.

[0148] In a possible implementation, it is determined whether the at least one parameter is all necessary parameters, if yes, step S209 is performed, and if not, step S210 is performed.

[0149] Step S209, in response to the at least one parameter being all necessary parameters for the music generation, calling a music generation service, wherein the music generation service means generating a real-time target audio stream through a large language model.

[0150] Step S210, in response to the at least one parameter being non-all necessary parameters for the music generation, sending a reacquisition instruction to the smart speaker, wherein the reacquisition instruction is used to control the smart speaker to reacquire user voice instruction information from the user.

[0151] In the embodiment of the present application, the processing flow of the smart speaker is as Figure 5 As shown in the figure, the Figure 5 is a method flow chart for music generation based on a smart speaker, specifically including the following steps:

[0152] Step S501, obtaining an audio instruction sent by a user, and generating an instruction audio stream.

[0153] Step S502, sending the instruction audio stream to a cloud server through a gateway, and determining that the user completes instruction issuing according to a microphone closing detection, and determining user voice instruction information sent by the user.

[0154] Step S503, converting the user voice instruction information into user text instruction.

[0155] In a possible implementation, the user voice instruction information is converted into the user text instruction through automatic speech recognition (ASR).

[0156] Step S504, sending the user text instruction to the cloud server through the gateway.

[0157] Specifically, the smart speaker sends the user text instruction to the cloud server through the gateway, and the cloud server can perform intent recognition according to the user text instruction sent by the smart speaker and the user text instruction generated by the cloud server.

[0158] Step S505, receiving a real-time target audio stream sent by the cloud server through the gateway, wherein the real-time target audio stream is generated by inputting a lyric vector feature, an audio vector feature and a target voiceprint feature into a large language model by the cloud server.

[0159] Specifically, the smart speaker receives the real-time target audio stream sent by the cloud server through the gateway, and plays the real-time target audio stream to the user.

[0160] In a possible implementation, before the step S505, other steps are further included, and specifically as shown in Figure 6 The method for generating music based on the smart speaker specifically includes the following steps: Figure 6 The method for generating music based on the smart speaker specifically includes the following steps:

[0161] Step S506, receiving a playing instruction sent by a content platform of the cloud server through the gateway.

[0162] Specifically, the smart speaker receives the playing instruction sent by the content platform of the cloud server through the gateway.

[0163] Step S507, sending a reading instruction to the content platform of the cloud server through the gateway, wherein the reading instruction is used to acquire the real-time target audio stream from the content platform of the cloud server.

[0164] In a possible implementation, after the step S504, other steps are further included, and specifically as follows. Figure 7 As shown in the figure, the Figure 7 A method flow chart for music generation based on a smart speaker, specifically including the following steps:

[0165] Step S508, receiving the re-acquisition instruction sent by the cloud server through the gateway.

[0166] Specifically, when the parameters identified by the cloud server are not all necessary parameters for music generation, the re-acquisition instruction needs to be sent to the smart speaker.

[0167] Step S509, sending follow-up information to the user to re-acquire the user voice instruction information.

[0168] Specifically, the smart speaker sends follow-up information to the user, for example, if the user's first voice instruction information does not contain emotional words, the follow-up information is "what style of track needs to be generated?", which is only an example.

[0169] In a possible implementation, before the step S501, other steps are further included, and specifically as follows. Figure 8 As shown in the figure, the Figure 8 A method flow chart for music generation based on a smart speaker, specifically including the following steps:

[0170] Step S510, receiving the wake-up word sent by the user.

[0171] Specifically, the wake-up word is a specific word set by the smart speaker, the smart speaker establishes a long link with the gateway, and the smart device is in an idle state when it is not applicable in order to save energy consumption. The smart speaker continuously monitors the wake-up word at low power consumption.

[0172] Step S511, switching from an idle state to a working state, wherein the idle state means that the smart speaker cannot receive the instruction audio stream, and the working state means that the smart speaker can receive the instruction audio stream.

[0173] Specifically, the smart speaker switches from the idle state to the working state and starts receiving the audio instruction sent by the user.

[0174] The interaction process between the cloud server and the smart speaker will be described in detail below through a complete embodiment, and specifically as follows. Figure 9 As shown in the figure, the specific process is as follows.

[0175] Step S901, the smart speaker acquires the wake-up word sent by the user, and switches from an idle state to a working state.

[0176] Step S902, the smart speaker acquires the audio instruction sent by the user, and generates an instruction audio stream.

[0177] Step S903, the smart speaker sends the instruction audio stream to the cloud server through a gateway.

[0178] Specifically, after the step S903 is completed, the step S906 is performed.

[0179] Step S904, the smart speaker performs a mute detection to determine that the user has completed the instruction issuance, and determines the user voice instruction information sent by the user.

[0180] Specifically, the step S903 and the step S904 are synchronously performed.

[0181] Step S905, the smart speaker converts the user voice instruction information into user text instruction, and sends the user text instruction to the cloud server.

[0182] Step S906, the cloud server converts the user voice instruction information into user text instruction through automatic speech recognition.

[0183] Step S907, the cloud server performs intent recognition on the user text instruction.

[0184] Step S908, the cloud server determines whether the result of the intent recognition is music generation, if yes, the step S909 is performed, if not, the process is ended or the user voice instruction information is re-acquired.

[0185] Step S909, the cloud server performs intent recognition on the user text instruction, and determines the at least one parameter, if the at least one parameter is all necessary parameters for the music generation, the step S910 is performed, if not, the step S911 is performed.

[0186] Step S910, the cloud server calls a music generation service.

[0187] Step S911, the cloud server sends a follow-up question information to the user, and re-acquires the user voice instruction information.

[0188] Step S912, the cloud server determines whether a song lyric is specified in the user text instruction, if yes, the step S913 is performed, if not, the step S914 is performed.

[0189] Step S913, the cloud server directly determines a song lyric text.

[0190] Step S914, the cloud server inputs the user text instruction into a base large model to generate a lyric text.

[0191] Specifically, after the step S913 and the step S914, a step S915 is performed.

[0192] Step S915, the cloud server inputs the lyric text into a text editor to encode and generate a lyric vector feature.

[0193] In a possible implementation, after the step S906 is performed, a step S916 is performed simultaneously.

[0194] Step S916, the cloud server extracts features from the user voice instruction information to determine a voiceprint feature of the user.

[0195] Step S917, the cloud server determines historical audio information of the user according to the voiceprint feature of the user, and extracts features from the user voice instruction information and the historical audio information to determine a target voiceprint feature of the user.

[0196] Specifically, the user instruction information is also saved as the historical audio information.

[0197] In a possible implementation, after the step S906 is performed, a step S918 is performed simultaneously.

[0198] Step S918, the cloud server determines an emotional word in the user text instruction.

[0199] Step S919, the cloud server queries a reference audio source library according to the emotional word to determine reference track information.

[0200] Step S920, the cloud server inputs the reference track information into an audio encoder to generate an audio vector feature.

[0201] Specifically, the lyric vector feature generated in the step S915, the target voiceprint feature determined in the step S917, and the audio vector feature generated in the step S920 are input into a large language model to perform a step S921.

[0202] Step S921, the large language model of the cloud server generates a real-time target audio stream.

[0203] Specifically, the real-time audio stream is sent to a content platform through the music generation service.

[0204] Step S922, the smart speaker plays the real-time target audio stream sent by the content platform through a gateway.

[0205] In the embodiment of the present application, during the entire generation process of the real-time target audio stream, the reference track information, lyrics and the like used need to meet the requirements of the copyright party, and the user text instructions and the generated real-time target audio stream are subjected to security compliance verification, and the generation is terminated when bad information is found.

[0206] In a possible implementation, after the real-time target audio stream is generated, the complete target audio can be saved first, and the complete target audio is optimized in terms of style and / or rhythm, and the optimized target audio is output.

[0207] In the embodiment of the present application, the system for music generation based on a smart sound box can be divided into a device layer, a service layer and an algorithm layer in terms of structure, wherein the device layer belongs to a smart sound box, and the service layer and the algorithm layer belong to a cloud server.

[0208] In a possible implementation, the device layer directly interacts with a user, and specifically includes an audio module, a voice module, a communication module and an edge computing module, wherein the audio module specifically implements a wake-up function, an echo cancellation function, a noise reduction function, a gain and correction function; the voice module specifically implements an endpoint detection, voice enhancement, voice recognition and voice activity detection function; the communication module specifically implements a long link communication, local area networking, data stream management and device discovery function; and the edge computing module specifically implements an acoustic model, a language model, a codec and an offline recognition function.

[0209] In a possible implementation, the service layer of the cloud server includes a gateway, a content service and a skill service, wherein the gateway specifically implements a connection, reason, protocol, traffic and service function; the content service specifically implements a content storage, content retrieval, content control and content distribution function; the skill service specifically implements a registration discovery, call decision, combination arrangement and response fallback function; and the algorithm layer of the cloud server includes tokenization, a music model, a voice algorithm and a language algorithm, wherein the tokenization specifically implements an embedding representation, audio segmentation, text encoding and codebook function, the music model includes a conditional generation model combined with a vector quantization technology and a variational autoencoder (Vector Quantized-Variational Autoencoder, VQ-VAE) and a Transformer decoder; the voice algorithm specifically implements a voiceprint recognition, speech recognition, speech synthesis and keyword detection function; and the language algorithm specifically implements a language understanding, intent recognition and keyword extraction function.

[0210] In one possible implementation, the music generation system based on the smart speaker also includes a security and compliance verification function, specifically including risk control, privacy, security, compliance, and copyright.

[0211] Through the above embodiments, generating real-time target audio streams using large language models firstly allows for the rapid capture of subtle features of the user's voice and lowers the barrier to entry for user voice recording, enabling users to complete high-quality voice recording in everyday conversations. This reduces data transmission and processing time and improves the speed and accuracy of feature extraction. Secondly, the adoption of a full-link streaming push architecture from the large language model to the gateway to the smart speaker achieves real-time data transmission and processing, effectively alleviating the problem of slow generation speed of large models. Users can quickly receive feedback after initiating music generation requests, significantly improving the interactive experience. The optimization of the entire link ensures low latency and high efficiency in data transmission, making the music generation process smoother and more fluid. Thirdly, it fully leverages the voice interaction capabilities of the smart speaker. Users can directly describe and adjust their music generation needs using natural language, and can personalize music through simple voice commands, such as "generate a relaxing background music" or "adjust the rhythm to be more upbeat," greatly improving the convenience of operation. The smart speaker can understand and respond to complex commands, making the music generation process more interactive and personalized, meeting diverse user needs.

[0212] In this embodiment of the invention, a device for music generation based on a smart speaker is provided, such as... Figure 10 As shown, the system specifically includes: a first acquisition unit 1001, a first processing unit 1002, a first generation unit 1003, a second generation unit 1004, a first determination unit 1005, and a third generation unit 1006; wherein, the first acquisition unit 1001 is used to acquire user voice command information; the first processing unit 1002 is used to convert the user voice command information into user text commands and determine the user's voiceprint features based on the user voice command information; the first generation unit 1003 is used to determine lyrics text based on the user text commands and generate lyrics vector features based on the lyrics text; the second generation unit 1004 is used to determine reference track information based on the user text commands and generate audio vector features based on the reference track information; the first determination unit 1005 is used to determine the user's historical audio information based on the user's voiceprint features, extract features from the user voice command information and the historical audio information, and determine the user's target voiceprint features; the third generation unit 1006 is used to input the lyrics vector features, audio vector features, and target voiceprint features into a large-scale language model to generate a real-time target audio stream.

[0213] Further, the apparatus further comprises:

[0214] An output unit configured to output the real-time target audio stream to the smart speaker through a gateway.

[0215] Further, the apparatus further comprises:

[0216] An intent recognition unit configured to perform intent recognition on the user text instruction, determine at least one parameter of the text instruction in response to a result of the intent recognition being music generation, and call a music generation service in response to the at least one parameter being all necessary parameters for the music generation, wherein the music generation service represents generating a real-time target audio stream through a large language model.

[0217] Further, the first obtaining unit is further configured to:

[0218] In response to the at least one parameter being non-all necessary parameters for the music generation, send a re-obtaining instruction to the smart speaker, wherein the re-obtaining instruction is used to control the smart speaker to re-obtain user voice instruction information from the user.

[0219] Further, the first generation unit is specifically configured to:

[0220] output the user text instruction to a basic large model to generate the lyric text;

[0221] input the lyric text to a text encoder to generate a lyric vector feature.

[0222] Further, the second generation unit is specifically configured to:

[0223] determine an emotional word in the user text instruction;

[0224] query a reference audio source library according to the emotional word to determine reference song information;

[0225] input the reference song information to an audio encoder to generate an audio vector feature.

[0226] Further, the first obtaining unit is specifically configured to:

[0227] receive an instruction audio stream sent by the smart speaker through the gateway;

[0228] determine the user voice instruction information issued by the user according to voice activity detection in response to the user completing the instruction issuance.

[0229] Further, the output unit is specifically configured to:

[0230] send the real-time target audio stream to a content platform;

[0231] The content platform sends a play instruction to the smart speaker through a gateway.

[0232] Upon receiving the read instruction sent by the smart speaker, the content platform sends the real-time target audio stream to the smart speaker.

[0233] In the embodiments of the present application, a device for music generation based on a smart speaker is provided, as shown in the accompanying drawings, which specifically comprises a second acquisition unit 1101, a second determination unit 1102, a second processing unit 1103, a first sending unit 1104, and a first receiving unit 1105. Figure 11 The second acquisition unit 1101 is configured to acquire an audio instruction sent by a user and generate an instruction audio stream. The second determination unit 1102 is configured to send the instruction audio stream to a cloud server through a gateway and determine that the user has completed the instruction issuance according to a microphone closing detection, and determine user voice instruction information sent by the user. The second processing unit 1103 is configured to convert the user voice instruction information into user text instructions. The first sending unit 1104 is configured to send the user text instructions to the cloud server through the gateway. The first receiving unit 1105 is configured to receive a real-time target audio stream sent by the cloud server through the gateway, wherein the real-time target audio stream is generated by inputting a lyric vector feature, an audio vector feature, and a target voiceprint feature into a large language model by the cloud server.

[0234] Further, the first sending unit is further configured to:

[0235] receive a re-acquisition instruction sent by the cloud server through the gateway;

[0236] The second acquisition unit is further configured to send a follow-up question information to the user to re-acquire the user voice instruction information.

[0237] Further, the first receiving unit is further configured to:

[0238] receive a wake-up word sent by the user;

[0239] The second processing unit is further configured to switch an idle state to a working state, wherein the idle state indicates that the smart speaker cannot receive the instruction audio stream, and the working state indicates that the smart speaker can receive the instruction audio stream.

[0240] Further, the first receiving unit is further configured to:

[0241] receive a play instruction sent by a content platform of the cloud server through the gateway;

[0242] The first sending unit is further configured to send a reading instruction to a content platform of the cloud server through the gateway, wherein the reading instruction is used to acquire the real-time target audio stream from the content platform of the cloud server.

[0243] Figure 12 is a structural schematic diagram of the electronic device in the embodiment of the present application. As shown in Figure 12 the memory 1202 is adapted to store instructions or programs executable by the processor 1201. The processor 1201 can be a single microprocessor or a collection of one or more microprocessors. Thus, the processor 1201 performs the processing of data and the control of other devices by executing the instructions stored in the memory 1202. The bus 1203 connects the above-mentioned components together, and connects the above-mentioned components to the display controller 1204 and the display device, and the input / output (I / O) device 1205. The input / output (I / O) device 1205 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a body sense input device, a printer, and other devices known in the art. Typically, the input / output device 1205 is connected to the system through the input / output (I / O) controller 1206.

[0244] The instructions stored in the memory 1202 are executed by the at least one processor 1201 to achieve: acquiring user voice instruction information; converting the user voice instruction information into user text instructions, and determining a voiceprint feature of the user according to the user voice instruction information; determining a song text according to the user text instructions, and generating a song vector feature according to the song text; determining reference song information according to the user text instructions, and generating an audio vector feature according to the reference song information; determining historical audio information of the user according to the voiceprint feature of the user, performing feature extraction on the user voice instruction information and the historical audio information, and determining a target voiceprint feature of the user; inputting the song vector feature, the audio vector feature, and the target voiceprint feature into a large language model, and generating a real-time target audio stream.

[0245] Specifically, the electronic device includes one or more processors 1201 and a memory 1202, Figure 12 Taking one processor 1201 as an example. The processor 1201 and the memory 1202 can be connected through a bus or other means, Figure 12The bus connection is taken as an example. The memory 1202 is a non-volatile computer readable storage medium, which can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The processor 1201 performs various function applications and data processing of the device by running the non-volatile software programs, instructions and modules stored in the memory 1202, that is, implements the method for determining music generation based on a smart speaker.

[0246] The memory 1202 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; and the data storage area can store an option list and the like. In addition, the memory 1202 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 1202 can optionally include a memory disposed remotely with respect to the processor 1201, and these remote memories can be connected to the external device through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0247] One or more modules are stored in the memory 1202, and when executed by the one or more processors 1201, perform the method for determining music generation based on a smart speaker in any method embodiment described above.

[0248] As those skilled in the art will appreciate, various aspects of the present embodiments can be implemented as a system, method or computer program product. Accordingly, various aspects of the present embodiments can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining software and hardware aspects that can all generally be referred to herein as a "circuit", "module" or "system". Furthermore, various aspects of the present embodiments can take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

[0249] Any combination of one or more computer readable medium can be utilized. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium can be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0250] A computer readable signal medium can include a propagated digital signal, with computer readable program code embodied in the propagated digital signal. Such a propagated signal can take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium can be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0251] Program code embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0252] Computer program code for carrying out operations for aspects of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). These network connections are

[0253] The flowchart illustrations and / or block diagrams in the above described methods, devices (systems) and computer program products according to embodiments of the present application illustrate the various aspects of the embodiments of the present application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0254] These computer program instructions can also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0255] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions executed on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0256] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0257] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal. The user refuses to process personal information other than the necessary information required for basic functions, which does not affect the user to use the basic functions.

Claims

1. A method for music generation based on a smart speaker, characterized in that, The method comprises: Obtaining user voice instruction information; Convert the user voice instruction information into user text instructions, and determine the user's voiceprint features according to the user voice instruction information; Determine the lyrics text according to the user text instructions, and generate the lyrics vector features according to the lyrics text; Determine the reference track information according to the user text instructions, and generate the audio vector features according to the reference track information; Determine the user's historical audio information according to the user's voiceprint features, extract features from the user voice instruction information and the historical audio information, and determine the user's target voiceprint features; Input the lyrics vector features, audio vector features and target voiceprint features into a large language model to generate a real-time target audio stream.

2. The method of claim 1, wherein, The method further comprises: Output the real-time target audio stream to a smart speaker through a gateway.

3. The method of claim 1, wherein, The method further comprises: Perform intent recognition on the user text instructions, and determine at least one parameter of the text instructions in response to the result of the intent recognition being music generation; In response to the at least one parameter being all necessary parameters for the music generation, call a music generation service, wherein the music generation service represents generating a real-time target audio stream through a large language model.

4. The method of claim 3, wherein, The method further comprises: In response to the at least one parameter being non-all necessary parameters for the music generation, send a reacquisition instruction to a smart speaker, wherein the reacquisition instruction is used to control the smart speaker to reacquire user voice instruction information from the user.

5. The method of claim 1, wherein, The method further comprises: Output the user text instructions to a basic large model to generate the lyrics text; Input the lyrics text into a text encoder to generate lyrics vector features.

6. The method of claim 1, wherein, The method further comprises: Determine emotional words in the user text instructions; Query a reference audio source library according to the emotional words to determine reference track information; Input the reference track information into an audio encoder to generate audio vector features.

7. The method of claim 1, wherein, The method further comprises: Receive an instruction audio stream sent by a smart speaker through a gateway; Determine that the user has completed instruction issuance according to voice activity detection, and determine the user voice instruction information issued by the user.

8. The method of claim 2, wherein, The method further comprises: Send the real-time target audio stream to a content platform; The content platform sends a play instruction to the smart speaker through a gateway; Receive a reading instruction sent by the smart speaker, and the content platform sends the real-time target audio stream to the smart speaker. 9.A method for music generation based on a smart speaker, characterized in that, The method comprises: Obtain an audio instruction sent by a user to generate an instruction audio stream; Send the instruction audio stream to a cloud server through a gateway, and determine that the user has completed instruction issuance according to mute detection to determine the user voice instruction information issued by the user; Convert the user voice instruction information into user text instructions; The user text instruction is sent to the cloud server through a gateway, so that the cloud server determines a voiceprint feature of the user according to the user voice instruction information, determines a lyric text and reference track information according to the user text instruction, generates a lyric vector feature according to the lyric text, generates an audio vector feature according to the reference track information, determines historical audio information of the user according to the voiceprint feature of the user, extracts features of the user voice instruction information and the historical audio information, and determines a target voiceprint feature of the user; The real-time target audio stream sent by the cloud server through the gateway is received, wherein the real-time target audio stream is generated by the cloud server by inputting the lyric vector feature, the audio vector feature and the target voiceprint feature into a large language model.

10. The method of claim 9, wherein, The method further comprises: The reacquisition instruction sent by the cloud server through the gateway is received; The user is sent a follow-up question information, and the user voice instruction information is reacquired.

11. The method of claim 9, wherein, The method further comprises: The wake-up word sent by the user is received; The idle state is switched to the working state, wherein the idle state indicates that the smart speaker cannot receive the instruction audio stream, and the working state indicates that the smart speaker can receive the instruction audio stream.

12. The method of claim 9, wherein, The method further comprises: The playing instruction sent by the content platform of the cloud server through the gateway is received; The reading instruction is sent to the content platform of the cloud server through the gateway, wherein the reading instruction is used to acquire the real-time target audio stream from the content platform of the cloud server.

13. An apparatus for smart speaker based music generation, the apparatus comprising: The device comprises: A first acquisition unit configured to acquire user voice instruction information; A first processing unit configured to convert the user voice instruction information into user text instruction, and determine a voiceprint feature of the user according to the user voice instruction information; A first generation unit configured to determine a lyric text according to the user text instruction, and generate a lyric vector feature according to the lyric text; A second generation unit configured to determine reference track information according to the user text instruction, and generate an audio vector feature according to the reference track information; A first determination unit configured to determine historical audio information of the user according to the voiceprint feature of the user, extract features of the user voice instruction information and the historical audio information, and determine a target voiceprint feature of the user; A third generation unit configured to input the lyric vector feature, the audio vector feature and the target voiceprint feature into a large language model, and generate a real-time target audio stream.

14. An apparatus for smart speaker based music generation, the apparatus comprising: The device comprises: A second acquisition unit configured to acquire audio instruction sent by the user, and generate an instruction audio stream; A second determination unit configured to send the instruction audio stream to the cloud server through a gateway, and determine that the user completes instruction issuing according to a mute detection, and determine user voice instruction information issued by the user; A second processing unit configured to convert the user voice instruction information into user text instruction; The first sending unit is configured to send the user text instruction to the cloud server through a gateway, so that the cloud server determines a voiceprint feature of the user according to the user voice instruction information, determines a lyric text and reference track information according to the user text instruction, generates a lyric vector feature according to the lyric text, generates an audio vector feature according to the reference track information, determines historical audio information of the user according to the voiceprint feature of the user, extracts features from the user voice instruction information and the historical audio information, and determines a target voiceprint feature of the user. The first receiving unit is configured to receive a real-time target audio stream sent by the cloud server through a gateway, wherein the real-time target audio stream is generated by the cloud server by inputting the lyric vector feature, the audio vector feature and the target voiceprint feature into a large language model.

15. A smart speaker based music generation system, characterized by, The system comprises: a cloud server and a smart speaker; wherein the cloud server is configured to execute any one of claims 1-8, and the smart speaker is configured to execute any one of claims 9-12.

16. An electronic device comprising a memory and a processor, characterized in that The memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method of any one of claims 1-12.

17. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-12.

Images