Music generation method, device and system based on intelligent loudspeaker box

Abstract

The embodiment of the invention discloses a music generation method, device and system based on an intelligent loudspeaker box. In the embodiment of the invention, user voice instruction information is acquired; converting the user voice instruction information into a user text instruction, and determining voiceprint features of the user according to the user voice instruction information; determining a lyric text according to the user text instruction, and generating lyric vector features according to the lyric text; determining reference track information according to the user text instruction, and generating audio vector features according to the reference track information; determining historical audio information of the user according to the voiceprint features of the user, performing feature extraction on the user voice instruction information and the historical audio information, and determining target voiceprint features of the user; and inputting the lyric vector feature, the audio vector feature and the target voiceprint feature into a large language model to generate a real-time target audio stream. Through the method, personalized music can be simply and quickly generated for the user, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more specifically, to a method, device and system for generating music based on a smart speaker. Background Art

[0002] In recent years, with the development of artificial intelligence technology, various smart devices have entered people's daily lives, providing a lot of convenience for people's lives. Taking smart speakers as an example, as a core component of smart homes, smart speakers have evolved from simple voice assistants to a multi-functional platform integrating entertainment, education, and life services; users' expectations for smart speakers are no longer limited to basic voice interaction and playback control, but hope to obtain a more personalized, immersive and multi-faceted experience; taking smart speakers to generate music as an example, traditional smart speakers rely on preset music libraries and streaming services, and lack personalized music creation capabilities.

[0003] In the prior art, in order to generate personalized music, sound-based music generation schemes and text-based music generation schemes are adopted. Among them, the above-mentioned sound-based music generation scheme requires users to record sounds for a long period of time. Although personalized creation can be performed based on this, the long sound recording process is a burden for users, which can easily cause users to feel bored or lose patience during the experience, thereby causing user loss; in the above-mentioned text-based music generation scheme, text is input into an intelligent agent or plug-in to generate music, but the text-based music generation performed by the intelligent agent or plug-in is limited by the system architecture. After the music generation task is triggered, the user cannot interact with the user for at least 2 to 3 minutes, and can only wait for the generation to end before playing the music, which seriously affects the user experience. In addition, the scheme cannot provide personalized timbre matching, which makes the user's sense of participation weak.

[0004] In summary, how to simply and quickly generate personalized music for users and improve user experience is a problem that needs to be solved at present. Summary of the invention

[0005] In view of this, the embodiments of the present invention provide a method, device and system for music generation based on a smart speaker, which can simply and quickly generate personalized music for users and improve the user experience.

[0006] In a first aspect, an embodiment of the present invention provides a method for generating music based on a smart speaker, the method comprising:

[0007] Get user voice command information;

[0008] Converting the user voice command information into a user text command, and determining the user's voiceprint feature according to the user voice command information;

[0009] Determining lyrics text according to the user text instruction, and generating lyrics vector features according to the lyrics text;

[0010] Determine reference track information according to the user text instruction, and generate audio vector features according to the reference track information;

[0011] Determine the historical audio information of the user according to the voiceprint feature of the user, extract features of the user's voice command information and the historical audio information, and determine the target voiceprint feature of the user;

[0012] The lyrics vector features, audio vector features and the target voiceprint features are input into a large language model to generate a real-time target audio stream.

[0013] Optionally, the method further includes:

[0014] The real-time target audio stream is output to the smart speaker through the gateway.

[0015] Optionally, the method further includes:

[0016] Performing intent recognition on the user text instruction, and in response to a result of the intent recognition being music generation, determining at least one parameter of the text instruction;

[0017] In response to the at least one parameter being all necessary parameters for the music generation, a music generation service is called, wherein the music generation service represents generating a real-time target audio stream through a large language model.

[0018] Optionally, the method further includes:

[0019] In response to the at least one parameter being non-all necessary parameters for generating the music, a re-acquisition instruction is sent to the smart speaker, wherein the re-acquisition instruction is used to control the smart speaker to re-acquire user voice command information from the user.

[0020] Optionally, determining the lyrics text according to the user text instruction, and generating lyrics vector features according to the lyrics text specifically includes:

[0021] Outputting the user text instruction to the basic large model to generate the lyrics text;

[0022] The lyrics text is input into a text encoder to generate lyrics vector features.

[0023] Optionally, determining reference track information according to the user text instruction, and generating an audio vector feature according to the reference track information specifically includes:

[0024] Determining sentiment words in the user text instruction;

[0025] According to the sentiment words, a reference sound source library is searched to determine reference music information;

[0026] The reference track information is input into an audio encoder to generate audio vector features.

[0027] Optionally, the obtaining of user voice command information specifically includes:

[0028] Receive the command audio stream sent by the smart speaker through the gateway;

[0029] The user completes issuing the command according to the voice activity detection, and determines the user voice command information issued by the user.

[0030] Optionally, outputting the real-time target audio stream to a smart speaker through a gateway specifically includes:

[0031] Sending the real-time target audio stream to a content platform;

[0032] The content platform sends a play instruction to the smart speaker through the gateway;

[0033] After receiving the read instruction sent by the smart speaker, the content platform sends the real-time target audio stream to the smart speaker.

[0034] In a second aspect, an embodiment of the present invention provides a method for generating music based on a smart speaker, the method comprising:

[0035] Obtain the audio command sent by the user and generate a command audio stream;

[0036] The command audio stream is sent to the cloud server through the gateway, and the user completes the command issuance according to the closed microphone detection, and the user voice command information issued by the user is determined;

[0037] Converting the user voice command information into a user text command;

[0038] Sending the user text instruction to the cloud server through the gateway;

[0039] A real-time target audio stream is received which is sent by the cloud server through a gateway, wherein the real-time target audio stream is generated by the cloud server by inputting lyrics vector features, audio vector features and target voiceprint features into a large language model.

[0040] Optionally, the method further includes:

[0041] Receiving a re-acquisition instruction sent by the cloud server through a gateway;

[0042] Send a follow-up message to the user to reacquire the user's voice command information.

[0043] Optionally, the method further includes:

[0044] Receive the wake-up word sent by the user;

[0045] Switch the idle state to the working state, wherein the idle state indicates that the smart speaker cannot receive the command audio stream, and the working state indicates that the smart speaker can receive the command audio stream.

[0046] Optionally, the method further includes:

[0047] Receiving a play instruction sent by the content platform of the cloud server through a gateway;

[0048] A read instruction is sent to the content platform of the cloud server through the gateway, wherein the read instruction is used to obtain the real-time target audio stream from the content platform of the cloud server.

[0049] In a third aspect, an embodiment of the present invention provides a device for music generation based on a smart speaker, the device comprising:

[0050] A first acquisition unit, used to acquire user voice command information;

[0051] A first processing unit, configured to convert the user voice command information into a user text command, and determine the user's voiceprint feature according to the user voice command information;

[0052] A first generating unit, configured to determine a lyric text according to the user text instruction, and generate a lyric vector feature according to the lyric text;

[0053] A second generating unit, configured to determine reference track information according to the user text instruction, and generate an audio vector feature according to the reference track information;

[0054] A first determining unit is used to determine the historical audio information of the user according to the voiceprint feature of the user, perform feature extraction on the user voice command information and the historical audio information, and determine the target voiceprint feature of the user;

[0055] The third generating unit is used to input the lyrics vector features, the audio vector features and the target voiceprint features into a large language model to generate a real-time target audio stream.

[0056] Optionally, the device further comprises:

[0057] The output unit is used to output the real-time target audio stream to the smart speaker through the gateway.

[0058] Optionally, the device further comprises:

[0059] An intention recognition unit is used to perform intention recognition on the user text instruction, and in response to the result of the intention recognition being music generation, determine at least one parameter of the text instruction; in response to the at least one parameter being all necessary parameters for the music generation, call a music generation service, wherein the music generation service represents generating a real-time target audio stream through a large language model.

[0060] Optionally, the first acquiring unit is further used for:

[0061] In response to the at least one parameter being non-all necessary parameters for generating the music, a re-acquisition instruction is sent to the smart speaker, wherein the re-acquisition instruction is used to control the smart speaker to re-acquire user voice command information from the user.

[0062] Optionally, the first generating unit is specifically configured to:

[0063] Outputting the user text instruction to the basic large model to generate the lyrics text;

[0064] The lyrics text is input into a text encoder to generate lyrics vector features.

[0065] Optionally, the second generating unit is specifically configured to:

[0066] Determining sentiment words in the user text instruction;

[0067] According to the sentiment words, a reference sound source library is searched to determine reference music information;

[0068] The reference track information is input into an audio encoder to generate audio vector features.

[0069] Optionally, the first acquiring unit is specifically configured to:

[0070] Receive the command audio stream sent by the smart speaker through the gateway;

[0071] The user completes issuing the command according to the voice activity detection, and determines the user voice command information issued by the user.

[0072] Optionally, the output unit is specifically used for:

[0073] Sending the real-time target audio stream to a content platform;

[0074] The content platform sends a play instruction to the smart speaker through the gateway;

[0075] After receiving the read instruction sent by the smart speaker, the content platform sends the real-time target audio stream to the smart speaker.

[0076] In a fourth aspect, an embodiment of the present invention provides a device for generating music based on a smart speaker, the device comprising:

[0077] A second acquisition unit is used to acquire an audio instruction sent by a user and generate an instruction audio stream;

[0078] A second determining unit is used to send the command audio stream to the cloud server through the gateway, and determine that the user has completed the command issuance according to the closed microphone detection, and determine the user voice command information issued by the user;

[0079] A second processing unit, used for converting the user voice instruction information into a user text instruction;

[0080] A first sending unit, used to send the user text instruction to the cloud server through a gateway;

[0081] The first receiving unit is used to receive the real-time target audio stream sent by the cloud server through the gateway, wherein the real-time target audio stream is generated by the cloud server by inputting lyrics vector features, audio vector features and target voiceprint features into a large language model.

[0082] Optionally, the first sending unit is further used for:

[0083] Receiving a re-acquisition instruction sent by the cloud server through a gateway;

[0084] The second acquisition unit is further used to send a follow-up message to the user to re-acquire the user voice command information.

[0085] Optionally, the first receiving unit is further used for:

[0086] Receive the wake-up word sent by the user;

[0087] The second processing unit is also used to: switch the idle state to the working state, wherein the idle state indicates that the smart speaker cannot receive the command audio stream, and the working state indicates that the smart speaker can receive the command audio stream.

[0088] Optionally, the first receiving unit is further used for:

[0089] Receiving a play instruction sent by the content platform of the cloud server through a gateway;

[0090] The first sending unit is further used to send a read instruction to the content platform of the cloud server through the gateway, wherein the read instruction is used to obtain the real-time target audio stream from the content platform of the cloud server.

[0091] In a fifth aspect, an embodiment of the present invention provides a music generation system based on a smart speaker, comprising: a cloud server and a smart speaker; wherein the cloud server is used to execute a method as described in the first aspect or any possible embodiment of the first aspect, and the smart speaker is used to execute a method as described in the second aspect or any possible embodiment of the second aspect.

[0092] In the sixth aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement a method as described in the first aspect, any possible method of the first aspect, the second aspect, or any possible method of the second aspect.

[0093] In the seventh aspect, an embodiment of the present invention provides a computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described in the first aspect, any possible method of the first aspect, the second aspect, or any possible method of the second aspect is implemented. In the embodiment of the present invention, by obtaining user voice instruction information; converting the user voice instruction information into a user text instruction, and determining the user's voiceprint features according to the user voice instruction information; determining the lyrics text according to the user text instruction, and generating lyrics vector features according to the lyrics text; determining reference track information according to the user text instruction, and generating audio vector features according to the reference track information; determining the user's historical audio information according to the user's voiceprint features, extracting features from the user voice instruction information and the historical audio information, and determining the user's target voiceprint features; inputting the lyrics vector features, the audio vector features, and the target voiceprint features into a large language model to generate a real-time target audio stream. Through the above method, personalized music that meets user needs can be generated for users simply and quickly, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0095] Figure 1 is a schematic diagram of a system for music generation based on a smart speaker in an embodiment of the present invention;

[0096] Figure 2is a flow chart of a method for generating music based on a smart speaker in an embodiment of the present invention;

[0097] Figure 3 is a flow chart of another method for generating music based on a smart speaker in an embodiment of the present invention;

[0098] Figure 4 is a flow chart of another method for generating music based on a smart speaker in an embodiment of the present invention;

[0099] Figure 5 is a flow chart of another method for generating music based on a smart speaker in an embodiment of the present invention;

[0100] Figure 6 is a flow chart of another method for generating music based on a smart speaker in an embodiment of the present invention;

[0101] Figure 7 is a flow chart of another method for generating music based on a smart speaker in an embodiment of the present invention;

[0102] Figure 8 is a flow chart of another method for generating music based on a smart speaker in an embodiment of the present invention;

[0103] Fig. 9 is a flow chart of interaction between a cloud server and a smart speaker in an embodiment of the present invention;

[0104] Fig.10 is a schematic diagram of a device for music generation based on a smart speaker in an embodiment of the present invention;

[0105] Fig.11 is a schematic diagram of another device for music generation based on a smart speaker in an embodiment of the present invention;

[0106] Fig.12 is a schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0107] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for those skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, flows, components and circuits are not described in detail.

[0108] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0109] Unless the context clearly requires otherwise, the words "include", "comprising" and similar words throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, the meaning is "including but not limited to".

[0110] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0111] In the prior art, in order to generate personalized music and provide users with an immersive experience, the following two methods are used:

[0112] Method 1: Sound-based music generation solution.

[0113] Specifically, some common music applications require users to record sound for a long period of time. The above-mentioned music applications generate music using the voiceprint features of the recorded sound. Although personalized creation can be carried out based on this, music that belongs to one's own voiceprint can be customized to enhance the interactive experience, the above-mentioned music applications generally require recording of audio samples of more than 60 seconds to ensure that the timbre of the generated music is similar to the user's timbre. The timbre is determined by the voiceprint features, and different voiceprint features will generate different timbres. However, the long sound recording process is a burden for users, which can easily cause users to feel bored or lose patience during the experience, thereby leading to user loss.

[0114] Method 2: Text-based music generation solution.

[0115] Specifically, some large language model providers have opened up agents or plug-ins, and text is input into the agents or plug-ins to generate music. However, the text-to-music generation through the agent or plug-in is limited by the system architecture. Users cannot interact with the agent for at least 2 to 3 minutes after triggering the music generation task. They can only wait for the generation to be completed before playing the music, which seriously affects the user experience. In addition, the solution cannot provide personalized tone matching, which makes the user's sense of participation and experience weak, and it is difficult to meet the user's needs for immediate rewards and interaction. The large-scale speech model represents the frontier of deep integration of artificial intelligence with various fields, and has demonstrated amazing results in different applications. In addition to the field of music, it has demonstrated better capabilities in language understanding, knowledge memory, text understanding, text generation, image recognition and even complex decision-making. The above-mentioned large-scale language model is a language model constructed by a deep neural network containing more than tens of billions of parameters. It is trained through a large amount of unlabeled text using a self-supervised learning method, and can handle a variety of natural language tasks. It is an important path to artificial intelligence. For example, the large-scale language model is applied to ChatGPT, which can have natural and fluent conversations with users, answer various questions, etc. In addition, the large-scale language model can also accurately capture and simulate a variety of human expressions, from text to sound, and even subtle emotional fluctuations. In the field of music, with the application of large-scale language models, smart speakers are moving towards a higher level of intelligence, opening up new possibilities for future human-computer interaction.

[0116] Therefore, how to simply and quickly generate personalized music for users and improve user experience is a problem that needs to be solved at present.

[0117] In order to solve the above problems, an embodiment of the present invention proposes a music generation system based on an intelligent speaker, specifically as follows: Figure 1 As shown, the system includes a cloud server 101 and a smart speaker 102. The execution steps of the cloud server 101 and the smart speaker 102 are described in detail below.

[0118] In a possible implementation, the smart speaker may also be other devices integrated with voice interaction functions, such as smart TVs, smart tablets, smart wearable devices, etc.

[0119] In the embodiment of the present invention, the processing flow of the cloud server is as follows: Figure 2 As shown, Figure 2 The flowchart is a method for generating music based on a smart speaker, which specifically includes the following steps:

[0120] Step S201: Obtain user voice command information.

[0121] Specifically, the command audio stream sent by the smart speaker through the gateway is received, and the user is determined to have completed the command issuance based on Voice Activity Detection (VAD), and the user voice command information issued by the user is determined.

[0122] For example, the voice command information issued by the user is "play a lyrical music". The voice command information is conveyed by the smart speaker in real time through the gateway. Every time the smart speaker receives a word of voice, it sends the word of voice to the cloud server through the gateway. That is, the command audio stream received by the cloud server is received in real time. According to the voice activity detection VAD, it is determined that the user's command is completely issued, and it is determined that the user voice command information issued by the user is "play a lyrical music", wherein the real-time voice command information is audio information.

[0123] In a possible implementation, the voice command information issued by the user may also generate instructions for other music, for example, "generate a relaxing background music", etc., which is determined according to actual conditions.

[0124] In a possible implementation, the language instruction information may, in addition to generating music, also perform functions such as "reciting a Tang poem", "telling a fairy tale suitable for a 3-year-old child", etc.

[0125] Step S202: convert the user voice command information into a user text command, and determine the user's voiceprint features according to the user voice command information.

[0126] Specifically, the user voice command information is converted into a user text command through Automatic Speech Recognition (ASR), and the user's voiceprint features can be extracted from the user voice command through voiceprint recognition (VPR), wherein the voiceprint features are used in the subsequent step S205 to obtain the user's historical audio information in a user-imperceptible manner under the premise of user authorization, and the historical audio information is the historical conversation record between the user of the voiceprint feature and the smart speaker, and the historical conversation record is stored in the cloud server.

[0127] Step S203: determine the lyrics text according to the user text instruction, and generate lyrics vector features according to the lyrics text.

[0128] Specifically, the user text instruction is output to the basic large model to generate the lyrics text; the lyrics text is input to the text encoder to generate lyrics vector features.

[0129] In a possible implementation, when the user does not specify lyrics, the user text instruction is input as a prompt into the basic large model, and the basic large model expands the user text instruction to generate lyrics text; wherein the basic large model is also a large language model.

[0130] In the embodiment of the present invention, the lyrics text is encoded according to the text editor to generate lyrics vector features.

[0131] In a possible implementation, when the user specifies lyrics, the lyrics text is encoded according to the text editor to generate lyrics vector features.

[0132] Step S204: determine reference track information according to the user text instruction, and generate audio vector features according to the reference track information.

[0133] Specifically, the sentiment words in the user text instruction are determined; a reference sound source library is queried according to the sentiment words to determine reference track information; and the reference track information is input into an audio encoder to generate audio vector features.

[0134] In an embodiment of the present invention, the reference sound source library includes a plurality of reference tracks of different styles, and different emotional words correspond to reference tracks of different styles. The query reference sound source library is searched according to the emotional words in the user text instructions to determine the reference tracks that match the style of the emotional words. For example, if the emotional word is "happy", the reference sound source library is searched for reference tracks with a happy or cheerful style; if the emotional word is "sad", the reference sound source library is searched for reference tracks with a sad style; and then the determined reference track information is input into the audio encoder for encoding to generate audio vector features.

[0135] In a possible implementation, the style and sentiment words may be modeled in advance, and after the sentiment words are input into the established model, the style corresponding to the sentiment words is directly generated to avoid the situation where the sentiment words and the style are not completely corresponding and the style cannot be determined by the sentiment words.

[0136] Step S205: determining the historical audio information of the user according to the voiceprint feature of the user, performing feature extraction on the user voice command information and the historical audio information, and determining the target voiceprint feature of the user.

[0137] Specifically, the voiceprint features of different users are completely different, which is the user's unique identification. The user can match his or her own historical audio information in the historical audio database through the unique identification. The cloud server can directly extract the voiceprint features through the user's voice command information and the historical audio information. The user does not need to deliberately record a long audio to extract the voiceprint features. The voice information saved by the user and the smart speaker during the historical interaction process is directly used to extract the user's target voiceprint features conveniently and quickly.

[0138] Step S206: input the lyrics vector features, audio vector features and the target voiceprint features into a large language model to generate a real-time target audio stream.

[0139] In an embodiment of the present invention, the lyrics vector features, audio vector features and the target voiceprint features are input into a large language model (LLM) to generate a real-time target audio stream that meets user needs, and the real-time target audio stream is generated by the user's voiceprint features, thereby realizing the user's personalized and diversified needs.

[0140] In a possible implementation, after determining that a user text instruction is given, determining the user's voiceprint features based on the user voice instruction information, generating lyrics vector features based on the lyrics text, and generating audio vector features based on the reference track information can be performed simultaneously.

[0141] In a possible implementation, after step S206, other steps are also included, specifically: Figure 3 As shown, Figure 3 This is a flow chart of another method for generating music based on a smart speaker, which specifically includes the following steps:

[0142] Step S207: output the real-time target audio stream to the smart speaker through the gateway.

[0143] Specifically, the real-time target audio stream is sent to the content platform; the content platform sends a play instruction to the smart speaker through a gateway; upon receiving a read instruction sent by the smart speaker, the content platform sends the real-time target audio stream to the smart speaker.

[0144] In an embodiment of the present invention, the content platform is a part of the cloud server. The cloud server generates a target audio stream through the large language model and then delivers it to the content platform. After receiving the target audio stream, the content platform sends a play instruction to the smart speaker through a gateway. The content platform sends the real-time target audio stream to the smart speaker. After the target audio stream is fully output, it is the target audio generated by the user's voiceprint features to meet the user's needs. The target audio is pushed in a full-link streaming manner, which realizes real-time transmission of the target audio, ensures low latency and high efficiency during target audio transmission, reduces the target audio generation speed after the user initiates the user voice command information, and improves the user experience.

[0145] In a possible implementation, after step S202, other steps are also included, specifically: Figure 4 As shown, Figure 4 This is a flow chart of another method for generating music based on a smart speaker, which specifically includes the following steps:

[0146] Step S208: perform intent recognition on the user text instruction, and in response to a result of the intent recognition being music generation, determine at least one parameter of the text instruction.

[0147] Specifically, the user text instruction is subjected to intent recognition through natural language understanding (NLU), and a result of the intent recognition is determined. In response to the result being music generation, it is necessary to extract parameters from the text instruction, wherein the parameters are prompt words and / or styles.

[0148] In a possible implementation, it is determined whether the at least one parameter is all necessary parameters, and if so, step S209 is executed, and if not, step S210 is executed.

[0149] Step S209: In response to the at least one parameter being all necessary parameters for generating the music, calling a music generation service, wherein the music generation service represents generating a real-time target audio stream through a large language model.

[0150] Step S210: In response to the at least one parameter being non-all necessary parameters for generating the music, a re-acquisition instruction is sent to the smart speaker, wherein the re-acquisition instruction is used to control the smart speaker to re-acquire user voice command information from the user.

[0151] In the embodiment of the present invention, the processing flow of the smart speaker is as follows: Figure 5 As shown, Figure 5 The flowchart is a method for generating music based on a smart speaker, which specifically includes the following steps:

[0152] Step S501: Acquire the audio instruction sent by the user and generate an instruction audio stream.

[0153] Step S502: Send the command audio stream to the cloud server through the gateway, determine whether the user has completed the command issuance based on the closed microphone detection, and determine the user voice command information issued by the user.

[0154] Step S503: Convert the user voice command information into a user text command.

[0155] In a possible implementation, the user voice instruction information is converted into a user text instruction through automatic speech recognition ASR.

[0156] Step S504: Send the user text instruction to the cloud server through the gateway.

[0157] Specifically, the smart speaker sends the user text instruction to the cloud server through the gateway, and the cloud server can jointly perform intent recognition based on the user text instruction sent by the smart speaker and the user text instruction generated by the cloud server.

[0158] Step S505: receiving a real-time target audio stream sent by the cloud server through the gateway, wherein the real-time target audio stream is generated by the cloud server by inputting lyrics vector features, audio vector features and target voiceprint features into a large language model.

[0159] Specifically, the smart speaker receives the real-time target audio stream sent by the cloud server through the gateway, and plays the real-time target audio stream to the user.

[0160] In a possible implementation, before step S505, other steps are also included, specifically: Figure 6 As shown, Figure 6 The flowchart is a method for generating music based on a smart speaker, which specifically includes the following steps:

[0161] Step S506: receiving a play instruction sent by the content platform of the cloud server through the gateway.

[0162] Specifically, the smart speaker receives a play instruction sent by the content platform of the cloud server through a gateway.

[0163] Step S507: sending a read instruction to the content platform of the cloud server through the gateway, wherein the read instruction is used to obtain the real-time target audio stream from the content platform of the cloud server.

[0164] In a possible implementation, after step S504, other steps are also included, specifically: Figure 7 As shown, Figure 7 The flowchart is a method for generating music based on a smart speaker, which specifically includes the following steps:

[0165] Step S508: receiving a re-acquisition instruction sent by the cloud server through the gateway.

[0166] Specifically, when the parameters recognized by the cloud server are not all the necessary parameters for music generation, it is necessary to send a re-acquisition instruction to the smart speaker.

[0167] Step S509: Send a follow-up message to the user to reacquire the user's voice command information.

[0168] Specifically, the smart speaker sends a follow-up question to the user. For example, if there is no emotional word in the voice command information sent by the user for the first time, the follow-up question is "What style of music do you need to generate?" This is only an exemplary explanation.

[0169] In a possible implementation, before step S501, other steps are also included, specifically: Figure 8 As shown, Figure 8 The flowchart is a method for generating music based on a smart speaker, which specifically includes the following steps:

[0170] Step S510: receiving a wake-up word sent by a user.

[0171] Specifically, the wake-up word is a specific word set for the smart speaker, the smart speaker establishes a long link with the gateway, and the smart device is in an idle state when not in use in order to save energy, and the smart speaker continuously monitors the wake-up word with low power consumption.

[0172] Step S511: switch the idle state to the working state, wherein the idle state indicates that the smart speaker cannot receive the command audio stream, and the working state indicates that the smart speaker can receive the command audio stream.

[0173] Specifically, the smart speaker switches from an idle state to a working state and starts receiving audio commands issued by the user.

[0174] The following is a complete example of an interaction process between the cloud server and the smart speaker. Fig. 9 As shown, the specific process is as follows:

[0175] Step S901: The smart speaker obtains a wake-up word sent by the user, and the smart speaker switches from an idle state to a working state.

[0176] Step S902: The smart speaker obtains the audio command sent by the user and generates a command audio stream.

[0177] Step S903: The smart speaker sends the command audio stream to the cloud server through the gateway.

[0178] Specifically, after step S903 is completed, step S906 is executed.

[0179] Step S904: The smart speaker performs closed microphone detection to determine whether the user has completed the instruction issuance and to determine the user voice instruction information issued by the user.

[0180] Specifically, step S903 and step S904 are executed synchronously.

[0181] Step S905: The smart speaker converts the user voice command information into a user text command, and sends the user text command to the cloud server.

[0182] Step S906: The cloud server converts the user voice command information into user text commands through automatic voice recognition.

[0183] Step S907: The cloud server performs intent recognition on the user text instruction.

[0184] Step S908: The cloud server determines whether the result of the intent recognition is music generation. If so, execute step S909; if not, end or re-acquire the user voice command information.

[0185] Step S909, the cloud server performs intent recognition on the user text instruction and determines the at least one parameter. If the at least one parameter is all the necessary parameters for generating the music, execute step S910; if not, execute step S911.

[0186] Step S910: The cloud server calls the music generation service.

[0187] Step S911: The cloud server sends a follow-up message to the user to reacquire the user's voice command information.

[0188] Step S912, the cloud server determines whether lyrics are specified in the user text instruction, if so, executes step S913, if not, executes step S914.

[0189] Step S913: The cloud server directly determines the lyrics text.

[0190] Step S914: The cloud server inputs the user text instruction into the basic large model to generate lyrics text.

[0191] Specifically, after step S913 and step S914, step S915 is performed together.

[0192] Step S915: The cloud server inputs the lyrics text into a text editor for encoding to generate lyrics vector features.

[0193] In a possible implementation, after executing step S906, step S916 is executed simultaneously.

[0194] Step S916: The cloud server extracts features from the user's voice command information to determine the user's voiceprint features.

[0195] Step S917: The cloud server determines the historical audio information of the user according to the voiceprint feature of the user, and performs feature extraction on the user voice command information and the historical audio information to determine the target voiceprint feature of the user.

[0196] Specifically, the user instruction information is also saved as the historical audio information.

[0197] In a possible implementation, after executing step S906, step S918 is executed simultaneously.

[0198] Step S918: The cloud server determines the sentiment words in the user text instruction.

[0199] Step S919: The cloud server queries a reference sound source library according to the sentiment word to determine reference track information.

[0200] Step S920: The cloud server inputs the reference track information into the audio encoder to generate audio vector features.

[0201] Specifically, the lyrics vector features generated in step S915, the target voiceprint features determined in step S917, and the audio vector features generated in step S920 are input into the large language model together, and step S921 is executed.

[0202] Step S921: The large language model of the cloud server generates a real-time target audio stream.

[0203] Specifically, the real-time audio stream is sent to the content platform through the music generation service.

[0204] Step S922: The smart speaker plays the real-time target audio stream sent by the content platform through the gateway.

[0205] In an embodiment of the present invention, during the entire generation process of the real-time target audio stream, the reference track information, lyrics, etc. used need to comply with the copyright holder's requirements, and security and compliance checks are performed on user text instructions and the generated real-time target audio stream. If bad information is found, the generation is terminated.

[0206] In a possible implementation, after the real-time target audio stream is generated, the complete target audio may be saved first, the style and / or rhythm of the complete target audio may be optimized, and the optimized target audio may be output.

[0207] In an embodiment of the present invention, the music generation system based on the smart speaker can be divided into a device layer, a service layer and an algorithm layer from a structural perspective, wherein the device layer belongs to the smart speaker, and the service layer and the algorithm layer belong to the cloud server.

[0208] In one possible implementation, the device layer directly interacts with the user, and specifically includes an audio module, a voice module, a communication module and an edge computing module, wherein the audio module specifically implements the wake-up function, echo cancellation function, noise reduction function, gain and correction function; the voice module specifically implements endpoint detection, voice enhancement, voice recognition and voice activity detection functions; the communication module specifically implements long link communication, local area networking, data flow management and device discovery functions; the edge computing module specifically implements the acoustic model, language model, codec and offline recognition function.

[0209] In one possible implementation, the service layer of the cloud server includes a gateway, a content service and a skill service, wherein the gateway specifically implements connection, reason, protocol, traffic and service functions; the content service specifically implements content storage, content retrieval, content broadcast control and content distribution functions; the skill service specifically implements registration discovery, call decision, combination orchestration and response fallback functions; the algorithm layer of the cloud server includes tokenization, music model, speech algorithm and language algorithm, wherein the tokenization specifically implements embedding representation, audio segmentation, text encoding and codebook functions, and the music model includes conditional generation of a deep learning model (Vector Quantized-Variational Autoencoder, VQ-VAE) and a Transformer decoder that combines vector quantization technology and variational autoencoders; the speech algorithm specifically implements voiceprint recognition, speech recognition, speech synthesis and keyword detection functions; the language algorithm specifically implements language understanding, intent recognition and keyword extraction functions.

[0210] In one possible implementation, the smart speaker-based music generation system also includes security and compliance verification functions, specifically including risk control, privacy, security, compliance and copyright.

[0211] Through the above embodiments, a real-time target audio stream is generated by a large language model. First, the subtle features of the user's voice can be quickly captured, and the threshold for user voice recording is lowered, so that users can complete high-quality voice recording in daily conversations, reduce the time for data transmission and processing, and improve the speed and accuracy of feature extraction; secondly, a full-link streaming push architecture from a large language model to a gateway to a smart speaker is adopted to realize real-time transmission and processing of data, effectively alleviate the problem of slow generation speed of a large model, and the user can quickly obtain feedback after initiating a music generation request, which significantly improves the interactive experience. The full-link optimization ensures low latency and high efficiency of data transmission, making the process of generating music smoother and more fluent; thirdly, the voice interaction capability of the smart speaker is fully utilized, and the user can use natural language to directly describe and adjust the music generation requirements, and the user can customize personalized music through simple voice commands, such as "generate a relaxing background music" or "adjust the rhythm to be more cheerful", which greatly improves the convenience of operation; the smart speaker can understand and respond to complex commands, making the music generation process more interactive and personalized, meeting the diverse needs of users.

[0212] In an embodiment of the present invention, a device for music generation based on an intelligent speaker is provided. Fig.10 As shown, it specifically includes: a first acquisition unit 1001, a first processing unit 1002, a first generation unit 1003, a second generation unit 1004, a first determination unit 1005 and a third generation unit 1006; wherein the first acquisition unit 1001 is used to acquire user voice instruction information; the first processing unit 1002 is used to convert the user voice instruction information into a user text instruction, and determine the user's voiceprint feature according to the user voice instruction information; the first generation unit 1003 is used to determine the lyrics text according to the user text instruction, and generate lyrics vector features according to the lyrics text; the second generation unit 1004 is used to determine the reference track information according to the user text instruction, and generate an audio vector feature according to the reference track information; the first determination unit 1005 is used to determine the user's historical audio information according to the user's voiceprint feature, perform feature extraction on the user voice instruction information and the historical audio information, and determine the user's target voiceprint feature; the third generation unit 1006 is used to input the lyrics vector feature, the audio vector feature and the target voiceprint feature into a large language model to generate a real-time target audio stream.

[0213] Furthermore, the device also includes:

[0214] The output unit is used to output the real-time target audio stream to the smart speaker through the gateway.

[0215] Furthermore, the device also includes:

[0216] An intention recognition unit is used to perform intention recognition on the user text instruction, and in response to the result of the intention recognition being music generation, determine at least one parameter of the text instruction; in response to the at least one parameter being all necessary parameters for the music generation, call a music generation service, wherein the music generation service represents generating a real-time target audio stream through a large language model.

[0217] Furthermore, the first acquiring unit is further used for:

[0218] In response to the at least one parameter being non-all necessary parameters for generating the music, a re-acquisition instruction is sent to the smart speaker, wherein the re-acquisition instruction is used to control the smart speaker to re-acquire user voice command information from the user.

[0219] Furthermore, the first generating unit is specifically configured to:

[0220] Outputting the user text instruction to the basic large model to generate the lyrics text;

[0221] The lyrics text is input into a text encoder to generate lyrics vector features.

[0222] Furthermore, the second generating unit is specifically used to:

[0223] Determining sentiment words in the user text instruction;

[0224] According to the sentiment words, a reference sound source library is searched to determine reference music information;

[0225] The reference track information is input into an audio encoder to generate audio vector features.

[0226] Furthermore, the first acquisition unit is specifically configured to:

[0227] Receive the command audio stream sent by the smart speaker through the gateway;

[0228] The user completes issuing the command according to the voice activity detection, and determines the user voice command information issued by the user.

[0229] Furthermore, the output unit is specifically used for:

[0230] Sending the real-time target audio stream to a content platform;

[0231] The content platform sends a play instruction to the smart speaker through the gateway;

[0232] After receiving the read instruction sent by the smart speaker, the content platform sends the real-time target audio stream to the smart speaker.

[0233] In an embodiment of the present invention, a device for generating music based on an intelligent speaker is provided. Fig.11 As shown, it specifically includes: a second acquisition unit 1101, a second determination unit 1102, a second processing unit 1103, a first sending unit 1104 and a first receiving unit 1105; wherein, the second acquisition unit 1101 is used to acquire the audio instruction sent by the user and generate an instruction audio stream; the second determination unit 1102 is used to send the instruction audio stream to the cloud server through the gateway, and determine that the user has completed the instruction issuance according to the closed microphone detection, and determine the user voice instruction information issued by the user; the second processing unit 1103 is used to convert the user voice instruction information into a user text instruction; the first sending unit 1104 is used to send the user text instruction to the cloud server through the gateway; the first receiving unit 1105 is used to receive the real-time target audio stream sent by the cloud server through the gateway, wherein the real-time target audio stream is generated by the cloud server by inputting lyrics vector features, audio vector features and target voiceprint features into a large language model.

[0234] Furthermore, the first sending unit is further used for:

[0235] Receiving a re-acquisition instruction sent by the cloud server through a gateway;

[0236] The second acquisition unit is further used to send a follow-up message to the user to re-acquire the user voice command information.

[0237] Furthermore, the first receiving unit is further configured to:

[0238] Receive the wake-up word sent by the user;

[0239] The second processing unit is also used to: switch the idle state to the working state, wherein the idle state indicates that the smart speaker cannot receive the command audio stream, and the working state indicates that the smart speaker can receive the command audio stream.

[0240] Furthermore, the first receiving unit is further configured to:

[0241] Receiving a play instruction sent by the content platform of the cloud server through a gateway;

[0242] The first sending unit is further used to send a read instruction to the content platform of the cloud server through the gateway, wherein the read instruction is used to obtain the real-time target audio stream from the content platform of the cloud server.

[0243] Fig.12 Schematic diagram of the structure of the electronic device in the embodiment of the present invention. Fig.12 As shown, it includes a general computer hardware structure, which at least includes a processor 1201 and a memory 1202. The processor 1201 and the memory 1202 are connected via a bus 1203. The memory 1202 is suitable for storing instructions or programs executable by the processor 1201. The processor 1201 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 1201 executes the instructions stored in the memory 1202, thereby executing the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices. The bus 1203 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 1204 and the display device and the input / output (I / O) device 1205. The input / output (I / O) device 1205 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 1205 is connected to the system through an input / output (I / O) controller 1206.

[0244] Among them, the instructions stored in the memory 1202 are executed by at least one processor 1201 to achieve: obtaining user voice command information; converting the user voice command information into user text commands, and determining the user's voiceprint features based on the user voice command information; determining the lyrics text based on the user text commands, and generating lyrics vector features based on the lyrics text; determining reference track information based on the user text commands, and generating audio vector features based on the reference track information; determining the user's historical audio information based on the user's voiceprint features, performing feature extraction on the user voice command information and the historical audio information, and determining the user's target voiceprint features; inputting the lyrics vector features, audio vector features, and the target voiceprint features into a large language model to generate a real-time target audio stream.

[0245] Specifically, the electronic device includes: one or more processors 1201 and a memory 1202, Fig.12 Take a processor 1201 as an example. The processor 1201 and the memory 1202 may be connected via a bus or other means. Fig.12In the example, the bus connection is used. The memory 1202 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The processor 1201 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions and modules stored in the memory 1202, that is, the above-mentioned method for determining the generation of music based on the smart speaker is realized.

[0246] The memory 1202 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store a list of options, etc. In addition, the memory 1202 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1202 may optionally include a memory remotely arranged relative to the processor 1201, and these remote memories may be connected to an external device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0247] One or more modules are stored in the memory 1202, and when executed by one or more processors 1201, the method for generating music based on the smart speaker in any of the above method embodiments is executed.

[0248] As will be appreciated by those skilled in the art, various aspects of the embodiments of the present invention may be implemented as a system, method or computer program product. Therefore, various aspects of the embodiments of the present invention may take the form of a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.), or an implementation that combines software aspects with hardware aspects, which may generally be referred to herein as a "circuit," "module," or "system." In addition, various aspects of the embodiments of the present invention may take the form of a computer program product implemented in one or more computer-readable media having a computer-readable program code implemented thereon.

[0249] Any combination of one or more computer-readable media can be utilized. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, (but not limited to) electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or apparatuses, or any suitable combination of the foregoing. More specific examples (non-exhaustive enumeration) of computer-readable storage media will include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of an embodiment of the present invention, a computer-readable storage medium can be any tangible medium that can contain or store a program used by an instruction execution system, device or apparatus or a program used in conjunction with an instruction execution system, device or apparatus.

[0250] A computer readable signal medium may include a propagated digital signal having a computer readable program code implemented therein, such as in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any of the following computer readable media: not a computer readable storage medium, and may communicate, propagate, or transmit a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0251] Program code embodied on a computer readable medium may be transmitted using any appropriate medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0252] The computer program code for performing operations for various aspects of the embodiments of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and conventional process programming languages ​​such as "C" programming language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer as a stand-alone software package; partially on the user's computer and partially on a remote computer; or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., by using the Internet of an Internet service provider).

[0253] The flowchart legend and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present invention described above describe various aspects of the embodiment of the present invention.It will be understood that each block of the flowchart legend and / or block diagram and the combination of the blocks in the flowchart legend and / or block diagram can be implemented by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that the instructions (executed by the processor of the computer or other programmable data processing device) create a device for implementing the function / action specified in the flowchart and / or block diagram block or block.

[0254] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing device, or other apparatus to operate in a particular manner, so that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in the flowchart and / or block diagram blocks or blocks.

[0255] The computer program instructions may also be loaded onto a computer, other programmable data processing device or other apparatus so that a series of operable steps are performed on the computer, other programmable device or other apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions / actions specified in the flowchart and / or block diagram block or blocks.

[0256] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0257] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portals for users to choose to authorize or refuse. Users' refusal to process personal information other than the necessary information required for basic functions will not affect the user's use of basic functions.

[0258] The embodiment of the present invention discloses A1, a method for generating music based on a smart speaker, the method comprising:

[0259] Get user voice command information;

[0260] Converting the user voice command information into a user text command, and determining the user's voiceprint feature according to the user voice command information;

[0261] Determining lyrics text according to the user text instruction, and generating lyrics vector features according to the lyrics text;

[0262] Determine reference track information according to the user text instruction, and generate audio vector features according to the reference track information;

[0263] Determine the historical audio information of the user according to the voiceprint feature of the user, extract features of the user's voice command information and the historical audio information, and determine the target voiceprint feature of the user;

[0264] The lyrics vector features, audio vector features and the target voiceprint features are input into a large language model to generate a real-time target audio stream.

[0265] A2. The method according to A1, further comprising:

[0266] The real-time target audio stream is output to the smart speaker through the gateway.

[0267] A3. The method according to A1, further comprising:

[0268] Performing intent recognition on the user text instruction, and in response to a result of the intent recognition being music generation, determining at least one parameter of the text instruction;

[0269] In response to the at least one parameter being all necessary parameters for the music generation, a music generation service is called, wherein the music generation service represents generating a real-time target audio stream through a large language model.

[0270] A4. The method according to A3, further comprising:

[0271] In response to the at least one parameter being non-all necessary parameters for generating the music, a re-acquisition instruction is sent to the smart speaker, wherein the re-acquisition instruction is used to control the smart speaker to re-acquire user voice command information from the user.

[0272] A5. The method according to A1, wherein determining the lyrics text according to the user text instruction and generating lyrics vector features according to the lyrics text, specifically comprises:

[0273] Outputting the user text instruction to the basic large model to generate the lyrics text;

[0274] The lyrics text is input into a text encoder to generate lyrics vector features.

[0275] A6. The method according to A1, wherein determining reference track information according to the user text instruction and generating an audio vector feature according to the reference track information specifically comprises:

[0276] Determining sentiment words in the user text instruction;

[0277] According to the sentiment words, a reference sound source library is searched to determine reference music information;

[0278] The reference track information is input into an audio encoder to generate audio vector features.

[0279] A7. According to the method described in A1, the step of obtaining the user voice command information specifically includes:

[0280] Receive the command audio stream sent by the smart speaker through the gateway;

[0281] The user completes issuing the command according to the voice activity detection, and determines the user voice command information issued by the user.

[0282] A8. The method according to A2, wherein the real-time target audio stream is output to the smart speaker through a gateway, specifically comprising:

[0283] Sending the real-time target audio stream to a content platform;

[0284] The content platform sends a play instruction to the smart speaker through the gateway;

[0285] After receiving the read instruction sent by the smart speaker, the content platform sends the real-time target audio stream to the smart speaker.

[0286] The embodiment of the present invention discloses B1, a method for generating music based on a smart speaker, the method comprising:

[0287] Obtain the audio command sent by the user and generate a command audio stream;

[0288] The command audio stream is sent to the cloud server through the gateway, and the user completes the command issuance according to the closed microphone detection, and the user voice command information issued by the user is determined;

[0289] Converting the user voice command information into a user text command;

[0290] Sending the user text instruction to the cloud server through the gateway;

[0291] A real-time target audio stream is received which is sent by the cloud server through a gateway, wherein the real-time target audio stream is generated by the cloud server by inputting lyrics vector features, audio vector features and target voiceprint features into a large language model.

[0292] B2. The method according to B1, further comprising:

[0293] Receiving a re-acquisition instruction sent by the cloud server through a gateway;

[0294] Send a follow-up message to the user to reacquire the user's voice command information.

[0295] B3. The method according to B1, further comprising:

[0296] Receive the wake-up word sent by the user;

[0297] Switch the idle state to the working state, wherein the idle state indicates that the smart speaker cannot receive the command audio stream, and the working state indicates that the smart speaker can receive the command audio stream.

[0298] B4. The method according to B1, further comprising:

[0299] Receiving a play instruction sent by the content platform of the cloud server through a gateway;

[0300] A read instruction is sent to the content platform of the cloud server through the gateway, wherein the read instruction is used to obtain the real-time target audio stream from the content platform of the cloud server.

[0301] The embodiment of the present invention discloses C1, a device for music generation based on a smart speaker, the device comprising:

[0302] A first acquisition unit, used to acquire user voice command information;

[0303] A first processing unit, configured to convert the user voice command information into a user text command, and determine the user's voiceprint feature according to the user voice command information;

[0304] A first generating unit, configured to determine a lyric text according to the user text instruction, and generate a lyric vector feature according to the lyric text;

[0305] A second generating unit, configured to determine reference track information according to the user text instruction, and generate an audio vector feature according to the reference track information;

[0306] A first determining unit is used to determine the historical audio information of the user according to the voiceprint feature of the user, perform feature extraction on the user voice command information and the historical audio information, and determine the target voiceprint feature of the user;

[0307] The third generating unit is used to input the lyrics vector features, the audio vector features and the target voiceprint features into a large language model to generate a real-time target audio stream.

[0308] The embodiment of the present invention discloses D1, a device for music generation based on a smart speaker, the device comprising:

[0309] A second acquisition unit is used to acquire an audio instruction sent by a user and generate an instruction audio stream;

[0310] A second determining unit is used to send the command audio stream to the cloud server through the gateway, and determine that the user has completed the command issuance according to the closed microphone detection, and determine the user voice command information issued by the user;

[0311] A second processing unit, used for converting the user voice instruction information into a user text instruction;

[0312] A first sending unit, used to send the user text instruction to the cloud server through a gateway;

[0313] The first receiving unit is used to receive the real-time target audio stream sent by the cloud server through the gateway, wherein the real-time target audio stream is generated by the cloud server by inputting lyrics vector features, audio vector features and target voiceprint features into a large language model.

[0314] The embodiment of the present invention discloses E1, a music generation system based on an intelligent speaker, comprising:

[0315] Cloud servers and smart speakers;

[0316] Among them, the cloud server is used to execute any one of A1-A8, and the smart speaker is used to execute any one of B1-B4.

[0317] An embodiment of the present invention discloses F1, an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement a method as described in any one of A1-A8 and B1-B4.

[0318] The embodiment of the present invention discloses G1, a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of A1-A8 and B1-B4 is implemented.

Claims

1. A method for music generation based on an intelligent speaker, characterized in that: The method comprises: Get user voice command information; Converting the user voice command information into a user text command, and determining the user's voiceprint feature according to the user voice command information; Determining lyrics text according to the user text instruction, and generating lyrics vector features according to the lyrics text; Determine reference track information according to the user text instruction, and generate audio vector features according to the reference track information; Determine the historical audio information of the user according to the voiceprint feature of the user, extract features of the user's voice command information and the historical audio information, and determine the target voiceprint feature of the user; The lyrics vector features, audio vector features and the target voiceprint features are input into a large language model to generate a real-time target audio stream.

2. The method according to claim 1, characterized in that The method further comprises: The real-time target audio stream is output to the smart speaker through the gateway.

3. The method according to claim 1, characterized in that The method further comprises: Performing intent recognition on the user text instruction, and in response to a result of the intent recognition being music generation, determining at least one parameter of the text instruction; In response to the at least one parameter being all necessary parameters for the music generation, a music generation service is called, wherein the music generation service represents generating a real-time target audio stream through a large language model.

4. The method according to claim 3, characterized in that The method further comprises: In response to the at least one parameter being non-all necessary parameters for generating the music, a re-acquisition instruction is sent to the smart speaker, wherein the re-acquisition instruction is used to control the smart speaker to re-acquire user voice command information from the user.

5. A method for music generation based on an intelligent speaker, characterized in that: The method comprises: Obtain the audio command sent by the user and generate a command audio stream; The command audio stream is sent to the cloud server through the gateway, and the user completes the command issuance according to the closed microphone detection, and the user voice command information issued by the user is determined; Converting the user voice command information into a user text command; Sending the user text instruction to the cloud server through the gateway; A real-time target audio stream is received which is sent by the cloud server through a gateway, wherein the real-time target audio stream is generated by the cloud server by inputting lyrics vector features, audio vector features and target voiceprint features into a large language model.

6. A device for music generation based on a smart speaker, characterized in that: The device comprises: A first acquisition unit, used to acquire user voice command information; A first processing unit, configured to convert the user voice command information into a user text command, and determine the user's voiceprint feature according to the user voice command information; A first generating unit, configured to determine a lyric text according to the user text instruction, and generate a lyric vector feature according to the lyric text; A second generating unit, configured to determine reference track information according to the user text instruction, and generate an audio vector feature according to the reference track information; A first determining unit is used to determine the historical audio information of the user according to the voiceprint feature of the user, perform feature extraction on the user voice command information and the historical audio information, and determine the target voiceprint feature of the user; The third generating unit is used to input the lyrics vector features, the audio vector features and the target voiceprint features into a large language model to generate a real-time target audio stream.

7. A device for music generation based on an intelligent speaker, characterized in that: The device comprises: A second acquisition unit is used to acquire an audio instruction sent by a user and generate an instruction audio stream; A second determining unit is used to send the command audio stream to the cloud server through the gateway, and determine that the user has completed the command issuance according to the closed microphone detection, and determine the user voice command information issued by the user; A second processing unit, used for converting the user voice instruction information into a user text instruction; A first sending unit, used to send the user text instruction to the cloud server through a gateway; The first receiving unit is used to receive the real-time target audio stream sent by the cloud server through the gateway, wherein the real-time target audio stream is generated by the cloud server by inputting lyrics vector features, audio vector features and target voiceprint features into a large language model.

8. A music generation system based on an intelligent speaker, characterized in that: include: Cloud servers and smart speakers; Wherein, the cloud server is used to execute any one of claims 1-4, and the smart speaker is used to execute any one of claim 5.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images