A smart speaker and display method thereof

Abstract

The present invention discloses a smart speaker and a display method thereof, which uses a negative refractive index material to image a virtual imaging body displayed inside the smart speaker outside, thereby saving the volume of the smart speaker and improving the user experience. The smart speaker includes a display screen located inside the smart speaker, a negative refractive index material located on the smart speaker, and a processor, wherein: the display screen is used for display; the negative refractive index material is used to refract the content displayed on the display screen to the outside of the smart speaker for imaging; the processor is configured to execute: determining the working scene of the smart speaker; determining the state of the virtual imaging body corresponding to the working scene according to the correspondence between the preset working scene and the state of the virtual imaging body; using the display screen inside the smart speaker as a light source, refracting the state of the virtual imaging body displayed on the display screen to the outside of the smart speaker for imaging through the negative refractive index material.

Description

Technical Field

[0001] The present invention relates to the technical field of smart speakers, and in particular to a smart speaker and a display method thereof. Background Art

[0002] Smart speakers are the upgraded version of speakers. They are a tool for home consumers to surf the Internet with voice, such as ordering songs, shopping online, or checking the weather forecast. They can also be used as a medium for users to control smart home devices with voice, such as opening curtains, setting refrigerator temperature, and heating up water heaters in advance. Existing smart speakers are mainly used for voice interaction and have poor visual interaction capabilities.

[0003] Although there are currently solutions that combine smart speakers with display technology and use display screens for display, since the smart speaker structure needs to add a display screen, it is necessary to separate the sound-emitting and sound-collecting components of the display screen and the speaker, which increases the size of the smart speaker. In addition, since the display screen occupies the original volume of the smart speaker, the sound-emitting component of the smart speaker is bound to have a worse surround effect than a cylindrical speaker, and the user experience is poor. Summary of the invention

[0004] The present invention provides a smart speaker and a display method thereof, which are used to use a negative refractive index material to image a virtual imaging body displayed inside the smart speaker externally, without the need to set a display screen outside the smart speaker for display, thereby saving the volume of the smart speaker and improving the user experience.

[0005] In a first aspect, an embodiment of the present invention provides a smart speaker, comprising a display screen located inside the smart speaker, a negative refractive index material located on the smart speaker, and a processor, wherein:

[0006] The display screen is used for displaying;

[0007] The negative refractive index material is used to refract the content displayed on the display screen to the outside of the smart speaker for imaging;

[0008] The processor is configured to perform:

[0009] Determine a working scenario of the smart speaker;

[0010] Determining the state of the virtual imaging body corresponding to the working scene according to the correspondence between the preset working scene and the state of the virtual imaging body;

[0011] The display screen inside the smart speaker is used as a light source, and the state of the virtual imaging body displayed on the display screen is refracted to the outside of the smart speaker for imaging through the negative refractive index material.

[0012] A smart speaker provided by an embodiment of the present invention can display outside the smart speaker through virtual imaging in the air. The virtual imaging body displayed by the display screen embedded in the smart speaker is projected into the air outside the smart speaker through a negative refractive index material, and can present multiple states. This solves the problem of increasing the volume of the smart speaker by adding a display screen outside the smart speaker for display purposes, and improves the user experience.

[0013] In some examples, the processor is specifically configured to perform:

[0014] If the smart speaker does not receive the local audio stream, determining the working state of the smart speaker as the working scene of the smart speaker; or

[0015] If the smart speaker receives a local audio stream in a non-standby state, the semantics of the local audio stream is determined, and the working scenario of the smart speaker is determined according to the semantics; or,

[0016] If the smart speaker receives a local audio stream in standby mode, wake-up information is identified for the local audio stream, and a working scenario of the smart speaker is determined based on whether the wake-up information is identified.

[0017] In some examples, the processor is further configured to perform:

[0018] Voiceprint recognition is performed on the local audio stream to determine a user preference corresponding to the result of the voiceprint recognition, so as to control the smart home device through the smart speaker in a manner corresponding to the user preference.

[0019] In some examples, after determining the state of the virtual imaging body corresponding to the working scene, the processor is further configured to execute:

[0020] If it is determined that the working scene is a broadcast dialogue scene, then according to a lip synchronization algorithm, the lip shape of the virtual image is controlled to be synchronized with the played chat audio stream; or

[0021] If it is determined that the working scene is a scene of playing non-music media, the acquired non-music media is played, and according to a lip synchronization algorithm, the lip shape of the virtual image is controlled to be synchronized with the played non-music media; or

[0022] If it is determined that the working scene is a scene of playing music media, the acquired music media is played, and dance movements corresponding to the played music media are determined, and the limbs of the virtual imaging body are controlled to dance according to the dance movements.

[0023] In some examples, the processor is specifically configured to perform:

[0024] According to the music beat algorithm, the beat points of the music being played are monitored to determine the dance movements corresponding to the beat points at different times; or,

[0025] According to the music and dance generation algorithm, the dance moves corresponding to the played music are determined.

[0026] In a second aspect, an embodiment of the present invention provides a display method for a smart speaker, comprising:

[0027] Determine the working scenarios of smart speakers;

[0028] Determining the state of the virtual imaging body corresponding to the working scene according to the correspondence between the preset working scene and the state of the virtual imaging body;

[0029] The display screen inside the smart speaker is used as the light source, and the state of the virtual imaging body displayed on the display screen is refracted to the outside of the smart speaker for imaging through the negative refractive index material on the smart speaker.

[0030] In some examples, determining the working scenario of the smart speaker includes:

[0031] If the smart speaker does not receive the local audio stream, determining the working state of the smart speaker as the working scene of the smart speaker; or

[0032] If the smart speaker receives a local audio stream in a non-standby state, the semantics of the local audio stream is determined, and the working scenario of the smart speaker is determined according to the semantics; or,

[0033] If the smart speaker receives a local audio stream in standby mode, wake-up information is identified for the local audio stream, and a working scenario of the smart speaker is determined based on whether the wake-up information is identified.

[0034] In some examples, the performing wake-up information recognition on the local audio stream further includes:

[0035] Voiceprint recognition is performed on the local audio stream to determine a user preference corresponding to the result of the voiceprint recognition, so as to control the smart home device through the smart speaker in a manner corresponding to the user preference.

[0036] In some examples, after determining the state of the virtual imaging body corresponding to the working scene, the method further includes:

[0037] If it is determined that the working scene is a broadcast dialogue scene, then according to a lip synchronization algorithm, the lip shape of the virtual image is controlled to be synchronized with the played chat audio stream; or

[0038] If it is determined that the working scene is a scene of playing non-music media, the acquired non-music media is played, and according to a lip synchronization algorithm, the lip shape of the virtual image is controlled to be synchronized with the played non-music media; or

[0039] If it is determined that the working scene is a scene of playing music media, the acquired music media is played, and dance movements corresponding to the played music media are determined, and the limbs of the virtual imaging body are controlled to dance according to the dance movements.

[0040] In some examples, determining a dance move corresponding to the played music includes:

[0041] According to the music beat algorithm, the beat points of the music being played are monitored to determine the dance movements corresponding to the beat points at different times; or,

[0042] According to the music and dance generation algorithm, the dance moves corresponding to the played music are determined.

[0043] In a third aspect, an embodiment of the present invention further provides a display device of a smart speaker, comprising:

[0044] Determine the scene unit, which is used to determine the working scene of the smart speaker;

[0045] A state determination unit, configured to determine the state of the virtual imaging body corresponding to the working scene according to a correspondence between the preset working scene and the state of the virtual imaging body;

[0046] The imaging display unit is used to use the internal display screen of the smart speaker as a light source, and refract the state of the virtual imaging body displayed on the display screen to the outside of the smart speaker for imaging through the negative refractive index material on the smart speaker.

[0047] In some examples, the determining scene unit is specifically used to:

[0048] If the smart speaker does not receive the local audio stream, determining the working state of the smart speaker as the working scene of the smart speaker; or

[0049] If the smart speaker receives a local audio stream in a non-standby state, the semantics of the local audio stream is determined, and the working scenario of the smart speaker is determined according to the semantics; or,

[0050] If the smart speaker receives a local audio stream in standby mode, wake-up information is identified for the local audio stream, and a working scenario of the smart speaker is determined based on whether the wake-up information is identified.

[0051] In some examples, the scene determination unit is further used to:

[0052] Voiceprint recognition is performed on the local audio stream to determine a user preference corresponding to the result of the voiceprint recognition, so as to control the smart home device through the smart speaker in a manner corresponding to the user preference.

[0053] In some examples, after determining the state of the virtual imaging body corresponding to the working scene, the control unit is further included to:

[0054] If it is determined that the working scene is a broadcast dialogue scene, then according to a lip synchronization algorithm, the lip shape of the virtual image is controlled to be synchronized with the played chat audio stream; or

[0055] If it is determined that the working scene is a scene of playing non-music media, the acquired non-music media is played, and according to a lip synchronization algorithm, the lip shape of the virtual image is controlled to be synchronized with the played non-music media; or

[0056] If it is determined that the working scene is a scene of playing music media, the acquired music media is played, and dance movements corresponding to the played music media are determined, and the limbs of the virtual imaging body are controlled to dance according to the dance movements.

[0057] In some examples, the control unit is specifically configured to:

[0058] According to the music beat algorithm, the beat points of the music being played are monitored to determine the dance movements corresponding to the beat points at different times; or,

[0059] According to the music and dance generation algorithm, the dance moves corresponding to the played music are determined.

[0060] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, it is used to implement the steps of the method described in the second aspect above.

[0061] These and other aspects of the present application will be more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0063] Figure 1 A schematic diagram of a smart speaker provided by an embodiment of the present invention;

[0064] Figure 2A schematic diagram of a smart speaker displaying a virtual imaging body provided by an embodiment of the present invention;

[0065] Figure 3 A schematic diagram of the imaging principle of a negative refractive index material provided by an embodiment of the present invention;

[0066] Figure 4 A flowchart of a control method for a virtual imaging body provided by an embodiment of the present invention;

[0067] Figure 5 A flowchart of a virtual display method for a smart speaker provided by an embodiment of the present invention;

[0068] Figure 6 A flowchart of a method for waking up a master smart speaker provided by an embodiment of the present invention;

[0069] Figure 7 A flowchart for determining a master smart speaker provided by an embodiment of the present invention;

[0070] Figure 8 A flowchart of another smart speaker wake-up method implementation provided by an embodiment of the present invention;

[0071] Fig. 9 A flow chart of an implementation method of a display method of a smart speaker provided by an embodiment of the present invention;

[0072] Fig.10 A schematic diagram of a display device of a smart speaker provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] In the embodiments of the present invention, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0075] The application scenarios described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. It is known to those skilled in the art that with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0076] Existing smart speakers are mainly used for voice interaction, and the visual interaction experience is poor. The embodiments of the present invention project the content displayed inside the smart speaker to the outside of the smart speaker based on the aerial projection display technology, and improve the user experience through the display of virtual imaging bodies, increasing the sense of technology of the product. In addition, the combination of aerial projection display technology and smart speakers can solve the conflict between the screen display and the sound and sound pickup structure outside the smart speaker, while maintaining the appearance and reducing the volume of the speaker.

[0077] like Figure 1 As shown, an embodiment of the present invention provides a smart speaker, including a display screen 100 located inside the smart speaker, a negative refractive index material 101 located on the smart speaker, and a processor 102, wherein:

[0078] The display screen 100 is used for display. The display method in the embodiment of the present invention is different from the prior art method of displaying by setting a display screen outside the smart speaker. The display method of this embodiment does not increase the volume of the smart speaker, and the display screen inside the smart speaker does not need to be provided to the user for viewing. Therefore, it only needs to have a display function and does not need to be viewable. It can be embedded in a smaller volume inside the smart speaker, thereby providing a display function without increasing the volume of the smart speaker and causing no conflict between the display screen and the sound-emitting structure and the sound pickup structure.

[0079] The negative refractive index material 101 is used to refract the content displayed on the display screen to the outside of the smart speaker for imaging; the negative refractive index material in the embodiment of the present invention can realize interactive aerial imaging technology, which uses the light field reconstruction principle through negative refractive index materials to re-converge the divergent light in the air, thereby forming a real image that does not require a medium to carry, and can also be combined with interactive technology to achieve direct interaction between people and real images in the air. The biggest advantage of interactive aerial imaging technology is that people can interact directly with the images in space without using auxiliary display devices like VR. This technology can present the picture directly in the air without any carrying medium, and the image is a real image, which is different from the virtual image presented by the common "holographic projection" in the prior art.

[0080] The processor 102 is configured to perform the following steps:

[0081] Step 1, determine the working scenario of the smart speaker; the working scenario in this embodiment is used to characterize the current working state or working mode of the smart speaker, wherein the working state or working mode can be determined before receiving the local audio stream, or after receiving the local audio stream, the working state is determined according to the local audio stream, and this embodiment does not impose too many restrictions on this. The working scenarios in this embodiment include but are not limited to one or more of the following: the default state after power-on; the state in the network distribution stage; the state of being awakened and waiting for commands; the broadcast dialogue scene; the scene of playing non-music media resources, such as the state of playing stories and poems; the scene of playing music media resources, such as the state of playing songs; the state of querying the weather; the state of not understanding the user's command or not hearing the user's command clearly; the state of controlling smart home appliances; the state when shutting down.

[0082] Step 2: Determine the state of the virtual imaging body corresponding to the working scene according to the correspondence between the preset working scene and the state of the virtual imaging body;

[0083] This embodiment provides a specific correspondence between a work scene and the state of a virtual imaged object. The correspondence in this embodiment is only an example of a specific description. The method protected by this embodiment includes but is not limited to this example. The correspondence between other work scenes and the state of a virtual imaged object obtained based on the same principle all belong to the protection scope of the present invention. The virtual imaged object in this embodiment can be a virtual character. The clothes and actions of the virtual character in different work scenes are different. For the convenience of description, the virtual character can be called Harry. The correspondence between the work scene and Harry's state is listed as follows:

[0084] If it is in the default state after powering on, Harry stands and greets people at set intervals, or kicks a ball; if it is in the network configuration stage, Harry wears an engineer-style hat and operates a router, and the router shows a WIFI signal, that is, Harry is busy configuring the network; if it is awakened and waiting for commands, Harry listens with his ears tilted, that is, he puts his hands on his ears; if it is playing songs, Harry dances happily by clapping his hands, or plays guitar or drums; if it is playing stories or poems, Harry wears glasses and holds a book; if it is checking the weather, Harry can hold an umbrella, wear a cotton-padded jacket, wear a windbreaker against the wind, sweat and drink cold drinks when it is too hot, etc. according to the current weather; if it does not understand the user's command or does not hear the user's command clearly, Harry shows a puzzled expression; if it is controlling smart home appliances, Harry stays at home and takes out the remote control to click; if it is shutting down, Harry disappears after performing a goodbye gesture.

[0085] Step 3: Using the internal display screen of the smart speaker as a light source, the state of the virtual imaging body displayed on the display screen is refracted to the outside of the smart speaker for imaging through the negative refractive index material.

[0086] In this embodiment, based on different working scenes, the state of the virtual imaging body corresponding to the working scene is pre-set, and the virtual imaging body can also be displayed in a 3D virtual state based on 3D rendering display technology (such as Unity 3D technology). It should be noted that the 3D virtual display in this embodiment is different from the holographic projection display. The display method in this embodiment is to first display the 3D virtual state on the internal display screen of the smart speaker, and project the currently displayed 3D virtual state to the air outside the smart speaker for imaging. The actual imaging state can be regarded as the state displayed on the internal display screen seen by the human eye, and the 3D virtual state can perform actions such as rotation according to the received instructions.

[0087] In some embodiments, Figure 2 As shown, an embodiment of the present invention provides a method for realizing a smart speaker displaying a virtual image body, wherein a negative refractive index material 200 is located above a smart speaker 201 and is formed integrally with the smart speaker 201, and a virtual image body 202 is located above the negative refractive index material 200. Figure 3 As shown, this embodiment provides a schematic diagram of the imaging principle of a negative refractive index material, wherein Figure 3 The light source in this embodiment is the internal display screen of the smart speaker, which can also be understood as all the contents displayed on the internal display screen. Figure 3 The imaging in is the virtual imaging body in this embodiment. In this embodiment, all contents displayed on the internal display screen can be refracted to the outside of the smart speaker through the negative refractive index material for imaging, thereby replacing the display function of the external display screen.

[0088] In some embodiments, the embodiments of the present invention can determine the current working scene of the smart speaker according to whether the local audio stream is received, wherein the method of determining the working scene of the smart speaker includes but is not limited to the following:

[0089] Mode 1: If the smart speaker does not receive the local audio stream, the working state of the smart speaker is determined as the working scene of the smart speaker;

[0090] In this embodiment, if the smart speaker does not receive the local audio stream, the current working state of the smart speaker is determined as the working scene, where the working state includes but is not limited to: the power-on state, the default state of not receiving the local audio stream after power-on, the network configuration state, the network configuration completed state, the song playing state, the story or poem playing state, the weather query state, the smart home appliance control state, the power-off state, etc.

[0091] Mode 2: If the smart speaker receives a local audio stream in a non-standby state, the semantics of the local audio stream is determined, and the working scenario of the smart speaker is determined according to the semantics;

[0092] In this embodiment, if a local audio stream is received, the working scene is determined according to the semantics of the local audio stream. For example, if an audio stream of playing a song is received, the working scene is determined to be playing a song.

[0093] In some embodiments, the semantics of the local audio stream is determined by:

[0094] 1) converting the local audio stream by a local speech recognition algorithm;

[0095] 2) Determine the semantics of the converted local audio stream through a local semantic understanding algorithm or a cloud semantic understanding algorithm.

[0096] During implementation, the local audio stream is converted by a local speech recognition algorithm to obtain a language that can be processed by a computer. If the converted semantics can be parsed by a local semantic understanding algorithm, the local semantic understanding algorithm is used to determine the semantics. If the converted semantics cannot be parsed by the local semantic understanding algorithm, a cloud semantic understanding algorithm is used to parse the converted semantics and determine the semantics, wherein the local semantic understanding algorithm is stored in a local server or a local smart speaker, and the cloud semantic understanding algorithm is stored in a cloud server.

[0097] In some scenarios, for example, when a local audio stream related to small talk is received, the local semantic understanding algorithm can be used to perform semantic analysis on the converted local audio stream, or when a local audio stream related to controlling smart home devices is received, the local semantic understanding algorithm can be used to perform semantic analysis to determine the working scene, and then determine the state of the virtual image (such as clothing, action, etc.) to maintain or switch the state. In some scenarios, for example, when a local audio stream related to querying the weather, playing songs, stories or poems is received, the cloud semantic understanding algorithm is used to perform semantic analysis, and then the corresponding media resources are obtained from the cloud for playback.

[0098] Method 3: If the smart speaker receives a local audio stream in standby mode, wake-up information is identified for the local audio stream, and the working scenario of the smart speaker is determined based on whether the wake-up information is identified.

[0099] In some embodiments, in order to save power or start up quickly, smart speakers are generally configured with a standby mode (i.e., standby state). In this mode, the smart speaker is in a standby or dormant state and will not parse the local audio stream. This implementation also provides a wake-up work scenario based on this mode, that is, when a local audio stream is received in the standby state, if a wake-up message, such as "Hello, Harry", is recognized, the smart speaker is switched to the wake-up state. At this time, the working state of the smart speaker is the wake-up state. If the wake-up message is not recognized, the state of the smart speaker remains unchanged, and the working scene remains unchanged accordingly. That is, if the wake-up message is recognized, the working scene is determined to be a scene of being awakened and waiting for a command, otherwise the working scene is determined to be a standby scene.

[0100] In some implementation scenarios, while performing speech recognition or semantic understanding on the local audio stream, or identifying the wake-up information, voiceprint recognition can also be performed on the local audio stream to determine the user preferences corresponding to the voiceprint recognition results. If the local audio stream is an audio stream related to controlling smart home devices, the smart home devices can be controlled through the smart speaker in a manner corresponding to the user preferences. For example, when family members A and B wake up the smart speaker and say "turn on the air conditioner", the smart speaker receives the local audio stream of "turn on the air conditioner" and performs semantic analysis, controls the air conditioner to turn on, and sets the air conditioner mode and temperature that A likes, where the preferences of user A can be set through the APP bound to A, or obtained from the historical interaction information between A and the smart speaker.

[0101] In some embodiments, the state of the virtual image in this embodiment also includes lip shape and action synchronized with the voice media of the broadcast. After determining the state of the virtual image corresponding to the working scene, this embodiment also includes one or more of the following display modes:

[0102] Mode 1: if it is determined that the working scene is a broadcast dialogue scene, then according to a lip synchronization algorithm, the lip shape of the virtual image is controlled to be synchronized with the played chat audio stream;

[0103] In implementation, the broadcast dialogue scene may be a chat scene. After determining the state (clothing, action) of the virtual imaged body corresponding to the broadcast dialogue scene, this embodiment also controls the lip shape of the virtual imaged body to be synchronized with the played chat audio stream based on the lip synchronization algorithm. That is, if the user chats with the smart speaker through voice, the virtual imaged body's lip shape will open and close according to the content of the played chat audio stream while the smart speaker is playing the chat audio stream.

[0104] Mode 2: If it is determined that the working scene is a scene of playing non-music media, the acquired non-music media is played, and the lip shape of the virtual image is controlled to be synchronized with the played non-music media according to a lip synchronization algorithm;

[0105] In implementation, it may be a non-music media scene such as querying the weather. After determining the state (clothing, action) of the virtual imaging body corresponding to the non-music media scene being played, this embodiment also controls the lip shape of the virtual imaging body to be synchronized with the non-music media being played based on the lip synchronization algorithm. In some examples, the scene also includes a special scene in which the smart speaker cannot obtain the semantics of the local audio stream after voice recognition and semantic understanding. The smart speaker will play special audio such as "Sorry, I didn't hear your instructions clearly, please say it again." That is, if the user inputs a local audio stream corresponding to a non-music media scene, the smart speaker will switch the state of the virtual imaging body to correspond to the non-music media scene. While playing the non-music media scene, the lip shape of the virtual imaging body can also be controlled to open and close according to the content being played.

[0106] In some examples, the basic principle of the lip synchronization algorithm in this embodiment is to control the mouth changes of the virtual imaging body by identifying the initial consonants, finals and the mouth shape numbers corresponding to the pronunciation of the initial consonants and finals in the chat audio stream or non-music media. Optionally, after obtaining the chat audio stream or non-music media, the initial consonants and finals in the chat audio stream or non-music media are identified, and the mouth shape number is determined by the mouth shape changes of the initial consonant and final combination, wherein the initial consonants and finals in the audio stream can be identified by a pre-trained deep learning algorithm. Optionally, the Chinese characters in the audio stream are identified by speech recognition technology, and the pinyin is directly obtained to determine the mouth shape corresponding to the pinyin.

[0107] Method 3: if it is determined that the working scene is a scene of playing music media, the acquired music media is played, and dance movements corresponding to the played music media are determined, and the limbs of the virtual imaging body are controlled to dance according to the dance movements.

[0108] In practice, it can be a music media scene such as playing songs. After determining the state (clothing, action) of the virtual image body corresponding to the music media scene, this embodiment can also generate dance movements based on the played music and control the virtual image body to dance according to the dance movements. For example, when the smart speaker receives the command "please play songs", the virtual image body presents dance movements according to the preset dance movements, or presents clapping and stomping movements according to the audio rhythm.

[0109] In some examples, one or more of the following methods for determining dance moves are included but are not limited to:

[0110] 1) According to the music beat algorithm, the beat points of the music being played are monitored, and the dance movements corresponding to the beat points at different times are determined;

[0111] In implementation, the existing librosa library can be used to monitor the beat_times of the music being played in real time through a thread. The beat_times variable is used as a Linux shared variable. When another thread detects that the current moment is a beat point, a clap or stomp signal is sent to the internal display screen to synchronize the played audio stream with the virtual imaging body.

[0112] 2) According to the music and dance generation algorithm, determine the dance moves corresponding to the music being played.

[0113] During the implementation, based on the existing music and dance matching data, the music and dance are divided into bars, and feature vectors are extracted for the music bars and dance bars respectively. Then, based on the pix2pix generative adversarial network (GAN) model, the dance clips corresponding to the feature vectors are determined, and finally dance movements are generated based on multiple dance clips.

[0114] In some examples, embodiments of the present invention pick up the local audio stream through the microphone or microphone array on the smart speaker, perform noise reduction processing, and then send it to the smart speaker processor for speech recognition and semantic understanding of the local audio stream. After the semantic information of the local audio stream is parsed using the local semantic understanding algorithm or the cloud semantic understanding algorithm, the corresponding audio stream is played according to the semantic information, and the state of the virtual imaging body corresponding to the semantic information is switched at the same time. Optionally, the local audio stream picked up by the smart speaker can be received by setting a service interface, and the audio stream to be played can be received to drive the speaker of the smart speaker to broadcast. For example, the following service interface content is set to indicate the semantic information for the virtual imaging body to perform a left turn:

[0115] Timestamp 20210311160052 Type 2 Data "Turn left"

[0116] Among them, Timestamp is the timestamp, indicating the moment when the current local audio stream is picked up; Type is the type, which is an enumeration value with values ​​of 1, 2, and 3. 1 represents system instructions, which are responsible for the startup, reset, standby and other instructions of the virtual imaging body; 2 represents semantic information. If Type is filled in as 2, it means that this interface requests the transmission of semantic information, which is used to switch the preset image of the virtual imaging body and set whether the acquired audio stream needs to use the lip synchronization algorithm or the music beat algorithm for lip synchronization or dance movement output; 3 represents audio content. When Type is 3, the audio stream is transmitted in Data. The audio stream data is processed accordingly using the lip synchronization algorithm or the music beat algorithm to control the lip shape or body movements.

[0117] In some examples, such as Figure 4 As shown, this embodiment also provides a control method for a virtual imaging body, and the implementation process of the method is as follows:

[0118] Step 400: Determine the semantics of the local audio stream received by the smart speaker;

[0119] Step 401, determining the state of the virtual imaging body according to the semantics, wherein the state includes a basic image of the virtual imaging body;

[0120] Step 402, determine whether the semantics is a broadcast dialogue semantics, if yes, execute step 403, otherwise execute step 404;

[0121] Step 403: According to the lip synchronization algorithm, the lip shape of the virtual image is controlled to be synchronized with the broadcasted dialogue audio;

[0122] Step 404: According to the music coordination algorithm, the limbs of the virtual image body are controlled to dance according to the broadcast audio.

[0123] The music collaboration algorithm includes but is not limited to the above-mentioned music beat algorithm or music and dance generation algorithm.

[0124] The smart speaker provided in this embodiment, in the default state of power-on, projects a virtual image in the air above the smart speaker, which can be presented as a standing state, and can randomly present actions such as walking, turning the head, and tidying up the schoolbag. When the smart speaker receives an instruction, such as "turn left", the virtual image will perform a left turn according to the preset left turn action corresponding to the instruction. When the user interacts with the smart speaker, if the smart speaker cannot recognize the user's instruction after voice recognition and semantic understanding, the smart speaker will broadcast "Sorry, I didn't hear your instruction clearly, please say it again", and while broadcasting, the mouth of the virtual image will open and close according to the content of the speech. When the smart speaker receives the instruction "Please play a song", the virtual image can present a dance action according to the preset dance action, or a dance action according to the audio rhythm, or a clapping and stamping action according to the audio rhythm. When the user inquires about the weather, the virtual image can adjust its own clothing according to the weather conditions such as rain, wind, haze, temperature, etc., and match it with the actual weather content broadcast.

[0125] In some examples, such as Figure 5 As shown, an embodiment of the present invention provides a virtual display method of a smart speaker, and the specific implementation process of the method is as follows:

[0126] Step 500: The smart speaker receives a local audio stream in a standby state;

[0127] Step 501: Perform wake-up information recognition and voiceprint recognition on the local audio stream;

[0128] Step 502: after determining that the wake-up information is recognized, wake up the smart speaker;

[0129] Step 503: receiving the local audio stream again;

[0130] Step 504: performing speech recognition and semantic understanding on the local audio stream;

[0131] Step 505: determine whether the semantics of the local audio stream can be determined according to the local semantic understanding algorithm, if so, execute step 506, otherwise execute step 510;

[0132] Step 506: Determine whether it is a chat scene according to the semantics, if yes, execute step 507, otherwise execute step 509;

[0133] Step 507: According to the lip synchronization algorithm, the lip shape of the virtual image is controlled to be synchronized with the played chat audio stream;

[0134] Step 508: Determine that the chat is over, and play the end chat audio;

[0135] Step 509: determine that the semantics is a control instruction, switch the state of the corresponding virtual imaging body according to the control instruction and execute corresponding control;

[0136] Step 510: determine whether the semantics of the local audio stream can be determined according to the cloud semantic understanding algorithm, if so, execute step 511, otherwise execute step 513;

[0137] Step 511: determining the state of the virtual imaging body according to the semantics, and acquiring corresponding media resources according to the semantics;

[0138] Step 512: Play the acquired media resource, and determine, according to the type of the media resource, to control the lip shape or movement of the virtual imaged object according to a lip synchronization algorithm or a music coordination algorithm.

[0139] Step 513: Play the state of the virtual image corresponding to the semantics of “incomprehensible” to convey the incomprehensible information to the user.

[0140] In some examples, if there are multiple smart speakers, that is, when the wake-up information spoken by the user is received by multiple smart speakers, this embodiment can also select one of the smart speakers to wake up. The principle of wake-up in this embodiment is to wake up nearby, that is, the smart speaker "closest" to the user is woken up. The wake-up method in this implementation is mainly applied to the wake-up of the smart speaker in any of the above methods. During the process of being woken up, the smart speaker can inform the user that it is being woken up through the state of the virtual imaging body. Figure 6 As shown, it is applied to the main smart speaker, and the specific implementation method of waking up in this embodiment is as follows:

[0141] Step 600: If a local audio stream is received in the standby state, wake-up information recognition is performed according to the received local audio stream;

[0142] Step 601: after identifying the wake-up information, trigger the generation of a wake-up request and send the wake-up request to itself, and receive the wake-up request sent by other smart speakers, wherein the wake-up request carries a parameter representing the quality of the wake-up information;

[0143] Step 602: Determine the smart speaker that needs to be awakened from the received wake-up request and other wake-up requests, and send an approval wake-up request to the smart speaker that needs to be awakened, so as to trigger the smart speaker that needs to be awakened to switch from the standby state to the awake state.

[0144] Among them, sending a wake-up request to the smart speaker that needs to be woken up also includes: sending a shut-down wake-up request to other smart speakers that do not need to be woken up.

[0145] In some examples, the wake-up request includes, but is not limited to, some or all of the following information: sound loudness, wake-up quality score, and timestamp. In this embodiment, the main smart speaker selects the wake-up request with the best wake-up information quality from its own and other smart speakers' wake-up requests based on the parameters representing the wake-up information quality contained in the wake-up request, thereby determining the smart speaker corresponding to the best wake-up request to wake up.

[0146] In some examples, during the wake-up process, the main smart speaker can use the state of the virtual imaging body to notify the user that the smart speaker that needs to be awakened is being selected at this time, so that the user can better interact with the smart speaker.

[0147] In some examples, the primary smart speaker is determined as follows:

[0148] After multiple smart speakers are powered on, they all generate a first random number; each smart speaker receives a second random number sent by other smart speakers, and compares the first random number with the second random number; based on the comparison result, it is determined whether it is the main smart speaker.

[0149] In some examples, such as Figure 7 As shown, taking two smart speakers as an example, the implementation process of determining the main smart speaker is as follows:

[0150] Step 700: Power on smart speaker 1 and smart speaker 2;

[0151] Step 701: Smart speaker 1 generates random number 1, and smart speaker 2 generates random number 2;

[0152] Step 702: Smart speaker 1 broadcasts random number 1 to the LAN port, and carries the IP address of smart speaker 1; smart speaker 2 broadcasts random number 2 to the LAN port, and carries the IP address of smart speaker 2;

[0153] Step 703: Smart speaker 1 receives random number 2, and smart speaker 2 receives random number 1;

[0154] Step 704: Smart speaker 2 compares the random numbers 1 and 2, and selects the smart speaker corresponding to the largest random number as the main smart speaker.

[0155] In some examples, the wake-up principle is the same as that of the main smart speaker mentioned above, such as Figure 8 As shown, this embodiment also provides a wake-up method applied to other smart speakers, and the implementation process of the method is as follows:

[0156] Step 800: If a local audio stream is received in the standby state, wake-up information recognition is performed according to the received local audio stream;

[0157] Step 801: after identifying the wake-up information, trigger the generation of a wake-up request and send it to the main smart speaker, so that the main smart speaker determines the smart speaker that needs to be awakened according to the received wake-up requests sent by the smart speaker and other smart speakers, and sends an approval wake-up request to the smart speaker that needs to be awakened, wherein the wake-up request carries a parameter representing the quality of the wake-up information;

[0158] Step 802: If a wake-up request is received from the main smart speaker, the standby state is switched to the wake-up state.

[0159] In some examples, whether the smart speaker is the master smart speaker is determined as follows:

[0160] The smart speaker generates a first random number after being powered on;

[0161] Receive a second random number sent by other smart speakers, and determine whether the smart speaker is the main smart speaker based on a comparison result of the first random number and the second random number.

[0162] In some examples, after the smart speaker switches from the standby state to the awake state, the smart speaker further includes:

[0163] The state of the virtual imaging body displayed on the internal display screen of the smart speaker is refracted to the outside of the smart speaker for imaging through the negative refractive index material on the smart speaker, wherein the state of the virtual imaging body is determined according to the wake-up state of the smart speaker.

[0164] Example 2: Based on the same inventive concept, the embodiment of the present invention also provides a display method for a smart speaker. Since this method is the method corresponding to the smart speaker in the embodiment of the present invention, and the principle of solving the problem by this method is similar to that of the smart speaker, the implementation of this method can refer to the implementation of the smart speaker, and the repeated parts will not be repeated.

[0165] like Fig. 9 As shown, the implementation process of this method is as follows:

[0166] Step 900: Determine the working scenario of the smart speaker;

[0167] Step 901: Determine the state of the virtual imaging body corresponding to the working scene according to the correspondence between the preset working scene and the state of the virtual imaging body;

[0168] Step 902: Using the display screen inside the smart speaker as a light source, the state of the virtual imaging body displayed on the display screen is refracted to the outside of the smart speaker for imaging through the negative refractive index material on the smart speaker.

[0169] In some examples, determining the working scenario of the smart speaker includes:

[0170] If the smart speaker does not receive the local audio stream, determining the working state of the smart speaker as the working scene of the smart speaker; or

[0171] If the smart speaker receives a local audio stream in a non-standby state, the semantics of the local audio stream is determined, and the working scenario of the smart speaker is determined according to the semantics; or,

[0172] If the smart speaker receives a local audio stream in standby mode, wake-up information is identified for the local audio stream, and a working scenario of the smart speaker is determined based on whether the wake-up information is identified.

[0173] In some examples, the performing wake-up information recognition on the local audio stream further includes:

[0174] Voiceprint recognition is performed on the local audio stream to determine a user preference corresponding to the result of the voiceprint recognition, so as to control the smart home device through the smart speaker in a manner corresponding to the user preference.

[0175] In some examples, after determining the state of the virtual imaging body corresponding to the working scene, the method further includes:

[0176] If it is determined that the working scene is a broadcast dialogue scene, then according to a lip synchronization algorithm, the lip shape of the virtual image is controlled to be synchronized with the played chat audio stream; or

[0177] If it is determined that the working scene is a scene of playing non-music media, the acquired non-music media is played, and according to a lip synchronization algorithm, the lip shape of the virtual image is controlled to be synchronized with the played non-music media; or

[0178] If it is determined that the working scene is a scene of playing music media, the acquired music media is played, and dance movements corresponding to the played music media are determined, and the limbs of the virtual imaging body are controlled to dance according to the dance movements.

[0179] In some examples, determining a dance move corresponding to the played music includes:

[0180] According to the music beat algorithm, the beat points of the music being played are monitored to determine the dance movements corresponding to the beat points at different times; or,

[0181] According to the music and dance generation algorithm, the dance moves corresponding to the played music are determined.

[0182] Example 3. Based on the same inventive concept, the embodiment of the present invention also provides a display device for a smart speaker. Since the device is the device of the method in the embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0183] like Fig.10 As shown, the device comprises:

[0184] A scene determination unit 1000 is used to determine a working scene of the smart speaker;

[0185] The state determination unit 1001 is used to determine the state of the virtual imaging body corresponding to the working scene according to the correspondence between the preset working scene and the state of the virtual imaging body;

[0186] The imaging display unit 1002 is used to use the internal display screen of the smart speaker as a light source, and refract the state of the virtual imaging body displayed on the display screen to the outside of the smart speaker for imaging through the negative refractive index material on the smart speaker.

[0187] In some examples, the determining scene unit is specifically used to:

[0188] If the smart speaker does not receive the local audio stream, determining the working state of the smart speaker as the working scene of the smart speaker; or

[0189] If the smart speaker receives a local audio stream in a non-standby state, the semantics of the local audio stream is determined, and the working scenario of the smart speaker is determined according to the semantics; or,

[0190] If the smart speaker receives a local audio stream in standby mode, wake-up information is identified for the local audio stream, and a working scenario of the smart speaker is determined based on whether the wake-up information is identified.

[0191] In some examples, the scene determination unit is further used to:

[0192] Voiceprint recognition is performed on the local audio stream to determine a user preference corresponding to the result of the voiceprint recognition, so as to control the smart home device through the smart speaker in a manner corresponding to the user preference.

[0193] In some examples, after determining the state of the virtual imaging body corresponding to the working scene, the control unit is further included to:

[0194] If it is determined that the working scene is a broadcast dialogue scene, then according to a lip synchronization algorithm, the lip shape of the virtual image is controlled to be synchronized with the played chat audio stream; or

[0195] If it is determined that the working scene is a scene of playing non-music media, the acquired non-music media is played, and according to a lip synchronization algorithm, the lip shape of the virtual image is controlled to be synchronized with the played non-music media; or

[0196] If it is determined that the working scene is a scene of playing music media, the acquired music media is played, and dance movements corresponding to the played music media are determined, and the limbs of the virtual imaging body are controlled to dance according to the dance movements.

[0197] In some examples, the control unit is specifically configured to:

[0198] According to the music beat algorithm, the beat points of the music being played are monitored to determine the dance movements corresponding to the beat points at different times; or,

[0199] According to the music and dance generation algorithm, the dance moves corresponding to the played music are determined.

[0200] Based on the same inventive concept, an embodiment of the present invention further provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the following steps are implemented:

[0201] Determine the working scenarios of smart speakers;

[0202] Determining the state of the virtual imaging body corresponding to the working scene according to the correspondence between the preset working scene and the state of the virtual imaging body;

[0203] The display screen inside the smart speaker is used as the light source, and the state of the virtual imaging body displayed on the display screen is refracted to the outside of the smart speaker for imaging through the negative refractive index material on the smart speaker.

[0204] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0205] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that has the functions specified in one or more boxes.

[0206] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0207] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0208] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A smart speaker, characterized in that: The invention comprises a display screen located inside a smart speaker, a negative refractive index material located on the smart speaker, and a processor, wherein: The display screen is used for displaying; The negative refractive index material is used to refract the content displayed on the display screen to the outside of the smart speaker for imaging; The processor is configured to perform: Determine a working scenario of the smart speaker; Determining the state of the virtual imaging body corresponding to the working scene according to the correspondence between the preset working scene and the state of the virtual imaging body; Using the display screen inside the smart speaker as a light source, the state of the virtual imaging body displayed on the display screen is refracted to the outside of the smart speaker through the negative refractive index material for imaging; When there are multiple smart speakers, the multiple smart speakers receive the user's wake-up information and select a smart speaker closest to the user for wake-up. During the wake-up process, the smart speaker informs the user that it is being woken up through the state of the virtual imaging body; The multiple smart speakers include a main smart speaker and other smart speakers; the main smart speaker is determined in the following manner: after the multiple smart speakers are powered on, they all generate a first random number; each smart speaker receives a second random number sent by other smart speakers, and compares the first random number with the second random number; and determines whether it is the main smart speaker based on the comparison result; The wake-up process of the main smart speaker is as follows: if a local audio stream is received in the standby state, the wake-up information is identified according to the received local audio stream; after identifying the wake-up information, a wake-up request is triggered and sent to itself, and the wake-up request sent by other smart speakers is received, wherein the wake-up request carries a parameter characterizing the quality of the wake-up information; from the received wake-up request of itself and other wake-up requests, the smart speaker that needs to be awakened is determined and a consent wake-up request is sent to the smart speaker that needs to be awakened, so as to trigger the smart speaker that needs to be awakened to switch from the standby state to the awakened state; wherein the consent wake-up request is sent to the smart speaker that needs to be awakened, and also includes: sending a shutdown wake-up request to other smart speakers that do not need to be awakened; during the awakening process, the main smart speaker notifies the user through the state of the virtual imaging body that the smart speaker that needs to be awakened is being selected at this time; The wake-up process of other smart speakers is as follows: if a local audio stream is received in the standby state, the wake-up information is identified according to the received local audio stream; after the wake-up information is identified, a wake-up request is triggered and sent to the main smart speaker, so that the main smart speaker determines the smart speaker that needs to be awakened according to the wake-up requests received from the smart speaker and other smart speakers, and sends a wake-up request to the smart speaker that needs to be awakened, wherein the wake-up request carries parameters characterizing the quality of the wake-up information; if the wake-up request sent by the main smart speaker is received, the standby state is switched to the wake-up state.

2. The smart speaker according to claim 1, characterized in that: The processor is specifically configured to execute: If the smart speaker does not receive the local audio stream, determining the working state of the smart speaker as the working scene of the smart speaker; or If the smart speaker receives a local audio stream in a non-standby state, determining the semantics of the local audio stream, and determining a working scenario of the smart speaker according to the semantics; or, If the smart speaker receives a local audio stream in standby mode, wake-up information is identified for the local audio stream, and a working scenario of the smart speaker is determined based on whether the wake-up information is identified.

3. The smart speaker according to claim 2, characterized in that: The processor is further configured to execute: Voiceprint recognition is performed on the local audio stream to determine a user preference corresponding to the result of the voiceprint recognition, so as to control the smart home device through the smart speaker in a manner corresponding to the user preference.

4. The smart speaker according to claim 1, characterized in that: After determining the state of the virtual imaging body corresponding to the working scene, the processor is further configured to execute: If it is determined that the working scene is a broadcast dialogue scene, then according to a lip synchronization algorithm, the lip shape of the virtual image is controlled to be synchronized with the played chat audio stream; or If it is determined that the working scene is a scene of playing non-music media, the acquired non-music media is played, and according to a lip synchronization algorithm, the lip shape of the virtual image is controlled to be synchronized with the played non-music media; or If it is determined that the working scene is a scene of playing music media, the acquired music media is played, and dance movements corresponding to the played music media are determined, and the limbs of the virtual imaging body are controlled to dance according to the dance movements.

5. The smart speaker according to claim 4, characterized in that: The processor is specifically configured to execute: According to the music beat algorithm, the beat points of the music being played are monitored to determine the dance movements corresponding to the beat points at different times; or, According to the music and dance generation algorithm, the dance moves corresponding to the played music are determined.

6. A display method for a smart speaker, characterized in that: The method includes: Determine the working scenarios of smart speakers; Determining the state of the virtual imaging body corresponding to the working scene according to the correspondence between the preset working scene and the state of the virtual imaging body; Using the display screen inside the smart speaker as a light source, the state of the virtual imaging body displayed on the display screen is refracted to the outside of the smart speaker through the negative refractive index material on the smart speaker for imaging; When there are multiple smart speakers, the multiple smart speakers receive the user's wake-up information and select a smart speaker closest to the user to wake up. During the process of being woken up, the smart speaker informs the user that it is being woken up through the state of the virtual imaging body; the multiple smart speakers include a main smart speaker and other smart speakers; the main smart speaker is determined in the following manner: after the multiple smart speakers are powered on, they all generate a first random number; each smart speaker receives a second random number sent by other smart speakers, and compares the first random number with the second random number; according to the comparison result, it is determined whether it is the main smart speaker; The wake-up process of the main smart speaker is as follows: if a local audio stream is received in the standby state, the wake-up information is identified according to the received local audio stream; after identifying the wake-up information, a wake-up request is triggered and sent to itself, and the wake-up request sent by other smart speakers is received, wherein the wake-up request carries a parameter characterizing the quality of the wake-up information; from the received wake-up request of itself and other wake-up requests, the smart speaker that needs to be awakened is determined and a consent wake-up request is sent to the smart speaker that needs to be awakened, so as to trigger the smart speaker that needs to be awakened to switch from the standby state to the awakened state; wherein the consent wake-up request is sent to the smart speaker that needs to be awakened, and also includes: sending a shutdown wake-up request to other smart speakers that do not need to be awakened; during the awakening process, the main smart speaker notifies the user through the state of the virtual imaging body that the smart speaker that needs to be awakened is being selected at this time; The wake-up process of other smart speakers is as follows: if a local audio stream is received in the standby state, the wake-up information is identified according to the received local audio stream; after the wake-up information is identified, a wake-up request is triggered and sent to the main smart speaker, so that the main smart speaker determines the smart speaker that needs to be awakened according to the wake-up requests received from the smart speaker and other smart speakers, and sends a wake-up request to the smart speaker that needs to be awakened, wherein the wake-up request carries parameters characterizing the quality of the wake-up information; if the wake-up request sent by the main smart speaker is received, the standby state is switched to the wake-up state.

7. The method according to claim 6, characterized in that Determining the working scenario of the smart speaker includes: If the smart speaker does not receive the local audio stream, determining the working state of the smart speaker as the working scene of the smart speaker; or If the smart speaker receives a local audio stream in a non-standby state, the semantics of the local audio stream is determined, and the working scenario of the smart speaker is determined according to the semantics; or, If the smart speaker receives a local audio stream in standby mode, wake-up information is identified for the local audio stream, and a working scenario of the smart speaker is determined based on whether the wake-up information is identified.

8. The method according to claim 7, characterized in that The performing wake-up information identification on the local audio stream further includes: Voiceprint recognition is performed on the local audio stream to determine a user preference corresponding to the result of the voiceprint recognition, so as to control the smart home device through the smart speaker in a manner corresponding to the user preference.

9. The method according to claim 6, characterized in that After determining the state of the virtual imaging body corresponding to the working scene, the method further includes: If it is determined that the working scene is a broadcast dialogue scene, then according to a lip synchronization algorithm, the lip shape of the virtual image is controlled to be synchronized with the played chat audio stream; or If it is determined that the working scene is a scene of playing non-music media, the acquired non-music media is played, and according to a lip synchronization algorithm, the lip shape of the virtual image is controlled to be synchronized with the played non-music media; or If it is determined that the working scene is a scene of playing music media, the acquired music media is played, and dance movements corresponding to the played music media are determined, and the limbs of the virtual imaging body are controlled to dance according to the dance movements.

10. The method according to claim 9, characterized in that The step of determining the dance movements corresponding to the played music includes: According to the music beat algorithm, the beat points of the music being played are monitored to determine the dance movements corresponding to the beat points at different times; or, According to the music and dance generation algorithm, the dance moves corresponding to the played music are determined.

Images