Intelligent sound box for generating and pushing music based on facial expressions

Abstract

The invention provides an intelligent sound box for generating and pushing music based on facial expressions, which comprises a sound box body and a control panel arranged in the sound box body, the sound box body is also provided with a camera, a first loudspeaker, a second loudspeaker, the camera and the digital panel are respectively connected with the control panel, the sound box body is also provided with a power interface, and the power interface is connected with the control panel. Electric energy supply is completed through a power interface, during emotion recognition, a camera is used for capturing high-definition pictures of the upper body posture and the facial expression of a person to serve as data input of an emotion recognition model, and after the model extracts needed data features and carries out feature fusion, the emotion state of a current user is recognized. According to the emotion recognition model, the parallel CNN encoder and Transform encoder are used, local features and global features of data can be well combined, a personalized recognition algorithm suitable for the user is formed, facial expressions of the user and music generation can be ingeniously combined together, and interestingness and immersion of man-machine interaction are enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of speakers, and in particular relates to an intelligent speaker that generates and pushes music based on facial expressions. Background Art

[0002] In today's society, music has become an indispensable part of people's lives. With the advancement of technology and the upgrading of people's consumption concepts, the market demand for personalized and intelligent music experience is growing. Facial expression is one of the most intuitive ways of expressing human inner thoughts, and it expresses a person's current emotions in the most natural way. Music is good at expressing emotions and is the easiest to resonate among all art forms. Therefore, facial expressions and music have the same emotional expression. As a new type of music generation device, smart speakers can quickly and accurately identify the user's emotional state, recommend suitable music for them, and promote human-computer collaboration. Meet the market's pursuit of personalized music experience.

[0003] In busy modern life, facing the pressure and troubles at work, professionals often want to find a way to release quickly. Although traditional music applications can provide music playback services, the search and selection process is cumbersome and time-consuming, and it is difficult to meet immediate needs. Smart speakers, with their efficient emotion recognition technology, can capture users' emotional changes in real time, and automatically generate and recommend music that matches their emotional state, cleverly integrating music into people's daily lives, bringing a new way of spiritual relaxation to professionals.

[0004] Smart speakers also play an important role in psychological counseling centers, nursing homes and other places. In the process of psychological counseling, music is often used as an auxiliary tool to help visitors regulate their emotions and relieve stress. Smart speakers can identify the emotional state of visitors in real time and recommend appropriate music according to their needs, providing more accurate and effective support for psychotherapy. In nursing homes, elderly residents often face emotional problems such as loneliness and anxiety. Smart speakers can provide them with automated and personalized music companionship, enriching their lives, alleviating negative emotions such as loneliness, and adding happiness and warmth to their lives.

[0005] Smart speakers are also suitable for children due to their automatic and intelligent features, who may lack the ability to choose music that suits their emotional state. The emotional music system provides personalized music generation and recommendations for these users through intelligent emotion recognition, allowing them to better understand and express their own emotional state while enjoying the pleasure and relaxation brought by music.

[0006] Based on this, an intelligent speaker based on facial expression generation and music push is proposed. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide an intelligent speaker based on facial expression generation and music push in view of the deficiencies of the above-mentioned prior art, so as to solve the problems raised in the above-mentioned background technology.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is: an intelligent speaker that generates and pushes music based on facial expressions, including a speaker box and a control panel arranged in the speaker box, the speaker box is also provided with a camera, a first speaker, and a second speaker, the first speaker and the second speaker are respectively connected to a digitizing tablet, the digitizing tablet is also provided with a digitizing tablet audio output interface, left and right channel volume knobs, a master volume knob, a subwoofer volume knob and a digitizing tablet power interface, the camera and the digitizing tablet are respectively connected to the control panel for signal connection, the speaker box is also provided with a power interface, and power supply is completed through the power interface.

[0009] As a further illustration of the present invention, when performing emotion recognition, a camera is used to capture high-definition pictures of a person's upper body posture and facial expression as data input for the emotion recognition model. The model extracts two data features and performs feature fusion to identify the emotional state of the current user.

[0010] As a further illustration of the present invention, the specific process of emotion recognition is as follows:

[0011] The camera captures high-definition pictures of a person's upper body posture and facial expression as data input for the emotion recognition model. In the first stage of the model, features are extracted from these two types of data using parallel CNN Encoder and Transformer Encoder.

[0012] The basic structure of CNN Encoder is convolution layer, activation layer, pooling layer. The convolution layer extracts the local features of the input data. The output feature map Y of the convolution operation is obtained by the local weighted sum of the input feature map X and one or more convolution kernels K.

[0013] For a two-dimensional input feature map X and a convolution kernel K of size K×K, the convolution operation can be expressed as shown in the following formula;

[0014]

[0015] Where Y(i,j) is the value of the output feature map Y at position (i,j),

[0016] X(i+m,j+n) is the value at (i+m,j+n) on the input feature map X;

[0017] After the convolution operation, the activation function ReLU is applied. The mathematical representation of ReLU is as follows;

[0018] Z(i,j)=max(0,Y(i,j))

[0019] The pooling layer reduces the amount of computation and prevents overfitting by reducing the spatial dimensions of the feature map. The mathematical expression of maximum pooling is shown below.

[0020]

[0021] Where P is the output feature map after pooling,

[0022] Y is the input feature map, the pooling window moves to calculate the maximum value, and the stride is used to determine the next position of the feature map.

[0023] Transformer Encoder extracts global features of the image, allowing the model to focus on the features of all other positions in the sequence when processing each position in the input sequence, which can effectively capture long-range dependencies.

[0024] As a further illustration of the present invention, the Transformer Encoder architecture improves the model's ability to learn different representations by extending the self-attention mechanism to multiple "heads". The multi-head attention mechanism solves this problem by using multiple self-attention heads at the same time, each head independently calculates and extracts information from different subspaces;

[0025] After the input image is embedded and positionally encoded, the image features are passed as input to the Transformer Encoder encoder to calculate the relationship between all blocks through self-attention. The self-attention mechanism allows the model to learn the dependencies between small blocks in the image, thereby capturing global features. The output is then passed through a feedforward neural network for nonlinear transformation. The Transformer Encoder can stack multiple encoder layers to gradually extract higher-level features. After being processed by multiple Transformer Encoder layers, the final output represents the global features of the image.

[0026] After the model extracts local and global features of the input data in the first stage, it goes to the second stage to fuse local and global features. In this stage, local attention is used to enhance the global features extracted by Transformer Encoder. The computational complexity can be reduced and the extraction efficiency can be improved by limiting the scope of attention calculation. The local attention mechanism is that for each element in the input feature sequence, it is only allowed to interact with a fixed number of elements around the element. The size of this window can be fixed or dynamically changed according to demand. Then, for each element x in the input X, i , local attention only calculates the element x iThe attention of the elements within the context window can be expressed as follows;

[0027]

[0028] Where K and V are the keys and values ​​that only consider the context window.

[0029] In the second stage, the features of the Transformer Encoder branch are fused with the features of the CNN Encoder branch. Macroscopically, the information obtained from the Transformer Encoder branch is equivalent to a global receptive field acting on the Transformer Encoder branch, while the information obtained from the CNN Encoder branch is equivalent to acting on the CNN Encoder branch in a local receptive field. The two interacting features L and G are fused from the CNN Encoder branch and the Transformer Encoder branch respectively. The core idea is to control whether the interacting information is retained or lost. First, the features of the two branches are spliced, and the features of the two branches are spliced ​​according to the following formula.

[0030] X=Concat(L,G)

[0031] Use two multi-layer perceptron MLP layers to act on X respectively to obtain two features restored to the original dimensions, and then connect them in series for information aggregation;

[0032] These operations can generate a feature Z to guide accurate and adaptive selection, and then use the softmax function to learn a selectivity factor α, which can select appropriate features from local and global aggregate features. The formula is shown below.

[0033] X f =Softmax(Z)⊙X

[0034] It means multiplying each element of input X by the corresponding element of Z after being processed by the softmax function;

[0035] The ECA module in the multi-layer perceptron MLP is channel attention;

[0036] The third stage of the model is dynamic feature fusion. In the model, facial expression image data is the main data, and upper body posture recognition image data is supplemented. The feature calculation formula after fusion is as follows;

[0037] W=αX expression +βX posture

[0038] X expression The representative is the facial expression picture feature,

[0039] X posture The representative is the upper body posture picture feature,

[0040] α and β correspond to the weights of two different feature fusions, α∈[0.51], β∈(0.00.5]). The weights of the two features in the fusion process are dynamically changed according to the model recognition effect.

[0041] As a further illustration of the present invention, a music generation model MusicGen based on deep learning is used for music generation.

[0042] MusicGen uses a single-stage autoregressive Transformer structure. This means that when generating music, the model generates one note or sequence of notes at a time, predicts the next note by taking the previously generated note as input, and uses a 32kHz EnCodec tagger to convert the audio signal into a discrete code form for easy processing and generation. EnCodec brings more efficient data representation and supports more complex audio generation tasks. MusicGen applies 4 codebooks that are sampled at a frequency of 50Hz. Unlike some existing methods, MusicGen does not rely on self-supervised semantic representation, which makes it more efficient when generating music.

[0043] During the generation process, MusicGen demonstrated the ability to predict codebooks in parallel by introducing small delays between them. This strategy significantly increased the generation speed, resulting in only 50 autoregressive steps per second of audio, thus optimizing for real-time audio generation.

[0044] As a further illustration of the present invention, when the smart speaker is in use, a camera is used to capture a person's upper body posture and facial expression high-definition emotional picture as data input for the emotion recognition model, the image data is sent to the server via network transmission, and the image is pre-processed to obtain a feature vector;

[0045] The server uses the emotion recognition model to process feature vectors and perform facial expression recognition and emotion analysis. It can accurately identify different expressions, thereby extracting the user's expression keywords and providing emotion tags for subsequent music generation.

[0046] The server uses the music generation algorithm MusicGen to generate music clips with corresponding emotions based on the emotion tags output by the emotion recognition model, providing users with a personalized music experience and pushing music.

[0047] The control board sends a POST request through Python's requests library to quickly upload the captured video or image data to the server to achieve fast data transmission. After the server-side processing is completed, the generated music is transmitted back to the control board in real time, and the control board controls the first speaker and the second speaker to complete the music playback.

[0048] Compared with the prior art, the present invention has the following advantages:

[0049] 1. The emotion recognition model in the present invention uses parallel CNN Encoder and Transformer Encoder, which can well combine the global features and local features of the data, make full use of intra-modal and inter-modal information, and improve the accuracy of emotion recognition. After the emotion recognition model identifies the emotional state, the user can provide feedback by comparing the actual emotional state. The model automatically adjusts the feature fusion weight according to the feedback to form a personalized recognition algorithm suitable for the user.

[0050] 2. The emotion recognition model in the present invention can capture the user's emotions in real time through the camera and generate music according to the emotions. The emotion recognition model does not recognize a single emotion but recognizes mixed emotions. It can cleverly combine the user's facial expressions with music creation, providing users with a new cross-sensory experience and enhancing the fun and immersion of human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0052] Figure 2 is a specific process diagram of emotion recognition in an embodiment of the present invention;

[0053] Figure 3 is a schematic diagram of a Transformer Encoder in an embodiment of the present invention;

[0054] Figure 4 Schematic diagram of a feature fusion module for features of a CNN Encoder branch in an embodiment of the present invention;

[0055] Figure 5 is a schematic diagram of a multi-layer perceptron MLP in an embodiment of the present invention;

[0056] Figure 6 is a schematic diagram of an ECA module in an embodiment of the present invention;

[0057] Figure 7 is a composition diagram of the SE module in an embodiment of the present invention;

[0058] Figure 8 It is a schematic diagram of the control principle of the smart speaker in an embodiment of the present invention.

[0059] Description of reference numerals:

[0060] 1- Camera; 2- Control panel; 3- First speaker; 4- Digital tablet; 5- Speaker box; 6- Second speaker; 7- Power interface. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] like Figure 1-8 As shown, the present invention provides a technical solution: a smart speaker for generating and pushing music based on facial expressions, comprising a speaker box 5 and a control board 2 arranged in a camera 1, wherein the control board 2 is specifically a development board, and the development board is specifically a Raspberry Pi, a RISC-V development board, an Arduino or an FPGA;

[0063] The speaker box 5 is also provided with a camera 1, a first speaker 3, and a second speaker 6. The first speaker 3 and the second speaker 6 are respectively connected to the digital tablet 4. The digital tablet 4 is also provided with a digital tablet audio output interface, left and right channel volume knobs, a master volume knob, a subwoofer volume knob and a digital tablet power interface. The camera 1 and the digital tablet 4 are respectively connected to the control board 2 for signals. The speaker box 5 is also provided with a power interface 7, and power supply is completed through the power interface 7.

[0064] When performing emotion recognition, camera 1 is used to capture high-definition pictures of a person's upper body posture and facial expression as data input for the emotion recognition model. The model extracts two data features and fuses the features to identify the current user's emotional state.

[0065] The specific process of emotion recognition is as follows: Figure 2 As shown:

[0066] Camera 1 captures high-definition pictures of a person's upper body posture and facial expression as the data input for the emotion recognition model. In the first stage of the model, the two types of data are subjected to feature extraction using parallel CNN Encoder and Transformer Encoder.

[0067] The basic structure of CNN Encoder is convolution layer, activation layer, and pooling layer. The convolution layer extracts the local features of the input data. The output feature map Y of the convolution operation is obtained by the local weighted sum of the input feature map X and one or more convolution kernels K.

[0068] For a two-dimensional input feature map X and a convolution kernel K of size K×K, the convolution operation can be expressed as shown in the following formula;

[0069]

[0070] Where Y(i,j) is the value of the output feature map Y at position (i,j),

[0071] X(i+m,j+n) is the value at (i+m,j+n) on the input feature map X;

[0072] After the convolution operation, the activation function ReLU is applied. The mathematical representation of ReLU is as follows;

[0073] Z(i,j)=max(0,Y(i,j))

[0074] The pooling layer reduces the amount of computation and prevents overfitting by reducing the spatial dimensions of the feature map. The mathematical expression of maximum pooling is shown below.

[0075]

[0076] Where P is the output feature map after pooling,

[0077] Y is the input feature map, the pooling window moves to calculate the maximum value, and the stride is used to determine the next position of the feature map.

[0078] Transformer Encoder extracts global features of the image, allowing the model to focus on the features of all other positions in the sequence when processing each position in the input sequence, which can effectively capture long-range dependencies.

[0079] Unlike traditional RNN and LSTM, Transformer Encoder can process image features in parallel, thus significantly improving training speed. Figure 3 shown.

[0080] The Transformer Encoder architecture improves the model's ability to learn different representations by extending the self-attention mechanism to multiple "heads". The multi-head attention mechanism solves this problem by using multiple self-attention heads at the same time, each of which calculates independently and extracts information from different subspaces;

[0081] After embedding and position encoding the input image, the resulting vector is passed as input to the Transformer Encoder and the relationship between all blocks is calculated through the self-attention mechanism. The self-attention mechanism allows the model to learn the dependencies between small blocks in the image, thereby capturing global features. The output is then passed through a feedforward neural network for nonlinear transformation. The Transformer Encoder can stack multiple encoder layers to gradually extract higher-level features. After being processed by multiple Transformer Encoder layers, the final output represents the global features of the image.

[0082] After the model extracts local and global features of the input data in the first stage, it goes to the second stage to fuse local and global features. In this stage, the local attention mechanism is used to enhance the local features of the global features extracted by the Transformer Encoder. The computational complexity can be reduced and the extraction efficiency can be improved by limiting the scope of attention calculation. The local attention mechanism is that for each element in the input feature sequence, it is only allowed to interact with a fixed number of elements around the element. The size of this window can be fixed or dynamically changed according to demand. Then, for each element x in the input X, i , the local attention mechanism only calculates the i The attention of the elements within the context window can be expressed as follows;

[0083]

[0084] Where K and V are the keys and values ​​that only consider the context window.

[0085] In the second stage, the features of the Transformer Encoder branch are fused with the features of the CNN Encoder branch. Macroscopically, the information obtained from the Transformer Encoder branch is equivalent to a global receptive field acting on the Transformer Encoder branch, while the information obtained from the CNN Encoder branch is equivalent to acting on the CNN Encoder branch in a local receptive field. The two interacting features L and G are fused from the CNN Encoder branch and the Transformer Encoder branch respectively. The core idea is to control whether the interacting information is retained or lost. First, the features of the two branches are spliced, and the features of the two branches are spliced ​​according to the following formula.

[0086] X=Concat(L,G)

[0087] Use two multi-layer perceptron MLP layers to act on X respectively to obtain two features restored to the original dimensions, and then connect them in series for information aggregation;

[0088] These operations can generate a feature Z to guide accurate and adaptive selection, and then use the softmax function to learn a selectivity factor α, which can select appropriate features from local and global aggregate features. The formula is shown below.

[0089] X f =softmax(Z)⊙X

[0090] The schematic diagram of the multi-layer perceptron MLP is as follows Figure 5 shown.

[0091] The ECA module in the multi-layer perceptron MLP is channel attention; the schematic diagram of the ECA module is as follows Figure 6 As shown;

[0092] The full name of the ECA module is channel attention, which is an improved version of the SE module Squeeze-and-Excitation. The SE module is composed of Figure 7 As shown in the figure, the SE module is mainly composed of a global average pooling layer (GAP), a fully connected layer (FC) and a sigmoid function. The two FC layers are used to capture nonlinear cross-channel interaction information (Relu layers are counted in the FC layers), and the first FC layer is used to reduce the dimension to control the complexity of the model. The idea of ​​the ECA module is very simple. It removes the FC layer in the original SE module and directly learns on the features after GAP through a 1D convolution that can share weights. The 1D convolution involves the hyperparameter k, which is the convolution kernel size, which represents the coverage of local cross-channel interactions.

[0093] The third stage of the model is dynamic feature fusion. In the model, facial expression image data is the main data, and upper body posture recognition image data is supplemented. The feature calculation formula after fusion is as follows;

[0094] W=αX expression +βX posture

[0095] X expression The representative is the facial expression picture feature,

[0096] X posture The representative is the upper body posture picture feature,

[0097] α and β correspond to the weights of two different feature fusions, α∈[0.5 1] and β∈(0.0 0.5], respectively. The weights of the two features in the fusion process are dynamically changed according to the recognition effect of the model.

[0098] For music generation, we use the deep learning-based music generation model MusicGen.

[0099] MusicGen uses a single-stage autoregressive Transformer structure. This means that when generating music, the model generates one note or sequence of notes at a time, predicts the next note by taking the previously generated note as input, and uses a 32kHz EnCodec tagger to convert the audio signal into a discrete code form for easy processing and generation. EnCodec brings more efficient data representation and supports more complex audio generation tasks. MusicGen applies 4 codebooks that are sampled at a frequency of 50Hz. Unlike some existing methods, MusicGen does not rely on self-supervised semantic representation, which makes it more efficient when generating music.

[0100] During the generation process, MusicGen demonstrated the ability to predict codebooks in parallel by introducing small delays between them. This strategy significantly increased the generation speed, resulting in only 50 autoregressive steps per second of audio, thus optimizing for real-time audio generation.

[0101] The logic control of smart speakers is as follows Figure 8 As shown, when in use, the control board 2 accesses the camera 1 through OpenCV, uses the camera 1 to capture the upper body posture and facial expression of the person as the data input of the emotion recognition model, sends the image data to the server through network transmission, pre-processes the image to obtain the feature vector, and scans the pixels to obtain the classification features of the target image;

[0102] The server uses the emotion recognition model to perform facial expression recognition and sentiment analysis, which can accurately identify different expression features, thereby extracting the user's expression keywords and providing emotional support for subsequent music generation;

[0103] The server uses the music generation algorithm MusicGen to generate music clips with corresponding emotions based on the emotion keywords output by the emotion recognition model, providing users with a personalized music experience and pushing music.

[0104] The control board 2 sends a POST request through Python's requests library to quickly upload the captured video or image data to the server to achieve fast data transmission. After the server-side processing is completed, the generated music is transmitted back to the control board 2 in real time, and the control board 2 controls the first speaker 3 and the second speaker 6 to complete the music playback.

[0105] The entire process from turning on the speaker to the end of music playback is automated as much as possible to reduce the user's operating burden. At the same time, the system realizes key technologies and lightweight face recognition tracking, and the system can be placed on portable and mobile devices and carried around, optimizing data transmission and processing processes, ensuring low latency of the system, and improving user experience. The cloud server is used for music synthesis and expression recognition processing to achieve resource management and performance optimization, providing guarantees for the stable operation of the system. Through the above solutions, an intelligent digital music player system integrating image capture, emotion recognition, music generation and data communication functions is built to provide users with personalized and intelligent music experience.

[0106] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0107] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent speaker based on facial expression generation and music push, characterized by: The invention comprises a speaker box (5) and a control panel (2) arranged in the speaker box (5); the speaker box (5) is also provided with a camera (1), a first speaker (3), and a second speaker (6); the first speaker (3) and the second speaker (6) are respectively connected to a digital board (4); the digital board (4) is also provided with a digital board audio output interface, left and right channel volume knobs, a master volume knob, a subwoofer volume knob and a digital board power interface; the camera (1) and the digital board (4) are respectively connected to the control panel (2); the speaker box (5) is also provided with a power interface (7); and electric energy is supplied through the power interface (7).

2. The smart speaker based on facial expression generation and music push according to claim 1, characterized in that When performing emotion recognition, a camera (1) is used to capture high-definition images of a person's upper body posture and facial expression as data input for an emotion recognition model. The model extracts required data features and performs feature fusion to identify the current user's emotional state.

3. The smart speaker based on facial expression generation and music push according to claim 2, characterized in that: The specific process of emotion recognition is as follows: The camera (1) captures high-definition images of a person's upper body posture and facial expression as data input for the emotion recognition model. In the first stage of the model, features are extracted from these two types of data using parallel CNN Encoder and Transformer Encoder. The basic structure of CNN Encoder is convolution layer, activation layer and pooling layer. The convolution layer extracts the local features of the input data. The output feature map Y of the convolution operation is obtained by the local weighted sum of the input feature map X and one or more convolution kernels K. For a two-dimensional input feature map X and a convolution kernel K of size K×K, the convolution operation can be expressed as shown in the following formula; Where Y(i,j) is the value of the output feature map Y at position (i,j), X(i+m,j+n) is the value at (i+m,j+n) on the input feature map X; After the convolution operation, the activation function ReLU is applied. The mathematical representation of ReLU is as follows; Z(i,j)=max(0,Y(i,j)) The pooling layer reduces the amount of computation and prevents overfitting by reducing the spatial dimensions of the feature map. The mathematical expression of maximum pooling is as follows; Where P is the output feature map after pooling, Y is the input feature map, the pooling window moves to calculate the maximum value, and the stride is used to determine the next position of the feature map; Transformer Encoder extracts global features of the image, allowing the model to focus on the features of all other positions in the sequence when processing each position in the input sequence, which can effectively capture long-range dependencies.

4. The smart speaker based on facial expression generation and music push according to claim 3, characterized in that: The Transformer Encoder architecture improves the model's ability to learn different feature representations by extending the self-attention mechanism to multiple "heads". The multi-head attention mechanism solves this problem by using multiple self-attention heads at the same time, each of which calculates independently and extracts information from different subspaces; After embedding and position encoding the input image, the resulting vector is passed as input to TransformerEncoder and the relationship between all blocks is calculated through the self-attention mechanism. The self-attention mechanism allows the model to learn the dependencies between small blocks in the image, thereby capturing global features. The output is then passed through a feedforward neural network for nonlinear transformation. Transformer Encoder can stack multiple encoder layers to gradually extract higher-level features. After processing by multiple Transformer Encoder layers, the final output represents the global features of the image. After the model extracts local and global features of the input data in the first stage, it goes to the second stage to fuse local and global features. In this stage, the local attention mechanism is used to enhance the local features of the global features extracted by the Transformer Encoder. The computational complexity can be reduced and the extraction efficiency can be improved by limiting the scope of attention calculation. The local attention mechanism is that for each element in the input feature sequence, it is only allowed to interact with a fixed number of elements around the element. The size of this window can be fixed or dynamically changed according to demand. Then, for each element x in the input X, i , the local attention mechanism only calculates the i The attention of the elements within the context window can be expressed as follows; Where K and V are the keys and values ​​that only consider the context window; In the second stage, the features of the Transformer Encoder branch are fused with the features of the CNN Encoder branch. Macroscopically, the information obtained from the Transformer Encoder branch is equivalent to a global receptive field acting on the Transformer Encoder branch, while the information obtained from the CNN Encoder branch is equivalent to acting on the CNN Encoder branch in a local receptive field. The two interacting features L and G are fused from the CNN Encoder branch and the Transformer Encoder branch respectively. The core idea is to control whether the interacting information is retained or lost. First, the features of the two branches are spliced ​​according to the following formula; X=concat ( L,G ) Use two multi-layer perceptron layers to act on X respectively, obtain two features restored to the original dimension, and then connect them in series for information aggregation; These operations can generate a feature Z to guide accurate and adaptive selection, and then use the softmax function to learn a selectivity factor α, which can select appropriate features from local and global aggregate features. The formula is as follows; X f =softmax(Z)⊙X It means multiplying each element of input X by the corresponding element of Z after being processed by the softmax function; The third stage of the model is dynamic feature fusion. In the model, facial expression image data is the main data, and upper body posture recognition image data is supplemented. The feature calculation formula after fusion is as follows; W=αX expression +βX posture X expression The representative is the facial expression picture feature, X posture The representative is the upper body posture picture feature, α and β correspond to the weights of two different feature fusions, α∈[0.51], β∈(0 0.5], and the weights of the two features in the fusion process are dynamically changed according to the model recognition effect.

5. The smart speaker based on facial expression generation and music push according to claim 4, characterized in that: For music generation, the deep learning-based music generation model MusicGen is used. MusicGen uses a single-stage autoregressive Transformer structure. When generating music, the model generates one note or a sequence of notes at a time, and predicts the next note by using the previously generated note as input. A 32kHz EnCodec marker is used to convert the audio signal into a discrete code form for easy processing and generation. EnCodec brings more efficient data representation and supports more complex audio generation tasks. MusicGen applies 4 code books, which are sampled at a frequency of 50Hz. During the generation process, MusicGen introduces a delay between the code books, so that only 50 autoregressive steps are required per second of audio, thereby optimizing real-time audio generation.

6. The smart speaker based on facial expression generation and music push according to claim 5, characterized in that: When the smart speaker is in use, the camera (1) is used to capture a person's upper body posture and facial expression high-definition picture as data input for the emotion recognition model, and the image data is sent to the server through network transmission, and the image is pre-processed to obtain a feature vector; The server uses the emotion recognition model to process feature vectors and perform facial expression recognition and emotion analysis. It can accurately identify different expressions, thereby extracting the user's expression keywords and providing emotion tags for subsequent music generation. The server uses the music generation algorithm MusicGen to generate music clips with corresponding emotions based on the emotion tags output by the emotion recognition model, providing users with a personalized music experience and pushing music. The control board (2) sends a POST request through the Python requests library to quickly upload the captured image data to the server, thereby realizing rapid data transmission. After the server-side processing is completed, the generated music is transmitted back to the control board (2) in real time, and the control board (2) controls the first speaker (3) and the second speaker (6) to complete the music playback.

Images