Dynamic picture generation method and device based on intelligent loudspeaker box

Abstract

The invention provides a dynamic picture generation method and device based on an intelligent loudspeaker box, and is suitable for the technical field of data processing, and the method comprises the steps: obtaining initial video frame information, initial lyric information and audio information; performing multi-dimensional reconstruction processing on the initial video frame information to generate target image information; performing text detection and error correction processing on the initial lyric information to obtain target lyric information; and obtaining a dynamic picture according to the target image information, the target lyric information and the audio information, and displaying the dynamic picture through a display screen of the intelligent sound box. According to the method, the uploaded video frame image is subjected to multi-dimensional reconstruction, self-definition of the lyric background image is realized, the visual effect of the background image is enhanced, error detection and error correction are performed on the obtained lyrics, a user is ensured to follow the correct lyrics to sing, and the user experience is improved. And the multi-dimensional reconstructed image and the corrected lyrics are combined and then dynamically output through a display screen of the intelligent sound box, so that the user experience is comprehensively improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular to a method and device for generating dynamic images based on a smart speaker. Background Art

[0002] With the development of society, various smart homes have gradually entered people's daily lives. Smart speakers, as a smart home device that can play various audio content, can meet the needs of modern people to play music in various scenarios, enriching modern people's work and life. The types and functions of smart speakers are endless.

[0003] In existing smart speakers, the display screen is only used to display a fixed screen or only display lyrics content and time information. The fixed screen refers to the lyrics display background, which is usually determined by the default factory configuration of the smart speaker. Users cannot change the fixed screen during use, which limits the applicability of the smart speaker. In addition, the displayed lyrics information often has errors, which reduces user satisfaction during use. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a method and device for generating dynamic images based on a smart speaker, which multi-dimensionally reconstructs the video frames uploaded by the user, corrects the lyrics and displays them in combination with the multi-dimensionally reconstructed images, so as to solve the problems in the prior art of being unable to change the background of the lyrics display and errors in the display of lyrics, so as to enrich the application scenarios of the smart speaker and enhance the user's immersion and pleasure in the process of using the smart speaker.

[0005] A first aspect of an embodiment of the present application provides a method for generating a dynamic picture based on a smart speaker, comprising:

[0006] Obtaining initial video frame information, initial lyrics information, and audio information;

[0007] Performing multi-dimensional reconstruction processing on the initial video frame information to generate target image information;

[0008] Performing text detection and error correction processing on the initial lyrics information to obtain target lyrics information;

[0009] A dynamic picture is obtained according to the target image information, the target lyrics information and the audio information, so as to be displayed through the display screen of the smart speaker.

[0010] A second aspect of an embodiment of the present application provides a dynamic picture generation device based on a smart speaker, comprising:

[0011] An information acquisition module, used to acquire initial video frame information, initial lyrics information and audio information;

[0012] A target image information generating module, used for performing multi-dimensional reconstruction processing on the initial video frame information to generate target image information;

[0013] A target lyrics information generation module is used to perform text detection and error correction on the initial lyrics information to obtain target lyrics information;

[0014] A dynamic picture generation module is used to obtain a dynamic picture based on the target image information, the target lyrics information and the audio information, so as to display it through the display screen of the smart speaker.

[0015] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the present application realizes the customization of the lyrics background by multi-dimensionally reconstructing the video frames uploaded by the user and using the multi-dimensionally reconstructed images as the lyrics background for the display screen to display. At the same time, semantic detection is performed on the acquired lyrics information, and erroneous lyrics can be corrected to ensure that the correct lyrics are displayed for the user to sing in real time, thereby increasing the application scenarios of the smart speaker and effectively enhancing the user's sense of immersion and pleasure during use. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0017] Figure 1 This is a schematic diagram of the implementation process of the method for generating a dynamic picture based on a smart speaker provided in the first embodiment of the present application;

[0018] Figure 2 This is a schematic diagram of the implementation process of the method for generating a dynamic picture based on a smart speaker provided in the second embodiment of the present application;

[0019] Figure 3 This is a schematic diagram of the implementation process of the method for generating a dynamic picture based on a smart speaker provided in Embodiment 3 of the present application;

[0020] Figure 4 This is a schematic diagram of the implementation process of the method for generating a dynamic picture based on a smart speaker provided in the fourth embodiment of the present application;

[0021] Figure 5 This is a schematic diagram of the implementation process of the method for generating a dynamic picture based on a smart speaker provided in Embodiment 5 of the present application;

[0022] Figure 6This is a schematic diagram of the implementation process of the method for generating a dynamic picture based on a smart speaker provided in Example 6 of the present application;

[0023] Figure 7 This is a schematic diagram of the implementation process of the method for generating a dynamic picture based on a smart speaker provided in Embodiment 7 of the present application;

[0024] Figure 8 This is a schematic diagram of the implementation process of the method for generating a dynamic picture based on a smart speaker provided in Embodiment 8 of the present application;

[0025] Fig. 9 This is a schematic diagram of the implementation process of the method for generating a dynamic picture based on a smart speaker provided in Embodiment 9 of the present application;

[0026] Fig.10 It is a structural schematic diagram of a dynamic picture generating device based on a smart speaker provided in an embodiment of the present application; DETAILED DESCRIPTION

[0027] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0028] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0029] Figure 1 The following is a flowchart of the implementation of the method for generating a dynamic picture based on a smart speaker provided in the first embodiment of the present application, which is described in detail as follows:

[0030] Step S101, obtaining initial video frame information, initial lyrics information and audio information.

[0031] In this embodiment, the initial video frame information can be uploaded by the user through a specific APP, that is, by establishing a network transmission between the mobile phone and the specific APP in the mobile phone, the server, and the smart speaker, the user can upload the picture or video file on the mobile phone to the smart speaker for display, so that the smart speaker can realize the function of dynamic scene customization of lyrics. The initial lyrics information can be obtained from the music software, or it can be uploaded by the user through a specific APP. The audio information can be obtained by real-time recording of the user's singing audio, or it can be obtained by playing an existing audio file on the Internet.

[0032] Step S102, performing multi-dimensional reconstruction processing on the initial video frame information to generate target image information.

[0033] In this embodiment, the multi-dimensional reconstruction processing of the initial video frame information refers to converting the two-dimensional video frame into a multi-dimensional image, which is the target image information, so that the user can intuitively observe the image content in different visual directions to present a multi-dimensional display effect. The video frame image uploaded by the user through the APP can be multi-dimensionally reconstructed by calling the function in OpenCV, or by calling the point cloud library, or by multi-dimensionally processing the video frame in the form of a neural radiation field.

[0034] Step S103, performing text detection and error correction processing on the initial lyrics information to obtain target lyrics information.

[0035] In this embodiment, text correction is performed on the initial lyric information by comparing the initial lyric information one by one with the lyric library on the Internet to detect and correct erroneous lyric texts, or by performing semantic analysis on the initial lyric information to avoid misdetection of the initial lyric information due to typos in the lyric library. For lyrics uploaded by users through a designated APP, users can also perform lyric text detection and correction on the APP. The lyrics that have undergone text error detection and text correction, i.e., the target lyrics, are displayed on the display screen of the smart speaker to prompt the user to sing according to the lyrics.

[0036] Step S104, obtaining a dynamic picture according to the target image information, the target lyrics information and the audio information, so as to display it through the display screen of the smart speaker.

[0037] In this embodiment, the target image information and the target lyrics information can be directly synthesized into a picture for real-time display on the display screen of the smart speaker. Among them, the audio information can be imported into a multi-layer perceptron or a long short-term memory network or a CNN model to perform emotional analysis through the frequency and pitch changes of the sound, so as to adjust the display content and switching frequency of the target image information according to the emotional analysis results, and realize the coordinated output of audio and picture, so as to provide a more suitable picture display for the user's emotions and better meet the user's emotional needs; it can also be through the conversion of audio information into text, and then the converted text is matched with the target lyrics content, so as to identify the lyrics that the user has sung, not sung, missed and sung wrongly, and display various lyrics in different forms during the singing process, so as to provide timely lyrics reminders for users in the use process, and at the same time, it can identify the "pause singing" and "switch songs" and other instructions in the audio information, and can automatically adjust the displayed dynamic picture in time based on the user's instructions.

[0038] The dynamic picture generation method based on the smart speaker provided in the embodiment of the present application realizes the customization of the lyrics background by multi-dimensionally reconstructing the video frames uploaded by the user, and using the multi-dimensionally reconstructed image as the lyrics background for the display screen to display. At the same time, the acquired lyrics information is semantically detected, and erroneous lyrics can be corrected to ensure that the correct lyrics are displayed for the user to sing in real time, thereby increasing the application scenarios of the smart speaker and effectively enhancing the user's immersion and pleasure during use.

[0039] Figure 2 The flowchart of the implementation of the method for generating a dynamic picture based on a smart speaker provided in the second embodiment of the present application is shown. The difference between the method and the first embodiment is that the step S102 specifically includes:

[0040] Step S201, obtaining a plurality of video frame pixel coordinates and multi-dimensional display position coordinates.

[0041] In this embodiment, the video frame pixel coordinates may be two-dimensional coordinates of all pixels in all video frame information extracted from the initial video frame information, and for the pixel coordinates in a single video frame, rectangular coordinates or polar coordinates may be extracted. Multi-dimensional display position coordinates refer to coordinates used to represent the display range of a multi-dimensional image, and may be three-dimensional coordinates or two-dimensional coordinates observed from different viewing angles.

[0042] In this embodiment, the polar coordinates of all video frame pixels may be optionally Combined with the three-dimensional coordinates (a, b, c) used to represent the display range of the multi-dimensional image into a five-dimensional coordinate Represent it, and substitute this five-dimensional coordinate representation into subsequent calculations. Among them, ω represents the polar radius and polar angle of the coordinates of the video frame pixel; a, b, c represent the coordinates of three dimensions in the display range of the multi-dimensional image, which are used to limit the display range of the multi-dimensional image and can be rectangular coordinates.

[0043] Step S202 , performing position coding processing on the video frame pixel coordinates and the multi-dimensional display position coordinates, establishing a corresponding relationship between the video frame pixel coordinates and the multi-dimensional display position coordinates, and obtaining video frame mapping coordinates and multi-dimensional display position mapping coordinates.

[0044] In this embodiment, because the size of the video frame may not be able to directly match the size of the smart speaker display screen, and the two-dimensional display of the video frame cannot directly adapt to the display pixels and display size of the multi-dimensional image, it is necessary to position encode the video frame pixel coordinates and the multi-dimensional display position coordinates, which are used to map the video frame pixel coordinates and the multi-dimensional display position coordinates to the same dimension for matching and subsequent calculations. The video frame pixel coordinates and the multi-dimensional display position coordinates can be simultaneously mapped to a high-dimensional space to facilitate better fitting of dynamically changing data. For the position encoding processing of the video frame pixel coordinates and the multi-dimensional display position coordinates, it can be performed by simultaneously importing the video frame pixel coordinates and the multi-dimensional display position coordinates into the Transformer model for position encoding processing, or it can be performed by simultaneously encoding the video frame pixel coordinates and the multi-dimensional display position coordinates using a rotation position encoding method. The coordinates calculated after the encoding processing of the video frame pixel coordinates are the video frame mapping coordinates, and the coordinates calculated after the encoding processing of the multi-dimensional display position coordinates are the multi-dimensional display position mapping coordinates.

[0045] In this embodiment, the polar coordinates of all video frame pixels can be optionally combined with the three-dimensional coordinates used to represent the display range of the multi-dimensional image to form a five-dimensional coordinate system. Import the Transformer model for position encoding, or convert (a, b, c) and Import the Transformer model for position encoding processing to obtain the video frame mapping coordinates δ(a, b, c) and the multi-dimensional display position mapping coordinates Where δ() represents the position encoding mapping function, which is used to represent the five-dimensional coordinates After layer-by-layer calculation of the Transformer model, the layer-by-layer calculation may be weighted summed with the layer-by-layer weights of the Transformer model, wherein the layer-by-layer weights of the Transformer model are determined after training.

[0046] In this embodiment, the five-dimensional coordinates may be optionally The Transformer model is used to calculate the position encoding, and the five-dimensional coordinates can also be After feature fusion or aggregation, the coordinate feature ε after feature fusion or aggregation is input into the Transformer model for position encoding processing. The formula for layer-by-layer encoding processing of the fused or aggregated coordinate feature ε through the Transformer model can be expressed as:

[0047] δ(ε)=(sin(2 0 πε),cos(2 0 πε),…,sin(2 K-1πε),cos(2 K-1 πε))

[0048] Among them, δ(ε) represents the position encoding mapping coordinates of the fused or aggregated features, K represents the number of encoding layers and decoding layers in the Transformer model, and then δ(ε) is converted into video frame mapping coordinates δ(a, b, c) and multi-dimensional display position mapping coordinates through dimensional decomposition.

[0049] Step S203 , performing nonlinear processing on the video frame mapping coordinates and the multi-dimensional display position mapping coordinates to obtain viewing angle display features.

[0050] In this embodiment, nonlinear calculation is introduced into the coordinate mapping calculation to extract features from the video frame mapping coordinates and the multi-dimensional display position mapping coordinates, and then the extracted features are analyzed and processed to obtain the viewing angle display features. Among them, the nonlinear calculation can be to use the video frame mapping coordinates and the multi-dimensional display position mapping coordinates as input data of the fully connected layer, and perform nonlinear calculation through the fully connected layer; it can also be to use the video frame mapping coordinates and the multi-dimensional display position mapping coordinates as input data of the residual network, and perform nonlinear calculation through the residual network; it can also be to use the video frame mapping coordinates and the multi-dimensional display position mapping coordinates as input data of the multi-layer perceptron, and perform nonlinear calculation through the multi-layer perceptron; it can also be to use the video frame mapping coordinates and the multi-dimensional display position mapping coordinates as input data of the convolutional neural network, and perform nonlinear calculation through the convolutional neural network, so that the calculation result is used as the viewing angle display feature, and the viewing angle display feature is used to display the pixel information on the multi-dimensional display position coordinates.

[0051] In this embodiment, optionally, the video frame mapping coordinates δ(a, b, c) are input into a fully connected layer having m layers, each layer having n channels, and each layer having an activation function for calculation, where m and n are both integers not less than 1, to obtain the intermediate video frame mapping coordinates Used to represent the three-dimensional coordinate variables generated during the calculation process, and then map the coordinates of the intermediate video frames Display location with multi-dimensional map coordinates Splice to get the intermediate five-dimensional mapping variable Then The input is input into a fully connected layer with x layers, y channels in each layer, and no activation function in each layer for calculation and feature fusion to obtain the viewing angle display feature R (μ, θ).

[0052] In this embodiment, optionally, m can be 8, n can be 256, x can be 5, and y can be 128.

[0053] Step S204: generating target image information according to the viewing angle display feature and the multi-dimensional display position coordinates.

[0054] In this embodiment, the viewing angle display feature can be used to directly generate an image, which can be to render the multi-dimensional display position coordinates according to the viewing angle display feature, or the coordinates of the viewing angle display feature can be conventionally transformed for rendering the image on the multi-dimensional display position coordinates, thereby obtaining the target image information, so that when the user observes the image from different viewing angles, he can see the image content with different contents displayed at the display positions of different viewing angles, so as to present the visual effect of a multi-dimensional picture. The target image refers to multiple two-dimensional images, that is, multiple two-dimensional images of different contents observed by the user in different visual directions.

[0055] In this embodiment, optionally, a plurality of two-dimensional viewing angle display features R (μ, θ) are used to represent the pixel display content of the two-dimensional image observed at each viewing angle, so that the pixel content represented by the viewing angle display feature R (μ, θ) is rendered to the multi-dimensional display position mapping coordinates. This enables different two-dimensional images to be observed from different viewing angles, creating a multi-dimensional display effect.

[0056] The dynamic picture generation method based on the smart speaker provided in the embodiment of the present application introduces nonlinear calculation to extract features of the video frame images uploaded by the user, so as to ensure that the visual impact points in the pixels of the video frame images can be accurately extracted and can display unique visual effects after multi-dimensional reconstruction, and map the extracted coordinate features and the display range coordinates of the display screen to ensure that the multi-dimensional display surface of the display screen is sufficient to present the image content of the video frame, thereby avoiding the problem of black edges or incomplete display caused by the mismatch between the display range of the display screen and the size of the image, and rendering the mapped image features to the display range coordinates, and generating different two-dimensional images for display in different dimensions, so that the user can see different picture contents from different perspectives during use, presenting the visual effect of multi-dimensional image display, which is convenient for setting off the atmosphere of the singing scene and increasing the sense of immersion and comfort.

[0057] Figure 3 The flowchart of the implementation of the method for generating a dynamic picture based on a smart speaker provided in the third embodiment of the present application is shown. The difference between the method and the second embodiment is that the viewing angle display feature includes a volume density feature and a color representation feature; the step S203 specifically includes:

[0058] Step S301 : performing nonlinear transformation on the multidimensional display position mapping coordinates according to the multidimensional display position mapping coordinates and a preset first nonlinear function to obtain a first display position representation feature.

[0059] In this embodiment, the preset first nonlinear function refers to a function whose derivative is not zero for introducing a nonlinear factor into the calculation of the multidimensional display position mapping coordinates, and is used to extract and enhance the features of the multidimensional display position mapping coordinates. The nonlinear conversion may be to perform multiple rounds of weighted summation processing on the multidimensional display position mapping coordinates through the first nonlinear function, so as to achieve multiple rounds of extraction and enhancement of the multidimensional display position mapping coordinates, ensure that important features in the multidimensional display position mapping coordinates are not omitted, and the final output calculation result is the first display position representation feature, which is used to represent the output feature result obtained after multiple rounds of feature extraction and feature enhancement of the multidimensional display position mapping coordinates, so as to facilitate subsequent matching with the perspective display feature to form a target image with rich visual effects.

[0060] In this embodiment, optionally, the preset first nonlinear function may be a ReLU function, a Sigmoid function, or a Tanh function.

[0061] Step S302: Perform feature enhancement processing on the first display position representation feature to obtain a second display position representation feature and volume density features corresponding to each multi-dimensional display position mapping coordinate.

[0062] In this embodiment, the feature enhancement processing performed on the first display position representation feature is used to re-screen and classify the data in the first display position representation feature, screen out more important feature data, and calculate various important data separately. The method of feature enhancement processing can be to calculate through a reinforcement learning model to achieve feature enhancement and feature selection, or to readjust the pixel coordinates of the first display position representation feature through histogram equalization, so as to facilitate the subsequent matching calculation with the viewing angle display feature to present a more prominent visual effect, or to use the first display position representation feature as an input of a fully connected layer, and perform weighted sum calculation on the first display position representation feature through the fully connected layer, and the output result is the second display position representation feature and the volume density feature. Among them, the second display position representation feature refers to the feature that limits the position of the multi-dimensional display range of the display screen, that is, the feature that represents the display position or display range, and the volume density feature refers to the feature that represents the opacity corresponding to the specific display position of the display screen. The smaller the volume density, the more transparent the display position.

[0063] In this embodiment, optionally, the first display position representation feature is used as the input of a fully connected layer with a multi-layer structure, 256 channels in each layer and no activation function, and the output result after calculation by the fully connected layer is a plurality of second display position representation features with a dimension of 256, and a plurality of volume density features corresponding one-to-one to the second display position representation features, wherein the volume density feature is used to represent the opacity of the second display position representation feature.

[0064] Step S303: splicing the video frame mapping coordinates with the second display position representation feature to obtain a third display position representation feature.

[0065] In this embodiment, the video frame mapping coordinates and the second display position representation feature are spliced, which can be understood as converting the video frame mapping coordinates and the second display position representation feature into arrays first, and splicing the arrays, which can be directly spliced, or merged using a recursive method, or the second display position representation feature is disassembled and added to the video frame mapping coordinates by a loop traversal method. The array after splicing is converted into a vector, and the obtained vector is the third display position representation feature.

[0066] Step S304: performing nonlinear transformation on the third display position representation feature according to the third display position representation feature and a preset second nonlinear function to obtain a color representation feature.

[0067] In this embodiment, the preset second nonlinear function refers to a function whose derivative is not zero for introducing a nonlinear factor into the calculation of the third display position representation feature, and is used to extract and enhance the features of the third display position representation feature. The nonlinear conversion can be a multi-round weighted summation process of the third display position representation feature by the second nonlinear function, so as to achieve multi-round extraction and enhancement of the third display position representation feature, ensuring that important features in the third display position representation feature are not omitted, and the final output calculation result is the color representation feature, which is used to represent the color of each pixel in the multidimensional image, so as to facilitate the subsequent matching with the position display feature to form a target image with rich visual effects.

[0068] In this embodiment, optionally, the preset second nonlinear function may be a ReLU function, a Sigmoid function, or a Tanh function.

[0069] In this embodiment, optionally, the third display position representation feature is calculated as a fully connected layer with a fixed number of layers and 128 channels in each layer, and the activation function of each layer is the ReLU function, that is, the second nonlinear function. Through layer-by-layer weighted summation calculation, the nonlinear transformation of the third display position representation feature is realized, and the features in the third display position representation feature are fully extracted to obtain the color representation feature, which can be used to render the display position representation coordinates, thereby generating a multi-dimensional image for display.

[0070] The dynamic picture generation method based on the smart speaker provided in the embodiment of the present application introduces nonlinear factors to fully extract and enhance the features of the video frame image, so as to obtain the color information and opacity information of the video frame image, so as to facilitate the subsequent matching of the extracted and enhanced color information and opacity information with the multi-dimensional display position coordinates of the display screen, thereby generating a multi-dimensional image with a stereoscopic visual effect on the display screen as a dynamically displayed singing background, which is used to set off the singing atmosphere and enhance the user's immersion and pleasure in the process of using the smart speaker.

[0071] Figure 4 The flowchart of the implementation of the method for generating a dynamic picture based on a smart speaker provided in the fourth embodiment of the present application is shown. The difference between the method and the third embodiment is that the step S204 specifically includes:

[0072] Step S401, decomposing and calculating the color representation feature to obtain a hue feature, a saturation feature, and a brightness feature.

[0073] In this embodiment, the color representation feature can be decomposed through the HSI model, and the color representation feature can be transformed from a unit cube based on a Cartesian rectangular coordinate system to a double cone based on a cylindrical polar coordinate system, and the brightness factor in the color representation feature can be separated. The separated brightness factor is the brightness feature, and the chromaticity factor is decomposed into hue and saturation, namely, the hue feature and the saturation feature.

[0074] Step S402, enhancing the saturation feature and the brightness feature to obtain a visual enhancement feature.

[0075] In this embodiment, the saturation feature and the brightness feature can be enhanced by adding interpolation values ​​or extrapolation values ​​respectively, or Gaussian noise can be added to the saturation feature and the brightness feature respectively to increase the diversity of the features and generate visual enhancement features to achieve feature enhancement effects, thereby improving the visual expression of the image.

[0076] Step S403: Rendering the multi-dimensional display position coordinates according to the color tone feature, the visual enhancement feature and the volume density to generate target image information.

[0077] In this embodiment, a volume rendering operation is performed on the multi-dimensional display position coordinates according to the hue feature, the visual enhancement feature and the volume density, wherein the volume density α(τ) can represent the differential probability that the ray perpendicular to the two-dimensional plane of the display generated based on the multi-dimensional display position coordinates terminates at an infinitesimal particle, and the visual ray β(t) from the observation angle is equal to o+td. Where o represents the visual starting point, and d represents the position distance of the display screen in the visual sense. The visual ray is at the boundary t of the screen display range. g and t f The visualization color of the presentation can be quantified using the color expectation value C(β):

[0078]

[0079] Among them, by calculating the integral for drawing the view for the continuous multi-dimensional display position coordinates, the color expected value C(β) is obtained as the value for quantizing the color displayed when drawing the view, and the function T(t) represents the ray perpendicular to the two-dimensional plane of the display generated based on the multi-dimensional display position coordinates from the boundary t of the screen display range g The cumulative transmittance along the ray from t to t is g The probability of reaching t without hitting other particles, that is, the probability of being projected directly without being blocked by other particles. g In the opposite direction, the ray from t g On the way to t, it passes through the other end boundary t f , the cumulative volume density The larger the value is, the smaller the cumulative transmittance T(t), and the lower the influence of factors such as occlusion on the color of the position, indicating that the higher the volume density α(β(t) of the current rendering area is, the greater the influence on the color is, but the pixels rendered immediately afterwards are insensitive to the influence of color, thereby highlighting the visual effect of color, increasing the contrast between the colors of the presented multi-dimensional image, and making the display effect more three-dimensional and rich.

[0080] The dynamic picture generation method based on the smart speaker provided in the embodiment of the present application decomposes the colors, adjusts the saturation and brightness of the image, improves the display contrast between colors, and combines the adjusted saturation and brightness to render the multi-dimensional display coordinates, thereby highlighting the visual effect of the color and making the generated multi-dimensional image more three-dimensional and rich. When the user uses the smart speaker to sing, the effect of the image on the singing atmosphere is enhanced, the display effect of the singing scene is enhanced, and the user experience is improved.

[0081] Figure 5 The flowchart of the implementation of the method for generating a dynamic picture based on a smart speaker provided in the fifth embodiment of the present application is shown. The difference between the method and the first embodiment is that the step S103 specifically includes:

[0082] Step S501, obtaining a lyrics text vector according to the initial lyrics information.

[0083] In this embodiment, the initial lyrics information can be converted into lyrics text vectors using methods such as One-hot encoding, TF-IDF, Word2Vec, Doc2Vec, BERT, ELMo, GPT, FastText, and GloVe.

[0084] Step S502, extracting text information from a preset lyrics library to obtain standard lyrics sentence information.

[0085] In this embodiment, the preset lyrics library is composed of various lyrics collected from the Internet. The lyrics library can be obtained directly from the Internet, or it can be formed by summarizing lyrics crawled from the Internet. The lyrics text information in the lyrics library includes text content information and its text content arrangement order information and context association information. The text content information extracted from the lyrics library can be used to automatically parse the text content arrangement order information and context association information contained in the text content information as standard lyrics sentence information for subsequent semantic judgment of the initial lyrics information, so as to ensure that the lyrics displayed on the smart speaker display screen are semantically correct and free of typos.

[0086] Step S503, obtaining a standard lyrics sentence vector according to the standard lyrics sentence information.

[0087] In this embodiment, One-hot encoding, TF-IDF, Word2Vec, Doc2Vec, BERT, ELMo, GPT and other methods can be used to convert standard lyrics sentence information into standard lyrics sentence vectors for subsequent analysis and calculation.

[0088] Step S504, analyzing and calculating the lyrics text vector and the standard lyrics sentence vector to obtain a distance measurement value.

[0089] In this embodiment, the distance measurement value can be obtained by directly calculating the Euclidean distance between two vectors. However, because directly calculating the Euclidean distance can easily lead to overly extreme semantic judgments, a word may have multiple correct spellings, but direct calculation of the Euclidean distance can easily lead to misjudgment of lyrics texts with multiple correct spellings. Therefore, the lyrics text vector and the standard lyrics sentence vector can also be used as input data of a specific neural network at the same time, and calculations can be performed through the specific neural network. The loss function obtained in the calculation process is used as the distance function value. Other measurement functions can also be introduced as reference values ​​to calculate the distance value between the lyrics text vector and the standard lyrics sentence vector relative to the reference value, and the difference between the two distance values ​​relative to the reference value is used as the distance measurement value between the lyrics text vector and the standard lyrics sentence vector.

[0090] In this embodiment, optionally, the specific neural network may be a convolutional neural network, a multi-layer perceptron, or a BP neural network.

[0091] Step S505, determining whether the distance measurement value is less than a preset distance threshold, if so, proceeding to step S506, if not, proceeding to step S507.

[0092] In this embodiment, the semantic consistency between the initial lyric information and the lyrics in the lyric library is quantified by the distance metric value, so that the semantic correctness of the initial lyric information is determined by judging the size relationship between the distance metric value and the preset distance threshold. The preset distance threshold can be set manually. When the distance metric value is less than the preset distance threshold, it means that the semantic consistency between the initial lyric information and the lyrics information in the lyric library is high, and the semantic correctness of the initial lyric information is determined; when the distance function value is greater than the preset distance threshold, it means that the semantic consistency between the initial lyric information and the lyrics information in the lyric library is low, and the semantic error of the initial lyric information is determined, indicating that there may be typos, word order reversals, or spelling errors in the lyrics.

[0093] Step S506: determining the initial lyrics information as the target lyrics information.

[0094] In this embodiment, when the distance measurement value is less than the preset distance threshold, it means that the initial lyric information has a high degree of semantic consistency with the lyrics in the lyric library, and the initial lyric information is judged to be semantically correct. The semantically correct initial lyric information is used as the target lyric information, and the dynamic image can be directly synthesized and displayed on the display screen of the smart speaker.

[0095] Step S507, performing error correction processing on the initial lyrics information to obtain target lyrics information.

[0096] In this embodiment, when the distance metric value is greater than the preset distance threshold, it means that the initial lyrics information has a low degree of semantic consistency with the lyrics in the lyrics library, and the initial lyrics information is determined to be semantically incorrect. The semantically incorrect initial lyrics information is regarded as incorrect lyrics and cannot be directly used to generate dynamic images for display by the smart speaker. If it is to be used for display by the smart speaker, the incorrect lyrics must be corrected. The corrected semantically correct lyrics can be used to generate dynamic images for display by the smart speaker. The correction of incorrect lyrics can be to locate the error position by automatic detection, and then replace the error position with the correct lyrics. The correct lyrics can be provided by the lyrics library or uploaded by the user through a specific APP.

[0097] The method for generating dynamic images based on a smart speaker provided in an embodiment of the present application extracts lyric text information from a lyric library and performs semantic analysis on the initial lyric information, calculates a distance metric value, and quantifies the degree of matching between the initial lyric information and the lyrics in the lyric library, thereby judging the correctness of the semantics of the initial lyric information, and then determining the erroneous lyrics in the initial lyric information, which is used to judge whether the initial lyric information can be directly used for generating dynamic images and for display on the display screen of the smart speaker, thereby ensuring the accuracy and effectiveness of the lyric text used for display on the smart speaker, ensuring that the user can sing according to the correct lyric prompts during the singing process, and improving the smoothness and satisfaction of the user during use.

[0098] Figure 6 The flowchart of the implementation of the method for generating a dynamic picture based on a smart speaker provided in the sixth embodiment of the present application is shown. The difference between the method and the fifth embodiment is that the step S504 specifically includes:

[0099] Step S601, obtaining a query vector, a key vector and a value vector according to the lyric text vector and a preset query weight matrix, a key weight matrix and a value weight matrix.

[0100] In this embodiment, the preset query weight matrix, key weight matrix and value weight matrix can be manually set. The lyrics text vector is respectively inner-producted with the preset query weight matrix, the preset key weight matrix and the preset value weight matrix to extract the text features in the initial lyrics information in multiple dimensions or multiple scales, and the vectors after the operation are respectively the query vector, the key vector and the value vector.

[0101] Step S602: multiply the query vector by the key vector to obtain a sentence similarity weight matrix.

[0102] In this embodiment, the query vector is multiplied by the key vector to calculate a sentence similarity matrix, and the numerical value in the sentence similarity matrix represents the similarity between the key vector and the query vector, which is used to assign weights to the semantic features in the initial lyric information according to the similarity between the key vector and the query vector, and is used to quantify the importance of the semantic features in the initial lyric information, and then the numerical values ​​in the sentence similarity matrix are normalized to limit the numerical values ​​in the sentence similarity matrix to the range of [0, 1], and the sentence similarity weight matrix is ​​calculated to simplify calculations, unify the statistical distribution of data, improve convergence speed and accuracy, and eliminate the influence of dimensions. The normalization method can be root mean square normalization, standard deviation normalization, or linear normalization.

[0103] Step S603: multiply the sentence similarity weight matrix by the value vector to obtain a sentence weight value matrix.

[0104] In this embodiment, the similarity weight matrix is ​​multiplied by the value vector to calculate the sentence weight value matrix, which is used to assign weights to the semantic features in the initial lyric information according to the similarity between the key vector and the query vector, thereby amplifying the semantic features with high importance in the initial lyric information, making the semantic feature extraction of the initial lyric information more accurate and effective, and ensuring that the obtained semantic features have stronger reference and effectiveness.

[0105] Step S604, performing nonlinear transformation on the lyrics text vector to obtain a nonlinear lyrics feature vector.

[0106] In this embodiment, the lyric text vector can be nonlinearly transformed through the ReLU function, introducing nonlinear factors into the overall calculation, which is used to map the semantic features in the initial lyric information to the nonlinear space, so that the computing network can adapt to the data distribution and decision boundaries in the initial lyric information, thereby enhancing the network's representation ability and ensuring that more complex semantic features in the initial lyric information can be calculated.

[0107] Step S605, summing the lyrics text vector and the nonlinear lyrics feature vector to obtain a lyrics weight connection matrix.

[0108] In this embodiment, the lyrics text vector is summed with the nonlinear lyrics feature vector to obtain a lyrics weight connection matrix. The lyrics weight connection matrix is ​​used to avoid the lyrics text vector from being directly operated with the nonlinear lyrics feature vector, to avoid the gradient vanishing problem caused by the calculated value exceeding the range of the data type during the calculation process, to enhance the performance and stability of the computing network, and to improve the accuracy of semantic analysis of the lyrics.

[0109] Step S606: Perform weighted summation on the lyrics weight connection matrix and the sentence weight value matrix to obtain a first intermediate lyrics vector.

[0110] In this embodiment, the lyrics weight connection matrix and the sentence weight value matrix are weighted and summed to obtain a first intermediate lyrics vector, which is used to avoid the lyrics text vector being directly operated with the sentence weight value matrix so that the weight value is too large and the weight change exceeds the expected range, thereby avoiding the consequences of data loss or error due to the calculation result exceeding the range that can be represented by the data type, so as to solve the problem of gradient explosion and promote the rapid convergence of the computing network.

[0111] Step S607: The first intermediate lyrics vector may be subjected to RMS normalization calculation to obtain a second intermediate lyrics vector.

[0112] In this embodiment, the first intermediate lyrics vector is subjected to RMS normalization calculation to obtain the second intermediate lyrics vector, which is used to avoid the lyrics text vector from being directly operated with the sentence weight value matrix, solve the problem of gradient disappearance, and enable the computing network to maintain effective learning and training at a deeper level, thereby improving the performance and stability of the computing network.

[0113] Step S608, training a preset lyrics text processing model according to the standard lyrics sentence vector to obtain a plurality of lyrics feature representation vectors.

[0114] In this embodiment, the preset lyrics text processing model can be trained by using the standard lyrics sentence vector to ensure that the lyrics text processing model has learned the semantic features in the standard lyrics sentence vector, thereby obtaining a trained lyrics text processing model and determining the feature representation vector of the lyrics text processing model for semantic processing and analysis calculation of the initial lyrics information.

[0115] In this embodiment, the lyrics text processing model can be an MLP model, a BERT model, a transformer model, or a GPT-3 model.

[0116] Step S609: performing weighted summation on the second intermediate lyrics vector and a plurality of the lyrics feature representation vectors to obtain a third intermediate lyrics vector.

[0117] In this embodiment, the second intermediate lyrics vector is weightedly summed with multiple lyrics feature representation vectors to obtain a third intermediate lyrics vector, which is used to extract features from the second intermediate lyrics vector and to enhance the contrast of the importance of semantic features in the second intermediate lyrics vector, thereby achieving nonlinear feature extraction of the initial lyrics information.

[0118] Step S610, calculating the Euclidean distance between the third intermediate lyrics vector and the lyrics text vector to obtain a distance metric value.

[0119] In this embodiment, the Euclidean distance between the third intermediate lyrics vector and the lyrics text vector is calculated to quantify the similarity or difference between the two vectors, which may be a sine distance or a cosine distance.

[0120] The dynamic picture generation method based on the smart speaker provided in the embodiment of the present application performs a nonlinear transformation on the lyrics text information, thereby mapping the semantic features in the lyrics text information to a nonlinear space, so that the semantic features in the lyrics can be represented and used for calculations, and the calculated values ​​are limited to a limited range by connecting the matrix, thereby avoiding the situation where the calculation results generated during the calculation process exceed the range that the data type can represent and cause data loss or errors, thereby improving the robustness of the lyrics semantic analysis process, ensuring the correctness of the lyrics displayed on the smart speaker, and facilitating users to sing according to the correct lyrics.

[0121] Figure 7 The flowchart of the implementation of the method for generating a dynamic picture based on a smart speaker provided in the seventh embodiment of the present application is shown. The difference between the method and the fifth embodiment is that the step S507 specifically includes:

[0122] Step S701, performing word segmentation processing on the initial lyrics information to obtain vocabulary information.

[0123] In this embodiment, the initial lyrics information is segmented into individual words. The segmentation can be based on a rule-based word segmentation algorithm that relies on predefined rules and dictionaries to segment the lyrics. The word segmentation is achieved by traversing the text content in the lyrics and searching for words or words in the dictionary.

[0124] Step S702, calculating the probability distribution of all characters in the vocabulary information corresponding to each character in the standard lyrics sentence information, and determining the lyrics information to be corrected and the correct lyrics information.

[0125] In this embodiment, all the characters in the vocabulary information are matched one by one with the characters of the specific lyrics in the standard lyrics sentence information to calculate the correct probability of each character in the vocabulary information. The probability distribution of each character in the vocabulary information corresponding to the correct character can be calculated by the NLP method. When the correct probability of a single character or a single word is higher than the preset probability, it means that the lyrics vocabulary is correct and is determined to be the correct lyrics information without correction. When the correct probability of a single character or a single word is lower than the preset probability, it means that the lyrics vocabulary is wrong and is determined to be an error in the lyrics information to be corrected, and the matched standard lyrics sentence information is used as the correct lyrics information of the lyrics for correction.

[0126] Step S703, replacing the lyrics information to be corrected according to the correct lyrics information to obtain the target lyrics information.

[0127] In this embodiment, the correct lyrics information is used to replace the incorrect part of the lyrics vocabulary to obtain the corrected lyrics information, and the original lyrics information with correct lyrics vocabulary is spliced ​​with the corrected lyrics to obtain the target lyrics information for generating a dynamic picture with the target image for display on the display screen of the smart speaker.

[0128] The dynamic picture generation method based on the smart speaker provided in the embodiment of the present application determines whether there are semantic errors in the initial lyric information, thereby determining whether the initial lyric information can be directly used for display on the smart speaker, and replaces the erroneous lyrics in the initial lyric information with the correct lyric information matched during the judgment process, thereby generating the target lyrics for display on the display screen of the smart speaker, ensuring that the user can sing according to the correct lyric prompts when using the smart speaker, thereby increasing the user's singing fluency and satisfaction.

[0129] Figure 8 The flowchart of the implementation of the method for generating a dynamic picture based on a smart speaker provided in the eighth embodiment of the present application is shown. The difference between the eighth embodiment and the first embodiment is that the step S104 specifically includes:

[0130] Step S801, matching the target lyrics information with the audio information to obtain normal display lyrics information and special effect display lyrics information.

[0131] In this embodiment, the audio information can be converted into text content through an automatic speech recognition model, so that the converted text content is matched one by one with the target lyrics content, and whether the content in the current audio mentions the content of the target lyrics is identified, which is used to identify whether the user has sung or has not sung or sung incorrectly for the target lyrics information currently used for display. When the text content converted by audio matches the target lyrics information, it means that the user has sung the content of the current target lyrics information and does not need to sing the target lyrics again, so the lyrics that have been sung are determined as ordinary display lyrics information; when the text content converted by audio does not match the target lyrics information, it indicates that the user may not have sung the target lyrics or sang incorrectly or missed them, then the target lyrics information can be determined as special effect display lyrics, which is used for the display screen of the smart speaker to display special effects to remind the user to sing. Among them, the special effect display effect can be bold display or color change display, etc. The lyrics that have been sung can also be displayed as special effects to affirm the user's singing, and the lyrics that have not been sung or sung incorrectly or missed are displayed normally, so as to remind the user to sing in time.

[0132] Step S802, synthesizing the target image, the normal display lyrics information and the special effect display lyrics information to obtain a dynamic picture.

[0133] In this embodiment, the target image may be directly synthesized with the normal display lyric information and the special effect display lyric information, the target image may be used as the lyrics background, and the lyrics information may be placed on the top display layer to generate a dynamic picture for real-time display on the display screen of the smart speaker.

[0134] The dynamic picture generation method based on the smart speaker provided in the embodiment of the present application identifies the lyrics information that the user has sung, has not sung, or has sung incorrectly or missed by matching the audio information with the target lyrics information, and then displays them separately from the lyrics that have already been sung to remind the user to sing in time, thereby ensuring the intuitiveness of the lyrics reminder when the user is singing, ensuring the fluency of the user during the singing process, and improving the user's performance ability in the usage scenario.

[0135] Fig. 9 The flowchart of the implementation of the method for generating a dynamic picture based on a smart speaker provided in the ninth embodiment of the present application is shown. The difference between the method and the eighth embodiment is that the step S801 specifically includes:

[0136] Step S901: extract features from the audio information to obtain an audio feature vector.

[0137] In this embodiment, the feature extraction of audio information can be performed by the MFCC algorithm, or by the method of calculating the linear prediction coding coefficient, or by the fast Fourier transform, or by the multi-head attention mechanism, or by the transformer model, so as to calculate the audio feature vector.

[0138] Step S902, obtaining a target lyrics text vector according to the target lyrics information.

[0139] In this embodiment, One-hot encoding, TF-IDF, Word2Vec, Doc2Vec, BERT, ELMo, GPT, FastText, GloVe and other methods can be used to convert the target lyrics information into a target lyrics text vector.

[0140] Step S903, obtaining audio text information according to the audio feature vector and a preset speech recognition model.

[0141] In this embodiment, the audio feature vector can be used as input data and substituted into a preset speech recognition model for calculation, and the calculation result is audio text information, which represents the specific content reflected in the audio information in the form of text. The preset speech recognition model can be a convolutional neural network, a recursive neural network, a natural language processing model, or a Google Cloud Speech-to-Text model.

[0142] Step S904: extract entity from the target lyrics information to obtain lyrics entities.

[0143] In this embodiment, text entity extraction is performed on the text in the target lyric information, which can be performed through the entity extraction model in natural language processing. The obtained lyric entities are nouns, verbs, adverbs, etc. in the lyrics. As lyric entities, they are used for subsequent matching with the text content converted from the audio, and for error checking and correction of the lyrics.

[0144] Step S905: extracting entities from the audio text information to obtain audio text entities.

[0145] In this embodiment, text entity extraction is performed on the text content in the audio text information, which can be performed through the entity extraction model in natural language processing. The obtained text entities are nouns, verbs, adverbs, etc. in the lyrics. As audio text entities, they are used to automatically match the target lyrics information to determine whether the target lyrics information contains lyrics that have been sung, not sung, wrongly sung, or missed.

[0146] Step S906, determine whether the lyrics entity matches the audio text entity, if so, proceed to step S907, if not, proceed to step S908.

[0147] In this embodiment, the matching degree between the lyrics entity and the audio text entity may be calculated, wherein the matching degree may be a similarity, which is used to characterize the degree of conformity between the lyrics entity and the audio text entity.

[0148] Step S907: display the lyrics entity as normal lyrics information.

[0149] In this embodiment, when the lyrics entity matches the audio text entity, it means that the lyrics entity has been sung, and the lyrics entity is displayed normally to remind the user that the lyrics have been sung and there is no need to sing them again.

[0150] Step S908, using the lyrics entity as a special effect to display lyrics information.

[0151] In this embodiment, when the lyrics entity does not match the audio text entity, it means that the lyrics entity has not been sung or has been sung incorrectly or missed, and the lyrics are displayed with special effects to remind the user that the lyrics need to be sung in time.

[0152] The dynamic picture generation method based on the smart speaker provided in the embodiment of the present application converts audio information into text information and matches it with the target lyric information to identify the lyrics that have been sung, the lyrics that have not been sung, or the lyrics that have been sung incorrectly or omitted, so that the lyrics that have been sung are displayed normally through the display screen of the smart speaker, prompting the user that there is no need to sing them again, and the lyrics that have not been sung or are sung incorrectly or omitted are displayed with special effects through the display screen of the smart speaker, prompting the user to sing them in time, thereby ensuring the user's fluency in the singing process and enhancing the user's immersive experience of the product.

[0153] Corresponding to the method of the above embodiment, Fig.10 A structural block diagram of a dynamic picture generation device based on a smart speaker provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown. Fig.10 The exemplary smart speaker-based dynamic picture generation device can be the execution entity of the smart speaker-based dynamic picture generation method provided in the aforementioned embodiment 1.

[0154] Reference Fig.10 , the dynamic picture generation device based on the smart speaker includes:

[0155] The information acquisition module 1010 is used to acquire initial video frame information, initial lyrics information and audio information;

[0156] The target image information generating module 1020 is used to perform multi-dimensional reconstruction processing on the initial video frame information to generate target image information;

[0157] The target lyrics information generating module 1030 is used to perform text detection and error correction on the initial lyrics information to obtain the target lyrics information;

[0158] The dynamic picture generation module 1040 is used to obtain a dynamic picture according to the target image information, the target lyrics information and the audio information, so as to display it through the display screen of the smart speaker.

[0159] The process of each module realizing its own function in the dynamic picture generation device based on the smart speaker provided in the embodiment of the present application can be specifically referred to the aforementioned Figure 1 The description of the first embodiment is not repeated here.

[0160] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0161] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0162] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0163] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0164] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish descriptions, and cannot be understood as indicating or suggesting relative importance. It should also be understood that although the terms "first", "second", etc. are used to describe various elements in some embodiments of the present application in the text, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first table can be named as the second table, and similarly, the second table can be named as the first table without departing from the scope of the various described embodiments. The first table and the second table are both tables, but they are not the same table.

[0165] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0166] The method for generating dynamic images based on a smart speaker provided in the embodiments of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiments of the present application do not impose any restrictions on the specific types of terminal devices.

[0167] For example, the terminal device can be a station (STAION, ST) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, a vehicle networking terminal, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite wireless device, a wireless modem card, a TV set top box (STB), a customer premises equipment (CPE) and / or other devices for communicating on a wireless system and a next-generation communication system, such as a mobile terminal in a 5G network or a mobile terminal in a future evolved Public Land Mobile Network (PLMN) network, etc.

[0168] As an example but not limitation, when the terminal device is a wearable device, the wearable device can also be a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not just hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are fully functional, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0169] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0170] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0171] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0172] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0173] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for generating dynamic images based on an intelligent speaker, characterized in that: include: Obtaining initial video frame information, initial lyrics information, and audio information; Performing multi-dimensional reconstruction processing on the initial video frame information to generate target image information; Performing text detection and error correction processing on the initial lyrics information to obtain target lyrics information; A dynamic picture is obtained according to the target image information, the target lyrics information and the audio information, so as to be displayed through the display screen of the smart speaker.

2. The method for generating dynamic images based on an intelligent speaker according to claim 1, characterized in that: The step of performing multi-dimensional reconstruction processing on the initial video frame information to generate target image information specifically includes: Obtaining pixel coordinates of multiple video frames and multi-dimensional display position coordinates; Performing position coding processing on the video frame pixel coordinates and the multi-dimensional display position coordinates, establishing a corresponding relationship between the video frame pixel coordinates and the multi-dimensional display position coordinates, and obtaining video frame mapping coordinates and multi-dimensional display position mapping coordinates; Performing nonlinear processing on the video frame mapping coordinates and the multi-dimensional display position mapping coordinates to obtain viewing angle display characteristics; The target image information is generated according to the viewing angle display feature and the multi-dimensional display position coordinates.

3. The method for generating dynamic images based on a smart speaker according to claim 2, characterized in that: The viewing angle display characteristics include volume density characteristics and color representation characteristics; The step of performing nonlinear processing on the video frame mapping coordinates and the multi-dimensional display position mapping coordinates to obtain viewing angle display features specifically includes: According to the multi-dimensional display position mapping coordinates and a preset first non-linear function, the multi-dimensional display position mapping coordinates are non-linearly transformed to obtain a first display position representation feature; Performing feature enhancement processing on the first display position representation feature to obtain a second display position representation feature and a volume density feature corresponding to each multi-dimensional display position mapping coordinate; The video frame mapping coordinates are spliced ​​with the second display position representation feature to obtain a third display position representation feature; According to the third display position representation feature and a preset second nonlinear function, the third display position representation feature is nonlinearly transformed to obtain a color representation feature.

4. The method for generating dynamic images based on a smart speaker according to claim 3, characterized in that: The step of generating target image information according to the viewing angle display feature and the multi-dimensional display position coordinates specifically includes: Decomposing and calculating the color representation feature to obtain a hue feature, a saturation feature, and a brightness feature; Performing enhancement processing on the saturation feature and the brightness feature to obtain a visual enhancement feature; The multi-dimensional display position coordinates are rendered according to the color tone feature, the visual enhancement feature, and the volume density to generate target image information.

5. The method for generating dynamic images based on a smart speaker according to claim 1, characterized in that: The step of performing text detection and error correction processing on the initial lyrics information to obtain the target lyrics information specifically includes: According to the initial lyrics information, a lyrics text vector is obtained; Extracting text information from a preset lyrics library to obtain standard lyrics sentence information; According to the standard lyrics sentence information, a standard lyrics sentence vector is obtained; Analyzing and calculating the lyrics text vector and the standard lyrics sentence vector to obtain a distance metric value; Determining whether the distance metric value is less than a preset distance threshold; If yes, the initial lyrics information is determined as the target lyrics information; If not, error correction is performed on the initial lyrics information to obtain target lyrics information.

6. The method for generating dynamic images based on a smart speaker according to claim 5, characterized in that: The step of analyzing and calculating the lyrics text vector and the standard lyrics sentence vector to obtain a distance metric value specifically includes: According to the lyrics text vector and the preset query weight matrix, key weight matrix and value weight matrix, a query vector, a key vector and a value vector are obtained; Multiplying the query vector by the key vector to obtain a sentence similarity weight matrix; Multiplying the sentence similarity weight matrix by the value vector to obtain a sentence weight value matrix; Performing a nonlinear transformation on the lyrics text vector to obtain a nonlinear lyrics feature vector; Summing the lyrics text vector and the nonlinear lyrics feature vector to obtain a lyrics weight connection matrix; Performing weighted summation on the lyrics weight connection matrix and the sentence weight value matrix to obtain a first intermediate lyrics vector; Performing root mean square normalization calculation on the first intermediate lyrics vector to obtain a second intermediate lyrics vector; Training a preset lyrics text processing model according to the standard lyrics sentence vector to obtain multiple lyrics feature representation vectors; Performing a weighted summation on the second intermediate lyrics vector and a plurality of the lyrics feature representation vectors to obtain a third intermediate lyrics vector; The Euclidean distance between the third intermediate lyrics vector and the lyrics text vector is calculated to obtain a distance metric value.

7. The method for generating dynamic images based on a smart speaker according to claim 5, characterized in that: The step of performing error correction processing on the initial lyrics information to obtain the target lyrics information specifically includes: Performing word segmentation processing on the initial lyrics information to obtain vocabulary information; Calculate the probability distribution of all characters in the vocabulary information corresponding to each character in the standard lyrics sentence information, and determine the lyrics information to be corrected and the correct lyrics information; According to the correct lyrics information, the lyrics information to be corrected is replaced to obtain the target lyrics information.

8. The method for generating dynamic images based on a smart speaker according to claim 1, characterized in that: The step of obtaining a dynamic picture according to the target image information, the target lyrics information and the audio information specifically includes: Matching the target lyrics information with the audio information to obtain normal display lyrics information and special effect display lyrics information; The target image, the normal display lyrics information and the special effect display lyrics information are synthesized to obtain a dynamic picture.

9. The method for generating dynamic images based on a smart speaker according to claim 8, characterized in that: The step of matching the target lyrics information with the audio information to obtain normal display lyrics information and special effect display lyrics information specifically includes: Extracting features from the audio information to obtain an audio feature vector; According to the target lyrics information, the target lyrics text vector is obtained; Obtaining audio text information according to the audio feature vector and a preset speech recognition model; Perform entity extraction on the target lyrics information to obtain lyrics entities; Performing entity extraction on the audio text information to obtain audio text entities; Determine whether the lyrics entity matches the audio text entity; if so, use the lyrics entity as normal display lyrics information; if not, use the lyrics entity as special effect display lyrics information.

10. A dynamic picture generation device based on an intelligent speaker, characterized in that: include: An information acquisition module, used to acquire initial video frame information, initial lyrics information and audio information; A target image information generating module, used for performing multi-dimensional reconstruction processing on the initial video frame information to generate target image information; A target lyrics information generation module is used to perform text detection and error correction on the initial lyrics information to obtain target lyrics information; A dynamic picture generation module is used to obtain a dynamic picture based on the target image information, the target lyrics information and the audio information, so as to display it through the display screen of the smart speaker.

Images