Speech recognition model training method, print generation method and print generation system

Abstract

The invention provides a speech recognition model training method, a print generation method and a print generation system, and relates to the field of financial science and technology or other related fields. The speech recognition model training method comprises the steps that pure picture content description information and pure emotional tendency information in sample audio data are extracted through orthogonal decomposition, the sample audio data are used for generating pictures, and the pure picture content description information describes image contents of the pictures corresponding to the sample audio data; the pure emotional tendency information describes the emotional tendency of a picture corresponding to the sample audio data; and calculating a loss value through a loss function according to the pure picture content description information and the pure emotional tendency information, and updating a weight parameter of the speech recognition model according to the loss value. Through orthogonal decomposition, the voice recognition model can decompose the audio signal into content-related and emotion-related features, so that the information is better combined when the picture is generated, and the generated style picture can meet user requirements.

Description

Technical Field

[0001] The present application relates to the field of financial technology or other related fields, and in particular to a speech recognition model training method, a printmaking generation method, and a printmaking generation system. Background Art

[0002] Prints are an investment option offered by the bank, including gold prints, silver prints, etc., which have both investment value and ornamental value.

[0003] In order to produce prints that meet user requirements, it is first necessary to produce style images based on the user's description. In the prior art, a speech recognition model is usually used to convert the audio input by the user into text, and then generate the corresponding style image based on the text. Therefore, whether the user's audio can be converted into text that can describe the user's needs is very important for generating style images that meet the user's requirements.

[0004] However, the speech recognition model used in the prior art cannot accurately recognize the picture content description information and user emotional tendency information in the audio, resulting in the generated style pictures failing to meet the needs of users. Summary of the invention

[0005] The present application provides a speech recognition model training method, a print generation method, and a print generation system to solve the technical problem that the speech recognition model used in the prior art cannot accurately recognize the picture content description information and user emotional tendency information in the audio, resulting in the generated style pictures failing to meet the user's needs.

[0006] In a first aspect, the present application provides a speech recognition model training method, comprising:

[0007] According to the sample audio data, extracting the pure picture content description information and the pure emotional tendency information in the sample audio data by orthogonal decomposition, the sample audio data is used to generate a picture, the pure picture content description information describes the image content of the picture corresponding to the sample audio data, and the pure emotional tendency information describes the emotional tendency of the picture corresponding to the sample audio data;

[0008] According to the pure picture content description information and the pure emotional tendency information, a loss value is calculated through a loss function, and the weight parameters of the speech recognition model are updated according to the loss value.

[0009] In a second aspect, the present application provides a print generation method, the print generation method is implemented based on a speech recognition model, the speech recognition model is trained by the method described in the first aspect, and the print generation method includes:

[0010] The audio data is converted into content description information and emotional tendency information through the speech recognition model, wherein the audio data is used to generate a picture, the content description information describes the image content of the picture corresponding to the audio data, and the emotional tendency information describes the emotional tendency of the picture corresponding to the audio data;

[0011] Generate a content picture according to the content description information and the emotional tendency information;

[0012] Extracting image content feature information of the content image and style feature information of the style image, and fusing the image content feature information and the style feature information to generate a target image;

[0013] A print is generated according to the target image.

[0014] In a third aspect, the present application provides a printmaking generation system, the system is implemented based on a speech recognition model, the speech recognition model is trained by the method described in the first aspect, and the system includes:

[0015] A speech recognition module, used to: convert audio data into content description information and emotional tendency information, wherein the audio data is used to generate a picture, the content description information describes the image content of the picture corresponding to the audio data, and the emotional tendency information describes the emotional tendency of the picture corresponding to the audio data;

[0016] An image generation module, used to generate a content image according to the content description information and the emotional tendency information;

[0017] Style transfer module, used to:

[0018] Extracting image content feature information of the content image and style feature information of the style image through a style transfer model, and fusing the image content feature information and the style feature information through the style transfer model to generate a target image;

[0019] The print generation module is used to generate a print according to the target image.

[0020] In a fourth aspect, the present application provides a speech recognition model training device, the device comprising:

[0021] An extraction module is used to: extract pure picture content description information and pure emotional tendency information from the sample audio data through orthogonal decomposition according to the sample audio data, wherein the pure picture content description information describes the image content of the generated picture, and the pure emotional tendency information describes the emotional tendency of the generated picture;

[0022] An updating module is used to: calculate a loss value through a loss function according to the pure picture content description information and the pure emotional tendency information, and update the weight parameters of the speech recognition model according to the loss value.

[0023] In a fifth aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0024] The memory stores computer-executable instructions;

[0025] The processor executes the computer-executable instructions stored in the memory to implement the method according to the first aspect, or the method according to the second aspect.

[0026] In a sixth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method as described in the first aspect, or the method as described in the second aspect.

[0027] In a seventh aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method as described in the first aspect, or the method as described in the second aspect.

[0028] The speech recognition model training method, printmaking generation method, and printmaking generation system provided in this application can decompose the audio signal into content-related and emotion-related features through orthogonal decomposition. This decomposition helps the model to process and optimize different types of information separately, improve the ability to understand complex audio signals, and thus better combine this information when generating pictures, so that the generated style pictures can meet user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0030] Figure 1 A flow chart of the speech recognition model training method provided for the implementation of this application;

[0031] Figure 2 A schematic diagram of the speech recognition model structure provided for the implementation of this application;

[0032] Figure 3 A schematic diagram of a method flow for extracting pure picture content description information and pure emotional tendency information provided in an embodiment of the present application;

[0033] Figure 4 A schematic diagram of the process flow of the print generation method provided in the embodiment of the present application;

[0034] Figure 5 A schematic diagram of the text understanding model structure provided in the embodiment of the present application;

[0035] Figure 6 A schematic diagram of the style transfer model structure provided in an embodiment of the present application;

[0036] Figure 7 A schematic diagram of the printmaking system structure and workflow provided in the embodiment of the present application;

[0037] Figure 8 A schematic diagram of the structure of a speech recognition model training device provided in an embodiment of the present application;

[0038] Fig. 9 A schematic diagram of the structure of the electronic device provided in this application.

[0039] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0040] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0041] It should be noted that the speech recognition model training method, printmaking method and printmaking system provided in the present application can be used in the field of financial technology, and can also be used in any field other than the field of financial technology. The application field of the speech recognition model training method, printmaking method and printmaking system in the present application is not limited.

[0042] Since prints have certain investment value, banks can set up machines specifically for customizing prints at their branches. When customizing prints, users need to first orally state their requirements for the prints. For example, users can say "a cute cat", where "cute" is the emotional tendency and the cat is the image content. After the machine receives the user's audio, it converts the audio into text information, and then generates a corresponding image based on the text information. Finally, the print is made from the generated image.

[0043] However, the audio actually input by the user may be much more complex than the above example. In this case, whether the description information of the image content and the description information of the image's emotional tendency in the audio can be accurately extracted is very important for generating prints that meet the user's requirements. However, the speech recognition model used in the prior art cannot accurately identify the image content description information and the user's emotional tendency information in the audio, resulting in the generated style pictures not meeting the user's needs. If the image content description information and the image emotional tendency information can be effectively separated in the sample audio data during the speech recognition model training stage, so that the model can learn to process the image content description information and the image emotional tendency information respectively, the accuracy of the model in processing the image content description information and the image emotional tendency information during the application stage can be improved.

[0044] Therefore, in response to the above technical problems, the technical concept of the present application is to extract pure picture content description information and pure emotional tendency information from the sample audio data through orthogonal decomposition during the training stage, and then calculate the loss value through the loss function based on the pure picture content description information and pure emotional tendency information, and update the speech recognition model weight according to the loss value.

[0045] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0046] Figure 1 A flow chart of a speech recognition model training method provided for the implementation of this application, which can be applied to a server, such as Figure 1 As shown, the method includes:

[0047] S101. Extracting pure picture content description information and pure emotional tendency information from the sample audio data by orthogonal decomposition according to the sample audio data, wherein the sample audio data is used to generate a picture, the pure picture content description information describes the image content of the picture corresponding to the sample audio data, and the pure emotional tendency information describes the emotional tendency of the picture corresponding to the sample audio data;

[0048] Specifically, in this step, audio features can be extracted from the sample data, and then the audio features can be decomposed through orthogonal decomposition technology, such as principal component analysis (PCA) or independent component analysis (ICA), so as to separate pure picture content description information related to the picture content description and another pure emotional tendency information related to the emotional tendency.

[0049] The working principles of PCA and ICA are explained below.

[0050] The goal of PCA is to project high-dimensional data into a low-dimensional space while preserving the variance of the data as much as possible. Specifically, it achieves this by finding the principal components of the data (i.e. the direction with the largest variance). PCA transforms the data into a new coordinate system through linear transformation, where the first principal component has the largest variance, the second principal component has the second largest variance, and so on. All principal components are orthogonal to each other. The implementation steps of PCA are as follows:

[0051] 1. Data centering: Subtract the mean of each feature of the data to make the data center;

[0052] 2. Calculate the covariance matrix: Calculate the covariance matrix of the centralized data;

[0053] 3. Eigenvalue decomposition: Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors;

[0054] 4. Select principal components: Select the eigenvector with the largest eigenvalue as the principal component;

[0055] 5. Data projection: Project the original data onto the selected principal components to obtain the reduced-dimensional data.

[0056] ICA assumes that the observed data is a linear mixture of multiple independent signals, so the goal of this method is to decompose the multivariate signal into statistically independent non-Gaussian signal components. Specifically, ICA achieves signal independence separation by maximizing the non-Gaussianity of the signal. The implementation steps of ICA are as follows:

[0057] 1. Data preprocessing: usually includes de-meaning and whitening (i.e. converting the data to unit variance);

[0058] 2. Choose a non-Gaussian metric: Choose a non-Gaussian metric (such as negative entropy) to evaluate the independence of the signal;

[0059] 3. Optimization algorithm: Use an optimization algorithm (such as gradient descent) to maximize the non-Gaussianity measure to find independent components;

[0060] 4. Signal separation: decompose the original data into independent components through linear transformation.

[0061] It should be noted that PCA reduces the dimension by finding the direction with the largest variance in the data. Therefore, if the image content description information and the emotional tendency information are expressed as different variance patterns in the data, PCA can separate them into different principal components, thereby achieving the separation of pure image content description information and pure emotional tendency information. ICA uses the non-Gaussianity of the signal to achieve separation. If the image content and emotional information are expressed in different non-Gaussian properties in the data, ICA can achieve the separation of pure image content description information and pure emotional tendency information.

[0062] S102. Calculate the loss value through the loss function according to the pure picture content description information and the pure emotional tendency information, and update the weight parameters of the speech recognition model according to the loss value.

[0063] Specifically, in this step, the weight parameters of the speech recognition model are updated according to the loss value calculated by the loss function by using the back propagation algorithm, and the value of the loss function is minimized by continuously adjusting the model parameters, thereby improving the performance of the model.

[0064] The speech recognition model training method provided in this embodiment can decompose the audio signal into content-related and emotion-related features through orthogonal decomposition. This decomposition helps the model to process and optimize different types of information separately, improve the ability to understand complex audio signals, and thus better combine this information when generating pictures, so that the generated style pictures can meet user needs.

[0065] Figure 2 The schematic diagram of the speech recognition model structure provided for the implementation of this application is as follows: Figure 2As shown in the figure, the model first passes the audio data through WAV2VEC and two CNN layers (convolutional layers). WAV2VEC is a model specially designed for audio data, which can extract useful feature representations from the original audio waveform. WAV2VEC can learn features from unlabeled audio data through self-supervised learning. These features can capture the timing information and spectral characteristics of the audio. In addition, WAV2VEC not only extracts low-level features, but also models the long-distance dependencies of audio signals through contextual networks (such as Transformer), so that the model can understand the complex patterns and structures in the audio. Furthermore, when processing audio data, the CNN layer can effectively capture local features. The convolution operation can identify local patterns in audio signals, such as specific frequency components or time domain features. Therefore, the speech recognition model can extract higher-level features layer by layer through the stacking of multiple convolutional layers. Therefore, the speech recognition model combines the characteristics of WAV2VE and CNN. The model can extract complex content description information in audio data, such as vocabulary and semantics in speech, and extract content-independent information in audio data, such as speaker characteristics, background noise, etc.

[0066] like Figure 2 As shown in the figure, after extracting complex content description information and content-irrelevant information, the complex content description information and content-irrelevant information are processed by Transformer respectively. Transformer can effectively capture long-distance dependencies and contextual information, making the feature representation richer and more comprehensive.

[0067] Afterwards, the content-irrelevant information processed by Transformer is orthogonally decomposed to generate pure image content description information (A), and then multiple ResBlock (residual block) layers, LSTM layers and fully connected layers are used to generate text for A; the complex content description information processed by Transformer is orthogonally decomposed to generate pure sentiment tendency information (B), and then multiple ResBlock layers are used to extract sentiment features from B, and the fully connected layer is used to map the features to the sentiment category space, and the Softmax layer is used to normalize the output, and the probability distribution of each sentiment category is calculated to perform sentiment recognition.

[0068] based on Figure 2 The speech recognition model shown, the embodiment of the present application provides a method for extracting pure picture content description information and pure emotional tendency information. Figure 3 A schematic flow chart of a method for extracting pure picture content description information and pure emotional tendency information provided in an embodiment of the present application is as follows: Figure 3 As shown, the method includes:

[0069] S301, obtaining complex content description information and content-irrelevant information of the sample audio data according to the sample audio data;

[0070] Specifically, the complex content description information includes the description information of the image content of the generated picture, and the content-irrelevant information does not include the description information of the image content of the generated picture. In this step, the WAV2VE and CNN layers of the speech recognition model can be used to extract the complex content description information and the content-irrelevant information, and then they are processed by Transformer respectively.

[0071] S302, calculating the projection of the complex content description information on the content-irrelevant information to obtain first impurity information, and subtracting the first impurity information from the complex content description information to obtain pure picture content description information;

[0072] Specifically, in this step, the following formula can be used for orthogonal decomposition:

[0073] A=A1-A2

[0074]

[0075] Among them, A represents pure image content description information, A1 represents complex content description information, A2 is first impurity information, B1 is content irrelevant information, and the first impurity information A2 is the projection of the complex content description information on the content irrelevant information. Specifically, by calculating the projection of the complex content description information on the content irrelevant information, the part of the complex content description information that is similar to the content irrelevant information can be identified, and this part is regarded as the first impurity information. By subtracting the first impurity information from the complex content description information, the components related to the content irrelevant information are removed, and the remaining part is the pure image content description information because it no longer contains the correlation with the content irrelevant information.

[0076] S303: Calculate the projection of the content-irrelevant information on the complex content description information to obtain second impurity information, and subtract the second impurity information from the content-irrelevant information to obtain pure sentiment tendency information.

[0077] Specifically, in this step, the following formula can be used for orthogonal decomposition:

[0078] B=B1-B2

[0079]

[0080] Among them, B is pure sentiment tendency information, B1 is content-irrelevant information, B2 is the projection of content-irrelevant information on complex content description information, and A1 represents complex content description information. Specifically, by calculating the projection of content-irrelevant information on complex content description information, the part of content-irrelevant information that is similar to complex content description information can be identified, and this part is regarded as the second impurity information. By subtracting the second impurity information from the content-irrelevant information, the components related to the complex content description information are removed, and the remaining part is the pure sentiment tendency information because it no longer contains the correlation with the complex content description information.

[0081] based on Figure 2 The speech recognition model shown, optionally, this embodiment provides a method for alternately training the speech recognition model, the method comprising:

[0082] Freeze the emotion feature extraction neural network of the speech recognition model, calculate the content-related loss value through the content-related loss function according to the pure image content description information, and update the weight parameter of the content feature extraction neural network of the speech recognition model according to the content-related loss value until the content-related loss value is less than the first loss threshold;

[0083] Freeze the content feature extraction neural network, calculate the emotion-related loss value through the emotion-related loss function according to the pure emotion tendency information, and update the weight parameters of the emotion feature extraction neural network according to the emotion-related loss value until the emotion-related loss value is less than the second loss threshold.

[0084] from Figure 2 It can be seen that the speech recognition model includes two major branches: the emotional feature extraction neural network and the content feature extraction neural network. Therefore, in the training process, an alternating training strategy is adopted. For example, as described above, the emotional feature extraction neural network can be frozen first, the content-related loss value is calculated through the content-related loss function, and the weight parameters of the content feature extraction neural network of the speech recognition model are updated through back propagation according to the content-related loss value. Then, the content feature extraction neural network is frozen, the emotional feature extraction neural network is unfrozen, and the emotional feature extraction neural network is calculated through the emotional-related loss function according to the pure emotional tendency information. The emotional-related loss value is calculated, and the weight parameters of the emotional feature extraction neural network are updated through back propagation according to the emotional-related loss value. Until the content-related loss value is less than the first loss threshold and the emotional-related loss value is less than the second loss threshold, the model training can be considered completed.

[0085] The method of alternately training a speech recognition model provided in this embodiment enables the model to focus more on the specific requirements of each task and reduce interference between tasks by optimizing the content and emotion feature extraction networks respectively; by using a specific loss function at each stage, the model can more accurately learn and extract relevant features and improve performance on respective tasks.

[0086] Optionally, the mean square error loss and CTC loss of text prediction can be calculated based on the pure image content description information through the mean square error loss function and the CTC loss function, and the weighted sum of the mean square error loss and the CTC loss of text prediction can be used as the content-related loss value. Specifically, the steps for calculating the mean square error loss of text prediction are as follows:

[0087] According to the pure image content description information, the text prediction value is obtained through the content feature extraction neural network, which can be a continuous value or a probability distribution;

[0088] Get the real text label corresponding to the text prediction value;

[0089] Calculate the error between each text prediction value and the true text label;

[0090] Each error is squared and the average of the squared errors of all samples is calculated to obtain the mean square error loss for text prediction.

[0091] The calculation steps of CTC loss are as follows:

[0092] Obtain the predicted probability distribution after Softmax from the content feature extraction neural network;

[0093] Get the target text sequence;

[0094] Use the CTC dynamic programming algorithm to calculate the total probability of all possible paths and map the predicted probability distribution to the target text sequence;

[0095] The negative logarithm of the total path probability of the target text sequence is calculated to obtain the CTC loss.

[0096] By minimizing the mean square error between the predicted value and the true value, the speech recognition model can more accurately learn and predict continuous content features and improve the accuracy of content description; by dealing with the sequence alignment problem, CTC loss enables the model to better handle variable-length inputs and enhance the robustness of the model in complex sequence prediction tasks.

[0097] Optionally, the cross entropy loss and L1 loss can be calculated based on the pure emotional tendency information through the cross entropy loss function and the L1 loss function, and the weighted sum of the cross entropy loss and the L1 loss is used as the emotional related loss value. Specifically, the steps for calculating the cross entropy loss are as follows:

[0098] The predicted probability distribution after Softmax processing is obtained from the emotional feature extraction neural network of the model. The predicted probability distribution represents the predicted probability of the model for each possible emotional category.

[0099] Get the corresponding true sentiment label, which can be in the form of one-hot encoding;

[0100] For each sample, the cross entropy loss between the true sentiment label and the predicted probability distribution is calculated through the cross entropy loss function.

[0101] The steps for calculating L1 loss are as follows:

[0102] Extracting the predicted sentiment value from the sentiment feature extraction neural network of the model, such as sentiment intensity or other continuous value representation;

[0103] Get the corresponding real sentiment value;

[0104] For each sample, the absolute difference between the predicted sentiment value and the true sentiment value is calculated, and the L1 loss is the average of these absolute differences.

[0105] By utilizing the cross entropy loss, the model is able to more accurately classify emotions and improve the accuracy of emotion recognition by minimizing the difference between the predicted probability distribution and the true distribution. The L1 loss enables the model to maintain accuracy when predicting numerical values ​​and provides a certain robustness when there are outliers in the sample data.

[0106] Until the content-related loss value is less than the first loss threshold and the emotion-related loss value is less than the second loss threshold, the model can be further fine-tuned as a whole. Optionally, if the content-related loss value is less than the first loss threshold and the emotion-related loss value is less than the second loss threshold, the speech recognition model is fine-tuned as a whole, and the total loss value is calculated by the content-related loss function and the emotion-related loss function, and the weight parameters of the speech recognition model are updated according to the total loss value until the total loss value is less than the third loss threshold. Specifically, the above-mentioned mean square error loss function, CTC loss function, cross entropy loss function, and L1 loss function can be used to perform weighted calculation to obtain the total loss value. When the total loss value is less than the third loss threshold, the model fine-tuning is considered to be completed. In the overall fine-tuning stage, by comprehensively considering content and emotional information, the speech recognition model can better generalize in a multi-task environment.

[0107] The speech recognition model trained based on the above training method can better extract the picture content description information and emotional tendency information in the user audio than the existing speech recognition model. Therefore, based on the speech recognition model of this embodiment, this embodiment provides a print generation method. Figure 4 A schematic diagram of the process flow of the print generation method provided in the embodiment of the present application is as follows: Figure 4 As shown, the method includes:

[0108] S401, converting audio data into content description information and emotional tendency information through a speech recognition model;

[0109] Specifically, the audio data is used to generate a picture, the content description information describes the image content of the picture corresponding to the audio data, and the emotional tendency information describes the emotional tendency of the picture corresponding to the audio data. Figure 2 As shown, the speech recognition model uses a content feature extraction neural network to extract content description information, and uses an emotion feature extraction neural network to extract emotion tendency information.

[0110] S402, generating a content image according to the content description information and the sentiment tendency information;

[0111] In this step, a text understanding model can be used to generate content images based on content description information and sentiment tendency information. Figure 5 A schematic diagram of the text understanding model structure provided in the embodiment of the present application is shown in FIG. Figure 5 As shown in the figure, specifically, the text understanding model generates multiple new text sequences through Permutation arrangement to increase the diversity of data and help the model generalize better during training; Word2Vec is used to convert the text sequence into a feature vector. This vectorized representation can capture the semantic relationship between words and facilitate subsequent feature extraction; two convolutional neural network (CNN) layers are used to perform preliminary processing on the feature vector. These convolutional layers extract more abstract features and simplify subsequent dimensional conversion; the emotional tendency information of the speech recognition model is used, and after aligning the dimensions through convolution, the Attention mechanism is applied. The Attention mechanism uses emotional features to guide the generation process, so that the generated images have specific emotional features.

[0112] Multiple DenseBlocks are used to enrich feature information. Through dense connections of features, DenseBlock can effectively extract and transmit information while reducing the number of parameters. Four layers of CNN convolution are used to align channels to ensure that the shape of the features is suitable for image generation. Finally, transposed convolution is used to convert the features into three-channel images (RGB) for easy display.

[0113] The loss function of the text understanding model in the training phase can take the weighted sum of L1 loss and VGG perceptual loss as the total loss, and update the model weight parameters through back propagation.

[0114] S403, extracting image content feature information of the content image and style feature information of the style image, and fusing the image content feature information and the style feature information to generate a target image;

[0115] In this step, the style image can be provided by the user. By applying the style transfer model, the image content feature information and the style feature information are fused to generate the target image.

[0116] S404: Generate a print according to the target image.

[0117] Specifically, this step can generate prints through a color printing mode or an engraving mode.

[0118] In color printing mode, you can select different colors such as gold and silver as the base of the print, and replace all areas of the target image that were originally displayed as white with the selected color. This replacement can be achieved through image processing technology. Specifically, you can use image editing tools or programming libraries (such as OpenCV) to identify and replace white areas. Alternatively, you can choose to completely cover with colored ink. In engraving mode, use the relief algorithm of the OpenCV tool library to convert the image into an engraving effect. This algorithm usually detects the edges of the image and creates a 3D relief effect, making the image look like it is engraved on the surface. This effect can be achieved through filtering and edge detection functions in OpenCV. In engraving mode, you can replace the base color of the image, and change the light strip color and print frame in the user interface.

[0119] Figure 6 A schematic diagram of the style transfer model structure provided in the embodiment of the present application is shown in FIG. Figure 6 As shown in the figure, the style feature extraction branch of the style transfer model includes a long-short range Attention structure, which specifically includes:

[0120] Feedforward network: Use a linear layer with ReLU activation, followed by another linear layer. This step is used for preliminary feature extraction;

[0121] Channel segmentation: Split the features into two according to the channel dimension to reduce the amount of calculation;

[0122] SelfAttention: Perform Self-Attention operation on half of the channels to capture the autocorrelation between features;

[0123] Convolution and Linear: The other half of the channels are convolved and then processed through a Linear layer to extract local features.

[0124] Channel splicing: splice the two features in the channel dimension;

[0125] Feed-forward network again: Through another Linear+ReLU+Linear network, further integrate features and focus on global relationships.

[0126] The content feature extraction branch of the style transfer model includes:

[0127] Transformer layer: used to obtain global correlation.

[0128] ResBlock stacking: Eight ResBlocks are stacked, each ResBlock is guided by style information, and the style and content are integrated through the Attention operation.

[0129] Skip connections: Keep content coherent.

[0130] Furthermore, the style transfer model uses two CNN layers to align the number of channels and uses transposed convolution to convert features into images.

[0131] During the model training phase, the high-frequency information of the model-generated image and the high-frequency distribution of the style image are used to calculate the GAN loss, and the low-frequency information of the generated image and the low-frequency information of the content image are used to calculate the MSE loss. It is also possible to extract the features of the generated image and the style image, calculate their Gram matrices, and compare the two Gram matrices to calculate the style loss to ensure that the generated image is consistent with the target image in style.

[0132] Based on the above style transfer model, the image content feature information of the content image and the style feature information of the style image are extracted, and the image content feature information and the style feature information are fused to generate the target image, including:

[0133] Convert style images into feature maps;

[0134] Performing a self-attention operation on the odd channels in the feature map to obtain first feature information, and performing a convolution operation and a full connection operation on the even channels in the feature map to obtain second feature information;

[0135] Performing a channel splicing operation on the first feature information and the second feature information to obtain fusion style feature information;

[0136] The fusion style feature information and the image content feature information are fused to generate a target image.

[0137] It should be noted that in this method, the channel segmentation function of the model is used to divide the feature map into two according to the channel dimension, divided into odd channels and even channels, and the channel splicing operation is implemented using the channel splicing function of the model. By performing self-attention operations on the odd channels of the feature map, long-distance dependencies between features can be captured, helping the model understand the global style features of the style image. Since convolution operations and full connection (linear) operations are performed on the even channels of the feature map, convolution operations are good at capturing local features and texture information, and channel splicing operations can combine the global and local style features of the style image.

[0138] Based on the above-mentioned print generation method, this embodiment provides a print generation system. Figure 7The structure and workflow diagram of the print generation system provided in the embodiment of the present application is as follows: Figure 7 As shown, the system includes:

[0139] The speech recognition module 701 is used to convert the audio data into content description information and emotional tendency information, the audio data is used to generate a picture, the content description information describes the image content of the picture corresponding to the audio data, and the emotional tendency information describes the emotional tendency of the picture corresponding to the audio data;

[0140] The image generation module 702 is used to generate a content image according to the content description information and the emotional tendency information;

[0141] The style transfer module 703 is used to:

[0142] Through the style transfer model, the image content feature information of the content image and the style feature information of the style image are extracted, and the image content feature information and the style feature information are fused through the style transfer model to generate the target image;

[0143] The print generation module 704 is used to generate a print according to the target image.

[0144] Figure 8 A schematic diagram of the structure of a speech recognition model training device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the device 80 includes:

[0145] Extraction module 801, used to: extract pure picture content description information and pure emotional tendency information from the sample audio data through orthogonal decomposition according to the sample audio data, wherein the pure picture content description information describes the image content of the generated picture, and the pure emotional tendency information describes the emotional tendency of the generated picture;

[0146] The updating module 802 is used to calculate the loss value through the loss function according to the pure picture content description information and the pure emotional tendency information, and update the weight parameters of the speech recognition model according to the loss value.

[0147] The speech recognition model training device provided in the embodiment of the present application can execute the method provided in the above-mentioned speech recognition model training method embodiment. Its implementation principle and technical effects are similar, and this embodiment will not be repeated here.

[0148] Fig. 9 This is a schematic diagram of the structure of the electronic device provided in this application. Fig. 9 As shown, the electronic device 90 provided in this embodiment includes: at least one processor 901 and a memory 902. Optionally, the device 90 further includes a communication component 903. The processor 901, the memory 902 and the communication component 903 are connected via a bus 704.

[0149] In a specific implementation process, at least one processor 901 executes the computer execution instructions stored in the memory 902, so that at least one processor 901 executes the above method.

[0150] The specific implementation process of the processor 901 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0151] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0152] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0153] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0154] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0155] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0156] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0157] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0158] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0159] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0160] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0161] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0162] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0163] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A speech recognition model training method, characterized in that: The method comprises: According to the sample audio data, extracting the pure picture content description information and the pure emotional tendency information in the sample audio data by orthogonal decomposition, the sample audio data is used to generate a picture, the pure picture content description information describes the image content of the picture corresponding to the sample audio data, and the pure emotional tendency information describes the emotional tendency of the picture corresponding to the sample audio data; According to the pure picture content description information and the pure emotional tendency information, a loss value is calculated through a loss function, and the weight parameters of the speech recognition model are updated according to the loss value.

2. The method according to claim 1, characterized in that The step of extracting pure picture content description information and pure emotional tendency information from the sample audio data by orthogonal decomposition according to the sample audio data includes: According to the sample audio data, complex content description information and content-irrelevant information of the sample audio data are obtained, wherein the complex content description information includes description information of the image content of the generated picture, and the content-irrelevant information does not include description information of the image content of the generated picture; Calculate the projection of the complex content description information on the content-irrelevant information to obtain first impurity information, and subtract the first impurity information from the complex content description information to obtain the pure picture content description information; The projection of the content-irrelevant information on the complex content description information is calculated to obtain second impurity information, and the second impurity information is subtracted from the content-irrelevant information to obtain the pure sentiment tendency information.

3. The method according to claim 1, characterized in that According to the pure picture content description information and the pure emotional tendency information, a loss value is calculated by a loss function, and a weight parameter of the speech recognition model is updated according to the loss value, including: Freeze the emotion feature extraction neural network of the speech recognition model, calculate a content-related loss value through a content-related loss function according to the pure image content description information, and update the weight parameter of the content feature extraction neural network of the speech recognition model according to the content-related loss value until the content-related loss value is less than a first loss threshold; Freeze the content feature extraction neural network, calculate the emotion-related loss value through the emotion-related loss function according to the pure emotion tendency information, and update the weight parameters of the emotion feature extraction neural network according to the emotion-related loss value until the emotion-related loss value is less than a second loss threshold.

4. The method according to claim 3, characterized in that The calculating the content-related loss value by using a content-related loss function according to the pure image content description information includes: According to the pure image content description information, the mean square error loss and CTC loss of text prediction are calculated by using the mean square error loss function and the CTC loss function, and the weighted sum of the mean square error loss of the text prediction and the CTC loss is used as the content-related loss value; The calculating the emotion-related loss value by using the emotion-related loss function according to the pure emotion tendency information includes: According to the pure emotional tendency information, the cross entropy loss and L1 loss are calculated by using the cross entropy loss function and the L1 loss function, and the weighted sum of the cross entropy loss and the L1 loss is used as the emotional related loss value.

5. The method according to claim 3 or 4, characterized in that: The method further comprises: If the content-related loss value is less than the first loss threshold, and the emotion-related loss value is less than the second loss threshold, the total loss value is calculated according to the pure picture content description information and the pure emotional tendency information through the content-related loss function and the emotion-related loss function, and the weight parameters of the speech recognition model are updated according to the total loss value until the total loss value is less than the third loss threshold.

6. A method for generating a print, characterized in that: The print generation method is implemented based on a speech recognition model, and the speech recognition model is trained by the method described in any one of claims 1 to 5. The print generation method includes: The audio data is converted into content description information and emotional tendency information through the speech recognition model, wherein the audio data is used to generate a picture, the content description information describes the image content of the picture corresponding to the audio data, and the emotional tendency information describes the emotional tendency of the picture corresponding to the audio data; Generate a content picture according to the content description information and the emotional tendency information; Extracting image content feature information of the content image and style feature information of the style image, and fusing the image content feature information and the style feature information to generate a target image; A print is generated according to the target image.

7. The printmaking method according to claim 6, characterized in that: The step of extracting the image content feature information of the content image and the style feature information of the style image, and fusing the image content feature information and the style feature information to generate a target image includes: Converting the style image into a feature map; Performing a self-attention operation on the odd channels in the feature map to obtain first feature information, and performing a convolution operation and a full connection operation on the even channels in the feature map to obtain second feature information; Performing a channel splicing operation on the first feature information and the second feature information to obtain fusion style feature information; A fusion operation is performed on the fusion style feature information and the image content feature information to generate the target image.

8. A printmaking system, characterized in that: The system is implemented based on a speech recognition model, and the speech recognition model is trained by the method described in any one of claims 1 to 5. The system comprises: A speech recognition module, used to: convert audio data into content description information and emotional tendency information, wherein the audio data is used to generate a picture, the content description information describes the image content of the picture corresponding to the audio data, and the emotional tendency information describes the emotional tendency of the picture corresponding to the audio data; An image generation module, used to generate a content image according to the content description information and the emotional tendency information; Style transfer module, used to: Extracting image content feature information of the content image and style feature information of the style image through a style transfer model, and fusing the image content feature information and the style feature information through the style transfer model to generate a target image; The print generation module is used to generate a print according to the target image.

9. A speech recognition model training device, characterized in that: The device comprises: An extraction module is used to: extract pure picture content description information and pure emotional tendency information from the sample audio data through orthogonal decomposition according to the sample audio data, wherein the pure picture content description information describes the image content of the generated picture, and the pure emotional tendency information describes the emotional tendency of the generated picture; An updating module is used to: calculate a loss value through a loss function according to the pure picture content description information and the pure emotional tendency information, and update the weight parameters of the speech recognition model according to the loss value.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 5, or the method according to any one of claims 6 or 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method according to any one of claims 1 to 5, or the method according to any one of claims 6 or 7.

12. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5, or the method according to any one of claims 6 or 7.

Images