Method and system for accurately recognizing voice commands for smart mirrors

Abstract

The present invention relates to the field of speech recognition technology, and more specifically to a method and system for accurately recognizing voice commands for smart mirrors. The method comprises: a second encoder of a speech recognition model outputs a simplified command recognition result, and a third encoder outputs a complete command recognition result; the third encoder has fewer parameters than the first encoder; when each speech sample is input into the speech recognition model, a first difference between the complete command recognition result and a dress code label is obtained, and a second difference between the complete command recognition result and the simplified command recognition result is obtained; the second difference is weightedly fused with the first difference to obtain a loss function; the data set and the loss function are used to train the speech recognition model and perform speech recognition. The present invention reduces speech recognition time while ensuring recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for accurately recognizing voice commands applicable to smart mirrors. Background Art

[0002] A smart mirror leverages artificial intelligence (AI) technology to interact with users and provide intelligent feedback. For example, users can generate outfits for specific scenarios by looking in the mirror and interacting with them through voice, thereby enhancing the user experience. Existing implementations use microphones to capture user voice and employ speech recognition models (such as ESPnet and WeNet) to generate outfit and makeup plans. However, this approach cannot perform speech recognition based on the user's current body shape, posture, current clothing, designated clothing areas (such as tops, bottoms, shoes, and socks), or weather conditions. This results in low speech recognition accuracy and the output outfit plans may not meet the user's expectations.

[0003] One readily conceivable solution is to utilize a large image-to-text and text-to-speech model to convert the user's body image and weather forecast into a speech signal. This signal, along with the user's voice, is then fed into a speech recognition model to generate an outfit and makeup plan. While this approach can accurately output the desired outfit plan, it requires cascading and stacking multiple AI models, resulting in long speech recognition times and significant latency. Furthermore, due to error propagation between models (e.g., model hallucinations), significant error amplification can occur, resulting in some outfit plans being clearly incorrect or unreasonable.

[0004] Therefore, when users generate outfits for specific scenarios by looking in the mirror and interacting with voice, there is a lack of fast and accurate voice recognition methods. Summary of the Invention

[0005] To solve the above problems, the present invention provides a method and system for accurately recognizing voice commands applicable to a smart mirror.

[0006] The method and system for accurately recognizing voice commands for smart mirrors of the present invention adopt the following technical solutions:

[0007] One embodiment of the present invention provides a method for accurately recognizing voice commands applicable to a smart mirror, the method comprising the following steps:

[0008] The constructed speech recognition model includes: inputting the collected speech into a first encoder, the output of the first encoder serving as input to a second encoder and a third encoder respectively, and the second encoder outputting a simplified command recognition result; using a fourth encoder, encoding the time command contained in the speech and the weather forecast and image collected under the body part command, and also serving as input to the third encoder; the third encoder outputting a complete command recognition result; the third encoder has fewer parameters than the first encoder;

[0009] The constructed dataset includes several speech samples and clothing labels corresponding to the speech samples. When each speech sample is input into a speech recognition model, a first difference between the complete command recognition result and the clothing label is obtained, and a second difference between the complete command recognition result and the simplified command recognition result is obtained. The second difference is weightedly fused with the first difference to obtain a loss function. The speech recognition model is trained using the dataset and the loss function to perform speech recognition.

[0010] Among them, the weight of the second difference in weighted fusion is positively correlated with the first difference;

[0011] When the weight obtained after the same speech sample is input into the speech recognition model twice adjacently increases, the change in the word vector in the complete instruction recognition result is recorded as the attention coefficient of the word vector, and the first difference is obtained by the attention coefficient.

[0012] Preferably, the specific steps of obtaining the first difference are as follows:

[0013] The complete instruction recognition result and the dress label are both vector sequences, and the vector sequence contains a number of word vectors;

[0014] Obtain the Euclidean distance of the word vectors in the same order in the complete instruction recognition result and the dressing label, and use the attention coefficient of each word vector in the complete instruction recognition result to perform weighted summation on the Euclidean distances of all word vectors to obtain a first difference.

[0015] Preferably, the specific steps of the second difference are as follows:

[0016] The complete instruction recognition result and the reduced instruction recognition result are both vector sequences, and the vector sequences contain word vectors;

[0017] Obtain the Euclidean distances of the word vectors in the same order in the complete instruction recognition result and the simplified instruction recognition result; and record the average of the Euclidean distances of all word vectors in the complete instruction recognition result as the second difference.

[0018] Preferably, when the weight obtained after the same voice sample is input into the voice recognition model twice adjacently increases, the change in the word vector in the complete instruction recognition result is recorded as the attention coefficient of the word vector, which includes the following specific steps:

[0019] Each speech sample is input into the speech recognition model once, which means that each speech sample is trained once;

[0020] For the same speech sample, and when the speech sample is trained for the current time, the weight obtained during the first training before the current training is obtained, which is recorded as , the obtained complete instruction recognition result is recorded as ;

[0021] Get the weight obtained by the second training of the speech sample before the current training, and record it as , the obtained complete instruction recognition result is recorded as ;

[0022] and The i-th word vector is recorded as 、 ; The attention coefficient and and The difference is positively correlated with 、 The similarity is negatively correlated.

[0023] Preferably, the loss function obtained by weighted fusion of the second difference and the first difference includes the following specific formula:

[0024]

[0025] Among them, S represents the loss function, Indicates the first difference between the complete instruction recognition result R1 and the dress label R, represents a second difference between the complete instruction recognition result R1 and the reduced instruction recognition result R2; represents the weight of the second difference, Represents the loss function for weather forecast and image.

[0026] Preferably, the specific steps for obtaining the loss function of the weather forecast and image are as follows:

[0027] Each speech sample contains time instructions and body part instructions;

[0028] Mark the body part represented by the body part instruction in each voice sample, and record it as the body part label m2; mark the time represented by the time instruction in each voice sample, and record it as the time label m1;

[0029] After each speech sample in the dataset is input into the speech recognition model, the first encoder outputs the time, recorded as M1, and the first encoder outputs the body part, recorded as M2;

[0030]

[0031] in represents the Euclidean distance between M1 and m1, Represents the function value of the cross entropy loss function between M2 and m2.

[0032] Preferably, the fourth encoder is used to encode the time instruction contained in the voice and the weather forecast and image collected under the body part instruction, and then serves as the input of the third encoder, including the following specific steps:

[0033] The speech includes time instructions and body part instructions;

[0034] After the voice is input into the first encoder, the first encoder outputs the time described by the time instruction and the body part described by the body part instruction; the time is then used to perform weather forecast, obtain weather voice instructions, and capture a full-body image of the user; the weather voice instructions and the body parts in the full-body image are then input into the fourth encoder to obtain voice-assisted recognition features, and the voice-assisted recognition features are used as input to the third encoder.

[0035] Preferably, the speech recognition comprises the following specific steps:

[0036] After the speech recognition model training is completed, the second encoder is deleted, and the user stands in front of the smart mirror. The smart mirror collects the user's voice and inputs the voice into the trained speech recognition model. The first encoder outputs the time and body parts in the voice, as well as the first feature map; then the weather voice command at the time is obtained, and the user's full-body image is collected at the same time. The weather voice command and the body parts in the full-body image are input into the fourth encoder, and the speech-assisted recognition feature is obtained. The first feature map and the speech-assisted recognition feature are then stacked together and input into the third encoder together to obtain the complete instruction recognition result. The vocabulary corresponding to all word vectors in the complete instruction recognition result is used as the dressing plan required by the user.

[0037] Preferably, the weight of the second difference in the weighted fusion is positively correlated with the first difference, and the specific formula included is as follows:

[0038] Where L0 represents the preset normalization coefficient, w represents the weight of the second difference, Indicates the first difference between the complete instruction recognition result R1 and the clothing label R.

[0039] Another embodiment of the present invention provides a system for accurately recognizing voice commands applicable to smart mirrors. The system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned method for accurately recognizing voice commands applicable to smart mirrors are implemented.

[0040] The beneficial effects of the technical solution of the present invention are:

[0041] The multimodal speech recognition model of the present invention is trained and used as a whole, which preliminarily avoids the problems of excessively long speech recognition time and inaccurate speech recognition caused by error propagation. In addition, the present invention allows the parameters of the third encoder and the fourth encoder to be changed to further shorten the speech recognition time. In order to further shorten the speech recognition time while ensuring the accuracy of speech recognition, the present invention first obtains a first difference between the complete instruction recognition result and the dress label, then obtains a second difference between the complete instruction recognition result and the simplified instruction recognition result, fuses the first difference and the second difference, and the weight of the second difference during fusion is positively correlated with the first difference. When there is a large training error (i.e., when the first difference is large), this process allows the simplified instruction recognition result to be closer to the complete instruction recognition result. Since the simplified instruction recognition result is an instruction recognition result obtained in advance by skipping the image and weather feature extraction process, this enables the first encoder to learn and extract a portion of speech features in advance, avoiding the situation where most of the speech features and all of the image and weather features (i.e., speech-assisted recognition features) are placed in the third encoder with a smaller number of parameters for fusion, which may lead to insufficient feature extraction or feature learning capabilities for speech, image, and weather, thereby resulting in large errors. This makes the speech recognition result of this embodiment more accurate (i.e., the generated dress scheme is more reliable).

[0042] Furthermore, in the present invention, when the weight obtained after the same speech sample is input into the speech recognition model twice in a row increases, the change in the word vector in the complete instruction recognition result is recorded as the attention coefficient of the word vector, and the first difference is obtained from the attention coefficient. This allows the first encoder to learn and extract a portion of the speech features in advance, thereby preventing the speech features in the third encoder from being overwhelmed by the speech-assisted recognition features, which would prevent the third encoder from being unable to fuse the speech features with the speech-assisted recognition features. This reduces the time required for speech recognition and further improves the accuracy of speech recognition.

[0043] In summary, the present invention provides a fast and accurate voice recognition method when a user generates an outfit for a specified scene by looking in a mirror and interacting with voice. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A flowchart of a method for accurately recognizing voice commands for a smart mirror according to an embodiment of the present invention;

[0046] Figure 2 A schematic diagram showing the structure of a multimodal speech recognition model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] To further illustrate the technical means and effectiveness of the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the method and system for accurately recognizing voice commands for smart mirrors, including their specific implementation, structure, features, and effectiveness. In the following description, references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0048] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0049] The specific scheme of the method and system for accurately recognizing voice commands applicable to smart mirrors provided by the present invention is described in detail below with reference to the accompanying drawings.

[0050] Comparative Example:

[0051] A voice acquisition module (e.g., a microphone) is installed in the smart mirror to collect the user's voice. The voice includes time instructions, body part instructions, and scene instructions. The time instructions and scene instructions indicate the scene at the corresponding time, and the body part instructions indicate the body part to dress up. For example, "I'm going out shopping with friends this afternoon. What pants should I wear? How should I touch up my makeup?", "I'm going to work tomorrow morning. What upper body clothes should I change into?" In these voice instructions, "this afternoon" and "tomorrow morning" are time instructions, "shopping" and "work" are scene instructions, and "what pants to wear", "touch up makeup", and "upper body" are body part instructions.

[0052] One way to implement this is to input speech into an existing large speech model (such as ESPnet) to generate a dressing plan.

[0053] However, this implementation method cannot obtain weather information under the time instruction, and body shape, posture and existing dressing information under the body part instruction, resulting in the generated dressing plan not being what the user needs.

[0054] Therefore, in this embodiment, a weather forecast module is installed in the smart mirror. This module uses a large speech model (such as ESPnet) to identify the time in the speech and then obtains the weather forecast for that time (a spoken weather report, referred to as a weather voice command). In addition, a camera is installed in the smart mirror to capture a full-body image of the user. The large speech model (such as ESPnet) is then used to identify the body parts in the speech. An image description generation model (such as the BLIP / BLIP-2 model) is then used to generate spoken descriptions of the body parts in the full-body image, referred to as body part voice commands.

[0055] The user's voice, the weather voice commands, and the body part voice commands are spliced ​​together and then input into the large voice model to generate a dressing plan.

[0056] In particular, when no time instruction or body part instruction is recognized in the collected user's voice, the time and body part are defaulted to "today" and "upper body".

[0057] In the above process, the large speech model, image description generation model, etc. are all well-known technologies and have mature existing applications. They will not be described in detail in this embodiment. The weather forecast module is also an existing technology and will not be described in detail in this embodiment.

[0058] This comparative example suffers from the following problems: Firstly, the need to cascade and stack multiple models (or the need to cascade the same model multiple times) results in excessively long speech recognition times. Secondly, due to the propagation of errors between models (e.g., model hallucination), significant error amplification occurs, resulting in significantly erroneous or unreasonable speech command recognition results.

[0059] Example 1:

[0060] like Figure 1 As shown, this embodiment provides a method for accurately recognizing voice commands applicable to a smart mirror, including:

[0061] Step S101: The constructed speech recognition model includes: a first encoder, a second encoder, a third encoder and a fourth encoder, the second encoder outputs a simplified instruction recognition result; the third encoder outputs a complete instruction recognition result.

[0062] This implementation builds a multimodal speech recognition model, which includes a first encoder, a second encoder, a third encoder, and a fourth encoder. Figure 2 As shown, the specific principles are as follows:

[0063] (1) The user's voice is collected and input into the first encoder. The first encoder extracts features from the voice, and the feature map output by the first encoder is recorded as the first feature map. The first feature map is input into the second encoder. The second encoder further extracts features from the voice and outputs a dressing plan, which is recorded as the simplified instruction recognition result.

[0064] The simplified command recognition result represents the dressing scheme obtained without relying on weather commands and body part commands.

[0065] (2) The first encoder also outputs the time and body part indicated by the weather instruction and body part instruction in the speech.

[0066] After inputting the time and body part, the weather forecast and image acquisition functions are triggered. The weather voice command for that time is predicted (also known as the weather voice command), and a full-body image of the user is captured. The weather voice command and the body parts in the full-body image are then fed into a fourth encoder for encoding and compression (or feature extraction), resulting in a voice-assisted recognition feature (essentially a feature map).

[0067] (3) Inputting the first feature map and the speech-assisted recognition feature into the third encoder together; for example, making the speech-assisted recognition feature and the first feature map have the same size, stacking the speech-assisted recognition feature and the first feature map together and then inputting them into the third encoder.

[0068] The third encoder outputs a complete instruction recognition result, which represents a dressing plan generated after considering the weather, body shape and existing clothing. The dressing plan is the dressing plan that the smart mirror in this embodiment needs to generate and the user needs.

[0069] In this step, the multiple models used in the comparative example are merged into a multimodal speech recognition model, which is trained and used as a whole to preliminarily avoid the problems of long speech recognition time and inaccurate speech recognition caused by error propagation.

[0070] As an example, the first encoder, the second encoder, the third encoder, and the fourth encoder all use the Transformer network. The Transformer network uses a multi-layer self-attention mechanism to achieve deep feature learning or feature extraction of speech or image data. It is a well-known technology. For example, the Transformer network is used in ESPnet. In this embodiment, the first encoder, the second encoder, the third encoder, and the fourth encoder each layer of the self-attention mechanism can use the self-attention mechanism in ESPnet; in addition, the GPT4 large model and the BLIP / BLIP-2 model both have application examples of the Transformer network, which can be used in this embodiment. Here, this embodiment will not further elaborate on the specific network structures of the first encoder, the second encoder, the third encoder, and the fourth encoder.

[0071] Furthermore, this embodiment takes into account that after the speech is input into the multimodal speech recognition model and the first feature map is obtained, the third encoder cannot immediately output the complete command recognition result; instead, it must wait for the weather acquisition, image acquisition, and fourth encoder processing to complete, resulting in a relatively long speech recognition time. To further reduce speech recognition time, the parameters of the fourth and third encoders in this embodiment should not be set too large.

[0072] As an example, the Transformer network in the first encoder contains 5 layers of self-attention mechanism, the Transformer networks of the fourth and third encoders each contain 2 layers of self-attention mechanism, and the second encoder contains 1 layer of self-attention mechanism.

[0073] In other examples, other methods may be used to set the parameter amounts of the fourth encoder and the third encoder, so that the parameter amounts of the fourth encoder and the third encoder are relatively small, especially ensuring that the parameter amount of the third encoder is smaller than the parameter amount of the first encoder.

[0074] Step S102: The constructed data set includes a number of speech samples and clothing labels corresponding to the speech samples.

[0075] The voices of different users are collected and recorded as voice samples. Each voice sample contains time instructions, body part instructions, and scene instructions.

[0076] A full-body color image of the user is captured. For the body part described by the body part command, only the pixels within that body part are retained (the RGB values ​​of pixels outside the body part are all set to 0). This full-body image is recorded as the full-body image of the voice sample. A weather report (e.g., including temperature, humidity, and weather conditions) at a random moment is obtained from historical weather forecasts and recorded as the weather voice command.

[0077] Using the method of the above-mentioned comparative embodiment, voice instructions for body parts are generated based on the full-body image, and a dressing plan (in text format) is generated based on the voice sample, weather voice instructions, and body part voice instructions. The word2vec technology is used to generate a word vector for each word (including punctuation marks) in the dressing plan (in this embodiment, all word vectors are normalized into unit vectors with a dimension of 128). The word vectors of all words in the dressing plan constitute a vector sequence, which is recorded as the dressing label of the voice sample.

[0078] It should be noted that in order to unify the length of the vector sequence, this embodiment sets the length of the vector sequence to 1000. If the total number of words in the dressing scheme is less than 1000, then the positions without word vectors in the vector sequence are filled with 128-dimensional zero vectors (the zero vectors represent empty characters that do not need to be displayed).

[0079] The dataset now contains several voice samples, each of which corresponds to a weather voice command, a full-body image (a full-body image that only retains the pixels in the body parts), and a clothing label.

[0080] In some other embodiments, the voice samples in the data set are manually screened, for example, voice samples with obvious dressing errors are deleted, or the dressing plan is fine-tuned and modified according to the user's existing clothing.

[0081] Step S103: After each speech sample is input into the speech recognition model, a first difference between the complete instruction recognition result and the dress label is obtained, and a second difference between the complete instruction recognition result and the simplified instruction recognition result is obtained; the second difference is weightedly fused with the first difference to obtain a loss function, and the speech recognition model is trained using the data set and the loss function.

[0082] The training process includes:

[0083] (1) Traverse all the speech samples in the data set. For the currently traversed speech sample, input the speech sample into the multimodal speech recognition model (i.e., the first encoder), input the weather voice command and full-body image corresponding to the speech sample into the fourth encoder, and then obtain the simplified command recognition result (denoted as R2) and the complete command recognition result (denoted as R1) output by the multimodal speech recognition model.

[0084] It should be noted that the simplified instruction recognition result or the complete instruction recognition result of this embodiment is also a vector sequence. Each vector in the vector sequence is a word vector, and each word vector represents a vocabulary. The vocabulary corresponding to all word vectors in a vector sequence describes the dressing plan required by the user.

[0085] (2) Obtain the loss function S based on the difference between the complete instruction recognition result R1 and the dress label of each speech sample.

[0086] As a comparative example, the loss function S is obtained as follows:

[0087]

[0088] in The loss function for the weather forecast and body part corresponding to the time and body part instructions in the speech sample is simply called the loss function for the weather forecast and image. R represents the vector sequence corresponding to the clothing label of each speech sample. Indicates the difference between R1 and R.

[0089] As an example, ,in, represents the i-th word vector in R, represents the i-th word vector in R1; N represents the number of word vectors in R (in this example, N=1000), express and The Euclidean distance.

[0090] Considering that in order to further shorten the speech recognition time in this embodiment, it is necessary to ensure that the parameter amount of the third encoder is relatively small (see step S101 for details), but this will lead to insufficient feature extraction or feature learning capabilities of the third encoder for speech, especially the third encoder needs to simultaneously fuse the first feature map and the speech auxiliary recognition feature, which results in shortening the speech recognition time while directly using the loss function S provided by the above comparison example to ensure the accuracy of speech recognition.

[0091] As a preferred example, the loss function S is obtained as follows:

[0092] Get the difference between R1 and the dress label R of each speech sample, recorded as the first difference, expressed as , and obtain the difference between R1 and R2 at the same time, recorded as the second difference The first difference and the second difference are weightedly fused, and the weight of the second difference is positively correlated with the first difference during weighted fusion.

[0093] That is to say . where w represents the weight of the second difference.

[0094] In this process, when the difference between R1 and the dress label R of each speech sample (that is, the first difference ) is large, it means that the multimodal speech recognition model has a large recognition error for each speech sample, or the multimodal speech recognition model has not deeply learned or extracted the accurate features of each speech sample. The reason may be that: in order to further reduce the time of speech recognition, the number of parameters of the third encoder is relatively small, resulting in insufficient feature extraction or feature learning capabilities for speech, image and weather when the third encoder fuses the first feature map and speech-assisted recognition features.

[0095] and The calculation formula is the same.

[0096] At this time (when the first difference When w is larger, more attention is paid to the second difference. The size of this embodiment is more The size of the simplified instruction recognition result allows the simplified instruction recognition result to be closer to the complete instruction recognition result. Since the simplified instruction recognition result skips the feature extraction process of the image and weather and obtains the result in advance (that is, the dressing plan that the user may need), this enables the first encoder to learn and extract a part of the speech features in advance, avoiding the situation where most of the speech features and all the image and weather features (that is, speech-assisted recognition features) are placed in the third encoder with a smaller number of parameters for fusion, and the feature extraction or feature learning capabilities of speech, image and weather are insufficient, which leads to large errors.

[0097] As an example, in weighted fusion, the weight of the second difference is positively correlated with the first difference, and the formula involved is:

[0098]

[0099] Where L0 represents the normalization coefficient. In this embodiment, It is set to 5. In other embodiments, it can be set to other values, preferably a value greater than 1, which is not limited in this embodiment.

[0100] As another example, the loss function S is obtained as follows:

[0101] During the first 20 training sessions of the same speech sample, the method of the comparative example is used to obtain the loss function S. During the subsequent training sessions of the same speech sample, the method of the preferred example is used to obtain the loss function S.

[0102] By pre-training the same speech sample, this example can ensure that the first encoder and the third encoder learn some speech features in advance, avoiding the situation where the second encoder learns and stores too many speech features when the loss function S is subsequently obtained using the method of the above preferred example, and the first encoder cannot learn and extract speech features in advance.

[0103] (3) Using the above dataset and the above loss function S, the multimodal speech recognition model is trained using the stochastic gradient descent algorithm.

[0104] The specific training process is well known and will not be described in detail in this embodiment.

[0105] Step S104: Perform speech recognition using the trained multimodal speech recognition model.

[0106] Obtain a trained multimodal speech recognition model, remove the second encoder, and run the model on the smart mirror's computer (e.g., an embedded development board like a Raspberry Pi). The user stands in front of the smart mirror, which collects their voice and inputs it into the trained multimodal speech recognition model. The model first outputs (also output by the first encoder) the time and body part of the speech, along with a first feature map. Then, a weather voice command for the specified time is obtained, while a full-body image of the user is simultaneously captured, retaining only the body parts in the full-body image. The weather voice command and full-body image are input into a fourth encoder to generate speech-assisted recognition features. The first feature map and speech-assisted recognition features are then input into a third encoder to obtain a complete command recognition result. The vocabulary corresponding to all word vectors in the complete command recognition result is used as the desired outfit plan for the user.

[0107] This concludes the present embodiment.

[0108] The multimodal speech recognition model of this embodiment is trained and used as a whole, which preliminarily avoids the problems of excessively long speech recognition time and inaccurate speech recognition caused by error propagation. In addition, this embodiment allows the parameters of the third encoder and the fourth encoder to be changed to further shorten the speech recognition time. In order to further shorten the speech recognition time while ensuring the accuracy of speech recognition, this embodiment first obtains a first difference between the complete instruction recognition result and the dress label, and then obtains a second difference between the complete instruction recognition result and the simplified instruction recognition result. The first difference and the second difference are fused, and the weight of the second difference during fusion is positively correlated with the first difference. When there is a large training error (i.e., when the first difference is large), this process allows the simplified instruction recognition result to be closer to the complete instruction recognition result. Since the simplified instruction recognition result is an instruction recognition result obtained in advance by skipping the image and weather feature extraction process, this enables the first encoder to learn and extract a portion of the speech features in advance, avoiding the situation where most of the speech features and all of the image and weather features (i.e., speech-assisted recognition features) are placed in the third encoder with a smaller number of parameters for fusion, which may lead to insufficient feature extraction or feature learning capabilities for speech, image, and weather, thereby resulting in large errors. This makes the speech recognition result of this embodiment more accurate (i.e., the generated dress scheme is more reliable).

[0109] Example 2:

[0110] This embodiment further considers that, in order to further reduce the speech recognition time in Example 1, in addition to ensuring that the number of parameters of the third encoder is relatively small, it is also necessary to ensure (or allow) that the number of parameters of the fourth encoder is small (see step S101 for details). This results in the fourth encoder being unable to perform in-depth feature extraction or learning of weather modal data and image modal data, resulting in a high noise level in the speech-assisted recognition features (i.e., containing too many features that are useless for speech recognition).

[0111] When obtaining the loss function S according to the method of step S103 in Example 1, the following situation may occur: due to the introduction of the weight w, the third encoder extracts and learns more abstract and specific high-dimensional speech features, but the speech-assisted recognition features are noisy and contain a lot of useless information, which causes the speech features in the third encoder to be submerged in the speech-assisted recognition features, causing the third encoder to ignore the speech features and excessively learn and extract dressing plans from the weather and images, resulting in inaccurate speech recognition results (for example, the generated dressing plan is only for the weather, user body shape and current clothing, and does not address the content described in the scene instructions in the user's voice).

[0112] To solve this problem and reduce the speech recognition time while further improving the accuracy of speech recognition, this embodiment provides another method for obtaining the loss function S, which includes:

[0113] When the same speech sample is trained several times (for example, 50 times) using the method in Example 1 (that is, after the speech sample is input into the multimodal speech recognition model 50 times):

[0114] For the same speech sample, and the weight obtained during the first training before the current training (that is, when the speech sample was last input into the multimodal speech recognition model) during the current training (that is, when the speech sample was currently input into the multimodal speech recognition model) , and the complete instruction recognition results .

[0115] Get the weight of the speech sample obtained during the second training before the current training (that is, the last time the speech sample was input into the multimodal speech recognition model), recorded as , the obtained complete instruction recognition result is recorded as . and Any word vector in (taking the i-th word vector as an example) is recorded as 、 .

[0116] when compared to When larger, 、 The smaller the similarity (or the larger the difference), it means that before the current training, the first encoder has learned and extracted some speech features in advance, resulting in a mutation in the word vector in the complete instruction recognition result. This may be because the speech features in the third encoder are submerged in the speech auxiliary recognition features. At this time, the feature extraction and learning process of the i-th word vector needs to be placed in the third encoder, that is, the third encoder needs to focus on learning the relevant features of the i-th word vector.

[0117] Therefore, when Greater than When, and exists 、 When the similarity is less than the first preset threshold th1, the loss function S is obtained again:

[0118]

[0119] in, Indicates the second difference between the complete instruction recognition result R1 and the simplified instruction recognition result R2 during the current training. Indicates the first difference between the complete instruction recognition result R1 and the clothing label R during the current training. Represents the weight of the second difference during current training.

[0120] The first difference , Represents the i-th word vector in the dress label of the speech sample, Represents the i-th word vector in R1. Representing word vectors With word vector The Euclidean distance is , and N represents the number of word vectors in the clothing label.

[0121] in represents the attention coefficient of the i-th word vector in the complete instruction recognition result, and and The difference is positively correlated with 、 The similarities are negatively correlated.

[0122] This embodiment is described by taking th1=0.45 as an example.

[0123] As an example, , express and Cosine similarity, where Describes the changes in word vectors in the complete instruction recognition results between two adjacent training sessions (i.e., the last and the previous training session). The larger the value, the more serious the mutation of the word vector. It describes the changes in the word vector in the complete instruction recognition result when the weight of the second difference increases. As mentioned above, the larger the value, the more it means that during the current training, the first encoder learned and extracted a part of the speech features in advance, which caused the word vector in the complete instruction recognition result to mutate. This may be because the speech features in the third encoder are submerged in the speech auxiliary recognition features. At this time, the feature extraction and learning process of the i-th word vector needs to be placed in the third encoder, that is, the third encoder needs to focus on learning the relevant features of the i-th word vector.

[0124] Special, When it is less than or equal to 0.05, let , the purpose of which is to avoid The value is too small, resulting in the denominator being 0.

[0125] when No more than When, or and When the cosine similarity of the word vectors in is not less than the first preset threshold th1, the loss function S is obtained according to the method in Example 1, which is equivalent to the loss function in this embodiment. Always equal to 1.

[0126] w represents the weight of the second difference, .

[0127] It should be noted that the present embodiment is run after several (for example, 50) trainings are performed using the method in the first embodiment, and the weights of the second difference obtained in the last and last trainings (i.e., w and ) to obtain the first difference during the current training , and then get the weight of the second difference during the current training , and then obtain the loss function of the current training. In this process, in particular, when the last or the previous training is less than 50 times, then the weights of the first difference and the second difference All were obtained according to the method of Example 1.

[0128] In summary, the first difference of this embodiment is The first difference from the first embodiment In this embodiment Compared with the embodiment 1 The former introduces the attention coefficient of the word vector, so that the first encoder learns and extracts a part of the speech features in advance, which to a certain extent avoids the speech features in the third encoder from being submerged in the speech-assisted recognition features, resulting in the third encoder being unable to fuse the speech features and the speech-assisted recognition features for learning. This further improves the accuracy of speech recognition while reducing the time of speech recognition.

[0129] Example 3:

[0130] This embodiment provides a method for obtaining a loss function of weather forecast and image , the method comprising:

[0131] In this embodiment, after collecting the user's voice and inputting it into the multimodal voice recognition model (i.e., the first encoder), the weather forecast function and image acquisition function need to be automatically triggered. Therefore, it is necessary to enable the first encoder to have the ability to recognize time and body parts from the voice. The specific method is to use the dataset to train the multimodal voice recognition model by constructing a loss function. , to enable the multimodal speech recognition model to learn the time and body parts in speech. The specific method is:

[0132] (1) For each speech sample in the dataset, the time indicated by the time instruction in each speech sample is manually marked and recorded as a time tag. In this embodiment, the time is a vector consisting of date and hour. For example, the time [a, b, c, d] represents a month b day c hour, where d represents a special time tag. When the time instruction includes relative times such as "today", "tomorrow", and "the day after tomorrow", d is equal to 0, 1, and 2 respectively (in this case, a and b are both equal to -1); when it does not include the above relative times, d = -1. In particular, when the time instruction includes general times such as "morning", "am", "noon", "afternoon", "evening", and "today", c is set to 7, 10, 12, 15, 20, and 12.

[0133] In other embodiments, other methods may be used to mark the time indicated by the time instruction. The specific marking method is common knowledge and will not be described in detail or limited in this embodiment.

[0134] In addition, the body part indicated by the body part command in each speech sample is manually annotated and recorded as the body part label. The annotated body part command areas are divided into multiple categories and represented by one-hot codes as 0001, 0010, 0100, and 1000, respectively. 0001 represents the head (face), 0010 represents the upper body, 0100 represents the legs, and 1000 represents the feet. Furthermore, the one-hot codes between the above four categories are combined through an "OR" operation to obtain other categories. For example, the "OR" operation of 0001 and 0010 produces 0011, which represents the head and upper body. Another example is the "OR" operation of the four one-hot codes 0001, 0010, 0100, and 1000 produces 1111, which represents the entire body.

[0135] In other embodiments, the speech may be input into an existing speech model (eg, ESPnet), and the speech model may be used to automatically recognize and generate the aforementioned time tags and body part tags.

[0136] (2) After traversing each speech sample in the dataset and inputting the speech sample into the first encoder, the time output by the first encoder is recorded as M1, and the body part output is recorded as M2. The time label and body part label corresponding to each speech sample are recorded as m1 and m2 respectively;

[0137]

[0138] in represents the Euclidean distance between M1 and m1, Represents the function value of the cross entropy loss function between M2 and m2.

[0139] So far, we have obtained the loss function , and then train the multimodal speech recognition model using the method of step S103 in embodiment 1.

[0140] As described in step S104 of Example 1: a user stands in front of a smart mirror, which collects the user's voice and inputs it into a trained multimodal speech recognition model. The model first outputs (i.e., outputs the first encoder) the time and body part of the speech, along with a first feature map. A weather voice command for the time is then acquired, while a full-body image of the user is simultaneously captured, retaining only the body parts in the full-body image. The weather voice command and full-body image are input into a fourth encoder to generate voice-assisted recognition features. The first feature map and voice-assisted recognition features are then input into a third encoder to generate a complete command recognition result. The vocabulary corresponding to all word vectors in the complete command recognition result is used as the user's desired outfit plan.

[0141] The model first outputs (i.e., the first encoder outputs) the time and body parts in the speech, specifically including:

[0142] The model output (i.e., the first encoder output) represents the time in the speech in vector format, again represented as [a, b, c, d], where a, b, c, and d are rounded to the nearest integer. When d is less than 0, the speech time represents a month, b day, and c hour. When d is greater than or equal to 0, the date represented by a month, b day is d days after the current date.

[0143] The body parts in the speech output by this model (that is, the output of the first encoder) are in one-hot encoding format. This encoding format is mapped to the body parts described above. After collecting a full-body image of the user, image recognition technology (such as YOLOV3 technology) is used to identify and select the body parts in the full-body image (the RGB values ​​of the pixels in the area outside the body part are all set to 0).

[0144] Finally, the weather voice command for month a, day b, and time c and the full-body image are input into the fourth encoder to obtain the voice-assisted recognition features. Then, the first feature map and the voice-assisted recognition features are input into the third encoder to obtain the complete command recognition result. The vocabulary corresponding to all word vectors in the complete command recognition result is used as the dressing plan required by the user.

[0145] Example 4:

[0146] This embodiment provides a method for inputting a weather voice instruction and a full-body image into a fourth encoder, the method comprising:

[0147] First, a Transformer network with a 2-layer attention mechanism is used to encode the weather voice command into a feature map (denoted as the weather feature map). Then, another Transformer network with a 2-layer attention mechanism is used to encode the full-body image into a feature map (denoted as the body feature map). These two feature maps have the same size. These two feature maps are stacked together to obtain a spliced ​​feature map. Finally, another Transformer network with a 1-layer attention mechanism is used to encode the spliced ​​feature map into the voice-assisted recognition feature.

[0148] The Transformer network used above, which includes a total of 5 layers of attention mechanisms, is the fourth encoder described in this embodiment.

[0149] Embodiment 5:

[0150] This embodiment provides a precise voice command recognition system for a smart mirror. The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps included in all the above embodiments are implemented. Furthermore, the system includes a smart mirror having a microphone for capturing user voice, a camera for capturing full-body images of the user, and a weather forecast module for weather forecasting. The memory and processor are mounted on the back of the smart mirror. Furthermore, a display screen is mounted on the mirror surface of the smart mirror to display the generated dressing plan.

[0151] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for accurately recognizing voice commands for a smart mirror, characterized in that: The method comprises the following steps: The constructed speech recognition model includes: inputting the collected speech into a first encoder, the output of the first encoder serving as the input of a second encoder and a third encoder respectively, and the word vector sequence output by the second encoder being recorded as a simplified instruction recognition result; the simplified instruction recognition result represents a dressing plan obtained when it does not rely on weather instructions and body part instructions, and is obtained by: collecting the user's speech, inputting the speech into the first encoder, the first encoder extracting features from the speech, and the feature map output by the first encoder being recorded as a first feature map; the first feature map is input into the second encoder, the second encoder further extracts features from the speech, and outputs a dressing plan, which is recorded as a simplified instruction recognition result; The fourth encoder encodes the time instructions contained in the speech, the weather forecast, and the image captured under the body part instructions, and then uses them as input to the third encoder. The word vector sequence output by the third encoder is recorded as the complete instruction recognition result. The third encoder has fewer parameters than the first encoder. The constructed dataset includes several speech samples and clothing labels corresponding to the speech samples. When each speech sample is input into a speech recognition model, a first difference between the complete command recognition result and the clothing label is obtained, and a second difference between the complete command recognition result and the simplified command recognition result is obtained. The second difference is weightedly fused with the first difference to obtain a loss function. The speech recognition model is trained using the dataset and the loss function to perform speech recognition. Among them, the weight of the second difference in weighted fusion is positively correlated with the first difference; When the weight obtained after the same speech sample is input into the speech recognition model twice in a row increases, the change in the word vector in the complete instruction recognition result is recorded as the attention coefficient of the word vector, and the first difference is obtained by the attention coefficient; After the speech recognition model training is completed, the second encoder is deleted, and the user stands in front of the smart mirror. The smart mirror collects the user's voice and inputs the voice into the trained speech recognition model. The first encoder outputs the time and body parts in the voice, as well as the first feature map; then the weather voice command at the time is obtained, and the user's full-body image is collected at the same time. The weather voice command and the body parts in the full-body image are input into the fourth encoder, and the speech-assisted recognition feature is obtained. The first feature map and the speech-assisted recognition feature are then stacked together and input into the third encoder together to obtain the complete instruction recognition result. The vocabulary corresponding to all word vectors in the complete instruction recognition result is used as the dressing plan required by the user.

2. The method for accurately recognizing voice commands for a smart mirror according to claim 1, characterized in that: The specific steps for obtaining the first difference are as follows: The complete instruction recognition result and the dress label are both vector sequences, and the vector sequence contains a number of word vectors; Obtain the Euclidean distance of the word vectors in the same order in the complete instruction recognition result and the dressing label, and use the attention coefficient of each word vector in the complete instruction recognition result to perform weighted summation on the Euclidean distances of all word vectors to obtain a first difference.

3. The method for accurately recognizing voice commands for a smart mirror according to claim 1, characterized in that: The specific steps of the second difference are as follows: The complete instruction recognition result and the reduced instruction recognition result are both vector sequences, and the vector sequences contain word vectors; Obtain the Euclidean distances of the word vectors in the same order in the complete instruction recognition result and the simplified instruction recognition result; and record the average of the Euclidean distances of all word vectors in the complete instruction recognition result as the second difference.

4. The method for accurately recognizing voice commands for a smart mirror according to claim 1, wherein: When the weight obtained after the same speech sample is input into the speech recognition model twice in a row increases, the change in the word vector in the complete instruction recognition result is recorded as the attention coefficient of the word vector, which includes the following specific steps: Each speech sample is input into the speech recognition model once, which means that each speech sample is trained once; For the same speech sample, and when the speech sample is trained for the current time, the weight obtained during the first training before the current training is obtained, which is recorded as , the obtained complete instruction recognition result is recorded as ; Get the weight obtained by the second training of the speech sample before the current training, and record it as , the obtained complete instruction recognition result is recorded as ; and The i-th word vector is recorded as 、 ; The attention coefficient and and The difference is positively correlated with 、 The similarity is negatively correlated.

5. The method for accurately recognizing voice commands for a smart mirror according to claim 1, wherein: The loss function is obtained by weighted fusion of the second difference and the first difference, and the specific formula included is as follows: Among them, S represents the loss function, Indicates the first difference between the complete instruction recognition result R1 and the dress label R, represents a second difference between the complete instruction recognition result R1 and the reduced instruction recognition result R2; represents the weight of the second difference, Represents the loss function for weather forecast and image.

6. The method for accurately recognizing voice commands for a smart mirror according to claim 5, characterized in that: The specific steps for obtaining the loss function of the weather forecast and image are as follows: Each speech sample contains time instructions and body part instructions; Mark the body part represented by the body part instruction in each voice sample, and record it as the body part label m2; mark the time represented by the time instruction in each voice sample, and record it as the time label m1; After each speech sample in the dataset is input into the speech recognition model, the first encoder outputs the time, recorded as M1, and the first encoder outputs the body part, recorded as M2; in represents the Euclidean distance between M1 and m1, Represents the function value of the cross entropy loss function between M2 and m2.

7. The method for accurately recognizing voice commands for a smart mirror according to claim 1, characterized in that: The fourth encoder is used to encode the time instruction contained in the speech, the weather forecast and the image collected under the body part instruction, and then serves as the input of the third encoder. The specific steps include the following: The speech includes time instructions and body part instructions; After the voice is input into the first encoder, the first encoder outputs the time described by the time instruction and the body part described by the body part instruction; the time is then used to perform weather forecast, obtain weather voice instructions, and capture a full-body image of the user; the weather voice instructions and the body parts in the full-body image are then input into the fourth encoder to obtain voice-assisted recognition features, and the voice-assisted recognition features are used as input to the third encoder.

8. The method for accurately recognizing voice commands for a smart mirror according to claim 1, wherein: The weight of the second difference in the weighted fusion is positively correlated with the first difference, and the specific formula included is as follows: Where L0 represents the preset normalization coefficient, w represents the weight of the second difference, Indicates the first difference between the complete instruction recognition result R1 and the clothing label R.

9. A precise voice command recognition system for a smart mirror, the system comprising: The invention comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method for accurately recognizing voice commands applicable to a smart mirror as described in any one of claims 1 to 8 are implemented.

Images