A speech recognition method, apparatus, electronic device, and storage medium

Abstract

The application relates to a speech recognition method and device, electronic equipment and a storage medium. Speech samples and personal attribute information of a user are collected, speech features in the speech samples are extracted, a pronunciation deviation degree table of the user is generated according to the speech features and the personal attribute information, in response to receiving a to-be-recognized speech of the user, multi-dimensional sensor data of the user is acquired, a context feature vector is generated based on the multi-dimensional sensor data, acoustic features of the to-be-recognized speech, the pronunciation deviation degree table and the context feature vector are input into a trained speech recognition model, and the speech recognition model outputs final speech recognition text. Based on dynamically creating the pronunciation deviation degree table of the user, precise adaptation of thousands of people with thousands of voices is realized, and the speech recognition accuracy of different individual users is improved. Meanwhile, based on real-time acquisition of multi-dimensional sensor data, context feature information is generated and deeply fused with the pronunciation features of the user, effectively solving the semantic ambiguity problem.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a speech recognition method, device, electronic device, and storage medium. Background Technology

[0002] With the development of the Internet and the popularization of smart wearable devices, portable smart voice interaction devices (such as smart bracelets) are small in size and limited in power consumption, and cannot rely on large cloud models to provide strong computing power support like smart speakers. Existing voice recognition solutions generally adopt the traditional architecture of general acoustic models and fixed language models, which can achieve natural voice interaction.

[0003] However, existing speech recognition methods are still severely affected by environmental noise, lack the ability to adapt to individual user differences, have rigid interaction logic, rely on a single speech signal for recognition, and cannot meet the real-time interaction requirements of evolving as you speak. Summary of the Invention

[0004] This application provides a speech recognition method, device, electronic device, and storage medium to solve the technical problems of existing speech recognition methods being susceptible to noise interference, lacking adaptive capabilities to individual user differences, relying solely on a single speech signal for recognition, and having rigid interaction logic.

[0005] In a first aspect, this application provides a speech recognition method, the method comprising:

[0006] Collect users' voice samples and personal attribute information, and extract voice features from the voice samples;

[0007] Based on the speech features and the personal attribute information, a pronunciation deviation table for the user is generated;

[0008] In response to receiving the user's voice to be recognized, the system acquires the user's current multi-dimensional sensor data and generates a contextual feature vector based on the multi-dimensional sensor data.

[0009] The acoustic features of the speech to be recognized, the pronunciation deviation table, and the contextual feature vector are input into a trained speech recognition model so that the speech recognition model outputs the final speech recognition text; wherein, the speech recognition model is configured to: compensate for speech deviations in the speech to be recognized based on the pronunciation deviation table, and perform semantic disambiguation on the recognition result of the speech to be recognized based on the contextual feature vector.

[0010] In one possible implementation, the collection of user voice samples and personal attribute information includes:

[0011] The system prompts the user to read aloud preset fixed text and open text, and collects the user's speech samples during the reading process; wherein, the fixed text is used to collect the user's phoneme features, and the open text is used to simulate the user's natural expression habits;

[0012] In response to the user's input, the system obtains the user's personal attribute information; wherein the personal attribute information includes at least age, gender, and permanent address information.

[0013] In one possible implementation, generating the user's pronunciation deviation table based on the speech features and the personal attribute information includes:

[0014] Based on the permanent address information in the personal attribute information, the corresponding standard dialect database is called from multiple preset dialect databases;

[0015] For each speech feature, the following processing is performed to generate the user's pronunciation deviation table:

[0016] Determine the difference measurement data between the standard dialect database and the speech features;

[0017] The pronunciation deviation parameter is obtained by weighted fusion of the difference measurement data and the age factor in the personal attribute information.

[0018] The pronunciation deviation parameter and the corresponding speech feature are stored in the pronunciation deviation table.

[0019] In one possible implementation, determining the difference measurement data between the standard dialect database and the speech features includes:

[0020] Obtain the standard speech features corresponding to the speech features from the standard dialect database;

[0021] The differences between the speech features and the standard speech features are compared to obtain difference measurement data; wherein, the difference measurement data includes at least the distance between the speech features and the standard speech features in the acoustic feature space, and the divergence of the speech features and the standard speech features in the distribution.

[0022] In one possible implementation, acquiring the user's current multi-dimensional sensor data includes:

[0023] Location data is obtained based on positioning technology;

[0024] Motion state data is acquired using inertial measurement tools;

[0025] Acoustic data of the current environment is obtained by collecting sound from the microphone, and the current timestamp data is also obtained.

[0026] The positioning data, motion state data, acoustic data, and timestamp data are identified as multi-dimensional sensor data.

[0027] In one possible implementation, generating the context feature vector based on the multi-dimensional sensor data includes:

[0028] Sensor data of different dimensions are mapped into feature vectors respectively;

[0029] Multiple feature vectors are concatenated to form a fused feature sequence;

[0030] The feature sequence is input into a neural network model, which outputs a contextual feature vector; wherein the neural network model is configured with an encoder to process the feature sequence.

[0031] In one possible implementation, the speech recognition model is trained in the following manner:

[0032] A training sample set is constructed based on the acoustic features of speech samples, their corresponding real text labels, pronunciation deviation features, and contextual feature vectors.

[0033] The acoustic features, pronunciation deviation features, and contextual feature vectors are used as model inputs, and the real text labels are used as training targets. The initial speech recognition model is trained using a masking loss function and a contextual completion function. The masking loss function is used to randomly mask some acoustic features or text labels during training, and the contextual completion function is used to predict the masked parts using the context and the contextual feature vectors.

[0034] Secondly, this application provides a speech recognition device, the device comprising:

[0035] The information collection module is used to collect users' voice samples and personal attribute information, and extract voice features from the voice samples;

[0036] The pronunciation deviation determination module is used to generate a pronunciation deviation table for the user based on the speech features and the personal attribute information;

[0037] The context feature vector generation module is used to, in response to receiving the user's voice to be recognized, acquire the user's current multi-dimensional sensor data, and generate a context feature vector based on the multi-dimensional sensor data;

[0038] The speech recognition module is used to input the acoustic features of the speech to be recognized, the pronunciation deviation table, and the contextual feature vector into a trained speech recognition model, so that the speech recognition model outputs the final speech recognition text; wherein, the speech recognition model is configured to: compensate for the speech deviation in the speech to be recognized based on the pronunciation deviation table, and perform semantic disambiguation on the recognition result of the speech to be recognized based on the contextual feature vector.

[0039] In one possible implementation, the information collection module is specifically used for:

[0040] The system prompts the user to read aloud preset fixed text and open text, and collects the user's speech samples during the reading process; wherein, the fixed text is used to collect the user's phoneme features, and the open text is used to simulate the user's natural expression habits;

[0041] In response to the user's input, the system obtains the user's personal attribute information; wherein the personal attribute information includes at least age, gender, and permanent address information.

[0042] In one possible implementation, the pronunciation deviation determination module includes:

[0043] The dialect database determination unit is used to call the corresponding standard dialect database from multiple preset dialect databases based on the permanent address information in the personal attribute information.

[0044] The pronunciation deviation table generation unit is used to perform the following processing for each speech feature to generate the user's pronunciation deviation table:

[0045] Determine the difference measurement data between the standard dialect database and the speech features;

[0046] The pronunciation deviation parameter is obtained by weighted fusion of the difference measurement data and the age factor in the personal attribute information.

[0047] The pronunciation deviation parameter and the corresponding speech feature are stored in the pronunciation deviation table.

[0048] In one possible implementation, the pronunciation deviation table generation unit is specifically used for:

[0049] Obtain the standard speech features corresponding to the speech features from the standard dialect database;

[0050] The differences between the speech features and the standard speech features are compared to obtain difference measurement data; wherein, the difference measurement data includes at least the distance between the speech features and the standard speech features in the acoustic feature space, and the divergence of the speech features and the standard speech features in the distribution.

[0051] In one possible implementation, the context feature vector generation module is specifically used for:

[0052] Location data is obtained based on positioning technology;

[0053] Motion state data is acquired using inertial measurement tools;

[0054] Acoustic data of the current environment is obtained by collecting sound from the microphone, and the current timestamp data is also obtained.

[0055] The positioning data, motion state data, acoustic data, and timestamp data are identified as multi-dimensional sensor data.

[0056] In one possible implementation, the context feature vector generation module is specifically used for:

[0057] Sensor data of different dimensions are mapped into feature vectors respectively;

[0058] Multiple feature vectors are concatenated to form a fused feature sequence;

[0059] The feature sequence is input into a neural network model, which outputs a contextual feature vector; wherein the neural network model is configured with an encoder to process the feature sequence.

[0060] In one possible implementation, the speech recognition model is trained in the following manner:

[0061] A training sample set is constructed based on the acoustic features of speech samples, their corresponding real text labels, pronunciation deviation features, and contextual feature vectors.

[0062] The acoustic features, pronunciation deviation features, and contextual feature vectors are used as model inputs, and the real text labels are used as training targets. The initial speech recognition model is trained using a masking loss function and a contextual completion function. The masking loss function is used to randomly mask some acoustic features or text labels during training, and the contextual completion function is used to predict the masked parts using the context and the contextual feature vectors.

[0063] Thirdly, this application provides an electronic device, including: a processor and a memory, wherein the processor is configured to execute a speech recognition program stored in the memory to implement the speech recognition method described in any one of the first aspects.

[0064] Fourthly, this application provides a storage medium storing one or more programs that can be executed by one or more processors to implement the speech recognition method described in any one aspect.

[0065] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application collects user voice samples and personal attribute information, extracts voice features from the voice samples, generates a user's pronunciation deviation table based on the voice features and personal attribute information, and in response to receiving the user's voice to be recognized, acquires the user's current multi-dimensional sensor data, generates a contextual feature vector based on the multi-dimensional sensor data, and inputs the acoustic features of the voice to be recognized, the pronunciation deviation table, and the contextual feature vector into a trained speech recognition model so that the speech recognition model outputs the final speech recognition text. Based on cold-start registration, the method dynamically creates the user's pronunciation deviation table, achieving precise adaptation to individual users and improving the accuracy of speech recognition for different individuals. Simultaneously, based on real-time collected multi-dimensional sensor data, contextual feature information is generated and deeply integrated with the user's pronunciation features, effectively solving semantic ambiguity problems and making interaction more natural and accurate. With limited hardware resources, a high level of intelligent voice interaction is achieved based on a lightweight speech recognition model. Attached Figure Description

[0066] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0067] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0069] Figure 1 A flowchart illustrating an embodiment of a speech recognition method provided in this application;

[0070] Figure 2 An embodiment flow of another speech recognition method provided in this application;

[0071] Figure 3A flowchart illustrating another embodiment of the speech recognition method provided in this application;

[0072] Figure 4 This is a schematic diagram of the structure of a speech recognition device provided in an embodiment of this application;

[0073] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0075] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0076] To address the shortcomings of existing speech recognition methods, such as susceptibility to background noise, lack of adaptability to individual user differences, reliance on single speech signals, and rigid interaction logic, this application provides a speech recognition method, device, electronic device, and storage medium. It enables dynamic creation of a user's pronunciation deviation table based on cold-start registration, achieving precise adaptation to individual user voices and improving the accuracy of speech recognition for diverse individual users. Simultaneously, based on real-time collected multi-dimensional sensor data, it generates contextual feature information and deeply integrates it with the user's pronunciation features, effectively resolving semantic ambiguity and making interaction more natural and accurate. With limited hardware resources, a high level of intelligent voice interaction is achieved based on a lightweight speech recognition model.

[0077] Figure 1 A flowchart illustrating an embodiment of a speech recognition method provided in this application is shown below. Figure 1 As shown, the method applied to smart voice interaction devices (such as smart bracelets worn on the body) includes the following steps:

[0078] Step 101: Collect users' voice samples and personal attribute information, and extract voice features from the voice samples.

[0079] The voice sample can be the voice data read by the user, which is essentially the voice recording spoken by the user for registering their identity and establishing a pronunciation model. It usually includes fixed texts covering all phonemes and open texts simulating natural conversations. For example, it can include fixed texts covering all Mandarin phonemes (such as "The mountains are bright and the waters are clear, and people live in harmony between heaven and earth") and open texts simulating daily expressions (such as "I want to go to the supermarket to buy fruits today"), which are used to capture the user's basic pronunciation habits.

[0080] Personal attribute information can be static data describing the inherent characteristics of the user, at least including age, gender, and permanent address. It is essentially the key basis for adapting to pronunciation characteristics.

[0081] Voice features can be a general term for digital features extracted from the original voice signal. It can include acoustic features such as MFCC (Mel-Frequency Cepstral Coefficients), which mainly characterize the pronunciation content (i.e., what sounds are spoken), and can also include speaker features such as x-vector voiceprint features, which mainly characterize the speaker's identity (i.e., who is speaking). It is essentially the digital features extracted from the voice sample that can characterize the pronunciation characteristics and can reflect the core acoustic information of the voice sample. For example, it can include the pronunciation of flat and curled tongues, tones, phoneme duration, etc., and can also include the flat and curled tongue confusion features where the user pronounces "supermarket" as "cao market".

[0082] In one embodiment, when the device is used for the first time, guide the user to enter the cold start registration process, collect the age, gender, and permanent region filled in or selected by the user on the supporting APP, record the voice sample of the user reading the relevant text, and after preprocessing the recorded voice sample, extract the voiceprint features.

[0083] Exemplarily, after the device is powered on for the first time, guide the user to open the supporting APP, fill in the age, gender, and residential address, synchronize the information to the local storage of the device to protect the user's privacy. At the same time, the device plays a guiding message: "Please read the following sentences to complete the registration". First, let the user read 3 fixed texts, such as "Dad holds the baby, and the baby laughs heartily" and "Four is four, ten is ten", covering all Mandarin phonemes. Then read 10 open texts, such as "I like to eat strawberry cakes" and "Mom takes me to the park to play", simulating daily expressions. The dual microphones collect voice data at a sampling rate of 16 kHz. Extract the x-vector voiceprint (for subsequent identity binding) and voice features (such as the pronunciation of flat and curled tongues, tone deviation, for example, this child often pronounces "sì" as "shì") from the collected voice samples.

[0084] Step 102: Generate a pronunciation deviation table for the user according to the voice features and personal attribute information.

[0085] The pronunciation deviation table can be a customized and quantified pronunciation medical record or accent profile for users. In the form of a data table, it systematically records the degree of difference between the user's pronunciation habits (phonetic features) and the standard pronunciation for each phoneme. In essence, it is an exclusive pronunciation correction reference table generated based on the user's phonetic features and personal attribute information, quantifying the deviation degree of each phoneme of the user from the standard Mandarin phonemes with specific numerical values, and at the same time marking the types of pronunciation deviations that frequently occur to the user, such as confusion between retroflex and alveolar consonants, and confusion between anterior and posterior nasal sounds.

[0086] A phoneme can be the smallest speech unit in linguistics and phonetics, referring to the smallest sound segment that can distinguish meanings in human language. Generally speaking, it is the smallest building block of speech. Different combinations of phonemes can form different syllables and words, thereby distinguishing different semantic meanings.

[0087] Exemplarily, in Mandarin, "爸 (bà)" consists of two phonemes, "b" and "à", and "妈 (mā)" consists of two phonemes, "m" and "ā". It is precisely the two different phonemes, the initial "b" and "m", that make the meanings of "爸" and "妈" essentially different. For example, "超 (chāo)" contains two phonemes, "ch" and "āo", and "草 (cǎo)" contains two phonemes, "c" and "ǎo". The phoneme difference between "ch" and "c" directly determines the different semantic meanings of the two words.

[0088] In speech recognition technology, phonemes are the core processing units. The system will first split the user's speech into individual phonemes, and then compare them with the standard phoneme library to judge the pronunciation deviation (such as calculating the deviation degree between the 'c' phoneme read by the user and the standard 'ch' phoneme), so as to achieve accurate speech recognition and correction.

[0089] In one embodiment, a dialect knowledge base associated with the permanent address filled in by the user is called, and the MFCC features generated when the user reads the text are compared with the MFCC feature template of the standard dialect phoneme by phoneme. For each phoneme, calculate its deviation degree from the target phoneme, and generate the pronunciation deviation table of this user.

[0090] Exemplarily, calculate the pronunciation deviation degree between the user's actual pronunciation / z / and the standard pronunciation / zh / , and store it in the corresponding pronunciation deviation table, as shown in Table 1 below:

[0091] Table 1

[0092]

[0093] The target phonemes can be standard phonemes from a standard dialect database; common actual pronunciations used to record phonemes that users habitually mispronounce; and the pronunciation deviation can be calculated using a preset calculation standard or formula. This is merely an example. In addition, the pronunciation deviation table can also include phoneme descriptions, historical frequency of occurrence, and typical vocabulary, etc., to record the user's pronunciation deviations. This application embodiment does not impose any limitations on this.

[0094] Step 103: In response to receiving the user's voice to be recognized, acquire the user's current multi-dimensional sensor data and generate a contextual feature vector based on the multi-dimensional sensor data.

[0095] Multi-dimensional sensor data can be raw data collected in real time from various sensors on a device, describing the user's current state. For example, multi-dimensional sensor data may include location information, motion state data, environmental acoustic fingerprints, and timestamps.

[0096] Context feature vectors can be low-dimensional, dense numerical vectors. They are the essence of multi-dimensional sensor data after being fused and encoded by deep neural networks, and are a readable digital description of the user, environment, and state.

[0097] In one embodiment, in response to a user speaking a voice command, the system synchronously activates all sensors to collect data, collects ambient sound through dual microphones, collects and calculates the user's current motion state, obtains positioning information based on GNSS (Global Navigation Satellite System), and records the current timestamp; the above data is input into the Transformer encoder, and a comprehensive, low-dimensional contextual feature vector is output.

[0098] Exemplarily, when the user utters a voice command, the system synchronously starts all sensors for data collection. For example, through GNSS and indoor Bluetooth beacons, it can be determined that the user is in the living room of the home. Based on the calculation and analysis of the data of the IMU (Inertial Measurement Unit) built into the device, it is judged that the current motion state of the user is stationary; based on the ambient sound collected by the microphone, after processing, there is the sound of TV playing and a small amount of conversation sound in the current environment. The above-mentioned collected sound and the portrait of the current user (such as a 70-year-old male) are input into a lightweight Transformer encoder, and finally a comprehensive and low-dimensional context feature vector is output, such as [0.7, 0.95, 0.1, 0.3, 0.6,...]. Among them, 0.7 in this vector can represent the time feature: weekend evening, leisure time; 0.95 can represent the space type: home living room, with a very high confidence level; 0.1 can represent the motion feature: stationary, almost no movement; 0.3 can represent the ambient volume: relatively quiet; 0.6 can represent the user age: elderly. Here is just an example, and the embodiments of this application are not limited thereto.

[0099] Step 104: Input the acoustic features of the voice to be recognized, the pronunciation deviation table, and the context feature vector into the trained voice recognition model, so that the voice recognition model outputs the final voice recognition text; wherein, the voice recognition model is configured to: compensate for the voice deviation in the voice to be recognized based on the pronunciation deviation table, and perform semantic disambiguation on the recognition result of the voice to be recognized based on the context feature vector.

[0100] The acoustic features can refer to the underlying acoustic information extracted from the voice to be recognized. Commonly used 13-dimensional MFCC features can reflect the frequency distribution, energy change, timbre characteristics, etc. of the voice, and are the basic input of the voice recognition model.

[0101] The voice recognition model can be a machine learning model that has been specially trained and has the ability to understand the context. Its core innovation lies in being configured to be able to simultaneously utilize: the pronunciation deviation table for voice deviation compensation (correcting personalized pronunciation problems), and the context feature vector for semantic disambiguation (understanding the true intention according to the scenario).

[0102] Voice deviation compensation can refer to that the model performs fault tolerance adaptation on the high-deviation phonemes (such as deviation > 0.5) in the voice to be recognized by the user based on the pronunciation deviation table. For example, if the user often reads "超 (chāo)" as "cao", the model will first match "cao" as "超" rather than "草".

[0103] Speech disambiguation refers to the model combining contextual feature vectors to eliminate ambiguity in the recognition results. For example, "cao shi" may correspond to "supermarket", "cao shi" or "cao shi". By using the context of "spatial type = outdoor street, historical command = query shopping mall", the correct meaning is determined to be "supermarket".

[0104] In one embodiment, acoustic features are extracted from the user's speech to be recognized, the user's pronunciation deviation table is retrieved from the cloud, and the newly generated context vector is input into the speech recognition model. This enables the speech recognition model to perform speech deviation compensation and semantic disambiguation on the acoustic features in the speech to be recognized, and output the final recognition result.

[0105] For example, if a user says the acoustic characteristics of "turn off the TV," the user's pronunciation deviation table retrieved from the cloud, along with the newly generated context vector, is input into the speech recognition model. The model initially identifies a vaguely pronounced syllable, somewhere between "turn off" and "close the station." Querying the user's pronunciation deviation table reveals a deviation of 0.62 for the / zh / sound, often pronounced as / z / . Therefore, the model tends to correct the vague initial consonant to / zh / , as this is the user's typical pronunciation habit. The model generates several candidate results: "turn off the TV" (initial score 0.7), "turn off the lights" (initial score 0.5), and "close the station" (initial score 0.1). At this point, the context vector begins to play a role. The model finds that the spatial type (living room) and the ambient acoustics (TV playback) in the current context feature vector have extremely high weights. Based on this, the model significantly increases the score of the candidate "turn off the TV" because it closely matches the current context, while suppressing "turn off the lights" (the living room lights might not be on) and the completely irrelevant "close the station." The final output of the user's speech recognition result is "Turn off the TV".

[0106] Furthermore, in this embodiment, the aforementioned speech recognition model can be trained in the following manner: a training sample set is constructed based on the acoustic features of the speech samples, their corresponding real text labels, pronunciation deviation features, and context feature vectors; the acoustic features, pronunciation deviation features, and context feature vectors are used as model inputs, and real text labels are used as training targets; a mask loss function and a context completion function are used to train the initial speech recognition model; wherein, the mask loss function is used to randomly mask part of the acoustic features or text labels during the training process, and the context completion function is used to perform prediction calculations on the masked part using the context and context feature vectors.

[0107] Exemplarily, the 13-dimensional MFCC features extracted from the speech sample, such as the MFCC feature matrix corresponding to the dialect version of the speech of a 60-year-old Sichuan user reading "I want to go to the supermarket" (pronounced as "I want to go to cao shi"). Obtain the pronunciation deviation table data exclusive to this user, including key information such as the deviation of the initial consonant / c / of "chao" from the standard / ch / being 0.51, and the deviation of the initial consonant / s / of "shi" from the standard / sh / being 0.53. Simulate the 16-dimensional context feature vector when the user pronounces, such as spatial type: outdoor street (0.8), historical instruction: query nearby shopping mall (0.9), content type: life service (0.85), user age: 60 years old (0.6), environmental volume: noisy (0.7). At the same time, obtain the standard text corresponding to each speech sample, that is, the true text label, for example, the true label corresponding to the above dialect speech is "I want to go to the supermarket". According to the above rules, collect multiple training samples covering different ages, regions, and scenarios to form a complete training sample set.

[0108] Furthermore, select a basic model using the Attention hybrid architecture. Based on the masked loss (L_Mask), set a masked probability of 15%. During the training process, randomly mask the high-deviation phonemes in the input acoustic features or text labels, forcing the model to rely on context and context information to complete. For example, mark the / c / phoneme in "cao shi" as "[MASK]". Train the model based on the joint loss function shown in the following formula (I):

[0109] ……………Formula (I)

[0110] Among them, L represents the total loss of the joint loss function, L_Mask represents the masked loss, represents the masked loss weight, which can balance the recognition accuracy and completion ability, and can take a value of 0.3; L_CTC represents the connectionist temporal classification loss function, which is responsible for making the model find the correct temporal correspondence between speech and text (solving the alignment problem); L_Attention represents the attention mechanism loss function, which is responsible for making the model generate a coherent and accurate text sequence (solving the semantic problem).

[0111] The context completion function can be defined as a matching rule based on the dialect phoneme knowledge base and the context vector. For example, when detecting the spatial type: outdoor street, historical instruction: query shopping mall, automatically retrieve the dialect rule of " / c / often corresponds to / ch / " for the prediction calculation of the masked part.

[0112] The training process of the above initial model can be divided into the following steps: First step: Input the input feature groups (acoustic features, pronunciation deviation features, context feature vectors) of the training set into the initial model. The model first learns the temporal correspondence between speech and text through the L_CTC branch, and captures the context semantic association through the L_Attention branch. Second step: In the forward propagation of each round of training, the masked loss function randomly masks 15% of the high-deviation phonemes. For example, transform "我要去cao市" into "我要去[MASK]市". The model needs to call the context completion function, combine the context vectors of "outdoor street, querying a shopping mall" and the dialect knowledge base, and predict that the masked position should be "超" instead of "草" or "曹". Third step: Update the model parameters (including encoder weights, L_CTC / L_Attention branch coefficients, masked loss weights, etc.) through backpropagation, and at the same time use the validation set to monitor the model performance in real time (such as phoneme recognition accuracy, semantic disambiguation accuracy). Fourth step: When the speech recognition accuracy of the validation set stabilizes above 92% (a 15% improvement compared to the base model) and the disambiguation scene error correction rate reaches 88%, stop the training and save the final trained speech recognition model.

[0113] The method provided in the embodiment of this application collects the user's speech samples and personal attribute information, extracts the speech features in the speech samples, generates the user's pronunciation deviation table according to the speech features and personal attribute information, responds to receiving the speech to be recognized by the user, obtains the user's current multi-dimensional sensor data, generates a context feature vector based on the multi-dimensional sensor data, and inputs the acoustic features, pronunciation deviation table, and context feature vector of the speech to be recognized into the trained speech recognition model, so that the speech recognition model outputs the final speech recognition text. Dynamically create the user's pronunciation deviation table based on cold start registration, achieve precise adaptation for thousands of different voices, and improve the speech recognition accuracy of individual users with differences; at the same time, based on the real-time collected multi-dimensional sensor data, generate context feature information and deeply fuse it with the user's pronunciation features, effectively solve the problem of semantic ambiguity, and make the interaction more natural and accurate. Based on a lightweight speech recognition model under limited hardware resources, high-level intelligent speech interaction is achieved.

[0114] Figure 2 For the flowchart of the embodiment of another speech recognition method provided in the embodiment of this application, based on the process shown in Figure 1 mainly describes how to collect the user's speech features and how to generate the corresponding pronunciation deviation table according to the speech features; as shown in Figure 2 it includes the following steps:

[0115] Step 201: Prompt the user to read aloud a preset fixed text and open text, collect the user's speech samples during the reading process, and extract the speech features from the speech samples; wherein, the fixed text is used to collect the user's phoneme features, and the open text is used to simulate the user's natural expression habits.

[0116] Fixed text can refer to a series of carefully designed, standardized sentences that cover all phonemes in a target language (such as Mandarin). Its purpose is to systematically collect pronunciation data from users for each basic phoneme.

[0117] Open text can refer to sentences that simulate everyday conversation, without fixed phoneme coverage requirements. Its purpose is to capture users' pronunciation habits, coherence, and prosodic features in natural speech.

[0118] Speech features refer to the digital representation extracted from the raw speech signal. This step primarily extracts two types of features: acoustic features (such as MFCC), used to analyze the content of the speech; and speaker features (such as x-vector voiceprint features), used to identify the user.

[0119] In one embodiment, when the device is activated for the first time, a guide is displayed on the screen or the accompanying APP, and a voice prompt is played to guide the user to record and read the relevant text. The user's recorded text is collected, preprocessed, and the MFCC feature sequence and x-vector feature vector are extracted.

[0120] For example, when the device is activated, the fixed text "He is browsing books in the library" is displayed. This fixed text covers multiple key vowels and consonants such as / a / , / eng / , / i / , / ian / , / u / , and / ue / . The open text can be determined by the user, and the system can capture the user's habits such as retroflex endings and elision in a natural and relaxed state. Based on the recording of the user reading the relevant fixed and open texts, the system generates an MFCC feature matrix and a 192-dimensional x-vector.

[0121] Step 202: In response to the user's input, obtain the user's personal attribute information; wherein, the personal attribute information includes at least age, gender and permanent address information.

[0122] Personal attribute information can be static data describing a user's inherent characteristics, serving as prior knowledge for building a personalized model. This information can include age, gender, and permanent address. Age is used to calculate the age factor, quantifying the impact of vocal cord aging or underdevelopment on pronunciation. Gender assists the voiceprint model, providing a priori range for acoustic features such as fundamental frequency. Permanent address is used to associate with a corresponding dialect knowledge base and is crucial for identifying the source of an accent.

[0123] In one embodiment, a form can be provided in the registration app, including drop-down menus, radio buttons, and geolocation authorization information. Users can manually select or fill in their age and personality, and authorize the app to access or manually select their city of residence. The system binds this information with the x-vector voiceprint features extracted in step 201 and stores it in the user's local configuration file.

[0124] For example, user Zhang San is a 60-year-old male from Shanghai. He submitted this information on the app. The system therefore knows that it needs to call up the Shanghai dialect database for this user and apply a higher age factor (such as 0.6) when calculating the deviation.

[0125] Step 203: Based on the permanent address information in the personal attribute information, call the corresponding standard dialect database from multiple preset dialect databases.

[0126] A standard dialect database can refer to a pre-built, expert-annotated speech database. It contains common pronunciation error patterns of speakers in specific dialect areas (such as Shanghai dialect or Guangdong dialect), the correspondence between phonemes and standard Mandarin, and typical acoustic feature templates.

[0127] In one embodiment, the system maintains a mapping table between a user's permanent address and a standard dialect database. When the user's permanent address information is obtained, the system looks up the mapping table based on the permanent address information to determine the corresponding standard dialect database, and then loads the corresponding standard dialect database from local or cloud storage.

[0128] For example, upon obtaining a user's permanent address (Shanghai), the system queries a mapping table to determine if the user belongs to the Shanghai area and then calls the Shanghai dialect knowledge base. For example, the Shanghai dialect database pre-defines common pronunciation rules such as " / sh / is often pronounced as / s / ", " / zh / is often pronounced as / z / ", and " / r / is often pronounced as / l / ". This is merely an illustrative example; the specific correspondence between permanent addresses and standard dialect databases is not limited in this embodiment.

[0129] Step 204: Perform the following processing for each speech feature to generate a pronunciation deviation table for the user: determine the difference measurement data between the standard dialect database and the speech feature; perform weighted fusion based on the difference measurement data and the age factor in the personal attribute information to obtain the pronunciation deviation parameter; store the pronunciation deviation parameter and the corresponding speech feature in the pronunciation deviation table.

[0130] Difference measurement data refers to indicators that quantify the difference between a user's pronunciation and standard pronunciation. The age factor can refer to an age-based weighting coefficient. For example, a weighting of 0.4 might be set for a 6-year-old child (indicating developmental immaturity), 0.6 for a 70-year-old (indicating vocal cord aging), and 1.0 for a 25-year-old standard user. The pronunciation deviation parameter can refer to a quantitative value that integrates all the above factors, representing the degree of deviation of a user from a specific phoneme.

[0131] In one embodiment, the difference measurement data between the standard dialect database and the speech features can be determined by: obtaining the standard speech features corresponding to the speech features in the standard dialect database; comparing the differences between the speech features and the standard speech features to obtain the difference measurement data; wherein the difference measurement data includes at least the distance between the speech features and the standard speech features in the acoustic feature space, and the divergence between the speech features and the standard speech features in the distribution.

[0132] For example, assuming the user is from Shanghai and has an age factor of 0.6, for each phoneme of the user (e.g., / sh / ), the corresponding MFCC template and feature distribution of the standard / sh / are found from the Shanghai dialect knowledge base. The Euclidean distance between the user's MFCC features and the standard template is calculated, and the KL (Kullback-Leibler Divergence) divergence between the MFCC feature distribution of all / sh / pronunciations of the user and the standard / sh / distribution is calculated. Based on the above calculations, the corresponding difference measure data is obtained. The user's age factor and the difference measure data are substituted into the following formula (II):

[0133] ……………Formula (II)

[0134] Where pij represents the pronunciation deviation parameter; KL divergence represents the KL divergence between the user's pronunciation features and the standard pronunciation distribution; MFCC Euclidean distance represents the Euclidean distance between the user's MFCC features and the standard template; and age factor represents a priori factors based on age-related vocal cord aging or development (e.g., 0.4 for a 6-year-old child and 0.6 for a 70-year-old). , , This represents the weight system, whose values ​​are dynamically updated based on the algorithm.

[0135] Furthermore, the calculated pij value and its corresponding factor relationship (such as target phoneme: / sh / , user's actual pronunciation: / s / ) are stored together in a data table, namely the user's personal pronunciation deviation table. See Table 1 above for details, which will not be repeated here.

[0136] Step 205: In response to receiving the user's voice to be recognized, acquire the user's current multi-dimensional sensor data and generate a contextual feature vector based on the multi-dimensional sensor data.

[0137] Step 206: Input the acoustic features, pronunciation deviation table, and contextual feature vector of the speech to be recognized into the trained speech recognition model so that the speech recognition model outputs the final speech recognition text; wherein, the speech recognition model is configured to: compensate for speech deviations in the speech to be recognized based on the pronunciation deviation table, and perform semantic disambiguation on the recognition result of the speech to be recognized based on the contextual feature vector.

[0138] For steps 205-206 above, please refer to the above. Figure 1 Detailed description of the relevant embodiments.

[0139] pass Figure 2 The description of the illustrated embodiment, based on differentiated fixed and open text, combines a systematic survey of user pronunciation with the capture of natural habits, providing a high-quality data source for generating a comprehensive and accurate pronunciation deviation table. By using age, gender, and residential address—three attributes strongly correlated with pronunciation characteristics—as core personal attribute information, subsequent algorithmic processing is equipped with profound prior knowledge. The system no longer blindly processes speech signals but can perform targeted analysis and reasoning based on user background. Dynamically calling standard dialect databases based on residential address transforms deviation calculation from a one-size-fits-all universal standard comparison to a customized diagnosis tailored to the user's dialect region. This significantly improves the efficiency of identifying and compensating for common regional pronunciation errors (such as retroflex consonants and nasal consonants). Furthermore, the generated pronunciation deviation table can realistically and dynamically reflect the user's unique pronunciation characteristics, providing a strong and reliable basis for efficient "deviation compensation" in subsequent ASR models. In addition, based on cold-start registration, users receive a significantly better recognition experience than general models upon first use, and this experience continues to improve over time. The entire process design takes into account both initialization efficiency and long-term performance gains, thereby improving user stickiness and satisfaction.

[0140] Figure 3 A flowchart illustrating another embodiment of the speech recognition method provided in this application is shown below. Figure 1 Based on the illustrated process, this section primarily describes how to acquire multi-dimensional sensor data from users and how to generate contextual feature vectors based on this data; see [link to documentation]. Figure 3 As shown, it includes the following steps:

[0141] Step 301: Collect users' voice samples and personal attribute information, and extract voice features from the voice samples.

[0142] Step 302: Generate a pronunciation deviation table for the user based on speech features and personal attribute information.

[0143] For steps 301-302 above, please refer to the above. Figure 1 Detailed description of the relevant embodiments;

[0144] Step 303: In response to receiving the user's voice to be recognized, acquire positioning data based on positioning technology; acquire motion state data based on inertial measurement tools; acquire acoustic data of the current environment by collecting sound from the current environment using a microphone, and acquire the current timestamp data; and determine the positioning data, motion state data, acoustic data, and timestamp data as multi-dimensional sensor data.

[0145] Location data describes a user's precise current geographic location and can achieve seamless indoor and outdoor positioning by incorporating GNSS (Global Navigation Satellite System) technology. Motion state data describes the user's body or device movement patterns and can be collected by the IMU (Inertial Measurement Unit) on smart devices, containing raw data from accelerometers and gyroscopes to identify user behavior. Acoustic data consists of raw sound signals from the current environment collected through a microphone array. It includes not only the user's voice but also rich ambient background noise. Timestamp data provides the precise time (e.g., year, month, day, hour, minute, second) at which an instruction occurred, providing temporal context.

[0146] In one embodiment, when the device's microphone detects that the user has started speaking, the system synchronously activates all relevant sensors, packages the collected multi-dimensional sensor data, aligns it on the timeline, adds a unified timestamp, and forms a complete multi-dimensional sensor data package.

[0147] For example, suppose a 6-year-old boy says "Looking for Mom" ​​in a shopping mall. GNSS and the mall's Bluetooth positioning system return location information (e.g., latitude: 39.9042, longitude: 116.4074, floor: 3), and semantically interpret this location information as "third floor of XX Shopping Mall in Beijing". The IMU collects continuous, irregularly high acceleration values, indicating that the boy is currently in motion. The microphone collects a mixed audio including the child's voice, crowd noise, and background music. The current timestamp data is collected as 2024-05-20-16:00:00, which is semantically interpreted as "Saturday afternoon 4 pm". The final output is a data structure containing the above four-dimensional raw data that is aligned on the timestamp.

[0148] Step 304: Map sensor data of different dimensions into feature vectors respectively.

[0149] Step 305: Concatenate multiple feature vectors to form a fused feature sequence.

[0150] Step 306: Input the feature sequence into the neural network model and output the context feature vector; wherein, the neural network model is configured with an encoder to process the feature sequence.

[0151] The following is a unified explanation of steps 304-306 above:

[0152] Feature vectors refer to the transformation of raw, messy, unstructured data into numerical, fixed-dimensional arrays through a series of signal processing and feature extraction methods. This is a form that machines can understand and process.

[0153] Feature sequences can concatenate feature vectors from multiple different sources in a specific order to form a longer, comprehensive vector sequence that integrates multi-dimensional information. This sequence serves as the input for subsequent deep fusion models.

[0154] In one embodiment, the received positioning data, motion state data, and acoustic data are mapped into different feature vectors. Multiple different feature vectors are then concatenated in a specific order to form a longer, comprehensive vector sequence that integrates multi-dimensional sensor data. This comprehensive vector sequence is then input into the lightweight Transformer encoder in the corresponding neural network model. Through multi-layer encoding, the model compresses and fuses the long sequence, which contains redundant information, into a low-dimensional contextual feature vector rich in semantic information.

[0155] For example, assume the location feature vector [0.9, 0.0, 0.0, 0.1, ...] is 10-dimensional. The motion feature vector [0.1, 0.6, 0.3, ...] is 8-dimensional. The acoustic feature vector is a 128-dimensional array, with the numerical pattern matching the acoustic template of a noisy shopping mall. The temporal feature vector [0.7, ...] is 6-dimensional. The concatenated feature sequence is a long vector of 10+8+128+6=152 dimensions: [0.9, 0.0, ..., 0.1, 0.6, 0.3, ..., 0.12, 0.45, ..., 0.7, ...], where the first 10 dimensions are location information, the next 8 dimensions are motion data, and so on. The concatenated vector is input into the corresponding neural network model, compressed and fused to output a 16-dimensional contextual feature vector, for example, [0.4, 0.9, 0.6, 0.8, 1.0, ...]. The final output context feature vector no longer corresponds to a single sensor in each dimension, but is a high-level semantic feature after deep fusion. Its values ​​comprehensively reflect the urgency of the scene, the probability of the user's state, etc., providing irreplaceable contextual basis for the speech recognition model.

[0156] Step 307: Input the acoustic features, pronunciation deviation table, and contextual feature vector of the speech to be recognized into the trained speech recognition model so that the speech recognition model outputs the final speech recognition text; wherein, the speech recognition model is configured to: compensate for speech deviations in the speech to be recognized based on the pronunciation deviation table, and perform semantic disambiguation on the recognition result of the speech to be recognized based on the contextual feature vector.

[0157] For step 307 above, please refer to the above. Figure 1 Detailed description of the relevant embodiments.

[0158] pass Figure 3 The description of the illustrated embodiment illustrates that by simultaneously acquiring four-dimensional sensor data—positioning, motion, acoustics, and time—the system no longer processes sound signals in isolation. Instead, it constructs multi-dimensional sensor data, accurately determining when, where, what the user is doing, and the environment, giving the speech recognition system a human-like environmental perception capability. In complex and noisy environments such as subways, shopping malls, and streets, the system effectively filters irrelevant noise and focuses on useful information, significantly improving voice wake-up rate and command recognition rate, overcoming the fatal weakness of traditional solutions where recognition rate drops sharply in complex environments. Utilizing the self-attention mechanism of the lightweight Transformer encoder, the model can automatically learn and capture complex and deep correlations between different sensor information. For example, it can automatically understand the combination of a child's identity with the features of shopping mall location and rapid movement, upgrading voice interaction from recognizing the words you speak to understanding the meaning you want to express in the current scenario. This significantly reduces misrecognition caused by similar pronunciations, abbreviated commands, or polysemous words, making the interaction more accurate, intelligent, and natural, especially suitable for groups such as the elderly and children whose expression may be irregular. Figure 3 The description of the illustrated embodiment greatly improves the robustness of the system in complex environments and achieves accurate insight into intent through deep semantic understanding, while also taking into account the deployment feasibility of terminal devices.

[0159] Figure 4 This is a schematic diagram of the structure of a speech recognition device provided in an embodiment of this application, as shown below. Figure 4 As shown, it includes:

[0160] The information collection module 41 is used to collect users' voice samples and personal attribute information, and extract voice features from the voice samples;

[0161] The pronunciation deviation determination module 42 is used to generate a pronunciation deviation table for the user based on the speech features and the personal attribute information;

[0162] The context feature vector generation module 43 is used to, in response to receiving the user's voice to be recognized, acquire the user's current multi-dimensional sensor data, and generate a context feature vector based on the multi-dimensional sensor data;

[0163] The speech recognition module 44 is used to input the acoustic features of the speech to be recognized, the pronunciation deviation table, and the context feature vector into the trained speech recognition model, so that the speech recognition model outputs the final speech recognition text; wherein, the speech recognition model is configured to: compensate for the speech deviation in the speech to be recognized based on the pronunciation deviation table, and perform semantic disambiguation on the recognition result of the speech to be recognized based on the context feature vector.

[0164] In one possible implementation, the information collection module 41 is specifically used for:

[0165] The system prompts the user to read aloud preset fixed text and open text, and collects the user's speech samples during the reading process; wherein, the fixed text is used to collect the user's phoneme features, and the open text is used to simulate the user's natural expression habits;

[0166] In response to the user's input, the system obtains the user's personal attribute information; wherein the personal attribute information includes at least age, gender, and permanent address information.

[0167] In one possible implementation, the pronunciation deviation determination module 42 includes:

[0168] The dialect database determination unit is used to call the corresponding standard dialect database from multiple preset dialect databases based on the permanent address information in the personal attribute information.

[0169] The pronunciation deviation table generation unit is used to perform the following processing for each speech feature to generate the user's pronunciation deviation table:

[0170] Determine the difference measurement data between the standard dialect database and the speech features;

[0171] The pronunciation deviation parameter is obtained by weighted fusion of the difference measurement data and the age factor in the personal attribute information.

[0172] The pronunciation deviation parameter and the corresponding speech feature are stored in the pronunciation deviation table.

[0173] In one possible implementation, the pronunciation deviation table generation unit is specifically used for:

[0174] Obtain the standard speech features corresponding to the speech features from the standard dialect database;

[0175] The differences between the speech features and the standard speech features are compared to obtain difference measurement data; wherein, the difference measurement data includes at least the distance between the speech features and the standard speech features in the acoustic feature space, and the divergence of the speech features and the standard speech features in the distribution.

[0176] In one possible implementation, the context feature vector generation module 43 is specifically used for:

[0177] Location data is obtained based on positioning technology;

[0178] Motion state data is acquired using inertial measurement tools;

[0179] Acoustic data of the current environment is obtained by collecting sound from the microphone, and the current timestamp data is also obtained.

[0180] The positioning data, motion state data, acoustic data, and timestamp data are identified as multi-dimensional sensor data.

[0181] In one possible implementation, the context feature vector generation module 43 is specifically used for:

[0182] Sensor data of different dimensions are mapped into feature vectors respectively;

[0183] Multiple feature vectors are concatenated to form a fused feature sequence;

[0184] The feature sequence is input into a neural network model, which outputs a contextual feature vector; wherein the neural network model is configured with an encoder to process the feature sequence.

[0185] In one possible implementation, the speech recognition model is trained in the following manner:

[0186] A training sample set is constructed based on the acoustic features of speech samples, their corresponding real text labels, pronunciation deviation features, and contextual feature vectors.

[0187] The acoustic features, pronunciation deviation features, and contextual feature vectors are used as model inputs, and the real text labels are used as training targets. The initial speech recognition model is trained using a masking loss function and a contextual completion function. The masking loss function is used to randomly mask some acoustic features or text labels during training, and the contextual completion function is used to predict the masked parts using the context and the contextual feature vectors.

[0188] like Figure 5As shown in the figure, this application provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0189] Memory 113 is used to store computer programs;

[0190] In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the speech recognition method provided in any of the foregoing method embodiments, including:

[0191] Collect users' voice samples and personal attribute information, and extract voice features from the voice samples;

[0192] Based on the speech features and the personal attribute information, a pronunciation deviation table for the user is generated;

[0193] In response to receiving the user's voice to be recognized, the system acquires the user's current multi-dimensional sensor data and generates a contextual feature vector based on the multi-dimensional sensor data.

[0194] The acoustic features of the speech to be recognized, the pronunciation deviation table, and the contextual feature vector are input into a trained speech recognition model so that the speech recognition model outputs the final speech recognition text; wherein, the speech recognition model is configured to: compensate for speech deviations in the speech to be recognized based on the pronunciation deviation table, and perform semantic disambiguation on the recognition result of the speech to be recognized based on the contextual feature vector.

[0195] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the speech recognition method provided in any of the foregoing method embodiments.

[0196] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0197] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0198] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also mean including the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0199] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A speech recognition method, characterized in that, The method includes: Collect users' voice samples and personal attribute information, and extract voice features from the voice samples; Based on the speech features and the personal attribute information, a pronunciation deviation table for the user is generated; In response to receiving the user's voice to be recognized, the system acquires the user's current multi-dimensional sensor data and generates a contextual feature vector based on the multi-dimensional sensor data. The acoustic features of the speech to be recognized, the pronunciation deviation table, and the contextual feature vector are input into a trained speech recognition model so that the speech recognition model outputs the final speech recognition text; wherein, the speech recognition model is configured to: compensate for the speech deviation in the speech to be recognized based on the pronunciation deviation table, and perform semantic disambiguation on the recognition result of the speech to be recognized based on the contextual feature vector; The step of generating the user's pronunciation deviation table based on the speech features and the personal attribute information includes: calling the corresponding standard dialect database from multiple preset dialect databases based on the permanent address information in the personal attribute information; performing the following processing for each speech feature to generate the user's pronunciation deviation table: determining the difference measurement data between the standard dialect database and the speech feature; performing weighted fusion based on the difference measurement data and the age factor in the personal attribute information to obtain a pronunciation deviation parameter; and storing the pronunciation deviation parameter and the corresponding speech feature in the pronunciation deviation table. The process of acquiring the user's current multi-dimensional sensor data includes: acquiring positioning data based on positioning technology; acquiring motion state data based on inertial measurement tools; acquiring acoustic data of the current environment by collecting sound from the current environment using a microphone, and acquiring current timestamp data; and determining the positioning data, motion state data, acoustic data, and timestamp data as multi-dimensional sensor data. The step of generating a context feature vector based on the multi-dimensional sensor data includes: mapping sensor data of different dimensions into feature vectors respectively; concatenating multiple feature vectors to form a fused feature sequence; inputting the feature sequence into a neural network model and outputting a context feature vector; wherein the neural network model is configured with an encoder for processing the feature sequence.

2. The method according to claim 1, characterized in that, The collection of users' voice samples and personal attribute information includes: The system prompts the user to read aloud preset fixed text and open text, and collects the user's speech samples during the reading process; wherein, the fixed text is used to collect the user's phoneme features, and the open text is used to simulate the user's natural expression habits; In response to the user's input, the system obtains the user's personal attribute information; wherein the personal attribute information includes at least age, gender, and permanent address information.

3. The method according to claim 1, characterized in that, The determination of the difference measurement data between the standard dialect database and the speech features includes: Obtain the standard speech features corresponding to the speech features from the standard dialect database; The differences between the speech features and the standard speech features are compared to obtain difference measurement data; wherein, the difference measurement data includes at least the distance between the speech features and the standard speech features in the acoustic feature space, and the divergence of the speech features and the standard speech features in the distribution.

4. The method according to claim 1, characterized in that, The speech recognition model was trained in the following way: A training sample set is constructed based on the acoustic features of speech samples, their corresponding real text labels, pronunciation deviation features, and contextual feature vectors. The acoustic features, pronunciation deviation features, and contextual feature vectors are used as model inputs, and the real text labels are used as training targets. The initial speech recognition model is trained using a masking loss function and a contextual completion function. The masking loss function is used to randomly mask some acoustic features or text labels during training, and the contextual completion function is used to predict the masked parts using the context and the contextual feature vectors.

5. A voice recognition device, characterized in that, The device includes: The information collection module is used to collect users' voice samples and personal attribute information, and extract voice features from the voice samples; The pronunciation deviation determination module is used to generate a pronunciation deviation table for the user based on the speech features and the personal attribute information; The context feature vector generation module is used to, in response to receiving the user's voice to be recognized, acquire the user's current multi-dimensional sensor data, and generate a context feature vector based on the multi-dimensional sensor data; A speech recognition module is used to input the acoustic features of the speech to be recognized, the pronunciation deviation table, and the contextual feature vector into a trained speech recognition model, so that the speech recognition model outputs the final speech recognition text; wherein, the speech recognition model is configured to: compensate for speech deviations in the speech to be recognized based on the pronunciation deviation table, and perform semantic disambiguation on the recognition result of the speech to be recognized based on the contextual feature vector; The pronunciation deviation determination module is further configured to: based on the permanent address information in the personal attribute information, call the corresponding standard dialect database from multiple preset dialect databases; perform the following processing for each speech feature to generate the user's pronunciation deviation table: determine the difference measurement data between the standard dialect database and the speech feature; perform weighted fusion based on the difference measurement data and the age factor in the personal attribute information to obtain the pronunciation deviation parameter; and store the pronunciation deviation parameter and the corresponding speech feature in the pronunciation deviation table. The context feature vector generation module is also used to: acquire positioning data based on positioning technology; acquire motion state data based on inertial measurement tools; acquire acoustic data of the current environment by collecting sound from the current environment using a microphone, and acquire the current timestamp data; and determine the positioning data, motion state data, acoustic data, and timestamp data as multi-dimensional sensor data. The speech recognition module is further configured to: map sensor data of different dimensions into feature vectors respectively; concatenate multiple feature vectors to form a fused feature sequence; input the feature sequence into a neural network model and output a contextual feature vector; wherein the neural network model is configured with an encoder for processing the feature sequence.

6. An electronic device, characterized in that, include: A processor and a memory, the processor being configured to execute a speech recognition program stored in the memory to implement the speech recognition method according to any one of claims 1-4.

7. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the speech recognition method according to any one of claims 1-4.

Images