Speech recognition methods, devices, electronic equipment, and temporal fusion language models

Abstract

This application discloses a speech recognition method, apparatus, electronic device, and temporal fusion language model, belonging to the field of artificial intelligence technology. The method includes: upon receiving speech information input by a user, performing speech recognition on the speech information to obtain a recognition result, wherein the speech information includes semantic information and temporal information; if the recognition result meets preset conditions, performing temporal information analysis on the speech information using a temporal fusion language model to obtain a temporal score; performing semantic information analysis on the speech information using the temporal fusion language model, and obtaining a target result based on the temporal score and semantic feature information obtained from the semantic information analysis.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a speech recognition method, device, electronic device, and temporal fusion language model. Background Technology

[0002] With the increasing use of voice input in electronic devices, the usage data for voice input continues to grow. Voice input serves various scenarios, making daily life more convenient.

[0003] However, the current voice input model is a general model. Due to its generality, the recognition result is often not the result that the user wants, but the result that is more commonly used by the public. This causes great confusion and inconvenience to the user. Summary of the Invention

[0004] The purpose of this application is to provide a speech recognition method, device, electronic device, and temporal fusion language model that can solve the problems of imperfections in existing speech input methods.

[0005] In a first aspect, this application provides a speech recognition method, the method comprising: upon receiving speech information input by a user, performing speech recognition on the speech information to obtain a recognition result, wherein the speech information includes semantic information and temporal information; if the recognition result meets preset conditions, performing temporal information analysis on the speech information using a temporal fusion language model to obtain a temporal score; performing semantic information analysis on the speech information using a temporal fusion language model, and obtaining a target result based on the temporal score and the analyzed semantic feature information.

[0006] Secondly, this application provides a temporal fusion language model, which includes: a temporal model and a semantic model; a temporal model and a semantic model;

[0007] The temporal model includes a first input embedding layer, a first multi-head attention layer, a first forward computation layer, a first linear layer, and a first activation function layer connected in sequence.

[0008] The first input embedding layer is the input layer of the temporal model, used to convert temporal information into a first feature vector; the first multi-head attention layer is used to extract features from the first feature vector to obtain temporal feature information; the first forward computation layer is used to perform weighted computation on the temporal feature information to obtain weighted temporal feature information; the first linear layer is used to perform dimensionality reduction computation on the weighted temporal feature information to obtain temporal confidence; the first activation function layer is the output layer of the temporal model, used to perform normalization computation on the temporal confidence to obtain a temporal score.

[0009] The semantic model includes a second input embedding layer, a second multi-head attention layer, a fusion module, a second forward computation layer, a second linear layer, and a second activation function layer connected in sequence; the first activation function layer is connected to the fusion module.

[0010] The second input embedding layer is the input layer of the semantic model, used to convert semantic information into a second feature vector; the second multi-head attention layer is used to extract features from the second feature vector to obtain semantic feature information; the fusion module is used to fuse the semantic feature information and the temporal score to obtain a fusion score; the second forward computation layer is used to perform weighted calculation on the fusion score to obtain a weighted fusion score; the second linear layer is used to perform dimensionality reduction calculation on the weighted fusion score to obtain a fusion score confidence level; the second activation function layer is the output layer of the semantic model, used to normalize the fusion score confidence level to obtain a fusion score and output the target text with the highest fusion score as the target result.

[0011] Thirdly, this application provides a speech recognition device, the device comprising: a recognition module, configured to perform speech recognition on the speech information received from a user, and obtain a recognition result, wherein the speech information includes semantic information and temporal information; a first processing module, configured to perform temporal information analysis on the speech information through a temporal fusion language model, and obtain a temporal score, provided that the recognition result meets preset conditions; and a second processing module, configured to perform semantic information analysis on the speech information through a temporal fusion language model, and obtain a target result based on the temporal score and the analyzed semantic feature information.

[0012] Fourthly, this application provides an electronic device including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method described in the first aspect.

[0013] Fifthly, this application provides a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0014] In a sixth aspect, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0015] In a seventh aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0016] In this application, by detecting user input, upon receiving the user's input voice information, speech recognition is performed on the voice information to obtain a recognition result. Conditional detection is then performed on the recognition result. If the recognition result meets preset conditions, a temporal fusion language model is used to analyze the speech information to obtain a temporal score. The temporal fusion language model can be triggered when the recognition result meets preset conditions. The speech information includes semantic and temporal information, integrating temporal information into the analysis and calculation of semantic information. The temporal fusion language model then performs semantic information analysis on the speech information. Based on the temporal score and the analyzed semantic feature information, a speech recognition result containing both temporal and semantic information can be obtained. This makes the target result more personalized and avoids the problem of overly standardized recognition results caused by relying solely on single semantic information. Attached Figure Description

[0017] Figure 1 This is a flowchart of the steps of a speech recognition method provided in this embodiment;

[0018] Figure 2 This is an example of the storage format of the dictionary database provided in this embodiment;

[0019] Figure 3 This is a schematic diagram of a temporal fusion language model provided in this embodiment;

[0020] Figure 4 This is a schematic diagram of the structure of a voice recognition device provided in this embodiment;

[0021] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this embodiment;

[0022] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0024] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0025] The speech recognition method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0026] refer to Figure 1 , Figure 1 Here is a flowchart of a speech recognition method, which includes the following steps S110 to S130:

[0027] S110. Upon receiving voice information input by the user, perform voice recognition on the voice information to obtain the recognition result.

[0028] In this embodiment, the voice information input by the user is as follows: the user opens the interface of a chat application, clicks the "Press and hold to speak" or "Voice input" button on the input method keyboard, and speaks using the voice input method in the input method, and obtains the spoken content as voice information.

[0029] In this embodiment, the voice information includes the current input content and the name of the current application in which the current input content is located. For example, if the name of the chat application opened by the user is A and the input chat content is "xi shua shua", then the voice information includes "A" and "xi shua shua".

[0030] In this embodiment, in order to fully meet the user's personalized needs, a dictionary database is established before receiving the user's voice information. This dictionary database is created based on the user's historical operation records.

[0031] In this embodiment, a dictionary database can be established by obtaining the user's historical operation records, classifying and storing the historical operation records according to preset rules, and thus obtaining a dictionary database.

[0032] Specifically, historical operation records include the user's historical operation records on different application software, including the user's input using Pinyin input method, handwriting input method, Wubi input method, or voice input method.

[0033] The historical operation records include the user's input content, the number of times of the input content, the input time, and the application name where the input content is located. The input content can be a certain word or phrase or sentence. The number of times of the input content can be the cumulative number of times of the input of a certain word or phrase or sentence. The historical operation records can be stored locally to ensure the security of the user's information.

[0034] In this embodiment, the historical operation records are classified and stored according to a preset rule, including: classifying the historical operation records according to the application name to obtain multiple sub-data, and the sub-data includes multiple sets; sorting the sub-data according to the chronological order of the input time, and sorting the sets according to the chronological order of the input time.

[0035] For example, the historical operation records are classified and stored in dictionary form. Classify them by each application to create multiple sub-data. Each sub-data includes multiple sets. Each set includes the user's input time, input content, and cumulative number of times of the input content. The storage format in dictionary form can be:

[0036] {Application 1: {1: Operation time 1, Input content 1, Cumulative number of times 1}

[0037] {2: Operation time 2, Input content 2, Cumulative number of times 2}},

[0038] {Application 2: {1: Operation time 1, Input content 1, Cumulative number of times 1}

[0039] {2: Operation time 2, Input content 2, Cumulative number of times 2}}

[0040] In a specific example, the storage format case can refer to Figure 2 the storage format case shown. Among them, the historical operation records include the operations of the user on Application 1, Application 2, and Application 3 in the past period of time. Then, the historical operation records are classified into 3 sub-data according to the application classification of Application 1, Application 2, and Application 3, and each set is sorted according to the chronological order of the input content. As Figure 2 shown, Set 1 of Application 1 includes the input time "20220321", the input content "Xi Shuashua", and the cumulative number of times "2". Similarly, Set 2 includes the input time "20220321", the input content "My Motherland and I", and the cumulative number of times "6". Set 3 includes the input time "20220321", the input content "Yesterday, Today, Tomorrow", and the cumulative number of times "3". The formats of other applications are the same and will not be described here.

[0041] In an example, the user can also edit the dictionary database, such as performing addition and deletion operations on the data in the dictionary database.

[0042] In this embodiment, a temporal fusion language model is created for each application based on the aforementioned dictionary database. In this embodiment, the speech information includes the current input content and the name of the current application containing the input content. Therefore, to construct a more accurate temporal fusion language model, a temporal fusion language model can be created for each application, with different speech recognition accuracies for different applications.

[0043] Specifically, the process of constructing a temporal fusion language model based on sub-data includes: querying the target sub-data corresponding to the current application name in the dictionary database; and training the temporal fusion language model based on the target sub-data to obtain the trained temporal fusion language model.

[0044] For example, if the current application is application 1, then the sub-data corresponding to application 1 in the dictionary database is used as the target sub-data. The temporal fusion language model is trained based on the sub-data corresponding to application 1, and the input time, input content, and cumulative number of input content in each set of the sub-data corresponding to application 1 are used as the input of the temporal fusion language model.

[0045] The temporal fusion language model can be pre-trained, or it can be trained after the user inputs voice information, based on the input content in the voice information and the target sub-data corresponding to the current application name in the dictionary database.

[0046] In this embodiment, speech recognition of speech information may include input speech, feature extraction, acoustic model processing, and language model processing. The input speech can be captured by a microphone and the audio data can be saved according to a specific format. Feature extraction involves extracting key feature data from the saved audio data. Acoustic model processing refers to the process where the input is feature data and the output is the probability of acoustic units. Language model processing involves further processing the results from the acoustic model units to obtain the final recognition result.

[0047] It is understandable that the dictionary database is created based on historical operation data. When the user inputs a new word, and the new word does not exist in the sub-data of the dictionary database, the output result will be consistent with the recognition result even through the temporal fusion language model. When the user inputs a word that has been previously input, the word will exist in the dictionary database, thus triggering the temporal fusion language model. Therefore, after obtaining the recognition result of the speech information, this embodiment needs to determine whether the recognition result is consistent with the input content in any set of sub-data to obtain the target result. Therefore, the preset condition of this embodiment includes that the recognition result is consistent with the input content in any set of sub-data in the dictionary database.

[0048] If the recognition result does not meet the preset conditions, that is, if the recognition result is inconsistent with the input content in any set of sub-data, the recognition result will be output as the target result.

[0049] For example, if the recognition result is "icebb", but the input content in any set of sub-data in the dictionary database does not contain "icebb", then the recognition result "icebb" will be output as the target result.

[0050] S120. If the recognition result meets the preset conditions, the speech information is analyzed by a temporal fusion language model to obtain a temporal score.

[0051] In this embodiment, if the recognition result meets preset conditions, indicating that the recognition result is consistent with the input content in any set of sub-data in the dictionary database, and indicating that the user has previously input the current speech, then the temporal information of the historical speech information can be integrated into the current semantics. It is understood that the same word may have different meanings at different times. Therefore, before fusing temporal and semantic information, it is necessary to analyze the temporal information to obtain a temporal score. The temporal score can be a score obtained based on the time when a certain speech information most recently triggered the temporal fusion language model and the number of times the temporal fusion language model has been triggered.

[0052] The speech information in this embodiment includes semantic information and temporal information. A temporal fusion language model is used to analyze the speech information to obtain a temporal score. This includes: converting the temporal information into a first feature vector, for example, converting dates and frequencies into feature vectors for easier feature extraction; extracting features from the first feature vector to obtain temporal feature information, facilitating the extraction of important information during temporal analysis; performing weighted calculations on the temporal feature information to obtain weighted temporal feature information, which has better temporal representativeness and is a high-dimensional feature vector; performing dimensionality reduction calculations on the weighted temporal feature information to obtain a temporal confidence score; and normalizing the temporal confidence score to obtain a temporal score.

[0053] Considering that users tend to have similar behavioral habits within the same time period or a short period of time, after obtaining the time-series score, this score can be combined with semantic information. This allows for the integration of the same user's behavioral characteristics at different time periods, and adding time-series features can better provide personalized services to users.

[0054] S130. Semantic information analysis of speech information is performed using a temporal fusion language model, and the target result is obtained based on the temporal score and semantic feature information obtained from the semantic information analysis.

[0055] In this embodiment, semantic information is converted into a second feature vector, which is a semantic feature vector; features are extracted from the second feature vector to obtain semantic feature information; the semantic feature information and the temporal score are fused to obtain a fusion score; the fusion score is weighted to obtain a weighted fusion score; the weighted fusion score is dimensionality reduced to obtain a fusion score confidence level; the fusion score confidence level is normalized to obtain a fusion score; and the target text with the highest fusion score is output as the target result.

[0056] The following describes the specific recognition process of the temporal fusion language model:

[0057] refer to Figure 3 The temporal fusion language model in this embodiment includes a temporal model and a semantic model. The temporal model includes a first input embedding layer, a first multi-head attention layer, a first forward computation layer, a first linear layer, and a first activation function layer connected in sequence. The first input embedding layer is the input layer of the temporal model, used to convert temporal information into a first feature vector. The first multi-head attention layer is used to extract features from the first feature vector to obtain temporal feature information. The first forward computation layer is used to perform weighted calculation on the temporal feature information to obtain weighted temporal feature information. The first linear layer is used to perform dimensionality reduction calculation on the weighted temporal feature information to obtain temporal confidence. The first activation function layer is the output layer of the temporal model, used to perform normalization calculation on the temporal confidence to obtain a temporal score.

[0058] The semantic model in this embodiment includes a second input embedding layer, a second multi-head attention layer, a fusion module, a second forward computation layer, a second linear layer, and a second activation function layer connected in sequence. The first activation function layer is connected to the fusion module. The second input embedding layer is the input layer of the semantic model, used to convert semantic information into a second feature vector. The second multi-head attention layer is used to extract features from the second feature vector to obtain semantic feature information. The fusion module is used to fuse the semantic feature information and the temporal score to obtain a fusion score. The second forward computation layer is used to perform weighted calculation on the fusion score to obtain a weighted fusion score. The second linear layer is used to perform dimensionality reduction calculation on the weighted fusion score to obtain a fusion score confidence level. The second activation function layer is the output layer of the semantic model, used to normalize the fusion score confidence level to obtain a fusion score and output the target text with the highest fusion score as the target result. Figure 3 In this context, Add&Nom indicates that the results of the layer in which the calculation is performed are weighted.

[0059] Traditional language models only consider the semantic information of hot words without taking into account frequency and the time of recent use. In contrast, the temporal fusion language model in this embodiment combines temporal and semantic models, taking into account both the semantic information of traditional language models and the temporal information of users. That is, the more recent the data, the more important it is. It takes into account both hot words and whether the hot words have been used frequently recently, thereby improving the personalized results of speech recognition.

[0060] The following uses a set {operation time "20220321", input content "yesterday, today, tomorrow", cumulative count "3"} as an example to describe the specific calculation process of the temporal fusion language model in this embodiment:

[0061] The input to the first input embedding layer includes the user's input time and the number of times the input content is entered, i.e., (20220321, 3). This is first processed by the first input embedding layer and transformed into a D-dimensional feature vector. The input embedding layer is a method of feature dimensionality enhancement, converting low-dimensional features into high-dimensional features, i.e., the first feature vector. The specific calculation formula is as follows:

[0062] embed 时序 =input×W 时序

[0063] Among them, embed 时序 The output of the first input embedding layer is the D-dimensional feature vector after dimensionality increase, W. 时序 The first input embedding layer contains pre-stored feature parameters, and input is the input.

[0064] Features are extracted after calculation by the first multi-head attention layer. The specific calculation formula is as follows:

[0065]

[0066] Among them, attention 时序 The output of the first multi-head attention layer function is the time sequence feature information, Q. 时序 K 时序 V 时序 for embed 时序 The new feature obtained after linear transformation, d k This represents the dimension of the feature.

[0067] After the calculation in the first multi-head attention layer, the matrix calculation and weighted calculation are performed in the first forward calculation layer. The specific formula is as follows:

[0068] FF 时序 =(attention) 时序 ×W1)×W2

[0069] FF 时序 W1 and W2 are the outputs of the first forward computation layer and the matrix parameters obtained during training.

[0070] The first linear layer then performs matrix multiplication to reduce dimensionality. The specific calculations for the linear layer are as follows:

[0071] L 时序 =FF 时序 ×W L

[0072] L 时序 The output of the first linear layer, i.e., the time-series confidence, W L These are the parameters of the time series matrix.

[0073] Finally, the original score is normalized and calculated using the first activation function layer. The Softmax formula is as follows:

[0074]

[0075] Score 时序 The score is the temporal score output by the first activation function layer. 时序 The data is output to the fusion module and will be used to fuse the semantics of the input content.

[0076] In this embodiment, the input to the second input embedding layer includes the user's input content, namely "yesterday, today, and tomorrow". After calculation by the second semantic input embedding layer, the input is transformed into a D-dimensional feature vector, namely the second feature vector. The formula is as follows:

[0077] embed 语义 =input×W 语义

[0078] Among them, embed 语义 The output of the second input embedding layer is the D-dimensional feature vector after dimensionality increase; W 语义 The input is the pre-stored feature parameters of the second input embedding layer.

[0079] After the second multi-head attention layer calculates and extracts features, semantic feature information is obtained. The specific calculation formula is as follows:

[0080]

[0081] Among them, attention 语义 Q is the output of the second multi-head attention layer function. 语义 K 语义 V 语义 for embed 语义The new features obtained after the linear transformation are only different in terms of the parameters of the linear transformation, d. k This represents the dimension of the feature, which is an integer divided by... It's for normalizing the model.

[0082] At this point, high-level semantic feature information is obtained, and fusion calculation begins through the fusion module, using an attention mechanism to calculate and fuse the score. 时序 and attention 语义 The specific formula for the fusion module is as follows:

[0083]

[0084] Among them, attention 融合分数 Q is the fusion score output by the fusion module. 语义特征 K 时序分数 V 时序分数 This involves fusing features. Specifically, the score is taken from the time-series model. 时序 K is obtained after linear transformation. 时序分数 Then from attention 语义 Q is obtained after linear transformation. 语义特征 .

[0085] The obtained fusion score attention 融合分数 After passing through the second forward computation layer for matrix calculation and weighted calculation, the weighted fusion score is obtained. The specific calculation formula is as follows:

[0086] FF 综合 =(attention) 融合分数 ×W1)×W2

[0087] Among them, FF 综合 W1 and W2 are the outputs of the second forward computation layer and matrix parameters.

[0088] Then, matrix multiplication is performed in the second linear layer to reduce the dimensionality, as shown in the following formula:

[0089] L 综合 =FF 综合 ×W L

[0090] Among them, L 综合 W is the output of the second linear layer, i.e., the fusion score confidence. L These are the matrix parameters obtained from model training.

[0091] Finally, the scores are normalized and calculated using the second activation function layer, as shown in the following formula:

[0092]

[0093] Score 综合 The output of the second activation function layer is the fusion score. The output scores are sorted from largest to smallest. The target text with the highest fusion score is output as the target result.

[0094] This embodiment incorporates temporal features into its temporal fusion language model, taking into account users' tendency to exhibit similar behavioral habits within the same or nearby time period. This enhances the personalization of different applications. For example, gaming applications may use specialized game terminology, while music and movie applications may involve different song and movie titles. By continuously learning and updating the model based on users' spoken language habits and personal preferences, it can both provide a better user experience through self-learning and effectively protect user privacy.

[0095] After obtaining the target result, the user may be dissatisfied with some aspects. Therefore, in order to improve the accuracy of the temporal fusion language model, this embodiment further includes, after outputting the target result: obtaining the user's modification information on the target result, including the modified target words; and updating the temporal fusion language model based on the input content of the target words and speech information.

[0096] This embodiment can use a pre-set discriminator to perform additional loss calculations and backpropagation to update the temporal fusion language model. It can also use generative adversarial methods to further optimize the model in real time based on user feedback.

[0097] The loss calculation is shown in the formula:

[0098]

[0099] Here, p corresponds to the target word, and q corresponds to the input speech information, i.e., the user's original word. If the user does not modify the target result, the model is not updated. If the user modifies the target result, the discriminator is calculated based on the target word and the input speech information, and the loss value is obtained. The model is then updated based on the loss, which can improve the accuracy of the temporal fusion language model.

[0100] This embodiment detects user input and performs speech recognition on the received voice information to obtain a recognition result. The recognition result is then subjected to conditional detection. If the recognition result meets preset conditions, a temporal fusion language model is used to analyze the speech information to obtain a temporal score. The temporal fusion language model can be triggered when the recognition result meets preset conditions. The speech information includes semantic and temporal information, which is integrated into the analysis and calculation of semantic information. The temporal fusion language model then performs semantic information analysis on the speech information. Based on the temporal score and the analyzed semantic features, a speech recognition result that simultaneously contains both temporal and semantic information can be obtained. This makes the target result more personalized and avoids the problem of overly standardized recognition results caused by relying solely on semantic information.

[0101] This embodiment provides a temporal fusion language model, referencing... Figure 3 Temporal fusion language models include: temporal models and semantic models;

[0102] The temporal model comprises a first input embedding layer, a first multi-head attention layer, a first forward computation layer, a first linear layer, and a first activation function layer connected in sequence.

[0103] The first input embedding layer is the input layer of the temporal model, used to convert temporal information into a first feature vector; the first multi-head attention layer is used to extract features from the first feature vector to obtain temporal feature information; the first forward computation layer is used to perform weighted computation on the temporal feature information to obtain weighted temporal feature information; the first linear layer is used to perform dimensionality reduction computation on the weighted temporal feature information to obtain temporal confidence; the first activation function layer is the output layer of the temporal model, used to perform normalization computation on the temporal confidence to obtain a temporal score.

[0104] The semantic model includes a second input embedding layer, a second multi-head attention layer, a fusion module, a second forward computation layer, a second linear layer, and a second activation function layer connected in sequence; the first activation function layer is connected to the fusion module.

[0105] The second input embedding layer is the input layer of the semantic model, used to convert semantic information into a second feature vector; the second multi-head attention layer is used to extract features from the second feature vector to obtain semantic feature information; the fusion module is used to fuse the semantic feature information and the temporal score to obtain a fusion score; the second forward computation layer is used to perform weighted calculation on the fusion score to obtain a weighted fusion score; the second linear layer is used to perform dimensionality reduction calculation on the weighted fusion score to obtain a fusion score confidence level; the second activation function layer is the output layer of the semantic model, used to normalize the fusion score confidence level to obtain a fusion score and output the target text with the highest fusion score as the target result.

[0106] The temporal fusion language model in this embodiment incorporates both semantic information from traditional language models and temporal information from the user. It then uses this model to recognize speech information and output the target result. This allows for more personalized results and avoids the problem of overly standardized existing language recognition models.

[0107] The speech recognition method provided in this application can be executed by a speech recognition device. This application uses a speech recognition device executing the speech recognition method as an example to illustrate the speech recognition device provided in this application.

[0108] This embodiment provides a speech recognition device, for reference... Figure 4 The voice recognition device 400 includes:

[0109] The recognition module 401 is used to perform speech recognition on the received voice information input by the user and obtain a recognition result, wherein the voice information includes semantic information and temporal information;

[0110] The first processing module 402 is used to perform temporal information analysis on the speech information through a temporal fusion language model to obtain a temporal score when the recognition result meets the preset conditions.

[0111] The second processing module 403 is used to perform semantic information analysis on the speech information through a temporal fusion language model, and obtain the target result based on the temporal score and the semantic feature information obtained from the semantic information analysis.

[0112] Optionally, the first processing module 402 is used to convert the time series information into a first feature vector; perform feature extraction on the first feature vector to obtain time series feature information; perform weighted calculation on the time series feature information to obtain weighted time series feature information; perform dimensionality reduction calculation on the weighted time series feature information to obtain time series confidence; and perform normalization calculation on the time series confidence to obtain time series score.

[0113] Optionally, the second processing module 403 is used to convert the semantic information into a second feature vector; extract features from the second feature vector to obtain semantic feature information; fuse the semantic feature information and the temporal score to obtain a fusion score; perform weighted calculation on the fusion score to obtain a weighted fusion score; perform dimensionality reduction calculation on the weighted fusion score to obtain a fusion score confidence level; perform normalization calculation on the fusion score confidence level to obtain a fusion score; and output the target text with the highest fusion score as the target result.

[0114] Optionally, the device includes a model creation 404 for acquiring the user's historical operation records, which include the user's input content, the number of times the input content was entered, the input time, and the application name where the input content is located; classifying and storing the historical operation records according to preset rules to obtain a dictionary database; and creating the temporal fusion language model based on each application according to the dictionary database.

[0115] Optionally, the model creation module 404 is further configured to: classify the historical operation records according to the application name to obtain multiple sub-data, the sub-data including multiple sets, each set including the user's input time, input content, and the number of times the input content is entered; sort the sub-data according to the chronological order of the input time, and sort the sets according to the chronological order of the input time.

[0116] Optionally, the recognition module 401 is further configured to output the recognition result as the target result if the recognition result does not meet the preset conditions; wherein the preset conditions include: the recognition result is consistent with the input content in any set of sub-data in the dictionary database.

[0117] Optionally, the speech recognition device further includes an update module 405, which is used to obtain user modification information of the target result after outputting the target result, the modification information including the modified target words; and update the temporal fusion language model according to the target words and the input content of the speech information.

[0118] This embodiment detects user input and performs speech recognition on the received voice information to obtain a recognition result. The recognition result is then subjected to conditional detection. If the recognition result meets preset conditions, a temporal fusion language model is used to analyze the speech information to obtain a temporal score. The temporal fusion language model can be triggered when the recognition result meets preset conditions. The speech information includes semantic and temporal information, which is integrated into the analysis and calculation of semantic information. The temporal fusion language model then performs semantic information analysis on the speech information. Based on the temporal score and the analyzed semantic features, a speech recognition result that simultaneously contains both temporal and semantic information can be obtained. This makes the target result more personalized and avoids the problem of overly standardized recognition results caused by relying solely on semantic information.

[0119] The voice input method optimization device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific device.

[0120] The voice input method optimization device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0121] The voice input method optimization device provided in this application embodiment can implement the various processes implemented in the above method embodiments. To avoid repetition, it will not be described again here.

[0122] Optionally, such as Figure 5As shown, this application embodiment also provides an electronic device 500, including a processor 501 and a memory 502. The memory 502 stores a program or instructions that can run on the processor 501. When the program or instructions are executed by the processor 501, they implement the various steps of the above-described speech recognition method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0123] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0124] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0125] The electronic device 1000 includes, but is not limited to, components such as: radio frequency unit 1001, network module 1002, audio output unit 1003, input unit 1004, sensor 1005, display unit 1006, user input unit 1007, interface unit 1008, memory 1009, and processor 1010.

[0126] Those skilled in the art will understand that the electronic device 1000 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1010 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0127] The input unit 1004 is used to receive voice information input by the user.

[0128] The processor 1010, upon receiving voice information input by a user, performs speech recognition on the voice information to obtain a recognition result, wherein the voice information includes semantic information and temporal information; if the recognition result meets preset conditions, it performs temporal information analysis on the voice information using a temporal fusion language model to obtain a temporal score; it then performs semantic information analysis on the voice information using the temporal fusion language model, and obtains a target result based on the temporal score and the semantic feature information obtained from the semantic information analysis.

[0129] Optionally, the display unit 1006 is used to output the target result.

[0130] The processor 1010 is further configured to convert the time-series information into a first feature vector; extract features from the first feature vector to obtain time-series feature information; perform weighted calculation on the time-series feature information to obtain weighted time-series feature information; perform dimensionality reduction calculation on the weighted time-series feature information to obtain time-series confidence; and perform normalization calculation on the time-series confidence to obtain a time-series score.

[0131] The processor 1010 is further configured to: convert the semantic information into a second feature vector; extract features from the second feature vector to obtain semantic feature information; fuse the semantic feature information and the temporal score to obtain a fusion score; perform weighted calculation on the fusion score to obtain a weighted fusion score; perform dimensionality reduction calculation on the weighted fusion score to obtain a fusion score confidence level; perform normalization calculation on the fusion score confidence level to obtain a fusion score; and output the target text with the highest fusion score as the target result.

[0132] The processor 1010 is further configured to, before receiving voice information input by the user, acquire the user's historical operation records, the historical operation records including the user's input content, the number of times the input content was entered, the input time, and the application name where the input content is located; classify and store the historical operation records according to preset rules to obtain a dictionary database; and create the temporal fusion language model based on each application according to the dictionary database.

[0133] The processor 1010 is further configured to classify the historical operation records according to the application name to obtain multiple sub-data, the sub-data including multiple sets, each set including the user's input time, input content, and the number of times the input content is entered; to sort the sub-data according to the chronological order of the input time, and to sort the sets according to the chronological order of the input time.

[0134] This embodiment detects user input and performs speech recognition on the received voice information to obtain a recognition result. The recognition result is then subjected to conditional detection. If the recognition result meets preset conditions, a temporal fusion language model is used to analyze the speech information to obtain a temporal score. The temporal fusion language model can be triggered when the recognition result meets preset conditions. The speech information includes semantic and temporal information, which is integrated into the analysis and calculation of semantic information. The temporal fusion language model then performs semantic information analysis on the speech information. Based on the temporal score and the analyzed semantic features, a speech recognition result that simultaneously contains both temporal and semantic information can be obtained. This makes the target result more personalized and avoids the problem of overly standardized recognition results caused by relying solely on semantic information.

[0135] It should be understood that, in this embodiment, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042. The GPU 10041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1006 may include a display panel 10061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here.

[0136] The memory 1009 can be used to store software programs and various data. The memory 1009 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1009 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1009 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0137] Processor 1010 may include one or more processing units; optionally, processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 1010.

[0138] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described speech recognition method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0139] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0140] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described speech recognition method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0141] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0142] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described speech recognition method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0143] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0145] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A speech recognition method, characterized in that, The method includes: Upon receiving voice information input by a user, speech recognition is performed on the voice information to obtain a recognition result. The voice information includes semantic information and temporal information, and the temporal information indicates the time and frequency at which the voice information triggers the temporal fusion language model. If the recognition result meets the preset conditions, the speech information is analyzed by the temporal fusion language model to obtain a temporal score. The speech information is semantically analyzed using the temporal fusion language model, and the target result is obtained based on the temporal score and the semantic feature information obtained from the semantic analysis. The step of performing temporal information analysis on the speech information using the temporal fusion language model to obtain a temporal score includes: The time series information is converted into a first feature vector; Feature extraction is performed on the first feature vector to obtain temporal feature information; The time-series feature information is weighted to obtain weighted time-series feature information; The weighted time-series feature information is subjected to dimensionality reduction calculation to obtain the time-series confidence score; The time series confidence score is calculated by normalizing the time series confidence score.

2. The method according to claim 1, characterized in that, The step of performing semantic information analysis on the speech information using a temporal fusion language model, and obtaining the target result based on the temporal score and semantic feature information obtained from the semantic information analysis, includes: The semantic information is converted into a second feature vector; Feature extraction is performed on the second feature vector to obtain semantic feature information; The semantic feature information and the temporal score are fused to obtain a fused score; The fusion scores are weighted to obtain the weighted fusion scores; The dimensionality reduction calculation is performed on the weighted fusion score to obtain the fusion score confidence level; The fusion score is obtained by normalizing the confidence level of the fusion score. Output the target text with the highest fusion score as the target result.

3. The method according to claim 1, characterized in that, Before receiving voice input from the user, the method further includes: Obtain the user's historical operation records, which include the user's input content, the number of times the input content was entered, the input time, and the application name where the input content was located; The historical operation records are classified and stored according to preset rules to obtain a dictionary database; Based on the dictionary database, a time-series fusion language model is created for each application.

4. The method according to claim 3, characterized in that, The step of classifying and storing the historical operation records according to preset rules includes: The historical operation records are categorized according to the application name to obtain multiple sub-data. The sub-data includes multiple sets, and each set includes the user's input time, input content, and the number of times the input content is entered. The sub-data is sorted according to the chronological order of the input times, and the set is sorted according to the chronological order of the input times.

5. The method according to claim 1, characterized in that, The temporal fusion language model includes: a temporal model and a semantic model; The temporal model includes a first input embedding layer, a first multi-head attention layer, a first forward computation layer, a first linear layer, and a first activation function layer connected in sequence. The first input embedding layer is the input layer of the time series model, used to convert time series information into a first feature vector; The first multi-head attention layer is used to extract features from the first feature vector to obtain temporal feature information; The first forward computation layer is used to perform weighted computation on the temporal feature information to obtain weighted temporal feature information; The first linear layer is used to perform dimensionality reduction calculation on the weighted temporal feature information to obtain the temporal confidence score; The first activation function layer is the output layer of the time series model, used to normalize the time series confidence to obtain the time series score; The semantic model includes a second input embedding layer, a second multi-head attention layer, a fusion module, a second forward computation layer, a second linear layer, and a second activation function layer connected in sequence; the first activation function layer is connected to the fusion module. The second input embedding layer is the input layer of the semantic model, used to convert semantic information into a second feature vector; The second multi-head attention layer is used to extract features from the second feature vector to obtain semantic feature information; The fusion module is used to fuse the semantic feature information and the temporal score to obtain a fusion score; The second forward computation layer is used to perform weighted calculation on the fusion score to obtain a weighted fusion score; The second linear layer is used to perform dimensionality reduction calculation on the weighted fusion score to obtain the fusion score confidence level; The second activation function layer is the output layer of the semantic model. It is used to normalize the confidence of the fusion score, obtain the fusion score, and output the target text with the highest fusion score as the target result.

6. A voice recognition device, characterized in that, The device includes: The recognition module is used to perform speech recognition on the received voice information input by the user and obtain a recognition result. The voice information includes semantic information and temporal information, and the temporal information indicates the time and frequency at which the voice information triggers the temporal fusion language model. The first processing module is used to perform temporal information analysis on the speech information through the temporal fusion language model to obtain a temporal score when the recognition result meets the preset conditions. The second processing module is used to perform semantic information analysis on the speech information through the temporal fusion language model, and obtain the target result based on the temporal score and the semantic feature information obtained from the semantic information analysis; The first processing module is used to convert the time-series information into a first feature vector; Feature extraction is performed on the first feature vector to obtain temporal feature information; The time-series feature information is weighted to obtain weighted time-series feature information; The weighted time-series feature information is subjected to dimensionality reduction calculation to obtain the time-series confidence score; the time-series confidence score is then normalized to obtain the time-series score.

7. The apparatus according to claim 6, characterized in that, The second processing module is used to convert the semantic information into a second feature vector; Feature extraction is performed on the second feature vector to obtain semantic feature information; The semantic feature information and the temporal score are fused to obtain a fused score; The fusion scores are weighted to obtain the weighted fusion scores; The weighted fusion score is dimensionality reduced to obtain the fusion score confidence level; the fusion score confidence level is then normalized to obtain the fusion score. Output the target text with the highest fusion score as the target result.

8. The apparatus according to claim 6, characterized in that, The device also includes a model creation module for acquiring the user's historical operation records, which include the user's input content, the number of times the input content was entered, the input time, and the application name where the input content was located. The historical operation records are classified and stored according to preset rules to obtain a dictionary database; Based on the dictionary database, a time-series fusion language model is created for each application.

9. The apparatus according to claim 6, characterized in that, The device also includes a model creation module, used to: classify historical operation records according to application name to obtain multiple sub-data, the sub-data including multiple sets, each set including user input time, input content, and the number of times the input content is input; The sub-data is sorted according to the chronological order of the input times, and the set is sorted according to the chronological order of the input times.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the speech recognition method as described in any one of claims 1-5.

11. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the speech recognition method as described in any one of claims 1-5.

Images