Generative speech recognition method and system for intelligent semantic understanding and calibration

Abstract

The present invention provides a generative speech recognition method and system for intelligent semantic understanding and calibration, including: extracting features from speech data during a call to obtain key speech features; inputting the key speech features into a deep learning model based on a long short-term memory network for semantic understanding to obtain a preliminary semantic result; obtaining a call scene during the call process, analyzing the multi-dimensional semantics associated with the preliminary semantic result in a database associated with the call scene to obtain a semantic association matrix; adaptively calibrating the preliminary semantic result according to the semantic association matrix to obtain a calibrated semantic result; and performing text generation processing on the calibrated semantic result based on a generative adversarial network to obtain a speech recognition text result. In the present invention, the text result finally generated is made more accurate and complete.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a generative speech recognition method and system for intelligent semantic understanding and calibration. Background Art

[0002] Traditional speech recognition methods usually only perform simple feature extraction and pattern matching on speech signals, which makes it difficult to accurately understand complex semantic content. Especially in different call scenarios, the accuracy and reliability of speech recognition are greatly limited due to the diversity of language expression, the complexity of the context, and possible interference factors such as accents and noise.

[0003] For example, in business negotiation scenarios, professional terms and specific negotiation language styles may be involved; in daily communication scenarios, colloquial expressions and casualness are more common. Traditional methods often cannot adapt well to these different call scenarios, and cannot accurately capture semantic information and perform effective calibration and text generation.

[0004] In addition, existing speech recognition technology lacks in-depth and multi-dimensional analysis in terms of semantic understanding, making it difficult to accurately calibrate preliminary semantic results, resulting in inaccurate generated text results or problems with not conforming to language habits. Summary of the invention

[0005] The main purpose of the present invention is to provide a generative speech recognition method and system with intelligent semantic understanding and calibration, aiming to solve the current technical problem of inaccurate speech-generated text.

[0006] To achieve the above object, the present invention provides a generative speech recognition method with intelligent semantic understanding and calibration, comprising the following steps:

[0007] Extract features from the voice data during the call to obtain key voice features;

[0008] Input key speech features into a deep learning model based on a long short-term memory network for semantic understanding and obtain preliminary semantic results;

[0009] Acquire a call scene of the call process, and analyze the multi-dimensional semantics associated with the preliminary semantic results in a database associated with the call scene to obtain a semantic association matrix;

[0010] Adaptively calibrating the preliminary semantic result according to the semantic association matrix to obtain a calibrated semantic result;

[0011] Based on a generative adversarial network, text generation processing is performed on the calibration semantic result to obtain a speech recognition text result.

[0012] Furthermore, in the database associated with the call scenario, the multi-dimensional semantics associated with the preliminary semantic results are analyzed to obtain a semantic association matrix, including:

[0013] Extracting multiple semantic dimension data related to the preliminary semantic result from a database associated with the call scenario; the semantic dimension data includes lexical semantic data, grammatical semantic data, contextual semantic data, emotional semantic data and intentional semantic data;

[0014] Perform similar word analysis on lexical semantic data to obtain lexical semantic association values; perform grammatical structure analysis on grammatical semantic data to obtain grammatical semantic association values; perform context matching and scene adaptation analysis on contextual semantic data to obtain contextual semantic association values; perform emotional tendency judgment on emotional semantic data to obtain emotional semantic association values; perform purpose recognition on intention semantic data to obtain intention semantic association values;

[0015] The association values ​​of each semantic dimension data are combined into a semantic association matrix.

[0016] Furthermore, the step of adaptively calibrating the preliminary semantic result according to the semantic association matrix to obtain a calibrated semantic result includes:

[0017] Calculating the similarity between the preliminary semantic result and each semantic dimension data in the semantic association matrix;

[0018] The preliminary semantic results are weighted and adjusted according to the weight coefficients corresponding to the data of different semantic dimensions and the corresponding similarities;

[0019] The weighted adjusted results are subjected to consistency check and semantic conflict resolution to obtain calibrated semantic results; if there is a semantic conflict, it is corrected according to the preset conflict resolution rules.

[0020] Furthermore, the text generation process is performed on the calibration semantic result based on the generative adversarial network to obtain the speech recognition text result, including:

[0021] Converting the calibration semantic result into an initial text sequence and inputting it into the generator in the generative adversarial network;

[0022] Encoding the initial text sequence based on the generator to extract a semantic feature vector;

[0023] Generate text content word by word based on semantic feature vectors; when generating each word, combine the generated words and semantic feature vectors to determine the next word with the highest probability through probability calculation; and in the generation process, iteratively adjust the generated text to make it conform to language habits; when the length of the generated text reaches a preset value or meets a specific termination condition, stop the generation process and use the generated text as the speech recognition text result.

[0024] Further, the acquiring the call scenario of the call process includes:

[0025] Obtain the relationship with the person you are talking to and identify keywords in the voice data;

[0026] Based on the character relationships and the keywords, corresponding call scenarios are matched in the database; wherein each call scenario has a set character relationship and call keywords.

[0027] Furthermore, after obtaining the speech recognition text result, the method further includes:

[0028] Get all the voice data during the call and save it in a preset folder;

[0029] Extracting multiple key information of the entire speech data, and performing multiple dimensional feature extraction on the speech recognition text result to obtain multi-dimensional features;

[0030] Generate a communication key based on the multiple key information and multi-dimensional features;

[0031] The preset folder is compressed, and the compressed file is encrypted based on the communication key and then transmitted to the management server for archiving.

[0032] The present invention also provides a generative speech recognition system with intelligent semantic understanding and calibration, comprising:

[0033] A feature extraction unit, used to extract features from voice data during a call to obtain key voice features;

[0034] The semantic understanding unit is used to input key speech features into the deep learning model based on the long short-term memory network for semantic understanding and obtain preliminary semantic results;

[0035] A semantic association unit, configured to obtain a call scenario of the call process, and analyze the multi-dimensional semantics associated with the preliminary semantic results in a database associated with the call scenario to obtain a semantic association matrix;

[0036] A semantic calibration unit, configured to adaptively calibrate the preliminary semantic result according to the semantic association matrix to obtain a calibrated semantic result;

[0037] The text generation unit is used to perform text generation processing on the calibration semantic result based on a generative adversarial network to obtain a speech recognition text result.

[0038] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0039] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0040] The generative speech recognition method and system for intelligent semantic understanding and calibration provided by the present invention include: extracting features from speech data during a call to obtain key speech features; inputting the key speech features into a deep learning model based on a long short-term memory network for semantic understanding to obtain a preliminary semantic result; obtaining a call scene during the call process, and analyzing the multi-dimensional semantics associated with the preliminary semantic result in a database associated with the call scene to obtain a semantic association matrix; adaptively calibrating the preliminary semantic result according to the semantic association matrix to obtain a calibrated semantic result; and performing text generation processing on the calibrated semantic result based on a generative adversarial network to obtain a speech recognition text result. In the present invention, by combining the call scene during the call process, the multi-dimensional semantics associated with the preliminary semantic result is expanded, and then the preliminary semantic result is adaptively calibrated, and then the text generation processing is performed, so that the final generated text result is more accurate and complete. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of a generative speech recognition method for intelligent semantic understanding and calibration in one embodiment of the present invention;

[0042] Figure 2 It is a schematic diagram of the steps of a generative speech recognition method for intelligent semantic understanding and calibration in one embodiment of the present invention;

[0043] Figure 3 is a block diagram of a generative speech recognition system for intelligent semantic understanding and calibration in one embodiment of the present invention;

[0044] Figure 4 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0045] The implementation, functional features and advantages of the present invention will be further described in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0047] Reference Figure 1 as well as Figure 2 In one embodiment of the present invention, a generative speech recognition method with intelligent semantic understanding and calibration is provided, comprising the following steps:

[0048] Step S1, extracting features from the voice data during the call to obtain key voice features;

[0049] Step S2, inputting the key speech features into a deep learning model based on a long short-term memory network for semantic understanding to obtain preliminary semantic results;

[0050] Step S3, obtaining the call scene of the call process, and analyzing the multi-dimensional semantics associated with the preliminary semantic results in a database associated with the call scene to obtain a semantic association matrix;

[0051] Step S4, adaptively calibrating the preliminary semantic result according to the semantic association matrix to obtain a calibrated semantic result;

[0052] Step S5: Based on a generative adversarial network, text generation processing is performed on the calibration semantic result to obtain a speech recognition text result.

[0053] In this embodiment, as described in step S1 above, the voice data collected during the call needs to be processed first. The purpose of feature extraction is to extract key features that effectively characterize the voice content from the original voice signal. The above features include the spectral characteristics of the voice, Mel-frequency cepstral coefficients (MFCC), and perceptual linear prediction coefficients (PLP). Through feature extraction, complex voice signals are converted into digital feature vectors that are easier to process and analyze, providing a basis for subsequent semantic understanding and recognition. For example, a continuous voice signal can be divided into several short-term voice frames by using techniques such as framing and windowing, and then each voice frame is processed by Fourier transform and the like to extract its frequency characteristics. The above features can reflect information such as the timbre and pitch of the voice, which helps to distinguish different voice contents.

[0054] As described in step S2 above, the long short-term memory network (LSTM) is a deep learning model specifically used to process sequence data, and is very suitable for processing data with time series characteristics such as speech. In this step, the key speech features extracted in step S1 are input into the pre-trained LSTM-based deep learning model. By learning a large amount of speech data and corresponding text semantic annotations, the model can automatically extract deep semantic information from the input speech features and generate preliminary semantic results. This preliminary semantic result can be a combination of one or more words, phrases or sentences, representing a preliminary understanding of the input speech. For example, if the input speech is "The weather is really good today", after being processed by the LSTM model, the preliminary semantic result may be "The weather is good today".

[0055] As described in step S3 above, different call scenarios may have an important impact on the semantic understanding of speech. For example, the same words have different meanings in business negotiation scenarios and daily chat scenarios. In this step, the call scenario of the current call process is first obtained, and the call scenario can be determined by analyzing the identities of the two parties to the call, the time, location, topic, etc. of the call. Then, in the database associated with the call scenario, a multi-dimensional semantic analysis is performed on the preliminary semantic results obtained in step S2. This multi-dimensional semantic analysis may include lexical semantics, grammatical semantics, contextual semantics, emotional semantics, and intentional semantics. By analyzing the semantics of these different dimensions, a semantic association matrix can be obtained. Each element in the semantic association matrix represents the degree of association between different words or phrases in the preliminary semantic results in different semantic dimensions. For example, in a business negotiation scenario, the two words "price" and "discount" may have a high degree of association in lexical semantics; in emotional semantics, if the speaker's tone is relatively strong, then the word "price" may have a high negative emotional tendency.

[0056] As described in the above step S4, the preliminary semantic result obtained in step S2 is calibrated using the semantic association matrix obtained in step S3. Specifically, according to the degree of association between each word or phrase in the semantic association matrix, the words in the preliminary semantic result are adjusted, supplemented or deleted to obtain a more accurate and complete calibrated semantic result. For example, if the semantic association matrix shows that a certain word is ambiguous or inaccurate in the current context, then the word can be corrected based on the semantic information of other related words. Or, if the semantic association matrix shows that an important word is missing in the preliminary semantic result, it can be supplemented based on words with a higher degree of association.

[0057] As described in step S5 above, the above-mentioned generative adversarial network (GAN) is composed of a generator and a discriminator. In this step, the calibration semantic result obtained in step S4 is processed for text generation using the generative adversarial network. First, the calibration semantic result is input into the generator, and then the generator generates the above-mentioned speech recognition text result based on the calibration semantic result. After the above processing, the generator generates more natural and fluent sentences based on the keywords and semantic information in the calibration semantic result, so that the speech recognition result is more accurate and easy to understand. The above-mentioned discriminator is mainly used in the training stage of the generative adversarial network, and it does not need to participate in text generation in the stage of generating text.

[0058] In the training phase of the above-mentioned generative adversarial network, the generator generates candidate texts based on the training data. Then, the discriminator receives the candidate texts generated by the generator and the real text data (i.e., the text corresponding to the training data), and makes a authenticity judgment on them. The generator continuously adjusts parameters based on the feedback from the discriminator to generate results that are closer to the real text. After multiple iterations, when the text generated by the generator meets the preset quality standards, a trained generative adversarial network is obtained.

[0059] In one embodiment, in the database associated with the call scenario, the multi-dimensional semantics associated with the preliminary semantic results are analyzed to obtain a semantic association matrix, including:

[0060] Extracting multiple semantic dimension data related to the preliminary semantic result from a database associated with the call scenario; the semantic dimension data includes lexical semantic data, grammatical semantic data, contextual semantic data, emotional semantic data and intentional semantic data;

[0061] Perform similar word analysis on lexical semantic data to obtain lexical semantic association values; perform grammatical structure analysis on grammatical semantic data to obtain grammatical semantic association values; perform context matching and scene adaptation analysis on contextual semantic data to obtain contextual semantic association values; perform emotional tendency judgment on emotional semantic data to obtain emotional semantic association values; perform purpose recognition on intention semantic data to obtain intention semantic association values;

[0062] The association values ​​of each semantic dimension data are combined into a semantic association matrix.

[0063] In this embodiment, the database associated with the call scenario is pre-built for a specific call scenario, and it contains various semantic information related to the scenario. Different call scenarios have different databases. For example, the database of a business negotiation scenario contains semantic content related to business terms and negotiation strategies; the database of a daily communication scenario may contain semantic content related to spoken expressions and life scenarios. The above database is constructed by collecting a large amount of voice data and corresponding text annotations in the scenario, and analyzing and organizing these data using natural language processing technology.

[0064] The above-mentioned multiple semantic dimension data include:

[0065] Lexical semantic data: mainly involves the basic meaning of words, synonyms, antonyms, etc. For example, in a call scenario, the lexical semantic data of the word "buy" may include synonyms such as "purchase" and "procurement".

[0066] Grammatical semantic data: Focus on the grammatical structure and grammatical rules of a sentence, including parts of speech, sentence components, grammatical relations, etc. For example, analyzing the subject-verb-object structure and modification relations of a sentence.

[0067] Contextual semantic data: considers the meaning of words and sentences in a specific context. The context can be the sentences before and after, the background of the speaker, the topic of the conversation, etc. For example, in a discussion about a technology product, the contextual semantics of the word "performance" may be related to the speed, stability, etc. of the product.

[0068] Sentimental semantic data: Determine the speaker's emotional tendency, such as positive, negative, neutral, etc. Sentimental semantics can be determined by analyzing the speaker's tone, word choice, etc. For example, the use of words such as "great" and "terrible" can reflect positive or negative emotional tendencies.

[0069] Intent semantic data: Identify the speaker’s purpose and intention. For example, in a business negotiation, the speaker’s intention may be to close a deal, get a better price, etc. In daily communication, the intention may be to ask for information, share feelings, etc.

[0070] In this embodiment, it is necessary to analyze data of different semantic dimensions to obtain associated values.

[0071] Specifically, the lexical semantic association value is determined by calculating the semantic similarity between words. Word vector technology can be used to represent words in vector form, and then the distance between vectors is calculated to measure the similarity. For example, the closer the vector distance between two words is, the higher their similarity is, and the greater the lexical semantic association value is. For example, if the word "mobile phone" is included in the preliminary semantic results, similar word analysis may result in words such as "smartphone" and "mobile phone", which have higher lexical semantic association values ​​with "mobile phone".

[0072] Perform grammatical analysis on the sentences in the preliminary semantic results to determine whether the grammatical structure of the sentences is correct and reasonable. For example, check whether the subject, predicate and object are complete, whether the modifiers are appropriate, etc. If the grammatical structure of the sentence conforms to the grammatical rules, then the grammatical semantic association value is high; conversely, if there are grammatical errors, the association value is low. For example, the sentence "I ate an apple" has a grammatical structure error and a low grammatical semantic association value; while "I ate an apple" has a correct grammatical structure and a high association value.

[0073] Match the words and sentences in the preliminary semantic results with the context of the call scenario to determine whether they are appropriate in the scenario. At the same time, the semantics are adapted considering the characteristics and requirements of the scenario. If the words and sentences are highly consistent with the context and meet the requirements of the scenario, the contextual semantic association value is high. For example, in a business meeting scenario, when the word "contract" is mentioned, if the context is discussing the terms of the contract, the contextual semantic association value is high; if the context is not related to the contract, the association value is low.

[0074] By analyzing the speaker's tone, word choice, etc., the emotional semantic tendency, such as positive, negative or neutral, can be determined. The emotional dictionary, machine learning algorithm and other methods can be used for judgment. If the preliminary semantic result expresses positive emotions, the emotional semantic association value is positive; if it is negative emotions, the association value is negative; the neutral emotional association value is close to zero. For example, "This solution is very good" expresses positive emotions and has a high emotional semantic association value; "This solution has many problems" expresses negative emotions and has a low association value.

[0075] By analyzing the speaker's speech content and behavior, their intention and purpose can be determined. The judgment can be made based on the call scene and context. For example, if the speaker proposes "lowering the price" in a business negotiation, then it can be judged that his intention is to strive for better transaction conditions, and the intention semantic association value is related to "negotiation strategy"; if he says "the weather is so nice today" in daily communication, the intention may be to share his feelings, and the association value is related to "social interaction".

[0076] The above-mentioned semantic association matrix is ​​a multidimensional matrix, in which each row and each column represents a semantic dimension, and the elements in the matrix represent the degree of association between different semantic dimensions. For example, the first row of the matrix can represent the lexical semantic dimension, the second row represents the grammatical semantic dimension, and so on. The elements in the matrix can be numerical values, and the size of the numerical value represents the degree of association between the two semantic dimensions. The obtained association values ​​of each semantic dimension are combined according to the structure of the matrix. The combination method of the association values ​​can be determined according to specific needs and algorithms. For example, the weighted average method can be used to weight the sum of the association values ​​of different semantic dimensions to obtain the final semantic association matrix. The determination of the weight can be adjusted according to factors such as the characteristics of the call scenario and the importance of the semantic dimension.

[0077] Through the above steps, we can obtain a semantic association matrix that can comprehensively reflect the degree of association of the preliminary semantic results in multiple semantic dimensions, providing an important basis for subsequent semantic calibration and text generation.

[0078] In one embodiment, the step of adaptively calibrating the preliminary semantic result according to the semantic association matrix to obtain a calibrated semantic result includes:

[0079] Calculating the similarity between the preliminary semantic result and each semantic dimension data in the semantic association matrix;

[0080] The preliminary semantic results are weighted and adjusted according to the weight coefficients corresponding to the data of different semantic dimensions and the corresponding similarities;

[0081] The weighted adjusted results are subjected to consistency check and semantic conflict resolution to obtain calibrated semantic results; if there is a semantic conflict, it is corrected according to the preset conflict resolution rules.

[0082] In this embodiment, the above-mentioned preliminary semantic result is usually a text representation consisting of words, phrases or sentences. It is obtained through the previous semantic understanding step and may still have some inaccuracies or incompleteness. For example, the preliminary semantic result may be "The weather is nice today, suitable for going out for a walk."

[0083] The above semantic association matrix contains data of multiple semantic dimensions, such as lexical semantic data, grammatical semantic data, contextual semantic data, emotional semantic data, and intentional semantic data, etc. In order to determine the similarity between the preliminary semantic results and the data of each semantic dimension, different similarity calculation methods can be used.

[0084] For the lexical semantic dimension, the cosine similarity of word vectors can be used to calculate the similarity between words. The words in the preliminary semantic results are represented as word vectors, and then compared with the word vectors in the lexical semantic data in the semantic association matrix to calculate the cosine similarity.

[0085] For the grammatical semantic dimension, the similarity can be determined by analyzing whether the grammatical structures of the sentences are similar, for example, determining whether the subject-predicate-object structures of two sentences are consistent.

[0086] For the contextual semantic dimension, the rationality and adaptability of the preliminary semantic results in a specific context can be considered, matched with the contextual semantic data in the semantic association matrix, and the similarity can be calculated.

[0087] For the emotional semantic dimension, the similarity can be calculated based on the consistency of the emotional tendency. For example, if the emotion expressed by the preliminary semantic result is the same as the emotional tendency of the emotional semantic data in the semantic association matrix, the similarity is high.

[0088] For the intent semantic dimension, the similarity can be calculated by judging the matching degree between the intent reflected by the preliminary semantic result and the intent semantic data in the semantic association matrix.

[0089] Different semantic dimensions have different importance in the semantic calibration process, so a weight coefficient needs to be assigned to each semantic dimension data. The weight coefficient can be adjusted according to call scenarios, application requirements, data statistics and other factors. For example, in business negotiation scenarios, the intention semantic dimension is more important, so it can be given a higher weight coefficient; while in daily communication scenarios, the emotional semantic dimension may be relatively important. Therefore, the weight coefficient can be determined by setting experience values, optimizing machine learning algorithms, and so on.

[0090] According to the calculated similarity and the corresponding weight coefficient, the preliminary semantic result is weighted and adjusted. Specifically, for each semantic dimension, its similarity is multiplied by the corresponding weight coefficient to obtain the contribution value of the semantic dimension to the preliminary semantic result. Then the contribution values ​​of each semantic dimension are added together to obtain the preliminary semantic result after weighted adjustment.

[0091] The consistency test is performed on the preliminary semantic results after weighted adjustment to check whether there are any contradictions or inconsistencies. For example, check whether the lexical semantics are contradictory, whether the grammatical structure is correct and consistent, whether the emotional tendency is consistent with the context, etc. The consistency test can be performed by formulating a series of rules and constraints. For example, it is stipulated that contradictory words cannot appear at the same time in the same sentence; the emotional semantic tendency should match the context semantics, etc.

[0092] If a semantic conflict is found during the consistency check, conflict resolution is required. Conflict resolution can be performed according to preset conflict resolution rules. These rules can be formulated according to specific application scenarios and requirements. For example, if there is a conflict in lexical semantics, a word that is more in line with the context can be selected; if there is a conflict in grammatical structure, it can be corrected according to grammatical rules. For example, if the two contradictory words "good" and "bad" appear in the preliminary semantic results at the same time, the specific emotional tendency can be judged according to the context, and one of the words can be selected to retain; if there is an error in the grammatical structure of the sentence, it can be adjusted according to the grammatical rules to make it conform to the correct grammatical structure.

[0093] After consistency check and semantic conflict resolution, the result is the calibrated semantic result. Compared with the preliminary semantic result, the calibrated semantic result is more accurate, complete and consistent, and can better reflect the semantic content during the call.

[0094] In one embodiment, the text generation process is performed on the calibration semantic result based on the generative adversarial network to obtain the speech recognition text result, including:

[0095] Converting the calibration semantic result into an initial text sequence and inputting it into the generator in the generative adversarial network;

[0096] Encoding the initial text sequence based on the generator to extract a semantic feature vector;

[0097] Generate text content word by word based on semantic feature vectors; when generating each word, combine the generated words and semantic feature vectors to determine the next word with the highest probability through probability calculation; and in the generation process, iteratively adjust the generated text to make it conform to language habits; when the length of the generated text reaches a preset value or meets a specific termination condition, stop the generation process and use the generated text as the speech recognition text result.

[0098] In this embodiment, first, the calibration semantic result is converted into an initial text sequence form suitable for input to the generator. This can be achieved by arranging and encoding words or phrases according to certain rules. For example, each word can be represented as a number or vector, and then the entire sentence can be represented as a sequence of numbers or vectors. Such a sequence can be used as the input of the generator.

[0099] Generative adversarial networks (GANs) consist of a generator and a discriminator. The initial text sequence is input into the generator to start the text generation process. The function of the generator is to generate new text content based on the input initial text sequence. The generator first encodes the input initial text sequence. The purpose of encoding is to convert the text sequence into an intermediate representation that can capture semantic information. Common encoding methods include the use of neural networks, such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs), or transformers. The above networks can learn the sequence features and semantic information of the text and encode them into a fixed-length vector, namely a semantic feature vector.

[0100] Through the encoding process, the generator extracts the semantic feature vector of the initial text sequence. This vector contains the semantic information of the text and can be used in the subsequent text generation process. The semantic feature vector can be regarded as an abstract representation of the input text, which can capture the characteristics of the text in terms of theme, sentiment, intention, etc. When generating each word, the generator combines the generated words and the semantic feature vector to determine the possibility of the next word. The generated words provide the context information of the current text, while the semantic feature vector contains the overall semantic information. The generator takes these two factors into consideration to determine the next most likely word. The generator determines the possibility of the next word through probability calculation. Specifically, it calculates the probability of each possible word according to the current state (generated words and semantic feature vector). Then, the word with the highest probability is selected as the next generated word. This process can be implemented through the output layer of the neural network. Each node of the output layer corresponds to a word, and the output value of the node represents the probability of the word appearing. During the generation process, the generator will continuously iteratively adjust the generated text to make it more in line with language habits. For example, a language model is used to evaluate the rationality and fluency of the generated text, and the generated words are adjusted according to the evaluation results. The generator can also learn language grammatical rules, vocabulary collocation and other patterns to generate more natural text.

[0101] In order to control the length of the generated text and ensure the termination of the generation process, a preset value and a specific termination condition are set. The preset value can be the upper limit of the length of the generated text, for example, the generation is stopped when the length of the generated sentence reaches a certain number of words. The termination condition can be that the generator automatically stops generating when it determines that the generated text is complete enough or meets a specific standard. When the length of the generated text reaches the preset value or meets a specific termination condition, the generator stops the generation process. At this point, the generated text is considered to be the final speech recognition text result.

[0102] In one embodiment, the acquiring the call scenario of the call process includes:

[0103] Obtain the relationship with the person you are talking to and identify keywords in the voice data;

[0104] Based on the character relationships and the keywords, corresponding call scenarios are matched in the database; wherein each call scenario has a set character relationship and call keywords.

[0105] In this embodiment, during a call, it is very important to determine the character relationship between the call objects. Character relationships may include family, friends, colleagues, customers, etc. Character relationships can be obtained in a variety of ways, such as obtaining contact information from an address book, analyzing the relationship between historical call objects through call records, or using social network data. For example, if the contact note of the call object is "Dad", then it can be determined that the character relationship is a family relationship. At the same time, keyword recognition is performed on the voice data during the call. Keywords are important words that can reflect the topic, scene or specific semantics of the call. Keyword extraction algorithms in natural language processing technology, such as statistical methods (such as TF-IDF) and machine learning methods (such as support vector machines, deep learning, etc.) can be used to identify keywords. For example, keywords such as "project progress" and "meeting arrangement" are identified in a voice, and these keywords may imply business work scenarios.

[0106] In this embodiment, a database containing various call scenarios is pre-built. Each call scenario has specific character relationships and call keyword settings. The construction of the database can be achieved by analyzing and annotating a large amount of call data. For example, some common call scenarios such as family chats, business negotiations, and friends' gatherings are classified, and typical character relationships and keywords in each scenario are determined. For example, the character relationship set in the family chat scenario is family members, and the keywords include "family gatherings", "birthdays", "housework", etc.; the character relationship set in the business negotiation scenario is colleagues or customers, and the keywords include "contracts", "prices", "plans", etc.

[0107] Based on the acquired character relationships and identified keywords, matching is performed in the database to determine the corresponding call scenario. The specific matching method can be to compare the character relationships and keywords with each call scenario in the database to find the most suitable scenario. Similarity calculation methods, such as cosine similarity, Jaccard similarity, etc., can be used to measure the degree of matching between character relationships and keywords and various call scenarios. For example, if the character relationship is colleagues and the keywords include "project progress" and "meeting arrangement", then when matching in the database, a business work scenario may be found as the most matching call scenario.

[0108] In one embodiment, after obtaining the speech recognition text result, the method further includes:

[0109] Get all the voice data during the call and save it in a preset folder;

[0110] Extracting multiple key information of the entire speech data, and performing multiple dimensional feature extraction on the speech recognition text result to obtain multi-dimensional features;

[0111] Generate a communication key based on the multiple key information and multi-dimensional features;

[0112] The preset folder is compressed, and the compressed file is encrypted based on the communication key and then transmitted to the management server for archiving.

[0113] In this embodiment, after obtaining the speech recognition text result, it is necessary to obtain all the speech data during the call. This can be achieved through the recording function of the call device or the audio acquisition module of the communication system. The speech data includes the sound signals of both parties of the call, which contains rich information and can not only be used for subsequent analysis and archiving, but also for review and verification when necessary.

[0114] All acquired voice data is saved in a preset folder. The preset folder can be a specific directory on a local storage device or a specified location in a network storage.

[0115] Analyze all voice data and extract multiple key information. These key information may include the speaker's identity, call time, call duration, voice frequency characteristics, volume, etc. For example, the speaker's identity can be determined through voice recognition technology; the call duration can be recorded through system time; and the call duration and voice frequency characteristics can be calculated through audio analysis algorithms.

[0116] Extract features from speech recognition text results in multiple dimensions. These features can include text length, vocabulary richness, grammatical structure, sentiment, topic keywords, etc. For example, the number of words in the text can be counted to determine the text length; natural language processing technology can be used to analyze the vocabulary diversity of the text; grammatical structure can be determined through grammatical analysis algorithms; sentiment analysis tools can be used to determine sentiment; and topic keywords can be extracted through keyword extraction algorithms.

[0117] Integrate multiple key information of all extracted voice data and multi-dimensional features of voice recognition text results. The above information together constitutes the feature set of the call. This information can be encoded or converted into a specific data format for subsequent processing.

[0118] Generate a communication key based on the integrated feature set. The communication key can be a combination of numbers, letters or symbols, which is used to encrypt and decrypt call data. The algorithm for generating the communication key can be a symmetric encryption algorithm (such as AES), an asymmetric encryption algorithm (such as RSA) or a hash function. These algorithms can ensure the security and uniqueness of the key.

[0119] Compress the preset folder that contains all voice data. Compression can reduce the size of the file and improve transmission and storage efficiency. You can use common compression algorithms such as ZIP, RAR, etc. to compress the folder. Encrypt the compressed file using the generated communication key. Encryption protects the security of call data and prevents unauthorized access. According to the selected encryption algorithm, the communication key is applied to the compressed file to encrypt the file content. Only with the correct communication key can the file content be decrypted and accessed.

[0120] The encrypted files are transferred to the management server for archiving. The management server is a server dedicated to storing call data, with high-capacity storage and security management functions. Encrypted files can be uploaded to the management server through network transmission protocols (such as FTP, HTTP, etc.). The management server can further manage and back up the uploaded files to ensure the security and reliability of the data.

[0121] In one embodiment, the multiple key information includes the total amount of voice data, the duration of the voice call, and the identity of the caller; the multi-dimensional features include text length, the number of words, multiple keywords, and deduplicated text after deduplication of the voice recognition text result;

[0122] In one embodiment, the generating of the communication key based on the multiple key information and the multi-dimensional features includes:

[0123] The total amount of voice data, the duration of the voice call, and the identity of the caller are sequentially combined to obtain a voice information combination;

[0124] The text length, the number of words and the deduplicated text are combined to obtain a text combination; the voice information combination is spliced ​​with the text combination to obtain a comprehensive information combination;

[0125] Obtaining a preset binary tree template from a preset template library, and adding the characters in the comprehensive information combination to each node of the binary tree template in sequence, to obtain a character binary tree;

[0126] Deduplication of the multiple keywords is performed to obtain deduplication characters; in the character binary tree, the node where each deduplication character is located is searched as the target node;

[0127] Connecting the target nodes in sequence to obtain a plurality of line segments; extending each line segment so that it touches the edge of the character binary tree to obtain a plurality of extended line segments;

[0128] According to the positional relationship between each node in the character binary tree and the plurality of extended line segments, a plurality of nodes are selected from the character binary tree as key nodes;

[0129] The characters on the key nodes are combined in sequence to obtain the communication key.

[0130] In this embodiment, the total amount of voice data refers to the total size of voice data recorded during the call, which can be measured in bytes. It reflects one aspect of the amount of information and duration of the call. The length of the voice call indicates the duration of the call, usually in seconds, minutes, etc. This key information has a certain reference value for understanding the duration and importance of the call. The identity of the caller clarifies the identity information of the two parties to the call, which can be a name, phone number, position, etc. It is helpful to analyze the background and purpose of the call.

[0131] The above-mentioned text length refers to the length of the speech recognition text result, which can be measured by the number of characters, words, etc. It reflects the size and complexity of the text. The above-mentioned number of words is the number of words in the speech recognition text result, which helps to understand the lexical richness of the text. The above-mentioned multiple keywords are important words extracted by analyzing the speech recognition text results, which can summarize the theme and key content of the text. The above-mentioned deduplicated text is the text obtained after deduplication processing of the speech recognition text result, removing the repeated content and reflecting the core information of the text more concisely.

[0132] In this embodiment, the three key information are first combined in a specific order. For example, the total amount of voice data can be expressed in digital form, then the value of the voice call duration can be added, and finally the identification string of the call object can be added to form a continuous combination string. Such a combination can comprehensively reflect the basic information of the voice call.

[0133] Similarly, the numerical value of the text length, the numerical value of the number of words, and the deduplicated text are combined in a preset manner. The numerical values ​​of the text length and the number of words can be concatenated together first, and then the string content of the deduplicated text can be added to form a text combination. This combination reflects the important characteristics of the speech recognition text result. The previously obtained voice information combination and text combination are concatenated to form a more comprehensive comprehensive information combination. The two combined character strings can be directly connected together to obtain a comprehensive character string containing voice and text features. This comprehensive information combination will serve as the basis for the subsequent generation of communication keys.

[0134] In this embodiment, a library containing various binary tree templates is pre-established. These templates have a binary tree framework with a specific structure and number of nodes. Characters are taken out one by one from the comprehensive information combination and added to each node of the binary tree template in a certain order. For example, starting from the root node, characters can be added to the nodes of the left subtree and the right subtree in sequence until all characters are added. In this way, a character binary tree filled with characters in the comprehensive information combination is obtained. There may be repeated words in the original multiple keywords. By performing text deduplication processing on these keywords, the repeated parts are removed to obtain a unique set of deduplicated text. For example, if the keywords include "project", "project progress" and "important project", after deduplication, "project progress is important" is obtained.

[0135] For each deduplicated character, a search is performed in the character binary tree to determine the node where the character is located. These nodes are marked as target nodes because they are related to keywords and play an important role in generating communication keys.

[0136] Connect the target nodes in order to form multiple line segments. For example, you can connect the corresponding target nodes according to the order in which the duplicate characters appear in the keywords. Extend each line segment so that it touches the edge of the binary tree of characters. This ensures that the line segment covers more nodes, increasing the randomness and complexity of generating the communication key.

[0137] By analyzing the positional relationship between each node and the extended line segment in the character binary tree, it is determined which nodes have a specific association with the line segment. For example, nodes that are passed through or close to the line segment can be selected as key nodes. These nodes are considered to have high randomness and representativeness and are suitable for generating communication keys. It should be noted that the above-mentioned target node is not used as a key node. In some embodiments, the minimum distance between each node and the above-mentioned line segment can be calculated, and when the minimum distance is less than a threshold, it is used as a key node.

[0138] Finally, the characters on the key nodes are combined in order to form a communication key. The above communication key is generated based on the key information and multi-dimensional features of the call, is unique and secure, and can be used to encrypt and decrypt call data.

[0139] Reference Figure 3 In another embodiment of the present invention, a generative speech recognition system with intelligent semantic understanding and calibration is provided, comprising:

[0140] A feature extraction unit, used to extract features from voice data during a call to obtain key voice features;

[0141] The semantic understanding unit is used to input key speech features into the deep learning model based on the long short-term memory network for semantic understanding and obtain preliminary semantic results;

[0142] A semantic association unit, configured to obtain a call scenario of the call process, and analyze the multi-dimensional semantics associated with the preliminary semantic results in a database associated with the call scenario to obtain a semantic association matrix;

[0143] A semantic calibration unit, configured to adaptively calibrate the preliminary semantic result according to the semantic association matrix to obtain a calibrated semantic result;

[0144] The text generation unit is used to perform text generation processing on the calibration semantic result based on a generative adversarial network to obtain a speech recognition text result.

[0145] In this embodiment, for the specific implementation of each unit in the above system embodiment, please refer to the description in the above method embodiment, which will not be repeated here.

[0146] Reference Figure 4 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0147] Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0148] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0149] In summary, the generative speech recognition method and system for intelligent semantic understanding and calibration provided in the embodiments of the present invention include: extracting features from speech data during a call to obtain key speech features; inputting the key speech features into a deep learning model based on a long short-term memory network for semantic understanding to obtain a preliminary semantic result; obtaining the call scene of the call process, and analyzing the multi-dimensional semantics associated with the preliminary semantic result in a database associated with the call scene to obtain a semantic association matrix; adaptively calibrating the preliminary semantic result according to the semantic association matrix to obtain a calibrated semantic result; and performing text generation processing on the calibrated semantic result based on a generative adversarial network to obtain a speech recognition text result. In the present invention, by combining the call scene of the call process, the multi-dimensional semantics associated with the preliminary semantic result is expanded, and then the preliminary semantic result is adaptively calibrated, and then the text generation processing is performed, so that the final generated text result is more accurate and complete.

[0150] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0151] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0152] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A generative speech recognition method with intelligent semantic understanding and calibration, characterized in that: The following steps are involved: Extract features from the voice data during the call to obtain key voice features; Input key speech features into a deep learning model based on a long short-term memory network for semantic understanding and obtain preliminary semantic results; Acquire a call scene of the call process, and analyze the multi-dimensional semantics associated with the preliminary semantic results in a database associated with the call scene to obtain a semantic association matrix; Adaptively calibrating the preliminary semantic result according to the semantic association matrix to obtain a calibrated semantic result; Based on a generative adversarial network, text generation processing is performed on the calibration semantic result to obtain a speech recognition text result.

2. The generative speech recognition method with intelligent semantic understanding and calibration according to claim 1, characterized in that: The multi-dimensional semantics associated with the preliminary semantic results are analyzed in the database associated with the call scenario to obtain a semantic association matrix, including: Extracting multiple semantic dimension data related to the preliminary semantic result from a database associated with the call scenario; the semantic dimension data includes lexical semantic data, grammatical semantic data, contextual semantic data, emotional semantic data and intentional semantic data; Perform similar word analysis on lexical semantic data to obtain lexical semantic association values; perform grammatical structure analysis on grammatical semantic data to obtain grammatical semantic association values; perform context matching and scene adaptation analysis on contextual semantic data to obtain contextual semantic association values; perform emotional tendency judgment on emotional semantic data to obtain emotional semantic association values; perform purpose recognition on intention semantic data to obtain intention semantic association values; The association values ​​of each semantic dimension data are combined into a semantic association matrix.

3. The generative speech recognition method with intelligent semantic understanding and calibration according to claim 2, characterized in that: The step of adaptively calibrating the preliminary semantic result according to the semantic association matrix to obtain a calibrated semantic result includes: Calculating the similarity between the preliminary semantic result and each semantic dimension data in the semantic association matrix; The preliminary semantic results are weighted and adjusted according to the weight coefficients corresponding to the data of different semantic dimensions and the corresponding similarities; The weighted adjusted results are subjected to consistency check and semantic conflict resolution to obtain calibrated semantic results; if there is a semantic conflict, it is corrected according to the preset conflict resolution rules.

4. The generative speech recognition method with intelligent semantic understanding and calibration according to claim 1, characterized in that: The step of performing text generation processing on the calibration semantic result based on the generative adversarial network to obtain a speech recognition text result includes: Converting the calibration semantic result into an initial text sequence and inputting it into the generator in the generative adversarial network; Encoding the initial text sequence based on the generator to extract a semantic feature vector; Generate text content word by word based on semantic feature vectors; when generating each word, combine the generated words and semantic feature vectors to determine the next word with the highest probability through probability calculation; and in the generation process, iteratively adjust the generated text to make it conform to language habits; when the length of the generated text reaches a preset value or meets a specific termination condition, stop the generation process and use the generated text as the speech recognition text result.

5. The generative speech recognition method with intelligent semantic understanding and calibration according to claim 1, characterized in that: The acquiring of the call scenario of the call process includes: Obtain the relationship with the person you are talking to and identify keywords in the voice data; Based on the character relationships and the keywords, corresponding call scenarios are matched in the database; wherein each call scenario has a set character relationship and call keywords.

6. The generative speech recognition method with intelligent semantic understanding and calibration according to claim 1, characterized in that: After obtaining the speech recognition text result, the method further includes: Get all the voice data during the call and save it in a preset folder; Extracting multiple key information of the entire speech data, and performing multiple dimensional feature extraction on the speech recognition text result to obtain multi-dimensional features; Generate a communication key based on the multiple key information and multi-dimensional features; The preset folder is compressed, and the compressed file is encrypted based on the communication key and then transmitted to the management server for archiving.

7. A generative speech recognition system with intelligent semantic understanding and calibration, characterized in that: include: A feature extraction unit, used to extract features from voice data during a call to obtain key voice features; The semantic understanding unit is used to input key speech features into the deep learning model based on the long short-term memory network for semantic understanding and obtain preliminary semantic results; A semantic association unit, configured to obtain a call scenario of the call process, and analyze the multi-dimensional semantics associated with the preliminary semantic results in a database associated with the call scenario to obtain a semantic association matrix; A semantic calibration unit, configured to adaptively calibrate the preliminary semantic result according to the semantic association matrix to obtain a calibrated semantic result; The text generation unit is used to perform text generation processing on the calibration semantic result based on a generative adversarial network to obtain a speech recognition text result.

8. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images