Speech recognition method and related apparatus, device, and storage medium

Abstract

The application discloses a speech recognition method and related device, equipment and storage medium, wherein the speech recognition method comprises: performing feature coding based on the acoustic characteristics of a speech signal to obtain first voiceprint characteristics and first semantic characteristics of the speech signal, and extracting second voiceprint characteristics of each candidate object from the memory data of each candidate object; selecting a candidate object as a target object based on the similarity between the first voiceprint characteristics and each second voiceprint characteristic, and performing feature conversion based on the first semantic characteristics to obtain input data suitable for a large language model; performing retrieval in the second data of the target object based on the input data to obtain reference data; constructing a first prompt instruction based on the first data of the target object, the reference data and the input data; and obtaining target recognition text output by the large language model in response to the first prompt instruction. The above scheme can improve the accuracy and dynamic adaptability of speech recognition.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a speech recognition method and related apparatus, devices and storage media. Background Technology

[0002] With the rapid development of artificial intelligence technology, speech recognition technology has made significant progress and has become an important way of human-computer interaction.

[0003] Current mainstream speech recognition systems typically rely on large-scale, general-purpose training data. While this ensures good recognition performance in general scenarios, it often performs poorly when faced with personalized user expressions. To address this issue, existing technologies introduce hotword mechanisms, allowing users to import specific vocabulary such as frequently used words and technical terms for special processing. However, this approach has limited personalization, handling only a small number of predefined words and failing to dynamically adapt to users' constantly changing language habits and usage scenarios. Therefore, improving the accuracy and dynamic adaptability of speech recognition has become a pressing problem to be solved. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a speech recognition method and related devices, equipment, and storage media that can improve the accuracy and dynamic adaptability of speech recognition.

[0005] To address the aforementioned technical problems, a first aspect of this application provides a speech recognition method, comprising: responding to a speech signal to be recognized, performing feature encoding based on the acoustic features of the speech signal to obtain a first voiceprint feature and a first semantic feature of the speech signal, and extracting a second voiceprint feature of each candidate object from memory data of each candidate object; wherein the memory data includes first data related to object profiles and second data related to interactive dialogue; selecting candidate objects as target objects based on the similarity between the first voiceprint feature and each of the second voiceprint features, and performing feature transformation based on the first semantic feature to obtain input data adapted to a large language model; retrieving reference data from the second data of the target object based on the input data; constructing a first prompt instruction based on the first data, reference data, and input data of the target object; wherein the first prompt instruction is used to instruct the large language model to refer to the first data and reference data of the target object and process the input data into speech recognition text; and obtaining the target recognition text output by the large language model in response to the first prompt instruction.

[0006] To address the aforementioned technical problems, a second aspect of this application provides a speech recognition device, comprising: an encoding extraction module, a selection and conversion module, a memory retrieval module, an instruction construction module, and a text acquisition module. The encoding extraction module is configured to, in response to a speech signal to be recognized, perform feature encoding based on the acoustic features of the speech signal to obtain a first voiceprint feature and a first semantic feature of the speech signal, and extract a second voiceprint feature of each candidate object from the memory data of each candidate object; wherein the memory data includes first data related to the object profile and second data related to the interactive dialogue; the selection and conversion module is configured to, based on the first voiceprint feature, respectively select and convert the second voiceprint feature of each candidate object and the second voiceprint feature of the third voiceprint feature. The similarity between the two voiceprint features is used to select candidate objects as target objects, and feature transformation is performed based on the first semantic feature to obtain input data adapted to the large language model; the memory retrieval module is used to retrieve reference data from the second data of the target object based on the input data; the instruction construction module is used to construct a first prompt instruction based on the first data, reference data and input data of the target object; wherein, the first prompt instruction is used to instruct the large language model to refer to the first data and reference data of the target object and process the input data into speech recognition text; the text acquisition module is used to acquire the target recognition text output by the large language model in response to the first prompt instruction.

[0007] To address the aforementioned technical problems, a third aspect of this application provides a computer device comprising at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor executes the program instructions to implement the speech recognition method of the first aspect.

[0008] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the speech recognition method of the first aspect described above.

[0009] The above scheme, in response to the speech signal to be recognized, performs feature encoding based on the acoustic features of the speech signal to obtain the first voiceprint feature and the first semantic feature of the speech signal. It then extracts the second voiceprint feature of each candidate object from the memory data of each candidate object, where the memory data includes first data related to the object profile and second data related to the interactive dialogue. Based on the similarity between the first voiceprint feature and each of the second voiceprint features, candidate objects are selected as target objects. Feature transformation is then performed based on the first semantic feature to obtain input data adapted to the large language model. Reference data is then retrieved from the second data of the target object based on the input data. A first prompt instruction is constructed based on the first data, reference data, and input data of the target object. This first prompt instruction instructs the large language model to refer to the first data and reference data of the target object, processing the input data into speech recognition text. Finally, the target recognition text output by the large language model in response to the first prompt instruction is obtained. This process involves extracting the first voiceprint feature of the speech signal and comparing it with... The second voiceprint features of each candidate are matched to select the candidate as the target. The subsequent speech recognition process is then guided by the target's memory data. This allows for the most accurate selection of the target voice signal from among numerous candidates based on the voiceprint features in the memory data, improving the accuracy of subsequent speech recognition. Furthermore, since the memory data is divided into first data related to the object profile and second data related to the interactive dialogue (first data is relatively stable, while second data is more dynamic), combining both types of data to assist the subsequent speech recognition process can maximize accuracy with the aid of both static and dynamic memory information. Additionally, after selecting a candidate as the target, further retrieval within the target's second data helps extract relevant data from the more dynamic memory information of the target as reference data. Combining this with the first data for speech recognition improves the adaptability of speech recognition to dynamic user changes. Therefore, this approach enhances both the accuracy and dynamic adaptability of speech recognition. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating an embodiment of the speech recognition method of this application; Figure 2 This is a schematic diagram of a process of an embodiment of the speech recognition method of this application; Figure 3 This is a schematic diagram of the framework of an embodiment of the speech recognition device of this application; Figure 4 This is a schematic diagram of the framework of an embodiment of the computer device of this application; Figure 5 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0011] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0012] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0013] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the slash " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper indicates two or more objects.

[0014] Please refer to the following: Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating an embodiment of the speech recognition method of this application. Figure 2 This is a schematic diagram illustrating an embodiment of the speech recognition method of this application. It should be noted that this embodiment can be executed by a computer device. The specific structure and type of the computer device can be found in subsequent related embodiments of this application, and will not be elaborated upon here. Specifically, this embodiment may include the following steps: Step S11: In response to the speech signal to be identified, feature encoding is performed based on the acoustic features of the speech signal to obtain the first voiceprint feature and the first semantic feature of the speech signal, and the second voiceprint feature of each candidate object is extracted from the memory data of each candidate object respectively.

[0015] In this embodiment of the disclosure, the memory data may include first data related to the object profile and second data related to the interactive dialogue. It should be noted that when it is necessary to read and / or save memory information, relevant prompt messages may be output in advance to inform the user, and the memory information may be read and / or saved only with the user's authorization.

[0016] In one implementation scenario, before extracting acoustic features from the speech signal, the speech signal can be pre-processed, such as noise reduction and endpoint detection, etc., which is not limited here.

[0017] In one implementation scenario, the acoustic features of a speech signal may include, but are not limited to, MFCC (Mel-Frequency Cepstral Coefficients), FBank (Filter Bank), etc. The specific type of acoustic feature is not limited here.

[0018] In an implementation scenario, such as Figure 2 As shown, the acoustic features of a speech signal can be extracted by the speech input module. For example, the speech input module can first perform front-end processing on the speech signal, such as noise reduction and endpoint detection, and then extract the acoustic features.

[0019] In one implementation scenario, as a possible approach, after obtaining the acoustic features of the speech signal, a voiceprint encoder can be used to encode the acoustic features of the speech signal to obtain the first voiceprint features. A semantic encoder can then be used to encode the acoustic features of the speech signal to obtain the first semantic features. For example, voiceprint encoders may include, but are not limited to, WeSpeaker, x-vector, i-vector, etc., and the specific type of voiceprint encoding is not limited here. Similarly, semantic encoders may include, but are not limited to, Whisper, WavLM, HuBERT (Hidden-Unit Bidirectional Encoder Representations from Transformers), etc., and the specific type of semantic encoder is not limited here. It should be noted that semantic encoders can extract features at different levels, such as, but are not limited to, low-level acoustic features, mid-level phoneme / syllable features, and high-level semantic features, to achieve multi-scale feature extraction.

[0020] In another implementation scenario, as another possible approach, after obtaining the acoustic features of the speech signal, voiceprint feature encoding can be performed based on the acoustic features of the speech signal to obtain the first voiceprint feature of the speech signal. Then, semantic feature encoding can be performed based on the acoustic features of the speech signal and the first voiceprint feature to obtain the first semantic feature. In the above method, the first semantic feature is extracted from the acoustic features of the speech signal with the assistance of the first voiceprint feature. This helps to dynamically adjust the encoding parameters according to the first voiceprint feature during the semantic feature encoding process, thereby improving the accuracy of the first semantic feature.

[0021] In a specific implementation scenario, the method for extracting the first voiceprint feature can be found in the aforementioned voiceprint encoder, and will not be repeated here.

[0022] In a specific implementation scenario, the method for extracting the first semantic feature can be found in the aforementioned semantic encoder, and will not be repeated here. It should be noted that, in order to dynamically adjust the encoding parameters based on the first voiceprint feature, the semantic encoder can also use parameter-efficient methods such as adapters or low-rank adaptation (LoRA). For details, please refer to the technical details of adapters or low-rank adaptation, which will not be repeated here.

[0023] In an implementation scenario, such as Figure 2 As shown, the first voiceprint feature and the first semantic feature can be obtained by encoding the acoustic features of the speech signal by the encoder module. For example, the encoder module may include a voiceprint encoder and a semantic encoder. The specific types of the voiceprint encoder and the semantic encoder can be found in the foregoing descriptions and will not be repeated here.

[0024] In one implementation scenario, the primary data of the candidate object may include, but is not limited to, the candidate object's basic information and personalized vocabulary. For example, the basic information may include, but is not limited to, user identification information, language style statistics, and basic profile information; the specific content of the basic information is not limited here. Specifically, user identification information may include, but is not limited to, object identifiers (used to uniquely identify the object), secondary voiceprint features (used for authentication), and object names (e.g., nicknames, names, etc.); language style statistics may include, but is not limited to, statistics on commonly used sentence patterns (e.g., sentence structures preferred by the candidate object), vocabulary selection preferences (e.g., synonyms and near-synonyms tended by the candidate object), speech rate and pause features (e.g., prosodic features helpful for speech processing); and basic profile information may include, but is not limited to, demographic information (e.g., age, gender, occupation, etc., can be obtained with authorization), object interest tags (e.g., the candidate object's interests can be extracted from historical conversations), and usage scenario preferences (e.g., the candidate object prefers to use the voice assistant in different scenarios such as driving, at home, or in the office). In addition, a personalized vocabulary database may include, but is not limited to: proper nouns (e.g., names of people, places, and organizations frequently mentioned by candidates), technical terms (e.g., professional vocabulary in the candidate's field), idioms (e.g., preferred expressions, abbreviations, and catchphrases of candidates), and vocabulary correction records (e.g., word pairs previously misidentified by the system but corrected by the user). It should be noted that the above examples are merely one possible case study for First Data; other possible scenarios are not limited here, nor will they be listed individually.

[0025] In one implementation scenario, the second data may include, but is not limited to, several rounds of historical dialogues and knowledge graphs built upon these dialogues, contextual state memories, recognition and error correction records, satisfaction feedback, and explicit preferences. The several rounds of historical dialogues can be stored in a structured manner, such as being organized by conversation and recording user input and system response for each round. The knowledge graph can be constructed based on entities extracted from the dialogues and the relationships between entities, and the dialogue history can also mark important events, dates, and agreements explicitly mentioned by the user. Contextual state memories can include the current dialogue state (e.g., key information confirmed in the current conversation), reference resolution history (e.g., recording reference relationships in previous dialogues to help understand the expression of references such as "it" and "this"), and task state (e.g., for task-oriented dialogues, recording the task completion progress). Recognition and error correction records can include user corrections to the speech recognition results (i.e., recognition and error correction records mainly correct speech recognition results, unlike the aforementioned vocabulary error correction records which mainly target personalized vocabulary). Satisfaction feedback can include user ratings or evaluations of the interaction. Explicit preferences can include user-expressed preference settings. It should be noted that the above example is only one possible example of the second data. Other possible situations are not limited here, nor will they be listed one by one.

[0026] In one implementation scenario, as a possible example, the first data can be stored in a structured manner, such as using a relational database or a graph database, to facilitate fast access and updates. As another possible example, the second data can adopt a hybrid storage strategy, such as storing the original dialogue text in a document database, while storing the extracted entity, relationship, and other structured information in a graph database for efficient retrieval.

[0027] In one implementation scenario, please refer to the following: Figure 2 The memory management system can include two levels of memory (Level 1 and Level 2), which can be used to manage user memory storage, updates, and retrieval. Level 1 memory can be considered as user profile memory, including but not limited to voiceprints, personalized vocabulary, and language style. For example, Level 1 memory can correspond to the aforementioned first data, and details can be found in the relevant description of the aforementioned first data. Level 2 memory can be considered as content-driven memory, including but not limited to dialogue history, entity relationships, and contextual states. For example, Level 2 memory can correspond to the aforementioned second data, and details can be found in the relevant description of the aforementioned second data. Furthermore, the method of constructing memory data can be found in the subsequent descriptions in the embodiments of this disclosure, and will not be repeated here.

[0028] Step S12: Based on the similarity between the first voiceprint feature and each of the second voiceprint features, select candidate objects as target objects, and perform feature transformation based on the first semantic feature to obtain input data adapted to the large language model.

[0029] In one implementation scenario, after obtaining the first voiceprint feature and the second voiceprint features of each candidate object, the similarity (e.g., cosine similarity) between the first voiceprint feature and the second voiceprint features of each candidate object can be obtained, and the candidate object can be selected as the target object to which the speech signal belongs based on the similarity. For example, the candidate object to which the second voiceprint feature with the highest similarity can be selected can be the target object.

[0030] In one implementation scenario, as a possible approach, the input data obtained after feature transformation of the first semantic feature may include a second semantic feature whose feature dimensions are adapted to the large language model. That is, the first semantic feature can be dimension-mapped based on the feature dimensions supported by the input layer of the large language model, thereby converting the feature dimensions of the first semantic feature into the feature dimensions supported by the aforementioned input layer to obtain the second semantic feature. For example, a projection layer or a cross-modal attention mechanism can be used to implement dimension mapping; the specific process can be found in the technical details of projection layers or cross-modal attention mechanisms, which will not be elaborated here.

[0031] In another implementation scenario, as another possible approach, to maximize the explicitness of the input data and make it more closely resemble the embedding of a large language model, the input data obtained from the feature transformation of the first semantic features may include the initial recognition text obtained by preliminary decoding of the first semantic features. That is, preliminary decoding can be performed based on the first semantic features to obtain the initial recognition text of the speech signal. For example, the initial recognition text can be obtained by preliminary decoding of the first semantic features based on a CTC (Connectionist Temporal Classification) decoder.

[0032] In another implementation scenario, as yet another possible approach, the input data obtained from the feature transformation of the first semantic feature may include a second semantic feature whose feature dimension is adapted to a large language model, and the initial recognition text obtained from the preliminary decoding of the first semantic feature. It should be noted that the specific methods for obtaining these two features can be found in the aforementioned descriptions, and will not be repeated here.

[0033] In one implementation scenario, please refer to the following: Figure 2 The input data can be obtained by transforming the first semantic features using the adapter module. It should be noted that the adapter module can act as a bridge between the aforementioned encoder module and the subsequent large language model module.

[0034] Step S13: Based on the input data, retrieve the reference data from the second data of the target object.

[0035] In one implementation scenario, as a possible implementation method, as mentioned above, the input data may include second semantic features whose feature dimensions are adapted to the large language model. At this time, the second semantic features can be used to retrieve several first corpora and the first confidence of each first corpus. Then, based on the correlation between the first corpora, several corpus sets are obtained, and each corpus set may contain at least one first corpus. Thus, the first corpora in the same corpus set can be fused to obtain second corpora and the second confidence of the second corpora. Then, based on the second confidence of each second corpus, at least one second corpus can be selected and included in the reference data.

[0036] In a specific implementation scenario, when performing retrieval based on the second semantic feature, vector retrieval can be performed across several rounds of historical dialogues based on the second semantic feature to obtain semantically similar historical dialogues as the first corpus, and the semantic similarity is used as the first confidence level. For example, dense vector retrieval can be used to find historical dialogues that are semantically similar to the second semantic feature and use them as the first corpus.

[0037] In a specific implementation scenario, after retrieving several first corpora and their first confidence scores, clustering can be performed on these first corpora to obtain several corpus sets. It should be noted that the purpose of clustering the first corpora is to group first corpora that are related to each other into the same corpus set based on the degree of correlation between them. On this basis, the first corpora located in the same corpus set can be merged (e.g., re-integrated into a new corpus) to obtain second corpora, and the second confidence score of the second corpus can be obtained by fusing the first confidence scores of each first corpus in the corresponding corpus set (e.g., averaging, weighting, etc.). Furthermore, the specific process of clustering several first corpora can be found in methods such as hierarchical clustering, spectral clustering, K-Means, and HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise), etc. The specific clustering algorithms are not limited here, nor will they be listed individually.

[0038] In a specific implementation scenario, after obtaining several second corpora and their second confidence levels, they can be sorted according to their second confidence levels, and the Top-K second corpora can be selected and included in the reference data. It should be noted that the specific value of K can be set according to the actual application requirements. For example, when a larger reference range is required, the specific value of K can be larger; conversely, when the reference range requirement is smaller, the specific value of K can be smaller. For example, K can take values ​​of 5, 6, 7, etc. The specific value of K is not limited here, and further examples will not be provided.

[0039] In another implementation scenario, as another possible implementation method, as mentioned above, the input data may include the initial recognition text obtained by preliminary decoding of the first semantic features. In this case, a search can be performed on the second data based on the initial recognition text to obtain several first corpora and a first confidence level for each first corpus. Then, based on the correlation between the first corpora, several corpus sets are obtained, and each corpus set may contain at least one first corpus. Therefore, the first corpora in the same corpus set can be fused to obtain second corpora and a second confidence level for each second corpus. Finally, based on the second confidence levels of each second corpus, at least one second corpus can be selected and included in the reference data. It should be noted that parts similar to the aforementioned implementation method can be directly referred to in the aforementioned implementation method and will not be repeated here.

[0040] In a specific implementation scenario, unlike the aforementioned implementation methods, this implementation uses the initial identified text for retrieval in the second data. Specifically, keyword retrieval can be performed based on the initial identified text through several rounds of historical dialogues to obtain historical dialogues with matching keywords as the first corpus, and the keyword matching degree is used as the first confidence level. For example, algorithms such as BM25 (Best Matching 25) can be used to perform precise matching through keywords to obtain historical dialogues related to the initial identified text from the second data as the first corpus.

[0041] In a specific implementation scenario, unlike the aforementioned implementation methods, this implementation uses the initial identified text for retrieval in the second data. Specifically, it can perform association reasoning based on a knowledge graph to obtain historical dialogues that are content-related to the initial identified text as the first corpus, and the content relevance is used as the first confidence level. For example, if there is an entity in the initial identified text (referred to as entity A for ease of distinction), and according to the knowledge graph, entity B is clearly related to entity A, and entity B comes from a certain historical dialogue, then this historical dialogue can be used as the first corpus.

[0042] In another implementation scenario, as another possible implementation method, as mentioned above, the input data may include second semantic features with feature dimensions adapted to the large language model and initial recognition text obtained by preliminary decoding of the first semantic features. In this case, retrieval can be performed on the second data based on the second semantic features and the initial recognition text to obtain several first corpora and the first confidence level of each first corpus. Then, based on the correlation between the various first corpora, several corpus sets are obtained, and each corpus set may contain at least one first corpus. Therefore, first corpora from the same corpus set can be fused to obtain second corpora and their second confidence levels. Finally, based on the second confidence levels of each second corpus, at least one second corpus can be selected and included in the reference data. It should be noted that parts similar to the aforementioned implementation methods can be directly referred to in the aforementioned implementation methods and will not be repeated here.

[0043] In a specific implementation scenario, unlike the aforementioned implementation method, this implementation method uses both the second semantic feature and the initial recognition text for retrieval. Specifically, vector retrieval can be performed in several rounds of historical dialogues based on the second semantic feature to obtain semantically similar historical dialogues as the first corpus, and the semantic similarity is used as the first confidence level. For details, please refer to the aforementioned relevant descriptions, which will not be repeated here.

[0044] In a specific implementation scenario, unlike the aforementioned implementation method, this implementation method uses both the second semantic feature and the initial identification text for retrieval. Specifically, keyword retrieval can be performed in several rounds of historical dialogues based on the initial identification text, and the historical dialogues with keyword matching can be used as the first corpus, with the keyword matching degree as the first confidence level. For details, please refer to the aforementioned relevant descriptions, which will not be repeated here.

[0045] In a specific implementation scenario, unlike the aforementioned implementation methods, this implementation method uses both the second semantic features and the initial identification text for retrieval. Specifically, it can perform association reasoning based on a knowledge graph to obtain historical dialogues that are content-related to the initial identification text as the first corpus, and the content association degree as the first confidence level. For details, please refer to the aforementioned relevant descriptions, which will not be repeated here.

[0046] It should be noted that the above-mentioned implementation methods are merely a few possible examples of retrieving reference data from the second data based on the input data in practical applications. Other possible implementation methods are not limited here, nor will they be listed one by one.

[0047] Step S14: Construct a first prompt instruction based on the first data, reference data, and input data of the target object.

[0048] In this embodiment of the disclosure, the first prompt instruction can be used to instruct the large language model to reference the first data and reference data of the target object, and process the input data into speech recognition text. It should be noted that if the input data includes a second semantic feature when performing a retrieval using the second data, then the input data also includes the second semantic feature when constructing the first prompt instruction. Similarly, if the input data includes initial recognition text when performing a retrieval using the second data, then the input data also includes the initial recognition text when constructing the first prompt instruction; similarly, if the input data includes both the second semantic feature and the initial recognition text when performing a retrieval using the second data, then the input data also includes both the second semantic feature and the initial recognition text when constructing the first prompt instruction.

[0049] In one implementation scenario, when constructing the first prompt instruction, the target object's initial data, reference data, and input data can be represented in a structured manner. For example, this can be organized into specific formats such as JSON (JavaScript Object Notation), XML (Extensible Markup Language), or tagged text. Furthermore, as a possible example, location or type markers can be added to data from different sources in the first prompt instruction to help large language models distinguish data types. As another possible example, reliability annotations can be applied to different data in the first prompt instruction to help large language models adjust their trust levels when referencing different data for speech recognition.

[0050] In another implementation scenario, besides constructing the first prompt instruction by combining the target object's first data, reference data, and input data, a search can also be performed on the target object's personalized vocabulary library based on the input data. Personalized words related to the input data can then be included in the reference data, which also includes the target object's basic information. In other words, before constructing the first prompt instruction, a search can be performed on the target object's personalized vocabulary library based on the input data to obtain personalized words related to the input data; this part will also be a component of the first prompt instruction. In addition, the target object's basic information will also be a component of the first prompt instruction. It should be noted that the specific meanings of the personalized vocabulary library and basic information can be found in the aforementioned description of the first data, and the specific process of searching the personalized vocabulary library using input data can be found in the aforementioned description of searching the second data using input data; these will not be repeated here.

[0051] Step S15: Obtain the target recognition text output by the large language model in response to the first prompt command.

[0052] In one implementation scenario, during the processing of the large language model in response to the first prompt command, recognition can be performed based on the input data to obtain the first recognized text of the speech signal. Then, based on the first data and reference data, the first recognized text is corrected to obtain the second recognized text. Finally, the second recognized text can be polished and a consistency check is performed to obtain the target recognized text. The above method, which executes a progressive decoding strategy of recognition, correction, polishing, and consistency checking in sequence, can make full use of memory information and helps to improve the recognition quality of the target recognized text.

[0053] In a specific implementation scenario, when the input data includes a second semantic feature, the large language model can perform a decoding operation based on the second semantic feature to obtain the first recognized text. For example, the large language model can perform a decoding operation on the second semantic feature through its own decoder based on model architectures such as Transformer to obtain the first recognized text. For specific details on the process, please refer to technical details such as decoders based on Transformer, which will not be elaborated here.

[0054] In a specific implementation scenario, when the input data includes initial recognition text, the large language model can further recognize the speech signal based on the prompts of the initial recognition text, obtaining the first recognized text of the speech signal. For example, the large language model can correct obviously abnormal word expressions in the initial recognition text based on its general understanding ability, thus obtaining the first recognized text of the speech signal. For instance, in a practical application, if the speech signal "I want to go to Nanjing" is initially recognized as "I want to go to Blue Whale" due to confusion between "n" and "l", the large language model can correct this error based on its own general understanding ability, obtaining the first recognized text "I want to go to Nanjing". Of course, the above example is only one possible example in practical applications, and other possible situations are not limited here, nor will they be listed one by one.

[0055] In a specific implementation scenario, when the input data includes a second semantic feature and an initial recognition text, the large language model can perform a decoding operation based on the second semantic feature to obtain the first recognition text. Furthermore, with the prompting from the initial recognition text, the large language model can further perform recognition to obtain another first recognition text. Based on this, the two first recognition texts can be combined to obtain the final first recognition text of the speech signal. For example, the large language model can also obtain a confidence level for the first recognition text by performing a decoding operation, and can obtain another confidence level for the first recognition text by performing further recognition. The text with the higher confidence level can then be selected as the final first recognition text.

[0056] In a specific implementation scenario, after obtaining the first recognized text of the speech signal, the first recognized text can be corrected based on the first data and reference data. As one possible example, relevant words in the first recognized text can be corrected based on a personalized vocabulary library in the first data. For example, based on professional terms in the personalized vocabulary library, words in the corresponding field in the first recognized text can be corrected (e.g., based on the professional term "vancomycin" in the personalized vocabulary library, the first recognized text "It is recommended to use stubborn mesotherapy" is corrected to "It is recommended to use vancomycin"); another example is based on word correction records in the personalized vocabulary library, relevant words in the first recognized text can be corrected (e.g., based on the word correction record "Artificial Intelligence → Artificial Intelligence" in the personalized vocabulary library, the first recognized text "Artificial Intelligence technology is developing rapidly" is corrected to "Artificial Intelligence technology is developing rapidly"). As another possible example, relevant words in the first recognized text can be corrected based on historical dialogues in the reference data (e.g., based on the historical dialogue "User feedback is mainly stuck in the semantic understanding stage," the first recognized text "Then we need to label the raincoat" is corrected to "Then we need to label the semantics"). As another possible example, relevant words in the first identified text can be corrected based on the recognition and correction records in the reference data (e.g., changing "data clearing" to "data cleaning" based on the recognition and correction record, thus correcting the first identified text "Xiao Wang is responsible for data clearing work" to "Xiao Wang is responsible for data cleaning work"). Of course, the above examples are merely a few possible examples of correcting the first identified text based on the first and reference data; other possible scenarios are not limited here, nor will they be listed one by one. Furthermore, although the above examples do not demonstrate a large language model, in practical applications, a large language model can correct the first identified text with the assistance of the first and reference data based on its general understanding capabilities.

[0057] In a specific implementation scenario, after correcting the first identified text to obtain the second identified text, polishing and consistency checks can be performed on the second identified text to obtain the target identified text. For example, the second identified text can be polished and consistency checked based on the general understanding capability of a large language model to obtain the target identified text.

[0058] In a specific implementation scenario, in addition to outputting the target recognition text, the large language model can also output the confidence score of the target recognition text (e.g., overall confidence and word-by-word confidence), memory utilization annotation (e.g., marking which parts utilize memory information), etc. Here, we will not limit the other possible outputs of the large language model, nor will we list them one by one.

[0059] In one implementation scenario, after acquiring the target recognition text output by the large language model in response to the first prompt command, the target session round to which the target recognition text belongs can be identified. This target session round can be either the current round or a new round. Information extraction is then performed based on the target recognition text to obtain the memory information of the target session round. Based on this memory information, the memory data of the target object can be updated. Furthermore, in response to preset trigger conditions, the updated memory data of the target object can be organized and compressed. This method, through a complete process including dialogue boundary detection, information extraction, conflict resolution, and memory compression, enables the automatic maintenance of the memory system.

[0060] In a specific implementation scenario, to identify the target conversation turn to which the target text belongs (i.e., whether it belongs to the current conversation or a new one), the first semantic feature of the speech signal can be selected as the semantic feature for identifying the target conversation turn. As a possible example, the second semantic feature can also be referenced to obtain the semantic feature for identifying the target conversation turn. For example, only the first semantic feature, only the second semantic feature, or both can be used to assist in identifying the target conversation turn. In addition, time features can be obtained based on the time-related feature information of the speech signal, and the feature information can at least include the time interval of the interactive dialogue. Based on this, classification prediction can be performed based on the acoustic features of the speech signal (e.g., acoustic cues such as silence segments and energy changes), semantic features (e.g., the coherence of dialogue semantics reflected in the semantic features), and time features to obtain the target conversation turn to which the target text belongs. For example, a lightweight classification model can be used to classify and predict the acoustic features, semantic features, and temporal features mentioned above (e.g., inputting the fused features of the acoustic features, semantic features, and temporal features into a lightweight classification model for classification and prediction) to obtain the probability values ​​of the target text belonging to the current round of the session and the new round of the session, so as to determine the target session round to which the target text belongs.

[0061] In a specific implementation scenario, as mentioned earlier, the first data may include vocabulary correction records, language style statistics, and usage scenario preferences; the second data may include several rounds of historical dialogue, a knowledge graph constructed based on the historical dialogue, and recognition and error correction records. During information extraction, a second prompt instruction can be constructed based on the target recognition text. This second prompt instruction can instruct the large language model to extract memory information from the target recognition text to update at least one of the following: vocabulary correction records, language style statistics, usage scenario preferences, several rounds of historical dialogue, knowledge graph, and recognition and error correction records. Based on this, the memory information of the target conversation round can be obtained from the output of the large language model in response to the second prompt instruction. As a possible example, the large language model can perform entity extraction on the target recognition text in response to the second prompt instruction. For example, it can use Named Entity Recognition (NER) to extract entities such as names of people, places, organizations, and times, and then identify the relationships between entities, such as "Zhang San—works at—a certain company." Based on this, memory information for updating the knowledge graph can be extracted. As another possible example, the large language model can extract facts from the target text in response to a second prompt, such as extracting factual information stated by the user (e.g., "I'm going to City A tomorrow"), thereby extracting memory information for updating language style statistics. As yet another possible example, the large language model can extract preferences from the target text in response to a second prompt, i.e., analyzing user preferences, such as "I don't like noisy places," thereby extracting memory information for updating usage scenario preferences. As yet another possible example, the large language model can perform error correction learning from the target text in response to a second prompt; i.e., when the user corrects the recognition result, it can extract "error-correct" word pairs, thereby extracting memory information for updating the recognition correction record. As yet another possible example, the large language model can determine the memory information for updating several rounds of historical dialogue based on the target conversation rounds stated in the target text; that is, based on the target conversation rounds, it can determine whether to use the target text as memory information for updating the current round of dialogue in several rounds of historical dialogue or as memory information for updating the next round of dialogue in several rounds of historical dialogue. It should be noted that the large language model used to extract memory information and the large language model used to obtain target recognition text can be the same large language model; or they can be different large language models. For example, the large language model used to extract memory information can be a lightweight dedicated model, a large language model obtained by distilling a large model, etc. There is no limitation on the large language model used to extract memory information here.

[0062] In a specific implementation scenario, in order to update memory data based on memory information, it is possible to detect whether there is a contradiction between the memory information of the target session round and the existing memory information in the memory data of the target object. In response to the existence of a contradiction, it is possible to determine whether to use the memory information of the target session round to replace the existing memory information in the memory data of the target object that contradicts it, based on the information confirmation flag, the information weight value, and the information time interval. The information confirmation flag can represent whether the memory information has been confirmed by the user, the information weight value can represent the recognition confidence of the recognition text from which the memory information originates, and the information time interval can represent the time interval between the memory information of the target session round and the existing memory information that contradicts it. For example, if there is a conflict between the memory information of the target session and the existing memory information in the target object's memory data, we can first check whether the two conflicting memory information have information confirmation marks. If one of the memory information has an information confirmation mark, the memory information with the information confirmation mark can be used. If neither memory information has an information confirmation mark, we can continue to check the information weight values ​​of the two conflicting memory information. If the information weight value of one memory information is greater than that of the other memory information, the memory information with the greater information weight value can be used. If the information weight values ​​of the two memory information are equal, we can continue to check the information time interval between the two conflicting memory information. If the information time interval exceeds the time interval threshold, the memory information with the newer time can be used. If the information time interval does not exceed the time interval threshold, we can actively ask the target object to confirm which memory information to use. In this way, the information conflict can be resolved and the memory information can be integrated and updated.

[0063] In a specific implementation scenario, as one possible example, to organize and compress the updated memory data of the target object, in response to the fact that the usage frequency of memory information in the updated memory data is lower than a frequency threshold, the information weight value of the memory information can be reduced, or the memory information can be summarized and merged. This achieves attenuation of unimportant information. It should be noted that the specific meaning of the information weight value can be found in the aforementioned descriptions, and will not be repeated here. As another possible example, to organize and compress the updated memory data of the target object, in response to the existence of multiple pieces of memory information with similarity higher than a similarity threshold in the updated memory data, these multiple pieces of memory information with similarity higher than the similarity threshold can be aggregated to obtain new memory information. This allows similar memory information to be aggregated to generate a more generalized representation. As another possible example, to organize and compress the memory data after the target object is updated, the memory data, especially the secondary data, can be stored in a hierarchical manner. For instance, it can be set to different levels such as short-term, medium-term, and long-term. Different storage strategies and retrieval priorities can be adopted for different levels of memory information. For example, short-term memory can be stored in the storage medium with the fastest read speed, medium-term memory in the storage medium with a moderate read speed, and long-term memory in the storage medium with the slowest read speed. Alternatively, short-term memory can have the highest retrieval priority, medium-term memory a moderate retrieval priority, and long-term memory the lowest retrieval priority. Of course, the above example is only one possible example in practical applications; other possible scenarios are not limited here, nor will they be listed in detail.

[0064] In one implementation scenario, a multi-stage training strategy can be employed to ensure, as far as possible, that... Figure 2The modules shown work collaboratively. The multi-stage training strategy can sequentially include a pre-training stage, a memory module training stage, and an end-to-end joint training stage. In the pre-training stage, large-scale speech recognition data can be used to pre-train the encoder and large language model (or, open-source encoders and large language models can be directly used). It should be noted that this stage does not involve personalized memory; its purpose is to establish a powerful general speech understanding capability. In the memory module training stage, a speaker recognition model can be trained using multi-speaker datasets, an information extraction model (e.g., for extracting entities, relationships, etc.) can be trained using labeled dialogue data, and a ranking retrieval model can be trained using manually constructed "relevant-irrelevant" dialogue pairs. In the end-to-end joint training stage, a complete training pipeline including a memory component can be constructed. Personalized dialogue data containing, but not limited to, user-customized vocabulary and expressions, context-related statements that require historical dialogue for correct understanding, and user corrections of recognition errors can be used. A multi-task loss function is used to perform end-to-end joint training of the overall model framework to simultaneously optimize speech recognition accuracy, memory retrieval relevance, and memory update accuracy.

[0065] In one implementation scenario, please continue to refer to [the relevant documentation]. Figure 2 ,like Figure 2 As shown, the left side represents the speech recognition backbone, consisting of an encoder module, an adapter module, and a large language model module. From bottom to top, these are the speech input module, encoder module, adapter module, large language model module, and output module. The right side represents the memory management system (including Level 1 user profile memory and Level 2 content-driven memory) and sub-modules such as memory construction and updating, and memory retrieval engine. The left and right parts are connected through four types of interactions: the adapter module generates retrieval queries based on the initial speech encoding (e.g., the aforementioned second semantic feature, initial recognized text); the memory retrieval engine injects relevant memories (e.g., the aforementioned reference data) as context into the large language model module; the speech recognition result (e.g., the aforementioned target recognized text) is automatically written back to the memory and update module; and the voiceprint features extracted by the encoder module (e.g., the aforementioned first voiceprint feature) trigger the Level 1 memory loading of the corresponding object (e.g., the aforementioned target object) (i.e., triggering the first data of the aforementioned target object). In other words, Figure 2The overall process can include: after the speech signal to be recognized is processed by the front end of the speech input module, it can be encoded by the encoder module to obtain the first semantic feature. The adapter module can adapt the first semantic feature to obtain input data adapted to the large language model and trigger memory retrieval. The retrieved relevant memory can be provided as context to the large language model module. The large language model module can combine multiple data to generate the final target recognition text, and can also extract memory information and update the memory management system accordingly. For details, please refer to the foregoing detailed description, which will not be repeated here. In addition, as a possible example, the above-mentioned memory management system can prioritize local storage on the user device. Memory retrieval and utilization can also be completed locally on the user device. For operations that need to be completed in the cloud, differential privacy technology can be used to protect user information. Users can request to delete specific memories or all memories, and the system can display the memory information stored in the system, allowing users to view, edit, and perform other operations on this memory information.

[0066] The above scheme, in response to the speech signal to be recognized, performs feature encoding based on the acoustic features of the speech signal to obtain the first voiceprint feature and the first semantic feature of the speech signal. It then extracts the second voiceprint feature of each candidate object from the memory data of each candidate object, where the memory data includes first data related to the object profile and second data related to the interactive dialogue. Based on the similarity between the first voiceprint feature and each of the second voiceprint features, candidate objects are selected as target objects. Feature transformation is then performed based on the first semantic feature to obtain input data adapted to the large language model. Reference data is then retrieved from the second data of the target object based on the input data. A first prompt instruction is constructed based on the first data, reference data, and input data of the target object. This first prompt instruction instructs the large language model to refer to the first data and reference data of the target object, processing the input data into speech recognition text. Finally, the target recognition text output by the large language model in response to the first prompt instruction is obtained. This process involves extracting the first voiceprint feature of the speech signal and comparing it with... The second voiceprint features of each candidate are matched to select the candidate as the target. The subsequent speech recognition process is then guided by the target's memory data. This allows for the most accurate selection of the target voice signal from among numerous candidates based on the voiceprint features in the memory data, improving the accuracy of subsequent speech recognition. Furthermore, since the memory data is divided into first data related to the object profile and second data related to the interactive dialogue (first data is relatively stable, while second data is more dynamic), combining both types of data to assist the subsequent speech recognition process can maximize accuracy with the aid of both static and dynamic memory information. Additionally, after selecting a candidate as the target, further retrieval within the target's second data helps extract relevant data from the more dynamic memory information of the target as reference data. Combining this with the first data for speech recognition improves the adaptability of speech recognition to dynamic user changes. Therefore, this approach enhances both the accuracy and dynamic adaptability of speech recognition.

[0067] Please see Figure 3 , Figure 3This is a schematic diagram of the framework of an embodiment of the speech recognition device of this application. The speech recognition device 30 includes: an encoding extraction module 31, a selection and conversion module 32, a memory retrieval module 33, an instruction construction module 34, and a text acquisition module 35. The encoding extraction module 31 is used to encode features based on the acoustic features of the speech signal in response to the speech signal to be recognized, to obtain a first voiceprint feature and a first semantic feature of the speech signal, and to extract a second voiceprint feature of each candidate object from the memory data of each candidate object; wherein, the memory data includes first data related to the object profile and second data related to the interactive dialogue; the selection and conversion module 32 is used to perform selection and conversion between the first voiceprint feature and each second voiceprint feature. The similarity is used to select candidate objects as target objects, and feature transformation is performed based on the first semantic features to obtain input data adapted to the large language model; the memory retrieval module 33 is used to retrieve reference data in the second data of the target object based on the input data; the instruction construction module 34 is used to construct a first prompt instruction based on the first data of the target object, the reference data and the input data; wherein, the first prompt instruction is used to instruct the large language model to refer to the first data and the reference data of the target object and process the input data into speech recognition text; the text acquisition module 35 is used to acquire the target recognition text output by the large language model in response to the first prompt instruction.

[0068] In the above scheme, the speech recognition device 30 responds to the speech signal to be recognized, performs feature encoding based on the acoustic features of the speech signal to obtain the first voiceprint feature and the first semantic feature of the speech signal, and extracts the second voiceprint feature of each candidate object from the memory data of each candidate object. The memory data includes first data related to the object profile and second data related to the interactive dialogue. Then, based on the similarity between the first voiceprint feature and each second voiceprint feature, the candidate object is selected as the target object, and feature transformation is performed based on the first semantic feature to obtain input data adapted to the large language model. Based on the input data, reference data is obtained by searching the second data of the target object. Based on the first data, reference data and input data of the target object, a first prompt instruction is constructed. The first prompt instruction is used to instruct the large language model to refer to the first data and reference data of the target object, process the input data into speech recognition text, and then obtain the target recognition text output by the large language model in response to the first prompt instruction. On the one hand, by extracting the first voiceprint of the speech signal... The system matches features with the second voiceprint features of each candidate object to select the candidate object as the target object. Then, it uses the target object's memory data to guide subsequent speech recognition processes. This allows for the most accurate selection of the target object from numerous candidates based on the voiceprint features in the memory data, improving the accuracy of subsequent speech recognition processes. Furthermore, since the memory data is divided into first data related to the object profile and second data related to the interactive dialogue (the first data is relatively stable, while the second data is more dynamic), combining both types of data to assist subsequent speech recognition processes can maximize accuracy with the aid of both static and dynamic memory information. Additionally, after selecting a candidate object as the target object, further searching within the target object's second data helps extract relevant data from the more dynamic memory information of the target object as reference data. Combining this with the first data for speech recognition improves the adaptability of speech recognition to dynamic user changes. Therefore, it enhances both the accuracy and dynamic adaptability of speech recognition.

[0069] In some disclosed embodiments, the input data includes second semantic features whose feature dimensions are adapted to a large language model and initial recognition text obtained by preliminary decoding of the first semantic features. The memory retrieval module 33 includes a data retrieval submodule, used to retrieve data from the second data based on the second semantic features and the initial recognition text, to obtain several first corpora and a first confidence level for each first corpus. The memory retrieval module 33 includes an association set submodule, used to obtain several corpus sets based on the association between the first corpora. The corpus sets contain at least one first corpus. The memory retrieval module 33 includes a corpus fusion submodule, used to fuse first corpora from the same corpus set to obtain second corpora and a second confidence level for the second corpora. The memory retrieval module 33 includes a corpus selection submodule, used to select at least one second corpus to be included in the reference data based on the second confidence level of each second corpus.

[0070] In some disclosed embodiments, the second data includes several rounds of historical dialogues and a knowledge graph constructed based on the historical dialogues. The data retrieval submodule includes a vector retrieval unit, used to perform vector retrieval in several rounds of historical dialogues based on the second semantic features, to obtain semantically similar historical dialogues as the first corpus, and the semantic similarity as the first confidence level; the data retrieval submodule includes a keyword retrieval unit, used to perform keyword retrieval in several rounds of historical dialogues based on the initial identification text, to obtain keyword-matched historical dialogues as the first corpus, and the keyword matching degree as the first confidence level; the data retrieval submodule includes a graph retrieval unit, used to perform association reasoning based on the knowledge graph, to obtain historical dialogues that have content association with the initial identification text as the first corpus, and the content association degree as the first confidence level.

[0071] In some disclosed embodiments, the first data includes basic information of the candidate object and a personalized vocabulary library. The memory retrieval module 33 is also used to perform a retrieval in the personalized vocabulary library of the target object based on the input data, and to obtain personalized words related to the input data and include them in the reference data. The reference data also includes basic information of the target object.

[0072] In some disclosed embodiments, the encoding extraction module 31 includes a voiceprint encoding submodule, used to encode voiceprint features based on the acoustic features of the speech signal to obtain the first voiceprint feature of the speech signal; the encoding extraction module 31 includes a semantic encoding submodule, used to encode semantic features based on the acoustic features of the speech signal and the first voiceprint feature to obtain the first semantic feature.

[0073] In some disclosed embodiments, the speech recognition device 30 includes a boundary detection module for identifying the target session round to which the target recognized text belongs; wherein the target session round is the current session round or a new session round; the speech recognition device 30 includes an information extraction module for extracting information based on the target recognized text to obtain memory information of the target session round; the speech recognition device 30 includes a memory update module for updating the memory data of the target object based on the memory information of the target session round; and the speech recognition device 30 includes a sorting and compression module for sorting and compressing the updated memory data of the target object in response to the fulfillment of a preset trigger condition.

[0074] In some disclosed embodiments, the boundary detection module includes a feature acquisition submodule, which selects a first semantic feature of the speech signal as a semantic feature for identifying the target conversation turn, and obtains a time feature based on the time-related feature information of the speech signal; wherein the feature information includes at least the time interval of the interactive dialogue; the boundary detection module includes a classification prediction submodule, which performs classification prediction based on the acoustic features, semantic features and time features of the speech signal to obtain the target conversation turn to which the target recognition text belongs.

[0075] In some publicly disclosed embodiments, the first data includes vocabulary correction records, language style statistics, and usage scenario preferences; the second data includes several rounds of historical dialogue, a knowledge graph constructed based on the historical dialogue, and recognition and error correction records; the information extraction module includes an instruction construction submodule, used to construct a second prompt instruction based on the target recognition text; wherein, the second prompt instruction is used to instruct the large language model to extract memory information from the target recognition text for updating at least one of the following: vocabulary correction records, language style statistics, usage scenario preferences, several rounds of historical dialogue, knowledge graph, and recognition and error correction records; the information extraction module includes an information acquisition submodule, used to obtain the memory information of the target conversation round based on the output content of the large language model in response to the second prompt instruction.

[0076] In some disclosed embodiments, the memory update module includes a contradiction detection submodule, used to detect whether there is a contradiction between the memory information of the target session round and the existing memory information in the memory data of the target object; the memory update module includes a contradiction resolution submodule, used to determine whether to use the memory information of the target session round to replace the existing memory information in the memory data of the target object that contradicts it, based on the information confirmation flag, the information weight value, and the information time interval in sequence, in response to the existence of a contradiction. Wherein, the information confirmation flag indicates whether the memory information has been confirmed by the user, the information weight value indicates the recognition confidence of the recognition text from which the memory information originates, and the information time interval indicates the time interval between the memory information of the target session round and the existing memory information that contradicts it.

[0077] In some disclosed embodiments, the sorting and compression module includes a first response submodule, used to reduce the information weight value of the memory information or to perform summary merging on the memory information in response to the memory information in the updated memory data having a usage frequency lower than a frequency threshold; the sorting and compression module includes a second response submodule, used to aggregate the memory information with a similarity higher than a similarity threshold in response to the existence of multiple memory information in the updated memory data to obtain new memory information.

[0078] In some disclosed embodiments, the text acquisition module 35 includes a recognition submodule for recognizing based on input data to obtain a first recognized text of the speech signal; the text acquisition module 35 includes a correction submodule for correcting the first recognized text based on first data and reference data to obtain a second recognized text; and the text acquisition module 35 includes a polishing and consistency check submodule for polishing and consistency checks based on the second recognized text to obtain a target recognized text.

[0079] Please see Figure 4 , Figure 4 This is a schematic diagram of a framework of an embodiment of the computer device of this application. The computer device 40 includes at least a memory 41 and a processor 42 coupled to each other. The memory 41 stores at least program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above-described embodiments of the speech recognition method. It should be noted that the computer device 40 can be an electronic device capable of executing computer instructions. Exemplarily, the computer device 40 can include, but is not limited to, servers, and terminal devices such as smartphones and tablets; the specific type of the computer device 40 is not limited herein.

[0080] Specifically, processor 42 controls itself and memory 41 to implement the steps in any of the above-described speech recognition method embodiments. Processor 42 can also be referred to as a CPU (Central Processing Unit). Processor 42 may be an integrated circuit chip with signal processing capabilities. Processor 42 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 42 can be implemented using integrated circuit chips.

[0081] In the above scheme, the computer device 40 responds to the speech signal to be recognized by performing feature encoding based on the acoustic features of the speech signal to obtain the first voiceprint feature and the first semantic feature of the speech signal. It then extracts the second voiceprint feature of each candidate object from the memory data of each candidate object, where the memory data includes first data related to the object profile and second data related to the interactive dialogue. Based on the similarity between the first voiceprint feature and each of the second voiceprint features, the candidate object is selected as the target object. Feature transformation is then performed based on the first semantic feature to obtain input data adapted to the large language model. Reference data is then retrieved from the second data of the target object based on the input data. A first prompt instruction is constructed based on the first data, reference data, and input data of the target object. This first prompt instruction instructs the large language model to refer to the first data and reference data of the target object, processing the input data into speech recognition text. Finally, the target recognition text output by the large language model in response to the first prompt instruction is obtained. This process is achieved by extracting the first voiceprint feature of the speech signal. The system matches features with the second voiceprint features of each candidate object to select the candidate object as the target object. Then, it uses the target object's memory data to guide subsequent speech recognition processes. This allows for the most accurate selection of the target object from numerous candidates based on the voiceprint features in the memory data, improving the accuracy of subsequent speech recognition processes. Furthermore, since the memory data is divided into first data related to the object profile and second data related to the interactive dialogue (the first data is relatively stable, while the second data is more dynamic), combining both types of data to assist subsequent speech recognition processes can maximize accuracy with the aid of both static and dynamic memory information. Additionally, after selecting a candidate object as the target object, further searching within the target object's second data helps extract relevant data from the more dynamic memory information of the target object as reference data. Combining this with the first data for speech recognition improves the adaptability of speech recognition to dynamic user changes. Therefore, it enhances both the accuracy and dynamic adaptability of speech recognition.

[0082] Please see Figure 5 , Figure 5 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor. The program instructions 51 are used to implement the steps in any of the above-described embodiments of the speech recognition method.

[0083] In the above scheme, the computer-readable storage medium 50 responds to the speech signal to be recognized by performing feature encoding based on the acoustic features of the speech signal to obtain the first voiceprint feature and the first semantic feature of the speech signal. It then extracts the second voiceprint feature of each candidate object from the memory data of each candidate object, where the memory data includes first data related to the object profile and second data related to the interactive dialogue. Based on the similarity between the first voiceprint feature and each of the second voiceprint features, the candidate object is selected as the target object. Feature transformation is then performed based on the first semantic feature to obtain input data adapted to the large language model. Reference data is then retrieved from the second data of the target object based on the input data. A first prompt instruction is constructed based on the first data of the target object, the reference data, and the input data. This first prompt instruction instructs the large language model to refer to the first data and the reference data of the target object, processing the input data into speech recognition text. Finally, the target recognition text output by the large language model in response to the first prompt instruction is obtained. This process involves extracting the first voiceprint feature of the speech signal... Voiceprint features are matched with the second voiceprint features of each candidate object to select the candidate object as the target object. The subsequent speech recognition process is then guided by the target object's memory data. This allows for the most accurate selection of the target object from numerous candidates based on the voiceprint features in the memory data, improving the accuracy of subsequent speech recognition processes. Furthermore, since the memory data is divided into first data related to the object profile and second data related to the interactive dialogue (first data is relatively stable, while second data is more dynamic), combining both types of data to assist the subsequent speech recognition process can maximize accuracy with the aid of both static and dynamic memory information. Additionally, after selecting a candidate object as the target object, further retrieval within the target object's second data helps extract relevant data from the more dynamic memory information of the target object as reference data. Combining this with the first data for speech recognition improves the adaptability of speech recognition to dynamic user changes. Therefore, this approach enhances both the accuracy and dynamic adaptability of speech recognition.

[0084] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0085] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0086] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0087] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0088] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0089] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0090] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. A speech recognition method, characterized in that, include: In response to the speech signal to be identified, feature encoding is performed based on the acoustic features of the speech signal to obtain the first voiceprint feature and the first semantic feature of the speech signal, and the second voiceprint feature of each candidate object is extracted from the memory data of each candidate object; wherein, the memory data includes first data related to the object profile and second data related to the interactive dialogue; Based on the similarity between the first voiceprint feature and each of the second voiceprint features, the candidate object is selected as the target object, and feature transformation is performed based on the first semantic feature to obtain input data adapted to the large language model. Based on the input data, a retrieval is performed in the second data of the target object to obtain reference data; Based on the first data of the target object, the reference data, and the input data, a first prompt instruction is constructed; wherein, the first prompt instruction is used to instruct the large language model to refer to the first data of the target object and the reference data to process the input data into speech recognition text; Obtain the target recognition text output by the large language model in response to the first prompt instruction.

2. The method according to claim 1, characterized in that, The input data includes second semantic features whose feature dimensions are adapted to the large language model and initial recognition text obtained by preliminary decoding of the first semantic features. The step of retrieving reference data from the second data of the target object based on the input data includes: Based on the second semantic features and the initial identified text, a search is performed in the second data to obtain several first corpora and the first confidence level of each first corpus; Based on the correlation between the various first corpora, several corpus sets are obtained; wherein, each corpus set contains at least one first corpus. The first corpus in the same corpus set is fused to obtain the second corpus and the second confidence level of the second corpus; Based on the second confidence level of each of the second corpora, at least one of the second corpora is selected and included in the reference data.

3. The method according to claim 2, characterized in that, The second data includes several rounds of historical dialogues and a knowledge graph constructed based on the historical dialogues. The second semantic features and the initial identified text are used to retrieve data from the second data to obtain several first corpora and a first confidence level for each first corpus, including: Based on the second semantic feature, vector retrieval is performed in the several rounds of historical dialogues to obtain semantically similar historical dialogues as the first corpus, and the semantic similarity is used as the first confidence level. Based on the initial identified text, keyword retrieval is performed in the several rounds of historical dialogues to obtain the historical dialogues with keyword matching as the first corpus, and the keyword matching degree is used as the first confidence degree. Based on the knowledge graph, association reasoning is performed to obtain historical dialogues that are content-related to the initial identified text as the first corpus, and the content association degree is used as the first confidence degree.

4. The method according to claim 1, characterized in that, The first data includes basic information and a personalized vocabulary of the candidate object. Before constructing the first prompt instruction based on the first data of the target object, the reference data, and the input data, the following is also included: Based on the input data, a search is performed in the personalized vocabulary database of the target object to obtain personalized vocabulary related to the input data and incorporate it into the reference data. The reference data also includes basic information about the target object.

5. The method according to claim 1, characterized in that, The feature encoding based on the acoustic features of the speech signal to obtain the first voiceprint feature and the first semantic feature of the speech signal includes: Based on the acoustic features of the speech signal, voiceprint feature encoding is performed to obtain the first voiceprint feature of the speech signal; Semantic feature encoding is performed based on the acoustic features of the speech signal and the first voiceprint feature to obtain the first semantic feature.

6. The method according to claim 1, characterized in that, After obtaining the target recognized text output by the large language model in response to the first prompt instruction, the method further includes: Identify the target session round to which the target text belongs; wherein, the target session round is the current session round or a new session round; Information is extracted based on the target-identified text to obtain the memory information of the target session rounds; Based on the memory information of the target session round, the memory data of the target object is updated; In response to the fulfillment of preset triggering conditions, the memory data of the target object after the update is organized and compressed.

7. The method according to claim 6, characterized in that, The identification of the target session round to which the target text belongs includes: A first semantic feature of the speech signal is selected as the semantic feature for identifying the target conversation turn, and a time feature is obtained based on the time-related feature information of the speech signal; wherein the feature information includes at least the time interval of the interactive dialogue; Based on the acoustic features, semantic features, and temporal features, classification prediction is performed to obtain the target session round to which the target-identified text belongs.

8. The method according to claim 6, characterized in that, The first data includes vocabulary correction records, language style statistics, and usage scenario preferences. The second data includes several rounds of historical dialogue, a knowledge graph constructed based on the historical dialogue, and recognition and error correction records. The step of extracting information based on the target recognition text to obtain the memory information of the target conversation rounds includes: Based on the target recognition text, a second prompt instruction is constructed; wherein, the second prompt instruction is used to instruct the large language model to extract memory information from the target recognition text for updating at least one of the following: the vocabulary correction record, the language style statistics, the usage scenario preference, the several rounds of historical dialogue, the knowledge graph, and the recognition error correction record; Based on the output of the second prompt instruction in response to the large language model, the memory information of the target session round is obtained.

9. The method according to claim 6, characterized in that, The updating of the memory data of the target object based on the memory information of the target session round includes: Detect whether there is a contradiction between the memory information of the target session round and the existing memory information in the memory data of the target object; In response to a contradiction, the system sequentially determines whether to use the memory information of the target session round to replace the existing memory information in the target object's memory data that contradicts it, based on the information confirmation marker, the information weight value, and the information time interval. The information confirmation marker indicates whether the memory information has been confirmed by the user, the information weight value indicates the recognition confidence of the text from which the memory information originates, and the information time interval indicates the time interval between the memory information of the target session round and the existing memory information that contradicts it.

10. The method according to claim 6, characterized in that, The process of organizing and compressing the memory data of the target object after the update includes: In response to the memory information in the updated memory data having a usage frequency lower than a frequency threshold, the information weight value of the memory information is reduced, or the memory information is summarized and merged. In response to the existence of multiple memory information entries with a similarity higher than the similarity threshold in the updated memory data, new memory information is obtained by aggregating these multiple memory information entries with a similarity higher than the similarity threshold.

11. The method according to claim 1, characterized in that, The processing of the large language model in response to the first prompt instruction also includes: Based on the input data, the first recognized text of the speech signal is obtained; The first identified text is corrected based on the first data and the reference data to obtain the second identified text; Based on the second identified text, polishing and consistency checks are performed to obtain the target identified text.

12. A voice recognition device, characterized in that, include: The encoding extraction module is used to respond to the speech signal to be recognized, perform feature encoding based on the acoustic features of the speech signal to obtain the first voiceprint feature and the first semantic feature of the speech signal, and extract the second voiceprint feature of each candidate object from the memory data of each candidate object respectively; wherein, the memory data includes first data related to the object profile and second data related to the interactive dialogue; The selection and conversion module is used to select the candidate object as the target object based on the similarity between the first voiceprint feature and each of the second voiceprint features, and to perform feature conversion based on the first semantic feature to obtain input data adapted to the large language model. The memory retrieval module is used to retrieve reference data from the second data of the target object based on the input data. The instruction construction module is used to construct a first prompt instruction based on the first data of the target object, the reference data, and the input data; wherein, the first prompt instruction is used to instruct the large language model to process the input data into speech recognition text by referring to the first data of the target object and the reference data; The text acquisition module is used to acquire the target recognition text output by the large language model in response to the first prompt instruction.

13. A computer device, characterized in that, It includes at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor is used to execute the program instructions to implement the speech recognition method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The device stores program instructions that can be executed by a processor, the program instructions being used to implement the speech recognition method according to any one of claims 1 to 11.

Images