Voice interaction task execution method and device based on large model, equipment and medium

Abstract

The invention discloses a voice interaction task execution method and device based on a large model, equipment and a medium, and relates to the field of artificial intelligence, and the method comprises the steps: carrying out the voice recognition of voice features, obtaining a text character sequence, and carrying out the optimization of the text character sequence through a preset large language model, and obtaining a target text character sequence; performing entity recognition on the target text character sequence by using a preset large language model to obtain an entity recognition result, determining a relationship type among entities in the target text character sequence according to the entity recognition result, and constructing a knowledge graph according to the relationship type among the entities; generating an initial triple based on the knowledge graph, and optimizing the initial triple by using a preset large language model to obtain a target triple; and fusing the target triple with the initial knowledge graph, and executing a voice interaction task in the target voice interaction scene based on the updated knowledge graph. According to the method and the device, the efficiency and the accuracy of extracting the structured knowledge from the Chinese speech are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method, device, equipment and medium for executing voice interaction tasks based on a large model. Background Art

[0002] Traditional methods often face challenges in processing speech signals, such as noise interference, dialect diversity, and colloquial expressions, which limit speech recognition accuracy. During text processing, extracting relationships between entities requires dealing with complex semantics and contextual dependencies. Existing technologies still need to improve their efficiency and accuracy when dealing with large-scale data and diverse semantics. Furthermore, effectively integrating extracted knowledge into knowledge graphs and enabling dynamic updates are also challenges that need to be addressed. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a method, device, equipment, and medium for executing voice interaction tasks based on a large model, which can improve the efficiency and accuracy of extracting structured knowledge from Chinese speech and promote the application and development of voice interaction systems in fields such as intelligent customer service and voice search. The specific solution is as follows:

[0004] In a first aspect, the present application provides a method for executing a voice interaction task based on a large model, comprising:

[0005] Performing pre-set speech signal preprocessing and feature extraction operations on the collected speech data to obtain speech features;

[0006] Performing speech recognition on the speech features based on a preset speech recognition model to obtain a semantic text character sequence corresponding to the speech features, and optimizing the text character sequence using a preset large language model to obtain a target text character sequence;

[0007] Performing entity recognition on the target text character sequence using the preset large language model to obtain an entity recognition result, determining the relationship type between entities in the target text character sequence based on the entity recognition result, and constructing a knowledge graph based on the relationship type between the entities;

[0008] Generating an initial triple based on the knowledge graph, and optimizing the initial triple using the preset large language model to obtain a target triple;

[0009] The target triplet is fused with the initial knowledge graph, and the voice interaction task in the target voice interaction scenario is performed based on the updated knowledge graph.

[0010] Optionally, performing a preset voice signal preprocessing and feature extraction operation on the collected voice data to obtain voice features includes:

[0011] Performing noise reduction processing on the collected voice data based on a preset deep learning algorithm to obtain first voice data, and removing silence data in the first voice data based on the energy and zero-crossing rate of the voice signal in the first voice data to obtain second voice data;

[0012] Mel-frequency cepstral coefficients are used to extract basic speech features in the second speech data, and a preset convolutional neural network is used to determine deep speech features based on the basic speech features.

[0013] Optionally, determining the relationship type between entities in the target text character sequence according to the entity recognition result includes:

[0014] Analyzing the entity recognition results using the attention mechanism of the preset large language model to determine the relationship type between entities in the target text character sequence;

[0015] The relationship types between the entities are tested and optimized by combining the preset large language model with the context information in the target text character sequence to obtain optimized relationship types between the entities.

[0016] Optionally, generating an initial triple based on the knowledge graph, optimizing the initial triple using the preset large language model to obtain a target triple; fusing the target triple with the initial knowledge graph, and performing a voice interaction task in a target voice interaction scenario based on the updated knowledge graph, including:

[0017] Using a predefined rule engine to preliminarily screen and combine the entities recorded in the knowledge graph and the relationship types between the entities to generate initial triples;

[0018] Checking whether the initial triples meet a preset logical standard based on the preset large language model, and correcting the initial triples that do not meet the preset logical standard to obtain optimized target triples;

[0019] Fusing the target triple with existing knowledge in an existing initial knowledge graph to update the initial knowledge graph to obtain an updated knowledge graph, and performing a voice interaction task in a target voice interaction scenario based on the updated knowledge graph;

[0020] Accordingly, the optimizing the initial triples by using the preset large language model to obtain target triples further includes:

[0021] A concurrent processing mechanism is adopted to perform batch processing on the optimization process of the initial triples.

[0022] Optionally, executing a voice interaction task in a target voice interaction scenario based on the updated knowledge graph includes:

[0023] Acquire a voice interaction task in a target voice interaction scenario; the voice interaction task includes a question request sent by a user terminal;

[0024] Determining the triple corresponding to the question request from the updated knowledge graph;

[0025] A request response corresponding to the question request is determined based on the triplet corresponding to the question request, and the request response is returned to the user terminal to complete the execution operation of the voice interaction task.

[0026] Optionally, after optimizing the text character sequence using a preset large language model to obtain a target text character sequence, the method further includes:

[0027] Sending the target text character sequence to a user terminal, and obtaining a feedback result returned by the user terminal with respect to the target text character sequence;

[0028] If the feedback result indicates that the target text character sequence is incorrect, the target text character sequence is corrected according to the feedback result to obtain a new target text character sequence, and the corresponding parameters of the preset large language model are adjusted to update the preset large language model.

[0029] Optionally, the training process of the preset large language model includes:

[0030] Training a preset large language model based on a historically collected annotated first target dataset;

[0031] and / or, fusing the collected speech data with different types of pre-collected background noise to obtain a second target data set, and using the second target data set to train the preset large language model;

[0032] And / or, adjusting the speaking speed and / or pitch of the collected speech data to obtain a third target data set, and using the third target data set to train the preset large language model.

[0033] In a second aspect, the present application provides a large-model-based voice interaction task execution device, comprising:

[0034] The feature acquisition module is used to perform pre-set speech signal preprocessing and feature extraction operations on the collected speech data to obtain speech features;

[0035] a sequence acquisition module, configured to perform speech recognition on the speech features based on a preset speech recognition model to obtain a semantic text character sequence corresponding to the speech features, and optimize the text character sequence using a preset large language model to obtain a target text character sequence;

[0036] A graph construction module is used to perform entity recognition on the target text character sequence using the preset large language model to obtain an entity recognition result, determine the relationship type between entities in the target text character sequence based on the entity recognition result, and construct a knowledge graph based on the relationship type between the entities;

[0037] A triple acquisition module, configured to generate an initial triple based on the knowledge graph, and optimize the initial triple using the preset large language model to obtain a target triple;

[0038] A graph updating module is used to fuse the target triples with the initial knowledge graph to obtain an updated knowledge graph;

[0039] A task execution module is used to execute the voice interaction task in the target voice interaction scenario based on the updated knowledge graph.

[0040] In a third aspect, the present application provides an electronic device, comprising:

[0041] Memory, used to store computer programs;

[0042] A processor is used to execute the computer program to implement the aforementioned large model-based voice interaction task execution method.

[0043] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned large-model-based voice interaction task execution method.

[0044] In this application, the collected voice data is subjected to a preset voice signal preprocessing and feature extraction operation to obtain voice features; voice recognition is performed on the voice features based on a preset voice recognition model to obtain a semantic text character sequence corresponding to the voice features, and the text character sequence is optimized using a preset large language model to obtain a target text character sequence; entity recognition is performed on the target text character sequence using the preset large language model to obtain an entity recognition result, and the relationship type between the entities in the target text character sequence is determined based on the entity recognition result, and a knowledge graph is constructed based on the relationship type between the entities; initial triples are generated based on the knowledge graph, and the initial triples are optimized using the preset large language model to obtain target triples; the target triples are fused with the initial knowledge graph, and the voice interaction task in the target voice interaction scenario is executed based on the updated knowledge graph. As can be seen from the above, this application performs a preset voice signal preprocessing and feature extraction operation on the collected voice data to accurately locate effective voice paragraphs, remove invalid silence, reduce redundant data, and improve subsequent processing efficiency and voice recognition accuracy. Optimizing text character sequences using a large language model yields more accurate, clear, and grammatically correct target text character sequences, significantly reducing input noise and semantic complexity in subsequent entity relationship extraction tasks, laying a solid foundation for high-precision extraction. Simultaneously, the large language model's powerful contextual understanding and reasoning capabilities allow for precise identification of entity boundaries and types, and the construction of a knowledge graph based on the relationship types between entities, significantly improving the accuracy of recognizing complex and diverse semantic relationships. Fusion of the optimized target triples with the initial knowledge graph enables dynamic updating and fusion of the knowledge graph, addressing the issues of knowledge integration and dynamic updating. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0046] Figure 1 This is a flow chart of a method for executing a voice interaction task based on a large model disclosed in this application;

[0047] Figure 2 This is a schematic diagram of a specific speech signal preprocessing and feature extraction module disclosed in this application;

[0048] Figure 3 This is a schematic diagram of a specific speech recognition and text conversion module disclosed in this application;

[0049] Figure 4 This is a schematic diagram of a specific text entity recognition and relationship extraction module disclosed in this application;

[0050] Figure 5 A schematic diagram of a specific triplet construction and optimization module disclosed in this application;

[0051] Figure 6 This is a schematic diagram of a specific method for executing a voice interaction task based on a large model disclosed in this application;

[0052] Figure 7 This is a schematic diagram of the structure of a large-scale model-based voice interaction task execution device disclosed in this application;

[0053] Figure 8 This is a schematic diagram of the structure of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0055] When processing speech signals, traditional methods often face challenges such as noise interference, dialect diversity, and colloquial expressions, which limit the accuracy of speech recognition. In the text processing stage, the extraction of entities and relationships between entities requires processing complex semantics and contextual dependencies. The efficiency and accuracy of existing technologies still need to be improved when faced with large-scale data and diverse semantics. In addition, how to effectively integrate the extracted knowledge into the knowledge graph and realize dynamic updates is also a problem that needs to be solved. To this end, the present application provides a method for executing speech interaction tasks based on a large model, which can improve the efficiency and accuracy of extracting structured knowledge from Chinese speech, and promote the application and development of speech interaction systems in the fields of intelligent customer service, voice search, etc.

[0056] See also Figure 1 As shown, the embodiment of the present application discloses a method for executing a voice interaction task based on a large model, comprising:

[0057] Step S11: Perform preset speech signal preprocessing and feature extraction operations on the collected speech data to obtain speech features.

[0058] This embodiment mainly consists of four modules. The first is the speech signal preprocessing and feature extraction module, which is the input end of the entire system and provides clean, high-quality speech feature data for subsequent speech recognition.

[0059] For details, see Figure 2 As shown, voice data can be collected first. For example, a high-precision microphone array or a smart device microphone can be used to collect voice signals to ensure clarity and integrity. Microphone arrays can effectively capture voice signals from multiple directions while suppressing interference from ambient noise. For smart device microphones, voice signal quality can be further improved by optimizing their hardware parameters and acquisition algorithms. Furthermore, during the acquisition process, voice signal metadata, such as sampling rate and channel information, can be recorded to provide a reference for subsequent processing.

[0060] The collected voice data can then be subjected to noise reduction processing based on a preset deep learning algorithm to obtain first voice data. Silence data in the first voice data can be removed based on the energy and zero-crossing rate of the voice signal in the first voice data to obtain second voice data. Basic voice features in the second voice data can be extracted using Mel-frequency cepstral coefficients, and deep voice features can be determined based on the basic voice features using a preset convolutional neural network.

[0061] For example Figure 2 As shown, an adaptive filter network is used to denoise speech signals. This method automatically learns noise characteristics and separates them from the speech signal, significantly reducing background noise interference. Secondly, voice activity detection (VAD) can be performed by analyzing features such as speech signal energy and zero-crossing rate to accurately locate speech segments and remove invalid silence. This allows valid speech segments to be identified and processing to be stopped, reducing redundant data and improving the efficiency of subsequent processing. For valid speech segments, Mel-Frequency Cepstral Coefficients (MFCCs) are used to extract basic features to reduce computational complexity. A feature extractor based on a convolutional neural network (CNN) is then used to further extract deeper features from the speech signal, ultimately generating speech feature data. CNNs can automatically learn local features and patterns in speech signals, improving feature representation and providing high-quality input data for subsequent operations.

[0062] Step S12: performing speech recognition on the speech features based on a preset speech recognition model to obtain a semantic text character sequence corresponding to the speech features, and optimizing the text character sequence using a preset large language model to obtain a target text character sequence.

[0063] In this embodiment, the high-quality speech feature data output by the speech signal preprocessing and feature extraction module is converted into text through the speech recognition and text conversion module, providing basic input for the subsequent entity recognition and relationship extraction modules.

[0064] See also Figure 3 As shown, a deep learning-based end-to-end speech recognition architecture, such as a Transformer- or Conformer-based speech recognition model, can be used to perform speech recognition on speech feature data. The speech feature data first passes through an encoder, such as a Transformer encoder or a Conformer encoder. The encoder uses a self-attention mechanism to model speech features, capturing long-range dependencies and contextual information in the speech signal. This generates an encoded feature sequence, which is then mapped into a text character sequence by the decoder.

[0065] Furthermore, after speech recognition is complete, the recognition results (i.e., the text character sequence) can be post-processed and optimized using a preset large language model. This involves using the language model portion of the preset large model to perform grammatical and semantic corrections on the recognition results. For example, the preset large language model can use its powerful contextual modeling capabilities to correct grammatical errors or unnatural expressions in the recognition results, ensuring that the generated text is more semantically accurate and natural. Furthermore, it can determine whether the recognition results require logical reasoning to disambiguate. If so, the logical reasoning capabilities of the preset large language model can be used to resolve ambiguities in the recognition results, further improving the accuracy and readability of the text.

[0066] After optimizing the text character sequence using the preset large language model to obtain a target text character sequence, the method may further include: sending the target text character sequence to a user terminal and obtaining feedback from the user terminal regarding the target text character sequence. If the feedback indicates that the target text character sequence is incorrect, the target text character sequence is corrected based on the feedback to obtain a new target text character sequence, and corresponding parameters of the preset large language model are adjusted to update the preset large language model.

[0067] This allows users to receive real-time recognition results based on a real-time feedback mechanism, allowing them to confirm or correct their responses. When users provide feedback on the results, the system incorporates this feedback into the model training process, dynamically adjusting model parameters and ultimately outputting an optimized target text character sequence. This optimizes recognition results and enhances the user experience.

[0068] Step S13: Use the preset large language model to perform entity recognition on the target text character sequence to obtain an entity recognition result, and determine the relationship type between the entities in the target text character sequence based on the entity recognition result, and construct a knowledge graph based on the relationship type between the entities.

[0069] In this embodiment, the text entity recognition and relationship extraction module is used to accurately identify entities and relationships in the text, providing key information for subsequent triple construction.

[0070] Specifically, a preset large language model can first be used to perform entity recognition on the target text character sequence to obtain entity recognition results. The preset large language model's attention mechanism is then used to analyze the entity recognition results to determine the relationship types between entities in the target text character sequence. Finally, the preset large language model is combined with contextual information in the target text character sequence to verify and optimize the relationship types between the entities, thereby obtaining optimized relationship types between the entities.

[0071] Among them, when performing entity recognition, large language models such as BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language model based on the Transformer architecture) and GPT (Generative Pre-trained Transformer) can be used. Through the context encoding capabilities of these large language models, entity recognition can be performed on the generated target text character sequence.

[0072] like Figure 4 As shown in the figure, after entity recognition, the large language model's relation extraction capabilities can be used to perform relation classification and extraction on the target text character sequence to identify the relationships between entities in the text. For example, the model identifies potential relationship types between entities in the text, such as "belongs to," "is located in," and "works for," and then further confirms these relationships using the model's relation extraction capabilities. Based on the relationship types between entities, the relationships between entities can be modeled using technologies such as graph neural networks (GNNs) to construct a knowledge graph.

[0073] To better perform entity recognition, the model can be trained. For example, a pre-trained language model suitable for Chinese text processing, such as BERT or its variant, RoBERTa, can be selected. Based on the pre-trained model, the model can be fine-tuned using an annotated Chinese NER (Named Entity Recognition) dataset to enable accurate recognition of various entities. Furthermore, the entity recognition task can be transformed into a sequence labeling task, using the annotated data to train the model, enabling it to identify entity boundaries and types in text.

[0074] It's important to note that a model's entity recognition and relationship extraction capabilities can be optimized simultaneously to achieve multi-task learning optimization. On the one hand, joint training can be performed within a multi-task learning framework, simultaneously training entity recognition and relationship extraction tasks. By sharing the feature representations of the underlying language model, model parameters can be reduced and training efficiency improved. On the other hand, a reasonable loss function can be designed to balance the weights of entity recognition and relationship extraction tasks, ensuring optimal performance for both tasks.

[0075] Step S14: generating an initial triple based on the knowledge graph, and optimizing the initial triple using the preset large language model to obtain a target triple.

[0076] In this embodiment, the target triple is obtained through the triple construction and optimization module.

[0077] For details, see Figure 5 As shown, first, the output results of the text recognition and relationship extraction modules are received, that is, the entities and the relationship types between entities recorded in the knowledge graph. The predefined rule engine is used to preliminarily screen and combine the entities and the relationship types between entities recorded in the knowledge graph, and automatically generate initial triples (head entity, relationship, tail entity). Then, the large language model can be called to perform semantic verification on the generated initial triples to ensure the logical rationality and semantic coherence of the initial triples. Furthermore, the initial triples can be checked based on the preset large language model to see whether they meet the preset logical standards, and the initial triples that do not meet the preset logical standards can be corrected to obtain the optimized target triples. Among them, optimizing the initial triples using the preset large language model to obtain the target triples can also include: adopting a concurrent processing mechanism to perform batch operations on the optimization process of the initial triples.

[0078] Understandably, see Figure 5As shown, if a triple passes verification, the large language model's logical reasoning capabilities can be used to check whether the triple conforms to common sense logic, such as whether "name-employment-company name" is reasonable. Inappropriate triples can then be corrected and optimized based on contextual information. For example, by retrieving similar text blocks, the large language model can be provided with richer context, helping it to more accurately understand and optimize triples.

[0079] Step S15: Fuse the target triple with the initial knowledge graph, and execute the voice interaction task in the target voice interaction scenario based on the updated knowledge graph.

[0080] See also Figure 5 As shown, in this embodiment, the target triple can be integrated with the existing knowledge in the existing initial knowledge graph to update the initial knowledge graph to obtain an updated knowledge graph, realize dynamic updating of knowledge, and execute the voice interaction task in the target voice interaction scenario based on the updated knowledge graph.

[0081] In this way, new triple information is fed back into the knowledge graph through incremental learning methods, realizing dynamic updating and optimization of knowledge, enabling the knowledge graph to continuously absorb new information, maintain the timeliness and accuracy of the knowledge graph, and provide richer knowledge support for the intelligent voice interaction system.

[0082] Executing a voice interaction task in a target voice interaction scenario based on the updated knowledge graph may include: first obtaining a voice interaction task in the target voice interaction scenario; the voice interaction task includes a question request sent by a user; then determining a triple corresponding to the question request from the updated knowledge graph; and finally, determining a request response corresponding to the question request based on the triple corresponding to the question request, and returning the request response to the user, thereby completing the execution of the voice interaction task.

[0083] It should be noted that this embodiment provides a standardized application interface that supports multiple output formats, such as JSON and XML. This outputs the optimized target triples to downstream application scenarios, such as intelligent customer service and voice search, facilitating integration with other systems. Furthermore, the output target triples can be customized and optimized based on the needs of downstream application scenarios, improving system adaptability and user experience.

[0084] In this embodiment, the training process for the preset large language model may include: training the preset large language model based on a historically collected annotated first target dataset; and / or fusing the collected speech data with pre-collected different types of background noise to obtain a second target dataset, and using the second target dataset to train the preset large language model; and / or adjusting the speaking speed and / or pitch of the collected speech data to obtain a third target dataset, and using the third target dataset to train the preset large language model. For example, the collected speech signal can be mixed with different types of background noise, such as street noise, office noise, etc., and the intensity of the noise can be adjusted to generate a diverse cluster of training samples. At the same time, by adjusting the speaking speed and pitch of the speech signal, the speech characteristics of different speakers can be simulated, further enhancing the generalization ability of the model.

[0085] On the other hand, during the model training stage, dialect-specific vocabularies can also be introduced so that the model can simultaneously learn the speech features and language patterns of multiple dialects and Mandarin. This allows the model to automatically identify the dialect or accent to which the input speech belongs based on its characteristics and make adaptive adjustments, thereby improving recognition accuracy and further optimizing the recognition effect of dialect speech.

[0086] As can be seen from the above, on the one hand, this embodiment uses a high-precision microphone array or a smart device microphone to collect voice, and uses a deep learning algorithm for noise reduction processing and voice activity detection, which significantly improves the clarity and integrity of the voice signal. On the other hand, the adoption of an end-to-end speech recognition architecture based on Transformer or Conformer, combined with the context-aware optimization of a large language model, effectively improves the accuracy and readability of speech recognition. Furthermore, by using a pre-large language model for entity recognition and relationship extraction, and by constructing a knowledge graph, entities in the text and the relationships between entities can be accurately identified. At the same time, triples are constructed through the semantic understanding capabilities of the rule engine and the large language model, and the knowledge base is dynamically updated using knowledge graph fusion and incremental learning methods to ensure the timeliness and accuracy of knowledge.

[0087] See also Figure 6 As shown, the technical solution in this application is explained below by taking the main modules involved in this application as an example.

[0088] This embodiment mainly includes four modules, namely, a speech signal preprocessing and feature extraction module, a speech recognition and text conversion module, a text entity recognition and relationship extraction module, and a triple construction and optimization module.

[0089] Speech signal preprocessing and feature extraction module: Use a high-precision microphone array or smart device microphone to collect speech signals, and determine whether the collected speech signals are complete and clear. If the speech signals are not complete and clear, re-collect the speech signals. If the speech signals are complete and clear, perform preprocessing operations such as noise reduction and voice activity detection on the speech signals, use MFCC to extract basic features in the speech signals, and use CNN to extract deep features in the speech signals to obtain speech feature data.

[0090] The Speech Recognition and Text Conversion Module uses a deep learning-based end-to-end speech recognition architecture to perform speech recognition on speech feature data, generating a converted text character sequence. If optimization of the recognition results is not required, entity recognition is performed directly on the converted text character sequence. If optimization is required, the recognition results are optimized based on the context-awareness of the pre-set large language model to obtain the optimized target text character sequence.

[0091] The Text Entity Recognition and Relationship Extraction module uses a pre-set large language model to perform entity recognition on the target text character sequence. The pre-set large language model's relationship extraction capabilities are then used to classify and extract relationships from the target text character sequence. Based on the relationship types between entities and using techniques such as graph neural networks, the relationships between entities are modeled to construct a knowledge graph.

[0092] The triple construction and optimization module receives the output of the text recognition and relationship extraction modules, namely, the entities and relationship types recorded in the knowledge graph. Using a predefined rule engine, it performs preliminary screening and combination of the entities and relationship types recorded in the knowledge graph, automatically generating initial triples. It then determines whether triple optimization is necessary. If not, it directly performs knowledge fusion. If optimization is required, it uses a large language model to perform semantic verification on the generated initial triples to ensure their logical rationality and semantic coherence. Furthermore, it checks whether the initial triples meet pre-set logical criteria based on the pre-set large language model. Initial triples that do not meet these criteria are corrected to obtain optimized target triples. The target triples are then integrated with existing knowledge in the initial knowledge graph to update the initial knowledge graph, resulting in an updated knowledge graph, enabling dynamic knowledge updating. Based on a standardized application interface, the optimized target triples are output to downstream application scenarios to execute voice interaction tasks in the target voice interaction scenario.

[0093] As can be seen from the above, this embodiment integrates advanced speech processing and natural language processing technologies through the speech signal preprocessing and feature extraction module, the speech recognition and text conversion module, the text entity recognition and relationship extraction module, and the triple construction and optimization module, effectively solving the limitations of traditional methods in noise interference, dialect diversity, and colloquial expressions, improving the efficiency and accuracy of extracting structured knowledge from Chinese speech, and having a certain driving effect on the application and development of intelligent voice interaction technology in the fields of intelligent customer service, voice search, etc.

[0094] See also Figure 7 As shown, the embodiment of the present application also discloses a large model-based voice interaction task execution device, including:

[0095] The feature acquisition module 11 is used to perform pre-set speech signal preprocessing and feature extraction operations on the collected speech data to obtain speech features;

[0096] A sequence acquisition module 12 is configured to perform speech recognition on the speech features based on a preset speech recognition model to obtain a semantic text character sequence corresponding to the speech features, and optimize the text character sequence using a preset large language model to obtain a target text character sequence;

[0097] A graph construction module 13 is configured to perform entity recognition on the target text character sequence using the preset large language model to obtain an entity recognition result, determine the relationship type between entities in the target text character sequence based on the entity recognition result, and construct a knowledge graph based on the relationship type between the entities;

[0098] A triple acquisition module 14 is configured to generate an initial triple based on the knowledge graph, and optimize the initial triple using the preset large language model to obtain a target triple;

[0099] A graph updating module 15 is configured to fuse the target triples with the initial knowledge graph to obtain an updated knowledge graph;

[0100] The task execution module 16 is used to execute the voice interaction task in the target voice interaction scenario based on the updated knowledge graph.

[0101] As can be seen from the above, this application performs pre-set speech signal preprocessing and feature extraction operations on the collected speech data to accurately locate valid speech segments, remove invalid silence, reduce redundant data, and improve subsequent processing efficiency and speech recognition accuracy. By optimizing the text character sequence using a large language model, a more accurate, clear, and grammatically compliant target text character sequence can be obtained, which significantly reduces the input noise and semantic complexity of subsequent entity relationship extraction tasks, laying a solid foundation for high-precision extraction; at the same time, the powerful context understanding and reasoning capabilities of the large language model are used to accurately identify entity boundaries and types, and a knowledge graph is constructed based on the relationship types between entities, which significantly improves the recognition accuracy of complex and diverse semantic relationships. The optimized target triples are integrated with the initial knowledge graph to achieve dynamic updating and fusion of the knowledge graph, solving the problem of knowledge integration and dynamic updating.

[0102] In some specific implementations, the feature acquisition module 11 includes:

[0103] a data acquisition unit, configured to perform noise reduction processing on the collected voice data based on a preset deep learning algorithm to obtain first voice data, and remove silence data in the first voice data based on the energy and zero-crossing rate of the voice signal in the first voice data to obtain second voice data;

[0104] A feature extraction unit is used to extract basic speech features in the second speech data using Mel-frequency cepstral coefficients, and to determine deep speech features based on the basic speech features using a preset convolutional neural network.

[0105] In some specific embodiments, the map construction module 13 includes:

[0106] a relationship determination unit, configured to analyze the entity recognition results using the attention mechanism of the preset large language model to determine the relationship type between entities in the target text character sequence;

[0107] The relationship optimization unit is used to test and optimize the relationship types between the entities by combining the preset large language model with the context information in the target text character sequence to obtain optimized relationship types between the entities.

[0108] In some specific implementations, the map updating module 15 includes:

[0109] An initial triple generation unit, configured to use a predefined rule engine to preliminarily screen and combine the entities recorded in the knowledge graph and the relationship types between the entities to generate initial triples;

[0110] a target triple generating unit, configured to check whether the initial triples meet a preset logical standard based on the preset large language model, and to modify the initial triples that do not meet the preset logical standard to obtain optimized target triples;

[0111] A graph updating unit, configured to fuse the target triple with existing knowledge in the existing initial knowledge graph to update the initial knowledge graph to obtain an updated knowledge graph;

[0112] Accordingly, the triplet acquisition module 14 further includes:

[0113] The optimization unit is configured to adopt a concurrent processing mechanism to perform batch processing operations on the optimization process of the initial triples.

[0114] In some specific implementations, the task execution module 16 includes:

[0115] A task acquisition unit is used to acquire a voice interaction task in a target voice interaction scenario; the voice interaction task includes a question request sent by a user terminal;

[0116] A triple determination unit, configured to determine the triple corresponding to the question request from the updated knowledge graph;

[0117] A task execution unit is used to determine a request response corresponding to the question request based on the triplet corresponding to the question request, and return the request response to the user terminal to complete the execution operation of the voice interaction task.

[0118] In some specific embodiments, the sequence acquisition module 12 further includes:

[0119] A result acquisition unit, configured to send the target text character sequence to a user terminal and acquire a feedback result returned by the user terminal with respect to the target text character sequence;

[0120] A model updating unit is configured to correct the target text character sequence according to the feedback result to obtain a new target text character sequence if the feedback result indicates that the target text character sequence is incorrect, and adjust the corresponding parameters of the preset large language model to update the preset large language model.

[0121] In some specific implementations, the training process of the preset large language model includes:

[0122] A first training unit is configured to train a preset large language model based on a historically collected annotated first target dataset;

[0123] a second training unit, configured to fuse the collected speech data with different types of pre-collected background noise to obtain a second target data set, and train the preset large language model using the second target data set;

[0124] The third training unit is used to adjust the speaking speed and / or pitch of the collected voice data to obtain a third target data set, and use the third target data set to train the preset large language model.

[0125] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 8 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0126] Figure 8 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the large model-based voice interaction task execution method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0127] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0128] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0129] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of implementing the large model-based voice interaction task execution method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program capable of completing other specific tasks.

[0130] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned method for executing a large-scale model-based voice interaction task. The specific steps of this method can be referred to the corresponding content disclosed in the aforementioned embodiments and will not be repeated here.

[0131] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0132] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0133] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0134] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0135] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for executing a voice interaction task based on a large model, characterized in that: include: Performing pre-set speech signal preprocessing and feature extraction operations on the collected speech data to obtain speech features; Performing speech recognition on the speech features based on a preset speech recognition model to obtain a semantic text character sequence corresponding to the speech features, and optimizing the text character sequence using a preset large language model to obtain a target text character sequence; Performing entity recognition on the target text character sequence using the preset large language model to obtain an entity recognition result, determining the relationship type between entities in the target text character sequence based on the entity recognition result, and constructing a knowledge graph based on the relationship type between the entities; Generating an initial triple based on the knowledge graph, and optimizing the initial triple using the preset large language model to obtain a target triple; The target triplet is fused with the initial knowledge graph, and the voice interaction task in the target voice interaction scenario is performed based on the updated knowledge graph.

2. The method for executing a voice interaction task based on a large model according to claim 1, characterized in that: The method of performing a preset voice signal preprocessing and feature extraction operation on the collected voice data to obtain voice features includes: Performing noise reduction processing on the collected voice data based on a preset deep learning algorithm to obtain first voice data, and removing silence data in the first voice data based on the energy and zero-crossing rate of the voice signal in the first voice data to obtain second voice data; Mel-frequency cepstral coefficients are used to extract basic speech features in the second speech data, and a preset convolutional neural network is used to determine deep speech features based on the basic speech features.

3. The method for executing a voice interaction task based on a large model according to claim 1, characterized in that: Determining the relationship type between entities in the target text character sequence according to the entity recognition result includes: Analyzing the entity recognition results using the attention mechanism of the preset large language model to determine the relationship type between entities in the target text character sequence; The relationship types between the entities are tested and optimized by combining the preset large language model with the context information in the target text character sequence to obtain optimized relationship types between the entities.

4. The method for executing a voice interaction task based on a large model according to claim 1, characterized in that: generating an initial triple based on the knowledge graph, and optimizing the initial triple using the preset large language model to obtain a target triple; The target triplet is integrated with the initial knowledge graph, and the voice interaction task in the target voice interaction scenario is performed based on the updated knowledge graph, including: Using a predefined rule engine to preliminarily screen and combine the entities recorded in the knowledge graph and the relationship types between the entities to generate initial triples; Checking whether the initial triples meet a preset logical standard based on the preset large language model, and correcting the initial triples that do not meet the preset logical standard to obtain optimized target triples; Fusing the target triple with existing knowledge in an existing initial knowledge graph to update the initial knowledge graph to obtain an updated knowledge graph, and performing a voice interaction task in a target voice interaction scenario based on the updated knowledge graph; Accordingly, the optimizing the initial triples by using the preset large language model to obtain target triples further includes: A concurrent processing mechanism is adopted to perform batch processing on the optimization process of the initial triples.

5. The method for executing a voice interaction task based on a large model according to claim 4, characterized in that: The performing of the voice interaction task in the target voice interaction scenario based on the updated knowledge graph includes: Acquire a voice interaction task in a target voice interaction scenario; the voice interaction task includes a question request sent by a user terminal; Determining the triple corresponding to the question request from the updated knowledge graph; A request response corresponding to the question request is determined based on the triplet corresponding to the question request, and the request response is returned to the user terminal to complete the execution operation of the voice interaction task.

6. The method for executing a voice interaction task based on a large model according to claim 1, characterized in that: After optimizing the text character sequence using the preset large language model to obtain a target text character sequence, the method further includes: Sending the target text character sequence to a user terminal, and obtaining a feedback result returned by the user terminal with respect to the target text character sequence; If the feedback result indicates that the target text character sequence is incorrect, the target text character sequence is corrected according to the feedback result to obtain a new target text character sequence, and the corresponding parameters of the preset large language model are adjusted to update the preset large language model.

7. The method for executing a voice interaction task based on a large model according to claim 1, characterized in that: The training process of the preset large language model includes: Training a preset large language model based on a historically collected annotated first target dataset; and / or, fusing the collected speech data with different types of pre-collected background noise to obtain a second target data set, and using the second target data set to train the preset large language model; And / or, adjusting the speaking speed and / or pitch of the collected speech data to obtain a third target data set, and using the third target data set to train the preset large language model.

8. A large-scale model-based voice interaction task execution device, characterized in that: include: The feature acquisition module is used to perform pre-set speech signal preprocessing and feature extraction operations on the collected speech data to obtain speech features; a sequence acquisition module, configured to perform speech recognition on the speech features based on a preset speech recognition model to obtain a semantic text character sequence corresponding to the speech features, and optimize the text character sequence using a preset large language model to obtain a target text character sequence; A graph construction module is used to perform entity recognition on the target text character sequence using the preset large language model to obtain an entity recognition result, determine the relationship type between entities in the target text character sequence based on the entity recognition result, and construct a knowledge graph based on the relationship type between the entities; A triple acquisition module, configured to generate an initial triple based on the knowledge graph, and optimize the initial triple using the preset large language model to obtain a target triple; A graph updating module is used to fuse the target triples with the initial knowledge graph to obtain an updated knowledge graph; A task execution module is used to execute the voice interaction task in the target voice interaction scenario based on the updated knowledge graph.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the large model-based voice interaction task execution method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, which, when executed by a processor, implements the large model-based voice interaction task execution method as described in any one of claims 1 to 7.

Images