Interaction method, training method, device and intelligent agent based on large model

Abstract

The present disclosure provides an interaction method, training method, device and intelligent agent based on a large model, which relates to the field of artificial intelligence, especially to technical fields such as speech recognition, speech interaction, deep learning, and large models, and can be applied to application scenarios such as knowledge search, autonomous driving, intelligent customer service, intelligent voice control, smart e-commerce, and AI medical care. The interaction method based on a large model includes: obtaining a demand voice; performing speech recognition on the demand voice to obtain a speech recognition feature that represents the semantics of the demand; using the large model to process the speech recognition feature to obtain a reply text, wherein the reply text includes a plurality of reply words arranged in sequence, and the target reply word among the plurality of reply words is determined based on the speech recognition feature and the associated reply word feature processed by the attention fusion layer of the large model, and the associated reply word feature is related to the associated reply word arranged before the target reply word.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to technical fields such as speech recognition, voice interaction, deep learning, and large models, and can be applied to application scenarios such as knowledge search, autonomous driving, intelligent customer service, intelligent voice control, smart e-commerce, and AI medical care. Background Art

[0002] With the rapid development of artificial intelligence technology, in the process of human-computer interaction, multimodal information such as text, voice, and video input by users can be processed based on artificial intelligence generated content (AIGC) technology to generate the information that users need to know. Summary of the Invention

[0003] The present disclosure provides an interaction method, training method, device and intelligent agent, electronic device, storage medium and program product based on a large model.

[0004] According to one aspect of the present disclosure, an interaction method based on a large model is provided, including: acquiring a demand voice; performing voice recognition on the demand voice to obtain a voice recognition feature representing the semantics of the demand; processing the voice recognition feature using the large model to obtain a reply text, wherein the reply text includes a plurality of reply words arranged in sequence, and a target reply word among the plurality of reply words is determined based on processing the voice recognition feature and the associated reply word feature by the attention fusion layer of the large model, and the associated reply word feature is related to the associated reply word arranged before the target reply word.

[0005] According to another aspect of the present disclosure, a training method for a large model is provided, including: obtaining sample demand speech and label reply text; performing speech recognition on the sample demand speech to obtain sample speech recognition features; using the initial large model to process the sample speech recognition features to obtain sample reply text, wherein the sample reply text includes a plurality of sample reply words arranged in sequence, and a sample target reply word among the plurality of sample reply words is determined based on the initial attention fusion layer of the initial large model processing the sample speech recognition features and the sample associated reply word features, and the sample associated reply word features are related to the associated reply words arranged before the sample target reply word; training the initial large model based on the reply text difference between the sample reply text and the label reply text to obtain the trained large model.

[0006] According to another aspect of the present disclosure, an interaction device based on a large model is provided, including: a first acquisition module for acquiring a demand voice; a first speech recognition module for performing speech recognition on the demand voice to obtain speech recognition features representing the semantics of the demand; a reply text acquisition module for processing the speech recognition features using the large model to obtain a reply text, wherein the reply text includes a plurality of reply words arranged in sequence, and a target reply word among the plurality of reply words is determined based on the speech recognition features and associated reply word features processed by the attention fusion layer of the large model, and the associated reply word features are related to the associated reply words arranged before the target reply word.

[0007] According to another aspect of the present disclosure, a training device for a large model is provided, including: a second acquisition module for acquiring sample demand speech and label reply text; a second speech recognition module for performing speech recognition on the sample demand speech to obtain sample speech recognition features; a sample reply text acquisition module for processing the sample speech recognition features using the initial large model to obtain sample reply text, wherein the sample reply text includes a plurality of sample reply words arranged in sequence, and the sample target reply word among the plurality of sample reply words is determined based on the initial attention fusion layer of the initial large model processing the sample speech recognition features and the sample associated reply word features, and the sample associated reply word features are related to the associated reply words arranged before the sample target reply word; a training module for training the initial large model based on the reply text difference between the sample reply text and the label reply text to obtain the trained large model.

[0008] According to another aspect of the present disclosure, an intelligent agent is provided, including: an input module for receiving input information; a processing module for determining a target task based on the input information received by the input module, determining a big model or an initial big model based on the target task, and obtaining output information by calling the big model to execute the big model-based interaction method provided by the embodiment of the present disclosure, or by calling the initial big model to execute the big model training method provided by the embodiment of the present disclosure; and an output module for outputting the output information obtained by the processing module.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided by an embodiment of the present disclosure.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method provided by an embodiment of the present disclosure.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the method provided in the embodiment of the present disclosure when executed by a processor.

[0012] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0014] Figure 1 Schematically illustrates an exemplary system architecture to which a large model-based interaction method and apparatus according to an embodiment of the present disclosure can be applied;

[0015] Figure 2 The flowchart of the interaction method based on the large model according to the embodiment of the present disclosure is schematically shown;

[0016] Figure 3 The following schematically illustrates an application scenario diagram of an interaction method based on a large model according to an embodiment of the present disclosure;

[0017] Figure 4 Schematically illustrates an application scenario diagram of an interaction method based on a large model according to another embodiment of the present disclosure;

[0018] Figure 5A The flowchart of the training method of the large model according to the embodiment of the present disclosure is schematically shown;

[0019] Figure 5B Schematically shows a principle diagram for determining an initial large model according to an embodiment of the present disclosure;

[0020] Figure 6 The following schematically shows a principle diagram of a large model training method according to an embodiment of the present disclosure;

[0021] Figure 7 Schematically shows a block diagram of an interactive device based on a large model according to an embodiment of the present disclosure;

[0022] Figure 8 A block diagram of a large model training device according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 9 Schematically shows a structural block diagram of an intelligent agent according to an embodiment of the present disclosure; and

[0024] Figure 10A schematic block diagram of an example electronic device that can be used to implement the large model interaction method and large model training method of an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0025] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0027] To facilitate understanding of the embodiments of the present disclosure, the meanings of English abbreviations or technical terms involved in the embodiments of the present disclosure may be explained based on the following content.

[0028] Artificial Intelligence Generated Content (AIGC) is a technology that uses artificial intelligence (AI) techniques, particularly large pre-trained models, to generate relevant content with appropriate generalization capabilities by learning from existing data and identifying patterns. The core concept of AIGC is to use AI algorithms to generate content with a certain level of creativity and quality. It can generate relevant articles, images, audio, and other content based on input conditions or guidance.

[0029] Large models can include deep learning models with numerous parameters and complex structures. Large models are capable of processing massive amounts of data and performing a variety of complex tasks, such as natural language processing, computer vision, and speech recognition, based on their large number of parameters and complex structure. These models can be built using deep neural networks and have billions or even hundreds of billions of parameters. By training on massive amounts of data, they learn complex patterns and features, possessing stronger generalization capabilities and enabling accurate predictions on unseen data. Large models can include large language models (LLMs). Large language models are based on machine learning and natural language processing techniques and are trained on large amounts of text data to develop capabilities that support human language understanding and generation. Large models can have billions or even hundreds of billions of parameters, enabling them to capture more linguistic knowledge and complex grammatical structures. For example, the network architecture of large language models can employ structures such as transformers. Large language models possess strong contextual awareness when processing text, enabling them to understand and generate text content that depends on previous context, enabling more accurate understanding of text content in conversations, article generation, and contextual understanding. The large model involved in the embodiments of the present disclosure may include a large language model, or may also include other types of generative large models in addition to the large language model.

[0030] During the human-computer interaction process, users represent their actual needs based on multimodal information such as voice and video, but the AIGC model finds it difficult to accurately understand the users' actual needs, resulting in a low degree of match between the generated response information and the users' actual needs, which in turn reduces the efficiency of human-computer interaction and affects the users' actual experience.

[0031] The embodiments of the present disclosure provide an interaction method, training method, device, intelligent agent, electronic device, storage medium and program product based on a large model. The interaction method based on the large model includes: obtaining a demand voice; performing speech recognition on the demand voice to obtain speech recognition features representing the semantics of the demand; using the large model to process the speech recognition features to obtain a reply text, wherein the reply text includes a plurality of reply words arranged in sequence, and a target reply word among the plurality of reply words is determined based on the speech recognition features and associated reply word features processed by the attention fusion layer of the large model, and the associated reply word features are related to the associated reply words arranged before the target reply word.

[0032] According to the embodiments of the present disclosure, by determining the speech recognition features from the demand voice and fusing the speech recognition features and the associated reply word features based on the attention fusion layer of the large model, deep feature alignment of the generated associated reply words and speech recognition features can be achieved, thereby improving the large model's deep semantic understanding ability of the demand text and reply words expressed in the demand voice, enabling the large model to reply under the condition of deep understanding of the demand intention, avoiding hallucinations of the large model, improving the matching degree between the reply text and the demand voice, and thereby improving the accuracy of replies to the demand voice and improving the user experience.

[0033] Figure 1 An exemplary system architecture to which the large model-based interaction method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0034] It should be noted that Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present disclosure may be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not imply that the embodiments of the present disclosure may not be applied to other devices, systems, environments, or scenarios. For example, in another embodiment, an exemplary system architecture to which the large-model-based interaction method and apparatus may be applied may include a terminal device, but the terminal device may implement the large-model-based interaction method and apparatus provided in the embodiments of the present disclosure without interacting with a server.

[0035] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0036] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software (for example only).

[0037] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0038] Server 105 may be a server that provides various services, such as a background management server (for example only) that supports content browsed by users using terminal devices 101, 102, and 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal device.

[0039] The server can be a cloud server, also known as a cloud computing server or cloud host. It is a hosting product within the cloud computing service system that addresses the management difficulties and poor scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or "VPS"). The server can also be a distributed system server or a server integrated with blockchain.

[0040] It should be noted that the interaction method based on the big model provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the interaction device based on the big model provided in the embodiment of the present disclosure can generally be set in the server 105. The interaction method based on the big model provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the interaction device based on the big model provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0042] Figure 2 The flowchart of the interaction method based on the large model according to the embodiment of the present disclosure is schematically shown.

[0043] like Figure 2 As shown, the large model-based interaction method includes operations S210 to S230.

[0044] In operation S210 , a demand voice is acquired.

[0045] In operation S220 , speech recognition is performed on the demand speech to obtain speech recognition features representing the semantics of the demand.

[0046] In operation S230 , the speech recognition features are processed using the large model to obtain a reply text.

[0047] According to embodiments of the present disclosure, a demand voice may be audio information that represents a user's demand intent. For example, it may be audio information representing a question asked by a user, or it may be audio information representing control instructions for controlling devices such as household appliances and assisted driving vehicles. However, this is not limited to these and may also be audio information representing other types of demands, such as a user's need to express emotion. The embodiments of the present disclosure do not limit the specific demand type represented by the demand voice.

[0048] According to embodiments of the present disclosure, the demand semantics may include, but are not limited to, the textual semantics of the demand speech. The demand semantics may also include audio attributes of the demand speech, such as intonation, voice, and speech rate. Speech recognition features may be vectors, scalars, or other data representing the demand semantics, such as the textual semantics and audio attributes of the demand speech.

[0049] According to embodiments of the present disclosure, speech recognition of the request speech may include processing the request speech based on a speech recognition algorithm such as a recurrent neural network algorithm or a hidden Markov model algorithm. The embodiments of the present disclosure do not limit the specific method for recognizing the request speech, as long as speech recognition features that represent the semantics of the request can be obtained.

[0050] In one example, a large speech recognition model can be used to process the demand speech for speech recognition. The speech recognition features can characterize the audio attributes and text semantics of the demand words corresponding to the demand speech, so as to improve the accuracy of the speech recognition features in characterizing the demand intent of the demand speech.

[0051] In one example, a large speech recognition model can be used to process the requested speech to more accurately capture the requested words corresponding to the requested text audio information from the requested speech with ambient audio information, thereby preventing the ambient audio information from interfering with the textual semantics of the requested text. Furthermore, the large speech recognition model can also integrate semantics related to the environmental attributes represented by the ambient audio information into the speech recognition features to enhance the semantic richness of the speech recognition feature expression. Furthermore, the large model can be used to process the semantically rich speech recognition features to more precisely control the text generation task, improving the expression accuracy of the reply text and the degree of match between the reply text and the requested speech.

[0052] According to an embodiment of the present disclosure, a reply text includes a plurality of reply words arranged in sequence. A target reply word among the plurality of reply words is determined based on speech recognition features and associated reply word features processed by an attention fusion layer of a large model, wherein the associated reply word features are related to associated reply words arranged before the target reply word.

[0053] According to the embodiment of the present disclosure, the reply word can represent any type of characters such as Chinese characters, English single words, English letters, logical symbols, punctuation marks, etc. The embodiment of the present disclosure does not limit the specific character type of the reply word.

[0054] According to an embodiment of the present disclosure, the associated reply word feature can be a hidden feature that characterizes the text semantics of the associated reply word. The large model can include multiple attention fusion layers. The attention fusion layer of the large model is used to perform attention fusion on the associated reply word feature and the speech recognition feature, which can achieve deep feature fusion of the demand speech in the speech modality and the associated reply word in the text modality, avoid fusing feature information of different modalities through feature embedding, improve the large model's deep understanding ability of multimodal feature information, and further improve the matching degree between the generated reply text and the demand intention expressed by the demand speech, thereby improving the accuracy of the reply text.

[0055] It is understandable that the target reply word in the reply text can be any specified reply word, and can be generated based on the speech recognition features and the associated reply word features corresponding to the generated associated reply words processed by the attention fusion layer. In this way, the generated target reply word can be used as the generated associated reply word, and the attention fusion layer can be used to cyclically process the generated associated reply word features and speech recognition features to obtain a new target reply word until all the reply words in the reply text are generated.

[0056] According to an embodiment of the present disclosure, using a large model to process speech recognition features to obtain a reply text may include: using a text feature fusion layer to process initial associated reply word features that represent associated reply words to obtain associated reply word features; using an attention fusion layer to process speech recognition features and associated reply word features to obtain target reply word features; and determining a target reply word based on the target reply word features.

[0057] According to an embodiment of the present disclosure, the large model further includes a text feature fusion layer. The large model may include one or more feature fusion networks. The initial associated reply word feature may be feature information characterizing the associated reply word. For example, the initial associated reply word feature A1 may be related to the associated reply word "A". By performing linear regression on the initial associated reply word feature A1, the initial associated reply word feature "A" may be obtained.

[0058] It should be noted that the initial associated reply word feature can be generated by the feature fusion network of the large model when the associated reply word is used as the target reply word. For example, the target reply word "B" can be generated based on the target reply word feature "B1". The target reply word "B" is used as the associated reply word, and the target reply word feature "B1" is used as the initial associated reply word feature. The initial associated reply word feature "B1" can be processed based on the large model to obtain a new target reply word feature and the corresponding target reply word.

[0059] In one example, the feature fusion network may include a cascaded text feature fusion layer, an attention fusion layer, and a feedforward layer. The text feature fusion layer may be constructed based on a self-attention algorithm and is used to perform attention fusion on one or more initial associated reply word features based on the self-attention mechanism, so that the associated reply word features can be integrated with the reply context semantics related to the target reply word.

[0060] According to the embodiments of the present disclosure, by using the attention fusion layer to process the speech recognition features and the associated reply word features, it is possible to achieve deep feature alignment of the associated reply word features and the speech recognition features representing the semantics of the demand intent in one or more feature fusion networks of the large model, so as to improve the depth of feature fusion, and enable the large model to accurately generate the target reply word under the condition of deep understanding of the semantics of the demand intent and the semantics of the associated reply words that have been generated, thereby avoiding the loss of semantic information and misunderstandings, reducing the probability of hallucinations in the large model, and improving the degree of matching of the target reply text with the demand intent represented by the demand speech.

[0061] According to an embodiment of the present disclosure, a large model may include multiple feature fusion networks. The feature fusion network may include a cascaded text feature fusion layer, an attention fusion layer, and a feedforward layer. The text feature fusion layer may be used to process multiple associated reply word features arranged before the target reply word, so as to achieve semantic fusion of the contextual semantics represented by the associated reply word features. By performing multi-level deep fusion of speech recognition features and initial associated reply word features based on the attention fusion layer, deep feature alignment between the demand intention semantics and the associated reply words can be achieved, so as to achieve deep intercommunication and deep fusion of information of different modalities in semantic attributes such as text semantics, emotional attributes, and contextual information, so that the large model can understand the content in the demand voice during the execution of complex interactive tasks and improve the accuracy of the reply text.

[0062] According to the embodiments of the present disclosure, by using the attention fusion layers of multiple feature fusion networks to perform layer-by-layer feature fusion of speech recognition features and associated reply word features, the large model not only aligns the surface features of information from different modalities, but also captures the complex contextual relationships between the requested speech and the reply text through the multi-level interaction of multiple attention fusion layers. In this way, when processing multi-step reasoning tasks or multi-round conversations, the large model can effectively model the long-term dependencies between different modalities, further improving the accuracy of the reply text.

[0063] According to an embodiment of the present disclosure, both the text feature fusion layer and the attention fusion layer include N layers, where N is an integer greater than 1. It should be understood that the feature fusion network may include N cascaded layers.

[0064] According to an embodiment of the present disclosure, using the attention fusion layer to process speech recognition features and associated reply word features to obtain target reply word features may include: using the nth attention fusion layer to process speech recognition features and the nth associated reply word features to obtain the nth intermediate fusion feature; and when n=N, determining the target reply word feature based on the Nth intermediate fusion feature.

[0065] According to an embodiment of the present disclosure, N≥n>1, where N can be an integer. The nth associated reply word feature is determined based on the nth text feature fusion layer processing the n-1th intermediate fusion feature, and the first associated reply word feature is determined based on the first text feature fusion layer processing the initial associated reply word feature.

[0066] In one example, the attention fusion layer is built based on the Cross-Attention algorithm. The attention fusion layer uses the cross-attention mechanism to deeply cross-fuse multiple associated reply word features and speech recognition features, thereby improving the large model's ability to understand information from different modalities.

[0067] Figure 3 The application scenario diagram of the large model-based interaction method according to an embodiment of the present disclosure is schematically shown.

[0068] like Figure 3 As shown, this application scenario may include a speech recognition large model 310 and a large model 320. Large model 320 may include a cascaded first feature fusion network 321 and a second feature fusion network 322. First feature fusion network 321 includes a cascaded first text feature fusion layer, a first attention fusion layer, and a first feed-forward layer. Second feature fusion network 322 includes a cascaded second text feature fusion layer, a second attention fusion layer, and a second feed-forward layer. It can be understood that N = 2.

[0069] The demand voice 301 can be voice audio information representing the demand text "What is artificial intelligence technology". The demand voice 301 is input into the large speech recognition model 310 to perform speech recognition on the demand voice 301, and speech recognition features are output. The initial associated reply word feature F301 can be related to the four reply words "artificial intelligence" in the already generated reply text.

[0070] The initial associated reply word feature F301 is input into the first text feature fusion layer, and the first associated reply word feature is output. The first associated reply word feature and the speech recognition feature are input into the first attention fusion layer, and the first intermediate fusion feature is output. In the case of n = 2, the first intermediate fusion feature is processed by the cascaded first feed-forward layer and the second text feature fusion layer to obtain the second associated reply word feature. The second associated reply word feature and the speech recognition feature are input into the second attention fusion layer, and the second intermediate fusion feature is output. The second intermediate fusion feature is input into the second feed-forward layer, and the target reply word feature F3011 is output. The target reply word feature F3011 can represent the target reply word "technique".

[0071] It should be understood that for the target reply word "technique", the five reply words "artificial intelligence technique" in the reply text 302 can be used as the associated reply words, and the associated reply word features related to "artificial intelligence technique" are processed based on the interaction method of the technical large model provided in the embodiments of the present disclosure to obtain new target reply words. Until all the reply words are generated in sequential order, the reply text 302 is obtained.

[0072] According to the embodiments of the present disclosure, performing speech recognition on the demand voice to obtain speech recognition features representing the demand semantics may include: extracting features from the demand voice to obtain initial speech features; decoding the initial speech features to obtain multiple initial decoding features; and fusing the multiple initial decoding features and the initial speech features based on an attention mechanism to obtain speech recognition features.

[0073] According to the embodiments of the present disclosure, the initial decoding features represent the recognition results of the demand words in the demand voice. For example, the initial decoding features may be the first speech recognition results corresponding to the demand words, and the multiple initial decoding features correspond one by one to the multiple demand words represented by the demand voice.

[0074] According to the embodiments of the present disclosure, any type of attention network algorithm such as Transformer can be used to process the initial decoding features and the initial speech features to obtain speech recognition features.

[0075] According to the embodiments of the present disclosure, the attention mechanism fuses multiple initial decoding features and initial speech features, so that the speech recognition features can include the text semantics of the required words corresponding to the speech segment of the required speech, as well as various types of speech attributes such as speaking speed, intonation, and emotional attributes represented by the required speech, so that the speech recognition features can accurately represent the text semantics of the required text while also including rich speech attributes, thereby improving the expression accuracy of the speech recognition features for the required intention.

[0076] According to an embodiment of the present disclosure, fusing multiple initial decoding features and initial speech features based on an attention mechanism to obtain speech recognition features may include: fusing the initial decoding features and the initial speech features based on an attention mechanism to obtain audio features of the demand word corresponding to the demand word; performing global feature fusion on multiple audio features of the demand word to obtain intermediate speech features; and fusing the intermediate speech features and multiple initial decoding features based on an attention mechanism to obtain speech recognition features.

[0077] According to an embodiment of the present disclosure, by fusing initial decoding features and initial speech features based on an attention mechanism, the initial decoding features corresponding to each demand word can be used as prior information, and the initial decoding features corresponding to each demand word can be fully fused with the audio clip corresponding to the initial speech features, so that the audio features of the demand words can more accurately represent the textual semantics and speech attributes of the audio clip corresponding to the demand words.

[0078] According to an embodiment of the present disclosure, the global feature fusion of multiple audio features of the demand words may include fusing multiple audio features of the demand words based on an attention network algorithm, and the obtained intermediate speech features may include intermediate speech word features corresponding to multiple demand words. From the audio information of the demand speech at the frame level, the speech feature information of unequal lengths is extracted to obtain the uniform audio features of equal length at the word level as the initial speech feature of the demand speech. Using multiple initial decoding features as a priori to fuse the initial speech features and multiple initial decoding features can solve the problem of inconsistent feature representation lengths of speech framing and improve the accuracy of speech recognition.

[0079] According to an embodiment of the present disclosure, the intermediate speech features and multiple initial decoding features are fused based on the attention mechanism to obtain speech recognition features, which may include using the initial decoding features as query features and the intermediate speech features as key features and value features of the attention algorithm. The speech recognition features are obtained by fusing the query features, key features and value features based on the attention algorithm. The speech recognition features can represent the text semantics and speech attributes represented by the speech audio segment corresponding to the demand word. By inputting the speech recognition features into multiple attention fusion layers of the large model to achieve multimodal feature fusion, the demand speech in the audio mode can be directly calculated with the associated reply word features after the modal conversion, avoiding the modal conversion calculations such as feature embedding on the demand speech in each network structure of the large model, and realizing deep calculations between different modal information. In this way, the speech recognition large model and the multi-level attention fusion layer can be used to more efficiently align the information represented by the speech modality and the text modality, thereby improving the response speed of the large model to the demand speech.

[0080] Figure 4 The following schematically illustrates an application scenario diagram of an interaction method based on a large model according to another embodiment of the present disclosure.

[0081] like Figure 4 As shown, this application scenario may include a speech recognition large model and a large model 430. The speech recognition large model may include a first speech encoder 411, a first speech decoder 412, a speech attention fusion layer 413, a second speech encoder 421, and a second speech decoder 422. The large model 430 includes N feature fusion networks 431. The feature fusion network 431 may include a cascade of text feature fusion layers, attention fusion layers, and feedforward layers.

[0082] The demand speech 401 is input into the first speech encoder 411, which performs feature extraction on the demand speech 401 and obtains an initial speech feature F401. The initial speech feature F401 may include multiple initial speech sub-features arranged in sequence, wherein the multiple initial speech sub-features are h1, h2, h3, ... hk. The multiple initial speech sub-features correspond to the demand word in the demand speech 401. The initial speech feature F401 is input into the first speech decoder 412 to obtain an initial decoded feature F402 representing the recognition result of the demand word. The initial speech feature F401 and the initial decoded feature F402 are input into the speech attention fusion layer 413 for attention fusion to obtain the demand word audio feature corresponding to the demand word. The multiple demand word audio features corresponding to the demand word are input into the second speech encoder 421 for global feature fusion to obtain an intermediate speech feature F403. The multiple intermediate speech sub-features in the intermediate speech feature F403 are u1, u2, u3, ... uk.

[0083] The fusion of intermediate speech features and multiple initial decoding features based on the attention mechanism may include inputting the initial decoding features F402 and the intermediate speech features F403 into the second speech decoder 422 and outputting speech recognition features. The second speech decoder 422 may, for example, include a forward decoder 4221 and a reverse decoder 4222. The forward decoder 4221 may, for example, decode the initial decoding features F402 and the intermediate speech features F403 based on the forward arrangement direction of the required words in the required text to obtain forward speech recognition features. The reverse decoder 4222 may, for example, decode the initial decoding features F402 and the intermediate speech features F403 based on the reverse arrangement direction of the required words in the required text to obtain reverse speech recognition features. By fusing the forward speech recognition features and the reverse speech recognition features based on the attention weight, speech recognition features may be obtained.

[0084] like Figure 4 As shown, the speech recognition features can be input into the attention fusion layer in multiple feature fusion networks in the large model 430, so that the text feature fusion layer and the attention fusion layer can perform multi-level deep fusion based on the initial associated reply word features and speech recognition features to obtain the reply text 402.

[0085] According to the embodiments of the present disclosure, in the process of multimodal interaction, the demand speech is processed by a large speech recognition model to obtain speech recognition features, and the attention network layer of the large model is used to deeply fuse the associated reply word features of different modalities with the speech recognition features, thereby avoiding the large computational overhead caused by the large model repeatedly performing multimodal feature alignment of speech-text embedding features and associated reply word features in the text feature fusion layer constructed based on the self-attention mechanism, thereby reducing the device computing cost of the interactive behavior and improving the computing efficiency of the device during the multimodal information interaction process.

[0086] Figure 5A The flowchart of the training method of the large model according to the embodiment of the present disclosure is schematically shown.

[0087] like Figure 5A As shown, the training method of the large model includes operations S510 to S540.

[0088] In operation S510 , a sample demand voice and a tag reply text are obtained.

[0089] In operation S520, speech recognition is performed on the sample demand speech to obtain a sample speech recognition feature.

[0090] In operation S530 , the sample speech recognition features are processed using the initial large model to obtain a sample reply text.

[0091] In operation S540 , an initial large model is trained based on the reply text difference between the sample reply text and the label reply text to obtain a trained large model.

[0092] According to embodiments of the present disclosure, the sample demand speech can be audio information representing the sample demand text. The sample demand speech can have any type of voice attributes, such as a specified speech rate, emotion, gender, etc. The embodiments of the present disclosure do not limit the specific voice attributes of the sample demand speech. The tag reply text can be text information used to reply to the sample demand speech.

[0093] According to an embodiment of the present disclosure, performing speech recognition on the sample demand speech may include processing the sample demand speech based on any type of speech recognition algorithm. The embodiment of the present disclosure does not limit the specific algorithm type for performing speech recognition.

[0094] According to an embodiment of the present disclosure, the sample reply text includes multiple sample reply words arranged in sequence, and the sample target reply word among the multiple sample reply words is determined based on the initial attention fusion layer of the initial large model processing the sample speech recognition features and the sample associated reply word features, and the sample associated reply word features are related to the associated reply words arranged before the sample target reply word.

[0095] The trained big model in the training method provided by the embodiment of the present disclosure can be applied to the big model-based interaction method provided by the embodiment of the present disclosure. For example, the speech recognition features corresponding to the demand voice can be processed based on the trained big model to obtain a reply text. The technical terms involved in the big model-based interaction method provided by the embodiment of the present disclosure, including but not limited to demand voice, speech recognition features, etc., have the same or corresponding meanings as the technical terms involved in the big model training method provided by the embodiment of the present disclosure, including but not limited to sample demand voice, sample speech recognition features, etc., and the embodiments of the present disclosure will not be repeated here.

[0096] According to an embodiment of the present disclosure, the reply text difference can represent the difference information between the sample reply text and the labeled reply text. Training the initial large model based on the reply text difference between the sample reply text and the labeled reply text can include determining a reply text loss value based on the reply text difference between the sample reply text and the labeled reply text. The reply text loss value is used to adjust the model parameters of the initial large model until the reply text loss value converges, thereby obtaining a trained large model.

[0097] According to an embodiment of the present disclosure, the sample speech recognition features are determined by processing the sample demand speech using a speech recognition large model. The speech recognition large model can be a speech modality large model constructed based on an attention network algorithm.

[0098] According to an embodiment of the present disclosure, training an initial large model based on the reply text difference between the sample reply text and the label reply text, and obtaining the trained large model may include: determining the reply text loss value based on the reply text difference between the sample reply text and the label reply text; and adjusting the model parameters of the speech recognition large model and the initial large model based on the reply text loss value and the demand feature loss value, to obtain the trained large model.

[0099] According to an embodiment of the present disclosure, the demand feature loss value characterizes the difference between the sample speech recognition feature and the preset label demand text feature. The label demand text feature can be obtained by processing the sample demand text using a preset large model. Using the preset large model as a text encoder to process the sample demand text can enable the obtained label demand text feature to fuse the text features between the sample demand words of the sample demand text and extract the demand intention semantics of the sample demand text. In this way, by using the difference between the label demand text feature and the sample speech recognition feature to determine the demand feature loss value, the demand feature loss value can more accurately represent the demand intention understanding ability of the speech recognition large model for the sample demand speech of the speech modality, so that the model parameters of the speech recognition large model and the initial large model can be adjusted based on the reply text loss value and the demand feature loss value, so that the expression ability of the speech recognition large model for the sample demand speech of different speech attributes can be more accurately adjusted during the training process, and the large model's understanding ability of the text semantic attributes and sample speech attributes expressed by the sample demand speech can be improved, thereby improving the understanding of the demand intention of the demand speech and outputting a more accurate reply text.

[0100] In one example, adjusting the model parameters of the speech recognition large model and the initial large model based on the reply text loss value and the required feature loss value may include determining a joint loss value based on the reply text loss value and the required feature loss value, and adjusting the model parameters of the speech recognition large model and the initial large model based on the joint loss value.

[0101] In one example, adjusting the model parameters of the speech recognition large model and the initial large model based on the reply text loss value and the demand feature loss value may include: in a first stage, adjusting the model parameters of the speech recognition large model based on the demand feature loss value until the demand feature loss value converges to obtain a trained speech recognition large model; and in a second stage, adjusting the model parameters of the initial large model based on the reply text loss value until the reply text loss value converges to obtain a trained large model.

[0102] According to an embodiment of the present disclosure, the sample demand speech includes sample demand text audio information representing the sample demand text, and sample environment audio information representing the speech environment sound.

[0103] According to an embodiment of the present disclosure, the sample demand text audio information can be understood as the sound audio corresponding to the sample demand words of the sample demand text, such as the sound audio segments corresponding to multiple sample demand words. The sample environment audio information can be understood as the background audio that generates the sample demand text audio information.

[0104] In one example, a large speech recognition model can be used to process sample demand speech to more accurately capture the audio information of the sample demand text from the sample demand speech with sample environmental audio information, thereby preventing the sample environmental audio information from interfering with the textual semantics of the sample demand text. Furthermore, the large speech recognition model can also integrate semantics related to environmental attributes into speech recognition features by identifying environmental attributes represented by the sample environmental audio information, thereby enhancing the semantic richness of the speech recognition feature expression.

[0105] According to an embodiment of the present disclosure, the sample demand speech includes a plurality of sample demand speech, and the plurality of sample demand speech have different sample speech attributes.

[0106] According to an embodiment of the present disclosure, the sample speech attribute includes at least one of the following: timbre attribute, speaking speed attribute, gender attribute, and accent attribute.

[0107] According to embodiments of the present disclosure, the timbre attribute can represent speech audio with different timbres. The speech rate attribute can represent the number of sample demand words represented by the sample demand speech within a preset duration. The gender attribute can represent the gender of the target audience of the sample demand speech. The accent attribute can be understood as the pronunciation style of the sample demand speech related to the accent.

[0108] According to the embodiments of the present disclosure, by utilizing a large speech recognition model to process sample demand speech with different sample speech attributes, the sample speech recognition features can be made to include diverse sample speech attribute semantics, so that the trained large model can learn sample speech recognition features with different sample speech attribute semantics, and combine the diverse sample speech attribute semantics and sample demand text semantics to perform text reply tasks, so that the large model can accurately capture the demand intention represented by the sample demand speech, so as to improve the matching degree between the sample reply text and the sample demand intention, and thereby improve the accuracy of the sample reply text.

[0109] According to embodiments of the present disclosure, the features of a tag requirement text are determined by processing a sample requirement text using a preset macromodel. The preset macromodel can be a general-purpose base macromodel or a specialized macromodel fine-tuned for a specific application scenario. Embodiments of the present disclosure do not limit the specific type of the preset macromodel.

[0110] According to embodiments of the present disclosure, the sample demand speech is determined based on the sample demand text. The sample demand text can be processed using a speech synthesis algorithm such as a Hidden Markov Model (HMM) to generate the sample demand speech. The sample demand speech can have one or more sample speech attributes.

[0111] According to the embodiments of the present disclosure, the text feature fusion layer of the large model is used to fuse the text semantics between the initial reply word features, and the attention fusion layer is used to perform deep feature alignment of the associated reply word features and speech recognition features of different modalities. This can improve the performance of the large model for multimodal information and reduce computing pressure. In addition, by performing deep feature alignment of feature information of different modalities, the large model can capture more accurate semantics, emotions, and contextual relationships between multimodal feature information, thereby fully utilizing the advantages of large-scale data training.

[0112] According to an embodiment of the present disclosure, the initial large model is determined by training the extended large model, and the extended large model is obtained by updating the network structure of the pre-trained basic large model based on the extended attention fusion layer.

[0113] According to the embodiments of the present disclosure, the basic big model may include a general base big model, or may also include a special big model that matches a specified scenario or task type. The embodiments of the present disclosure do not limit the specific type of the basic big model, as long as it is a big model that has undergone a pre-training process. By updating the extended big model obtained by the basic big model based on the extended attention fusion layer, the basic model parameters and model structure of the basic big model can be reused, reducing the construction difficulty and operational complexity of the subsequent construction of the initial big model. By training the extended big model to obtain the initial big model, the initial big model can have a model structure that deeply aligns the feature information of different modalities, while reducing the time required to perform model parameter adjustments on the initial big model during training, thereby improving training efficiency.

[0114] It should be noted that the large model or initial large model involved in the embodiments of the present disclosure may be a large model for processing text modal information. For example, the large model and the initial large model may be a large language model and an initial large language model. The speech recognition large model involved in the embodiments of the present disclosure may be a large model for processing audio modal information, and the embodiments of the present disclosure will not be repeated here.

[0115] According to an embodiment of the present disclosure, the basic large model includes a multi-level basic feature fusion network, and the basic feature fusion network includes a cascaded basic text feature fusion layer and a basic feedforward layer.

[0116] According to an embodiment of the present disclosure, the extended large model includes a multi-stage extended feature fusion network, which includes a cascaded basic text feature fusion layer, an extended attention fusion layer, and a basic feedforward layer. The extended large model can be constructed by adding an extended attention fusion layer between the basic text feature fusion layer and the basic feedforward layer of the basic large model.

[0117] According to an embodiment of the present disclosure, the initial big model can be determined based on the following training operations: obtaining an extended big model and a pre-trained basic big model, as well as a sample extended demand text and a labeled extended reply text; using the basic big model to process the sample extended demand text to obtain the sample extended demand text features; using the extended big model to process the sample extended demand text features to obtain the sample extended reply text; and based on the difference between the sample extended reply text and the labeled extended reply text, training the extended big model to obtain the initial big model.

[0118] According to an embodiment of the present disclosure, the sample extended demand text and the tag extended reply text may include question-answer pairs extracted from real user interaction scenarios, so that the tag extended reply text can more accurately respond to the demand intention of the sample extended demand text.

[0119] According to an embodiment of the present disclosure, the difference between the sample expanded reply text and the labeled expanded reply text can be characterized by a loss value obtained by processing the sample expanded reply text and the labeled expanded reply text based on a loss function. One or more model parameters of the expanded large model are adjusted based on the loss value between the loss values ​​obtained from the sample expanded reply text and the labeled expanded reply text until the loss value converges, thereby obtaining a trained initial large model.

[0120] Figure 5B The schematic diagram shows a principle diagram of determining an initial large model according to an embodiment of the present disclosure.

[0121] like Figure 5B As shown, the basic large model 510 may include N basic feature fusion networks 511, which include a cascaded basic text feature fusion layer 5111 and a basic feedforward layer 5112. The extended large model 520 may include N extended feature fusion networks 521, which include a cascaded basic text feature fusion layer 5111, an extended attention fusion layer 5211, and a basic feedforward layer 5112. The initial large model is trained based on the following operations.

[0122] Sample extended requirement text 501 is input into the basic large model 510, which outputs sample extended requirement text features F501. The N basic feature fusion networks 511 of the basic large model 510 can act as a text encoder to perform feature extraction and text semantic fusion on the sample extended requirement text 501, so that the sample extended requirement text features F501 more accurately represent the text semantics of each word in the sample extended requirement text 501 and the overall text semantics of the sample extended requirement text 501.

[0123] Inputting the sample extended demand text feature F501 into the extended attention fusion layer 5211 of each of the N extended feature fusion networks 521 of the extended large model 520 enables the N extended feature fusion networks 521 to learn the textual semantics of the sample extended demand text 501 and the textual semantics expressed by the sample extended reply text 502. The model parameters of the extended large model 520 are adjusted by the loss value between the labeled extended reply text and the sample extended reply text until the loss value converges, thereby obtaining a trained initial large model, thereby achieving pre-training of the initial large model using text modality information.

[0124] According to the embodiments of the present disclosure, after preliminary pre-training of the extended large model with sample extended demand text and label extended reply text in pure text mode, the early pre-training can be performed on the basis of the preliminary text understanding ability of the basic large model, so that the dialogue interaction ability of the trained initial large model can be consistent with the general basic large model ability, thereby improving the generalization ability of the initial attention fusion layer for different types of text understanding, reducing the training cycle of the initial large model, improving the training efficiency, and at the same time improving the text generation task execution ability of the initial large model obtained after training to respond to the demand intention of the sample extended demand text.

[0125] Figure 6 The principle diagram of the training method of the large model according to the embodiment of the present disclosure is schematically shown.

[0126] like Figure 6 As shown, the sample demand text 601 is input into the speech synthesis model 610, which outputs the sample demand speech 602 with the sample speech attributes. The sample demand speech 602 is input into the speech recognition model 620, which outputs the sample speech recognition feature F602. The speech synthesis model 610 can be constructed based on the speech synthesis algorithm.

[0127] The sample requirement text 601 is input into a preset large model 630, which outputs a label requirement text feature F601. The preset large model 630 may include N cascaded preset feature fusion networks 631, which may include a cascaded preset text feature fusion layer and a preset feedforward layer. Using the preset large model 630 as a text encoder allows the label requirement text feature F601 to more accurately represent the textual semantics of the sample requirement text 601.

[0128] The initial large model 640 includes N cascaded initial feature fusion networks 641. The initial feature fusion network 641 can have a cascaded initial text feature fusion layer, an initial attention fusion layer, and an initial feedforward layer. Inputting the sample speech recognition feature F602 into the initial attention fusion layer of each of the N initial feature fusion networks 641 can enable the initial large model to deeply align the sample speech recognition features of different modalities and the sample associated reply word features in the sample reply text 603 until all the sample target reply words are generated, thereby obtaining the sample reply text 603.

[0129] The sample reply text 603 and the labeled reply text 604 are processed based on the loss function to obtain a reply text loss value. The sample speech recognition features F602 and the labeled requirement text features F601 are processed based on the loss function to obtain a requirement feature loss value. A joint loss value is determined based on the reply text loss value and the requirement feature loss value. The model parameters of the speech recognition large model and the initial large model are adjusted based on the joint loss value. Once the joint loss value converges, the trained large model is obtained.

[0130] According to the embodiments of the present disclosure, the initial large model is trained based on the reply text loss value and the demand feature loss value, which can realize the multimodal loss value generated based on different modal information to ensure the feature alignment capability of the large model in the deep integration of different modal information, and realize the ability to understand information of different modalities, so that the large model can reach a convergence state more stably.

[0131] According to the embodiments of the present disclosure, by performing speech synthesis on the sample demand text to generate the sample demand speech, the sample demand speech can be made to represent the demand intention expressed by the user in different scenarios, so as to improve the authenticity of the sample demand speech and improve the accuracy of the sample speech recognition feature in representing the demand intention. By using the initial attention fusion layer to deeply fuse the sample speech recognition features with the sample associated reply word features, the trained large model can be adapted to diverse speech environments or user expressions such as noise and noisy environments, thereby improving the robustness and generalization ability of the large model in the interaction process. By generating the sample demand speech based on speech synthesis and combining it with real sample demand speech to construct training data, the convergence speed of the model parameters for the large model can be accelerated, and redundant computing tasks can be reduced, thereby reducing the overall training cost.

[0132] Figure 7 A block diagram of an interaction device based on a large model according to an embodiment of the present disclosure is schematically shown.

[0133] like Figure 7 As shown, the large model-based interaction device 700 includes: a first acquisition module 710 , a first speech recognition module 720 and a reply text acquisition module 730 .

[0134] The first acquisition module 710 is configured to acquire the required voice.

[0135] The first speech recognition module 720 is configured to perform speech recognition on the demand speech to obtain speech recognition features representing the semantics of the demand.

[0136] The reply text acquisition module 730 is used to use the large model to process the speech recognition features to obtain the reply text, wherein the reply text includes multiple reply words arranged in sequence, and the target reply word among the multiple reply words is determined based on the speech recognition features and the associated reply word features processed by the attention fusion layer of the large model, and the associated reply word features are related to the associated reply words arranged before the target reply word.

[0137] According to an embodiment of the present disclosure, the reply text obtaining module 730 includes: an associated reply word feature obtaining submodule, a target reply word feature obtaining submodule, and a target reply word obtaining submodule.

[0138] The associated reply word feature acquisition submodule is used to use the text feature fusion layer to process the initial associated reply word features that represent the associated reply words to obtain the associated reply word features. The large model also includes a text feature fusion layer.

[0139] The target reply word feature acquisition submodule is used to use the attention fusion layer to process the speech recognition features and the associated reply word features to obtain the target reply word features.

[0140] The target reply word obtaining submodule is used to determine the target reply word based on the target reply word features.

[0141] According to an embodiment of the present disclosure, the text feature fusion layer and the attention fusion layer each include N layers, where N is an integer greater than 1.

[0142] According to an embodiment of the present disclosure, the target reply word feature obtaining submodule includes: a first obtaining unit and a target reply word feature determining unit.

[0143] The first acquisition unit is used to use the nth attention fusion layer to process the speech recognition features and the nth associated reply word features to obtain the nth intermediate fusion feature, N≥n>1, the nth associated reply word feature is determined based on the nth text feature fusion layer processing the n-1th intermediate fusion feature, and the 1st associated reply word feature is determined by using the first text feature fusion layer to process the initial associated reply word feature.

[0144] The target reply word feature determination unit is used to determine the target reply word feature based on the Nth intermediate fusion feature when n=N.

[0145] According to an embodiment of the present disclosure, the first speech recognition module 720 includes: an initial speech feature acquisition submodule, a first decoding submodule, and a speech recognition feature acquisition submodule.

[0146] The initial speech feature acquisition submodule is used to extract features from the required speech to obtain initial speech features.

[0147] The first decoding submodule is used to decode the initial speech features to obtain a plurality of initial decoding features, where the initial decoding features represent the required words in the required speech.

[0148] The speech recognition feature acquisition submodule is used to fuse multiple initial decoding features and initial speech features based on the attention mechanism to obtain speech recognition features.

[0149] According to an embodiment of the present disclosure, the speech recognition feature acquisition submodule includes: a demand word audio feature acquisition unit, an intermediate speech feature acquisition unit, and a speech recognition feature acquisition unit.

[0150] The required word audio feature acquisition unit is used to fuse the initial decoding features and the initial speech features based on the attention mechanism to obtain the required word audio features corresponding to the required word.

[0151] The intermediate speech feature acquisition unit is used to perform global feature fusion on multiple required word audio features to obtain intermediate speech features.

[0152] The speech recognition feature acquisition unit is used to fuse intermediate speech features and multiple initial decoding features based on the attention mechanism to obtain speech recognition features.

[0153] Figure 8 A block diagram of a large model training device according to an embodiment of the present disclosure is schematically shown.

[0154] like Figure 8 As shown, the large model training device 800 includes: a second acquisition module 810, a second speech recognition module 820, a sample reply text acquisition module 830 and a training module 840.

[0155] The second acquisition module 810 is used to acquire sample demand voice and label reply text.

[0156] The second speech recognition module 820 is used to perform speech recognition on the sample demand speech to obtain the sample speech recognition features.

[0157] The sample reply text acquisition module 830 is used to use the initial large model to process the sample speech recognition features to obtain the sample reply text, wherein the sample reply text includes multiple sample reply words arranged in sequence, and the sample target reply word among the multiple sample reply words is determined based on the initial attention fusion layer of the initial large model to process the sample speech recognition features and the sample associated reply word features, and the sample associated reply word features are related to the associated reply words arranged before the sample target reply word.

[0158] The training module 840 is used to train the initial large model based on the reply text difference between the sample reply text and the label reply text to obtain a trained large model.

[0159] According to an embodiment of the present disclosure, the initial large model is determined by training the extended large model, and the extended large model is obtained by updating the network structure of the pre-trained basic large model based on the extended attention fusion layer.

[0160] According to an embodiment of the present disclosure, the basic large model includes a multi-level basic feature fusion network, which includes a cascaded basic text feature fusion layer and a basic feedforward layer; the extended large model includes a multi-level extended feature fusion network, which includes a cascaded basic text feature fusion layer, an extended attention fusion layer and a basic feedforward layer.

[0161] According to an embodiment of the present disclosure, the initial big model is determined based on the following training operations: obtaining an extended big model and a pre-trained basic big model, as well as a sample extended demand text and a labeled extended reply text; using the basic big model to process the sample extended demand text to obtain the sample extended demand text features; using the extended big model to process the sample extended demand text features to obtain the sample extended reply text; and based on the difference between the sample extended reply text and the labeled extended reply text, training the extended big model to obtain the initial big model.

[0162] According to an embodiment of the present disclosure, the sample speech recognition features are determined by processing the sample demand speech using a large speech recognition model.

[0163] The training module includes: a reply text loss value determination submodule and a training submodule.

[0164] The reply text loss value determination submodule is used to determine the reply text loss value based on the reply text difference between the sample reply text and the label reply text.

[0165] The training submodule is used to adjust the model parameters of the speech recognition large model and the initial large model based on the reply text loss value and the demand feature loss value to obtain the trained large model, wherein the demand feature loss value represents the difference between the sample speech recognition feature and the preset label demand text feature.

[0166] According to an embodiment of the present disclosure, the label requirement text features are determined by processing the sample requirement text using a preset large model, and the sample requirement speech is determined based on the sample requirement text.

[0167] According to an embodiment of the present disclosure, the sample demand speech includes sample demand text audio information representing the sample demand text, and sample environment audio information representing the speech environment sound.

[0168] According to an embodiment of the present disclosure, the sample demand speech includes multiple sample demand speech, and the multiple sample demand speech has different sample speech attributes, and the sample speech attribute includes at least one of the following: timbre attribute, speaking speed attribute, gender attribute, and accent attribute.

[0169] Figure 9 The structural block diagram of an intelligent agent according to an embodiment of the present disclosure is schematically shown.

[0170] In the embodiments of the present disclosure, Figure 9 As shown, the AI ​​agent 900 may include an input module 910 , a processing module 920 and an output module 930 .

[0171] Input module 910, for receiving input information;

[0172] Processing module 920, configured to determine a target task based on input information received by the input module, determine a large model based on the target task, and obtain output information by invoking the large model to execute the large model-based interaction method provided according to an embodiment of the present disclosure, or by invoking the large model to execute the large model training method provided according to an embodiment of the present disclosure;

[0173] The output module 930 is used to output the output information obtained by the processing module.

[0174] According to an embodiment of the present disclosure, the input module 910 is responsible for receiving or perceiving information such as queries, requests, instructions, signals, or data from the outside world (e.g., a user or the external environment) and converting it into a format that can be understood and processed by the AI ​​agent 900. The input module 910 is the primary link for the AI ​​agent 900 to interact with the outside world. It enables the AI ​​agent 900 to efficiently and accurately obtain the necessary "sensory" information from the outside world and respond to this information.

[0175] In an example, the input module 910 may input the demand speech or sample demand speech, sample demand speech features, demand speech features, etc. described above.

[0176] In the example, the processing module 920 is the core support for the AI ​​agent 900 to handle complex tasks. The processing module 920 can execute the large model-based interaction method and large model training method described above.

[0177] In this example, the performance of processing module 920 may be closely related to the large model underlying AI agent 900. To fully leverage the capabilities of the large model, the internal structure of processing module 920 may be designed to be highly configurable and extensible to handle a variety of different tasks and requirements in real-world scenarios.

[0178] In the example, after the AI ​​agent 900 obtains the required voice, the processing module 920 can use the speech recognition large model to process the required voice to obtain speech recognition features. The large model processes the speech recognition features to obtain a reply text, and passes the reply text to the output module 930.

[0179] Understandably, while the large model possesses excellent language understanding and generation capabilities, like humans, it can only perform limited tasks without the aid of tools. However, once AI Agent 900 is empowered with tool-based capabilities, it can perform tasks such as mathematical calculations using a calculator, data analysis using Python, and weather forecasting using search engines.

[0180] In an example, the output module 930 may output the reply text or the trained large model described above.

[0181] The AI ​​agent 900 according to the embodiment of the present disclosure can simply and effectively improve the level of intelligence, and enhance flexibility and versatility.

[0182] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0183] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0184] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described above.

[0185] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method described above.

[0186] Figure 10 A schematic block diagram of an example electronic device that can be used to implement the interactive method of the large model and the training method of the large model of an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0187] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. RAM 1003 may also store various programs and data required for the operation of device 1000. Computing unit 1001, ROM 1002, and RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to bus 1004.

[0188] Various components in device 1000 are connected to I / O interface 1005, including an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, optical disk, etc.; and a communication unit 1009, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0189] The computing unit 1001 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the large-model-based interaction method and the large-model training method. For example, in some embodiments, the large-model-based interaction method and the large-model training method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the large-model-based interaction method and the large-model training method described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute a large model-based interaction method or a large model training method in any other appropriate manner (eg, by means of firmware).

[0190] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0191] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0192] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0193] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0194] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0195] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0196] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0197] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. An interactive method based on a large model, comprising: Get demand voice; Performing speech recognition on the demand speech to obtain speech recognition features representing the demand semantics; The speech recognition features are processed using a large model to obtain a reply text, wherein the reply text includes multiple reply words arranged in sequence, and a target reply word among the multiple reply words is determined based on the speech recognition features and associated reply word features processed by the attention fusion layer of the large model, and the associated reply word features are related to the associated reply words arranged before the target reply word.

2. The method according to claim 1, wherein The method of using a large model to process the speech recognition features to obtain a reply text includes: Using a text feature fusion layer to process initial associated reply word features representing the associated reply word to obtain the associated reply word features, the large model including the text feature fusion layer; Processing the speech recognition features and the associated reply word features using the attention fusion layer to obtain target reply word features; and The target reply word is determined based on the target reply word feature.

3. The method according to claim 2, wherein: The text feature fusion layer and the attention fusion layer each include N layers, where N is an integer greater than 1; The step of processing the speech recognition features and the associated reply word features using the attention fusion layer to obtain target reply word features includes: Processing the speech recognition feature and the nth associated reply word feature using an nth attention fusion layer to obtain an nth intermediate fusion feature, where N≥n>1, the nth associated reply word feature is determined based on the nth text feature fusion layer processing the n-1th intermediate fusion feature, and the first associated reply word feature is determined by processing the initial associated reply word feature using the first text feature fusion layer; and In the case of n=N, the target reply word feature is determined based on the Nth intermediate fusion feature.

4. The method according to claim 1, wherein The performing speech recognition on the demand speech to obtain speech recognition features representing the demand semantics includes: Extracting features of the required speech to obtain initial speech features; Decoding the initial speech features to obtain a plurality of initial decoding features, wherein the initial decoding features represent the required words in the required speech; The speech recognition feature is obtained by fusing a plurality of the initial decoding features and the initial speech features based on an attention mechanism.

5. The method according to claim 4, wherein The speech recognition feature obtained by fusing the plurality of the initial decoding features and the initial speech features based on the attention mechanism includes: fusing the initial decoding features and the initial speech features based on an attention mechanism to obtain an audio feature of the required word corresponding to the required word; Performing global feature fusion on the audio features of the plurality of demand words to obtain an intermediate speech feature; The intermediate speech feature and the plurality of initial decoding features are fused based on an attention mechanism to obtain the speech recognition feature.

6. A large model training method comprising: Get sample demand voice and label reply text; Performing speech recognition on the sample demand speech to obtain sample speech recognition features; Processing the sample speech recognition features using the initial large model to obtain a sample reply text, wherein the sample reply text includes a plurality of sample reply words arranged in sequence, a sample target reply word among the plurality of sample reply words is determined based on processing the sample speech recognition features and sample associated reply word features by an initial attention fusion layer of the initial large model, and the sample associated reply word features are related to associated reply words arranged before the sample target reply word; The initial large model is trained based on the reply text difference between the sample reply text and the label reply text to obtain a trained large model.

7. The method according to claim 6, wherein: The initial large model is determined by training the extended large model, and the extended large model is obtained by updating the network structure of the pre-trained basic large model based on the extended attention fusion layer.

8. The method according to claim 7, wherein: The basic large model includes a multi-level basic feature fusion network, and the basic feature fusion network includes a cascaded basic text feature fusion layer and a basic feedforward layer; The extended large model includes a multi-stage extended feature fusion network, and the extended feature fusion network includes a cascaded basic text feature fusion layer, an extended attention fusion layer and a basic feedforward layer.

9. The method according to claim 7 or 8, wherein The initial large model is determined based on the following training operations: Obtain the extended large model and pre-trained basic large model, as well as sample extended demand text and labeled extended response text; Processing the sample extended requirement text using the basic large model to obtain features of the sample extended requirement text; Processing the features of the sample extended demand text using the extended large model to obtain a sample extended reply text; and Based on the difference between the sample extended reply text and the label extended reply text, the extended large model is trained to obtain the initial large model.

10. The method according to claim 6, wherein: The sample speech recognition feature is determined by processing the sample demand speech using a speech recognition large model; The training of the initial large model based on the reply text difference between the sample reply text and the label reply text to obtain the trained large model includes: Determining a reply text loss value based on a reply text difference between the sample reply text and the label reply text; and Based on the reply text loss value and the requirement feature loss value, the model parameters of the speech recognition large model and the initial large model are adjusted to obtain the trained large model, wherein the requirement feature loss value represents the difference between the sample speech recognition feature and the preset label requirement text feature.

11. The method according to claim 10, wherein: The label requirement text features are determined by processing the sample requirement text using a preset large model, and the sample requirement speech is determined based on the sample requirement text.

12. The method according to claim 11, wherein The sample demand speech includes sample demand text audio information representing the sample demand text and sample environment audio information representing speech environment sound.

13. The method according to claim 11, wherein The sample demand speech includes a plurality of samples, each of which has different sample speech attributes, and the sample speech attribute includes at least one of the following: Tone attributes, speaking speed attributes, gender attributes, and accent attributes.

14. An interactive device based on a large model, comprising: The first acquisition module is used to acquire the required voice; A first speech recognition module is used to perform speech recognition on the demand speech to obtain speech recognition features representing the demand semantics; A reply text acquisition module is used to use a large model to process the speech recognition features to obtain a reply text, wherein the reply text includes multiple reply words arranged in sequence, and a target reply word among the multiple reply words is determined based on the attention fusion layer of the large model processing the speech recognition features and associated reply word features, and the associated reply word features are related to the associated reply words arranged before the target reply word.

15. The device according to claim 14, wherein The reply text obtaining module includes: an associated reply word feature acquisition submodule, configured to process initial associated reply word features representing the associated reply word using a text feature fusion layer to obtain the associated reply word features, wherein the large model further includes the text feature fusion layer; a target reply word feature acquisition submodule, configured to process the speech recognition features and the associated reply word features using the attention fusion layer to obtain target reply word features; and The target reply word obtaining submodule is used to determine the target reply word based on the target reply word feature.

16. The device according to claim 15, wherein The text feature fusion layer and the attention fusion layer each include N layers, where N is an integer greater than 1; The target reply word feature acquisition submodule includes: a first obtaining unit, configured to process the speech recognition feature and the nth associated reply word feature using an nth attention fusion layer to obtain an nth intermediate fusion feature, where N≥n>1, the nth associated reply word feature is determined based on the nth text feature fusion layer processing the n-1th intermediate fusion feature, and the first associated reply word feature is determined by processing the initial associated reply word feature using the first text feature fusion layer; and The target reply word feature determination unit is used to determine the target reply word feature based on the Nth intermediate fusion feature when n=N.

17. The device according to claim 14, wherein The first speech recognition module includes: An initial speech feature acquisition submodule is used to extract features from the required speech to obtain initial speech features; A first decoding submodule is configured to decode the initial speech features to obtain a plurality of initial decoding features, wherein the initial decoding features represent the required words in the required speech; The speech recognition feature acquisition submodule is used to fuse multiple initial decoding features and the initial speech features based on the attention mechanism to obtain the speech recognition feature.

18. The device according to claim 17, wherein The speech recognition feature acquisition submodule includes: a demand word audio feature obtaining unit, configured to fuse the initial decoding feature and the initial speech feature based on an attention mechanism to obtain a demand word audio feature corresponding to the demand word; An intermediate speech feature obtaining unit, configured to perform global feature fusion on the audio features of the plurality of demand words to obtain an intermediate speech feature; The speech recognition feature acquisition unit is used to fuse the intermediate speech feature and the multiple initial decoding features based on the attention mechanism to obtain the speech recognition feature.

19. A large model training device comprising: The second acquisition module is used to obtain sample demand voice and label reply text; A second speech recognition module is used to perform speech recognition on the sample demand speech to obtain sample speech recognition features; a sample reply text acquisition module, configured to process the sample speech recognition features using the initial large model to obtain a sample reply text, wherein the sample reply text includes a plurality of sample reply words arranged in sequence, a sample target reply word among the plurality of sample reply words being determined by processing the sample speech recognition features and sample associated reply word features based on the initial attention fusion layer of the initial large model, wherein the sample associated reply word features are related to associated reply words arranged before the sample target reply word; A training module is used to train the initial large model based on the reply text difference between the sample reply text and the label reply text to obtain a trained large model.

20. The device according to claim 19, wherein The initial large model is determined by training the extended large model, and the extended large model is obtained by updating the network structure of the pre-trained basic large model based on the extended attention fusion layer.

21. The device according to claim 20, wherein The basic large model includes a multi-level basic feature fusion network, and the basic feature fusion network includes a cascaded basic text feature fusion layer and a basic feedforward layer; The extended large model includes a multi-stage extended feature fusion network, and the extended feature fusion network includes a cascaded basic text feature fusion layer, an extended attention fusion layer and a basic feedforward layer.

22. The device according to claim 20 or 21, wherein The initial large model is determined based on the following training operations: Obtain the extended large model and pre-trained basic large model, as well as sample extended demand text and labeled extended response text; Processing the sample extended requirement text using the basic large model to obtain features of the sample extended requirement text; Processing the features of the sample extended demand text using the extended large model to obtain a sample extended reply text; and Based on the difference between the sample extended reply text and the label extended reply text, the extended large model is trained to obtain the initial large model.

23. The apparatus according to claim 19, wherein The sample speech recognition feature is determined by processing the sample demand speech using a speech recognition large model; The training module includes: a reply text loss value determining submodule, configured to determine a reply text loss value based on a reply text difference between the sample reply text and the label reply text; and The training submodule is used to adjust the model parameters of the speech recognition large model and the initial large model based on the reply text loss value and the requirement feature loss value to obtain the trained large model, wherein the requirement feature loss value represents the difference between the sample speech recognition feature and the preset label requirement text feature.

24. The device according to claim 23, wherein The label requirement text features are determined by processing the sample requirement text using a preset large model, and the sample requirement speech is determined based on the sample requirement text.

25. The apparatus according to claim 24, wherein The sample demand speech includes sample demand text audio information representing the sample demand text and sample environment audio information representing speech environment sound.

26. The apparatus according to claim 24, wherein The sample demand speech includes a plurality of samples, each of which has different sample speech attributes, and the sample speech attribute includes at least one of the following: Tone attributes, speaking speed attributes, gender attributes, and accent attributes.

27. An intelligent agent comprising: An input module, used for receiving input information; a processing module, configured to determine a target task based on the input information received by the input module, determine a large model based on the target task, and obtain output information by executing the method of any one of claims 1 to 5 by calling the large model, or executing the method of any one of claims 6 to 13 by calling the initial large model; An output module is used to output the output information obtained by the processing module.

28. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

29. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 13.

30. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images