Interaction method and device based on large model, training method and device and intelligent agent

Abstract

The invention provides an interaction method, a training method, a device and an intelligent agent based on a large model, relates to the field of artificial intelligence, in particular to the technical fields of voice recognition, voice interaction, deep learning, large models and the like, and can be applied to application scenes such as knowledge search, automatic driving, intelligent customer service, intelligent voice control, intelligent e-commerce, AI medical treatment and the like. The interaction method based on the large model comprises the following steps: acquiring required voice; performing voice recognition on the required voice to obtain voice recognition features representing required semantics; processing the speech recognition features by using a large model to obtain a reply text, the reply text comprising a plurality of reply words arranged in sequence, and a target reply word in the plurality of reply words being determined based on the attention fusion layer processing speech recognition features and associated reply word features of the large model, the associated reply word feature is associated with an associated reply word arranged prior to the target reply word.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to technical fields such as speech recognition, voice interaction, deep learning, and large models, and can be applied to application scenarios such as knowledge search, autonomous driving, intelligent customer service, intelligent voice control, smart e-commerce, and AI medical care. Background Art

[0002] With the rapid development of artificial intelligence technology, in the process of human-computer interaction, multimodal information such as text, voice, video, etc. input by users can be processed based on artificial intelligence generated content (AIGC) technology to generate the information that users need to know. Summary of the invention

[0003] The present disclosure provides an interaction method, training method, device and intelligent agent, electronic device, storage medium and program product based on a large model.

[0004] According to one aspect of the present disclosure, a large model-based interaction method is provided, including: acquiring a demand voice; performing voice recognition on the demand voice to obtain voice recognition features representing the demand semantics; using the large model to process the voice recognition features to obtain a reply text, wherein the reply text includes a plurality of reply words arranged in sequence, a target reply word among the plurality of reply words is determined based on the large model's attention fusion layer processing voice recognition features and associated reply word features, and the associated reply word features are related to the associated reply words arranged before the target reply word.

[0005] According to another aspect of the present disclosure, a training method for a large model is provided, including: obtaining sample demand speech and labeled reply text; performing speech recognition on the sample demand speech to obtain sample speech recognition features; using an initial large model to process the sample speech recognition features to obtain sample reply text, wherein the sample reply text includes a plurality of sample reply words arranged in sequence, a sample target reply word among the plurality of sample reply words is determined based on the initial attention fusion layer of the initial large model processing the sample speech recognition features and the sample associated reply word features, and the sample associated reply word features are related to the associated reply words arranged before the sample target reply word; training the initial large model based on the reply text difference between the sample reply text and the labeled reply text to obtain a trained large model.

[0006] According to another aspect of the present disclosure, there is provided an interactive device based on a large model, comprising: a first acquisition module, for acquiring a demand voice; a first speech recognition module, for performing speech recognition on the demand voice to obtain speech recognition features representing the semantics of the demand; a reply text acquisition module, for processing the speech recognition features using the large model to obtain a reply text, wherein the reply text comprises a plurality of reply words arranged in sequence, a target reply word among the plurality of reply words is determined based on speech recognition features and associated reply word features processed by the attention fusion layer of the large model, and the associated reply word features are related to the associated reply words arranged before the target reply word.

[0007] According to another aspect of the present disclosure, a training device for a large model is provided, including: a second acquisition module, used to acquire sample demand speech and label reply text; a second speech recognition module, used to perform speech recognition on the sample demand speech to obtain sample speech recognition features; a sample reply text acquisition module, used to process the sample speech recognition features using the initial large model to obtain sample reply text, wherein the sample reply text includes a plurality of sample reply words arranged in sequence, and a sample target reply word among the plurality of sample reply words is determined based on the initial attention fusion layer of the initial large model processing the sample speech recognition features and the sample associated reply word features, and the sample associated reply word features are related to the associated reply words arranged before the sample target reply word; a training module, used to train the initial large model based on the reply text difference between the sample reply text and the label reply text to obtain the trained large model.

[0008] According to another aspect of the present disclosure, an intelligent agent is provided, including: an input module for receiving input information; a processing module for determining a target task based on the input information received by the input module, determining a big model or an initial big model based on the target task, and obtaining output information by calling the big model to execute the big model-based interaction method provided by an embodiment of the present disclosure, or by calling the initial big model to execute the big model training method provided by an embodiment of the present disclosure; and an output module for outputting the output information obtained by the processing module.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided by an embodiment of the present disclosure.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method provided by an embodiment of the present disclosure.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the method provided by the embodiment of the present disclosure when executed by a processor.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 An exemplary system architecture to which a large model-based interaction method and device can be applied according to an embodiment of the present disclosure is schematically shown;

[0015] Figure 2 A flowchart of an interaction method based on a large model according to an embodiment of the present disclosure is schematically shown;

[0016] Figure 3 The application scenario diagram of the interaction method based on the large model according to the embodiment of the present disclosure is schematically shown;

[0017] Figure 4 The following schematically shows an application scenario diagram of an interaction method based on a large model according to another embodiment of the present disclosure;

[0018] Figure 5A A flowchart of a large model training method according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 5B The schematic diagram of determining the initial large model according to an embodiment of the present disclosure is shown;

[0020] Figure 6 The principle diagram of the training method of the large model according to the embodiment of the present disclosure is schematically shown;

[0021] Figure 7 A block diagram of an interactive device based on a large model according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 8 A block diagram of a large model training device according to an embodiment of the present disclosure is schematically shown;

[0023] Fig. 9 A schematic diagram of a structure of an intelligent agent according to an embodiment of the present disclosure is shown; and

[0024] Fig.10A schematic block diagram of an example electronic device that can be used to implement the large model interaction method and large model training method of an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] In the technical solution disclosed in the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures are taken, and do not violate public order and good morals.

[0027] To facilitate understanding of the embodiments of the present disclosure, the meanings of English abbreviations or technical terms involved in the embodiments of the present disclosure may be explained based on the following content.

[0028] Artificial Intelligence Generated Content (AIGC) is a technology that uses artificial intelligence technology, especially large pre-trained models, to generate relevant content with appropriate generalization capabilities through learning and pattern recognition of existing data. The core idea of ​​AIGC technology is to use artificial intelligence algorithms to generate content with certain creativity and quality. It can generate relevant articles, images, audio, etc. based on input conditions or guidance.

[0029] Large models can include deep learning models with a large number of parameters and complex structures. Large models can process massive amounts of data and perform various types of complex tasks based on large-scale model parameters and complex model structures, such as natural language processing, computer vision, speech recognition, and other complex tasks. Large models can be built based on deep neural networks and have billions or even hundreds of billions of parameters. They learn complex patterns and features by training massive amounts of data, have stronger generalization capabilities, and can make accurate predictions on unseen data. Large models can include large language models (LLMs), which can be models based on machine learning and natural language processing technologies. Large models can learn to serve human language understanding and generation capabilities by training on large amounts of text data. Large models can have billions or even hundreds of billions of parameters, which enables them to capture more language knowledge and complex grammatical structures. In terms of the network structure of large language models, for example, network structures such as transformers can be used. Large language models have strong context-awareness when processing text, can understand and generate text content that depends on previous text, and can achieve a relatively accurate understanding of text content in terms of dialogue, article generation, and context understanding. The big model involved in the embodiments of the present disclosure may include a big language model, or may also include other types of generative big models in addition to the big language model.

[0030] During the human-computer interaction process, users represent their actual needs based on multimodal information such as voice and video, but the AIGC model finds it difficult to accurately understand the users' actual needs, resulting in a low match between the generated response information and the users' actual needs, which in turn reduces the efficiency of human-computer interaction and affects the users' actual experience.

[0031] The embodiments of the present disclosure provide an interaction method, training method, device and intelligent agent, electronic device, storage medium and program product based on a large model. The interaction method based on a large model includes: obtaining a demand voice; performing speech recognition on the demand voice to obtain speech recognition features that characterize the demand semantics; using the large model to process the speech recognition features to obtain a reply text, wherein the reply text includes a plurality of reply words arranged in sequence, and a target reply word among the plurality of reply words is determined based on the speech recognition features and associated reply word features processed by the attention fusion layer of the large model, and the associated reply word features are related to the associated reply words arranged before the target reply word.

[0032] According to the embodiments of the present disclosure, by determining the speech recognition features from the demand voice and fusing the speech recognition features and the associated reply word features based on the attention fusion layer of the large model, it is possible to achieve deep feature alignment of the generated associated reply words and speech recognition features, thereby improving the deep semantic understanding ability of the large model for the demand text and reply words expressed in the demand voice, enabling the large model to reply under the condition of deep understanding of the demand intention, avoiding hallucinations of the large model, improving the matching degree between the reply text and the demand voice, thereby improving the accuracy of replies to the demand voice and improving the user experience.

[0033] Figure 1 An exemplary system architecture to which a large model-based interaction method and apparatus can be applied according to an embodiment of the present disclosure is schematically shown.

[0034] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, an exemplary system architecture to which the interactive method and apparatus based on a large model can be applied may include a terminal device, but the terminal device may implement the interactive method and apparatus based on a large model provided by the embodiments of the present disclosure without interacting with a server.

[0035] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0036] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0037] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0038] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for the content browsed by the user using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device.

[0039] The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server can also be a server of a distributed system, or a server combined with blockchain.

[0040] It should be noted that the interaction method based on the big model provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the interaction device based on the big model provided in the embodiment of the present disclosure can generally be set in the server 105. The interaction method based on the big model provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the interaction device based on the big model provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0042] Figure 2 The flowchart of the interaction method based on a large model according to an embodiment of the present disclosure is schematically shown.

[0043] like Figure 2 As shown, the large model-based interactive method includes operations S210 to S230.

[0044] In operation S210, a demand voice is acquired.

[0045] In operation S220, speech recognition is performed on the demand speech to obtain speech recognition features representing the semantics of the demand.

[0046] In operation S230, the speech recognition features are processed using the large model to obtain a reply text.

[0047] According to the embodiments of the present disclosure, the demand voice may be audio information that represents the user's demand intention. For example, it may be a voice representing a question asked by the user, or it may be a voice representing a control instruction for controlling home appliances, assisted driving vehicles and other equipment. However, it is not limited to this, and it may also be audio information used to represent other types of demands such as the user's emotional expression needs. The embodiments of the present disclosure do not limit the specific demand type represented by the demand voice.

[0048] According to an embodiment of the present disclosure, the demand semantics may include the text semantics of the demand speech, but is not limited thereto, and the demand semantics may also include the audio attributes of the demand speech, such as intonation, voice, and speech speed. The speech recognition feature may be data information such as vectors and scalars that characterize the demand semantics, such as the text semantics and audio attributes of the demand speech.

[0049] According to the embodiments of the present disclosure, speech recognition of the demand speech may include processing the demand speech based on speech recognition algorithms such as recurrent neural network algorithms and hidden Markov model algorithms. The embodiments of the present disclosure do not limit the specific method of recognizing the demand speech, as long as speech recognition features that characterize the semantics of the demand can be obtained.

[0050] In one example, a large speech recognition model can be used to process the demand speech for speech recognition. The speech recognition features can characterize the audio attributes and text semantics of the demand words corresponding to the demand speech, so as to improve the accuracy of the speech recognition features in characterizing the demand intent of the demand speech.

[0051] In one example, a large speech recognition model can be used to process the demand speech to more accurately capture the demand words corresponding to the demand text audio information from the demand speech with environmental audio information, so as to avoid the interference of environmental audio information with the text semantics in the demand text. In addition, the large speech recognition model can also integrate the speech recognition features with the semantics related to the environmental attributes by identifying the environmental attributes represented by the environmental audio information, so as to improve the semantic richness of the speech recognition feature expression, and then use the large model to process the semantically rich speech recognition features to more accurately control the text generation task, improve the expression accuracy of the reply text and the matching degree between the reply text and the demand speech.

[0052] According to an embodiment of the present disclosure, the reply text includes a plurality of reply words arranged in sequence. A target reply word among the plurality of reply words is determined based on the speech recognition feature and the associated reply word feature processed by the attention fusion layer of the large model, and the associated reply word feature is related to the associated reply word arranged before the target reply word.

[0053] According to the embodiments of the present disclosure, the reply word can represent any type of characters such as Chinese characters, English words, English letters, logical symbols, punctuation marks, etc. The embodiments of the present disclosure do not limit the specific character type of the reply word.

[0054] According to an embodiment of the present disclosure, the associated reply word feature may be a hidden feature that characterizes the text semantics of the associated reply word. The large model may include multiple attention fusion layers. By using the attention fusion layer of the large model to perform attention fusion on the associated reply word feature and the speech recognition feature, it is possible to achieve deep feature fusion of the demand speech in the speech modality and the associated reply word in the text modality, avoid fusing feature information of different modalities through feature embedding, and improve the large model's deep understanding of multimodal feature information, thereby improving the matching degree between the generated reply text and the demand intent expressed by the demand speech, and improving the accuracy of the reply text.

[0055] It is understandable that the target reply word in the reply text can be any specified reply word, and the target reply word can be generated based on the speech recognition feature processed by the attention fusion layer and the associated reply word feature corresponding to the generated associated reply word. In this way, the generated target reply word can be used as the generated associated reply word, and the attention fusion layer is used to cyclically process the generated associated reply word feature and speech recognition feature to obtain a new target reply word until all the reply words in the reply text are generated.

[0056] According to an embodiment of the present disclosure, using a large model to process speech recognition features to obtain a reply text may include: using a text feature fusion layer to process initial associated reply word features that represent associated reply words to obtain associated reply word features; using an attention fusion layer to process speech recognition features and associated reply word features to obtain target reply word features; and determining a target reply word based on the target reply word features.

[0057] According to an embodiment of the present disclosure, the large model also includes a text feature fusion layer. The large model may include one or more feature fusion networks. The initial associated reply word feature may be feature information characterizing the associated reply word, for example, the initial associated reply word feature A1 may be related to the associated reply word "A". Linear regression is performed on the initial associated reply word feature A1 to obtain the initial associated reply word feature "A".

[0058] It should be noted that the initial associated reply word feature can be generated by the feature fusion network of the large model when the associated reply word is used as the target reply word. For example, the target reply word "B" can be generated based on the target reply word feature "B1". The target reply word "B" is used as the associated reply word, and the target reply word feature "B1" is used as the initial associated reply word feature. The initial associated reply word feature "B1" can be processed based on the large model to obtain a new target reply word feature and the corresponding target reply word.

[0059] In one example, the feature fusion network may include a cascaded text feature fusion layer, an attention fusion layer, and a feedforward layer. The text feature fusion layer may be constructed based on a self-attention algorithm, and is used to perform attention fusion on one or more initial associated reply word features based on a self-attention mechanism, so that the associated reply word features can fuse the reply context semantics related to the target reply word.

[0060] According to the embodiments of the present disclosure, by using the attention fusion layer to process speech recognition features and associated reply word features, it is possible to achieve deep feature alignment of associated reply word features and speech recognition features representing the semantics of demand intent in one or more feature fusion networks of the large model, so as to enhance the depth of feature fusion, and enable the large model to accurately generate target reply words under the condition of deep understanding of the semantics of demand intent and the semantics of associated reply words that have been generated, thereby avoiding loss of semantic information and misunderstandings, reducing the probability of hallucinations in the large model, and enhancing the degree of matching of the target reply text with the demand intent represented by the demand speech.

[0061] According to an embodiment of the present disclosure, a large model may include multiple feature fusion networks. The feature fusion network may include a cascaded text feature fusion layer, an attention fusion layer, and a feedforward layer. The text feature fusion layer may be used to process multiple associated reply word features arranged before the target reply word, so as to achieve semantic fusion of the contextual semantics represented by the associated reply word features. Based on the attention fusion layer, multi-level deep fusion of speech recognition features and initial associated reply word features may be performed to achieve deep feature alignment between demand intent semantics and associated reply words, so as to achieve deep intercommunication and deep fusion of information of different modalities in semantic attributes such as text semantics, emotional attributes, and contextual information, so that the large model can understand the content in the demand voice in the process of performing complex interactive tasks and improve the accuracy of the reply text.

[0062] According to the embodiments of the present disclosure, by using the attention fusion layers of multiple feature fusion networks to fuse the speech recognition features and the associated reply word features layer by layer, the large model can not only align the surface features of information of different modalities, but also capture the complex contextual relationship between the demand voice and the reply text through the multi-level interaction of the multi-level attention fusion layer. In this way, when processing multi-step reasoning tasks or multi-round conversations, the large model can effectively model the long-term dependencies between different modalities and further improve the accuracy of the reply text.

[0063] According to an embodiment of the present disclosure, the text feature fusion layer and the attention fusion layer each include N layers, where N is an integer greater than 1. It should be understood that the feature fusion network may include N layers in cascade.

[0064] According to an embodiment of the present disclosure, using an attention fusion layer to process speech recognition features and associated reply word features to obtain target reply word features may include: using the nth attention fusion layer to process speech recognition features and the nth associated reply word features to obtain the nth intermediate fusion feature; and when n=N, determining the target reply word feature based on the Nth intermediate fusion feature.

[0065] According to an embodiment of the present disclosure, N≥n>1, and N can be an integer. The nth associated reply word feature is determined based on the nth text feature fusion layer processing the n-1th intermediate fusion feature, and the first associated reply word feature is determined by processing the initial associated reply word feature using the first text feature fusion layer.

[0066] In one example, the attention fusion layer is built based on the Cross-Attention algorithm. The attention fusion layer deeply cross-fuses multiple related reply word features and speech recognition features based on the cross-attention mechanism, thereby improving the large model's ability to understand information of different modalities.

[0067] Figure 3 The application scenario diagram of the large model-based interaction method according to an embodiment of the present disclosure is schematically shown.

[0068] like Figure 3 As shown, the application scenario may include a speech recognition large model 310 and a large model 320. The large model 320 may include a cascaded first feature fusion network 321 and a second feature fusion network 322. The first feature fusion network 321 includes a cascaded first text feature fusion layer, a first attention fusion layer, and a first feed-forward layer. The second feature fusion network 322 includes a cascaded second text feature fusion layer, a second attention fusion layer, and a second feed-forward layer. It can be understood that N=2.

[0069] The demand voice 301 can be voice audio information representing the demand text "What is artificial intelligence technology". The demand voice 301 is input into the large speech recognition model 310 to perform speech recognition on the demand voice 301, and speech recognition features are output. The initial associated reply word feature F301 can be related to the four reply words "artificial intelligence" in the already generated reply text.

[0070] The initial associated reply word feature F301 is input into the first text feature fusion layer, and the first associated reply word feature is output. The first associated reply word feature and the speech recognition feature are input into the first attention fusion layer, and the first intermediate fusion feature is output. In the case of n = 2, the first intermediate fusion feature is processed by the cascaded first feedforward layer and the second text feature fusion layer to obtain the second associated reply word feature. The second associated reply word feature and the speech recognition feature are input into the second attention fusion layer, and the second intermediate fusion feature is output. The second intermediate fusion feature is input into the second feedforward layer, and the target reply word feature F3011 is output. The target reply word feature F3011 can represent the target reply word "technique".

[0071] It should be understood that for the target reply word "technique", the five reply words "artificial intelligence technique" in the reply text 302 can be used as the associated reply words, and the associated reply word features related to "artificial intelligence technique" are processed based on the interaction method of the technology large model provided in the embodiments of the present disclosure to obtain new target reply words. Until all the reply words are generated in sequential order to obtain the reply text 302.

[0072] According to the embodiments of the present disclosure, performing speech recognition on the demand voice to obtain speech recognition features representing the demand semantics may include: extracting features from the demand voice to obtain initial speech features; decoding the initial speech features to obtain a plurality of initial decoding features; and fusing the plurality of initial decoding features and the initial speech features based on an attention mechanism to obtain speech recognition features.

[0073] According to the embodiments of the present disclosure, the initial decoding features represent the recognition results for the demand words in the demand voice. For example, the initial decoding features may be the first speech recognition results corresponding to the demand words, and the plurality of initial decoding features correspond one by one to the plurality of demand words represented by the demand voice.

[0074] According to the embodiments of the present disclosure, any type of attention network algorithm such as Transformer can be used to process the initial decoding features and the initial speech features to obtain speech recognition features.

[0075] According to the embodiments of the present disclosure, the attention mechanism fuses multiple initial decoding features and initial speech features, so that the speech recognition features can include the text semantics of the required words corresponding to the speech fragment of the required speech, as well as various types of speech attributes such as speaking speed, intonation, and emotional attributes represented by the required speech, so that the speech recognition features can accurately represent the text semantics of the required text while also including rich speech attributes, thereby improving the expression accuracy of the speech recognition features for the demand intent.

[0076] According to an embodiment of the present disclosure, fusing multiple initial decoding features and initial speech features based on an attention mechanism to obtain speech recognition features may include: fusing the initial decoding features and the initial speech features based on an attention mechanism to obtain a demand word audio feature corresponding to the demand word; performing global feature fusion on multiple demand word audio features to obtain an intermediate speech feature; fusing the intermediate speech feature and multiple initial decoding features based on an attention mechanism to obtain a speech recognition feature.

[0077] According to the embodiments of the present disclosure, by fusing the initial decoding features and the initial speech features based on the attention mechanism, the initial decoding features corresponding to each required word can be used as prior information, and the initial decoding features corresponding to each required word can be fully fused with the audio clip corresponding to the initial speech features, so that the audio features of the required words can more accurately represent the textual semantics and speech attributes of the audio clip corresponding to the required words.

[0078] According to an embodiment of the present disclosure, the global feature fusion of multiple audio features of the required words may include fusing multiple audio features of the required words based on an attention network algorithm, and the obtained intermediate speech features may include intermediate speech word features corresponding to multiple required words. From the audio information of the required speech at the frame level, the speech feature information of unequal lengths is extracted to obtain the uniform audio features of equal length at the word level as the initial speech features of the required speech. Using multiple initial decoding features as a priori to fuse the initial speech features and multiple initial decoding features can address the defect of inconsistent feature representation lengths of speech framing and improve the accuracy of speech recognition.

[0079] According to an embodiment of the present disclosure, the intermediate speech features and multiple initial decoding features are fused based on the attention mechanism to obtain speech recognition features, which may include using the initial decoding features as query features and the intermediate speech features as key features and value features of the attention algorithm. The speech recognition features are obtained by fusing the query features, key features and value features based on the attention algorithm. The speech recognition features can characterize the text semantics and speech properties represented by the speech audio segment corresponding to the demand word. By inputting the speech recognition features into multiple attention fusion layers of the large model to achieve multimodal feature fusion, the demand speech of the audio modality can be directly calculated with the associated reply word features after the modal conversion, avoiding the modal conversion calculations such as feature embedding on the demand speech in each network structure of the large model, and realizing deep calculations between different modal information. In this way, the speech recognition large model and the multi-level attention fusion layer can be used to more efficiently align the information represented by the speech modality and the text modality, and improve the response speed of the large model to the demand speech.

[0080] Figure 4 An application scenario diagram of a large model-based interaction method according to another embodiment of the present disclosure is schematically shown.

[0081] like Figure 4 As shown, the application scenario may include a speech recognition large model and a large model 430. The speech recognition large model may include a first speech encoder 411, a first speech decoder 412, a speech attention fusion layer 413, a second speech encoder 421 and a second speech decoder 422. The large model 430 includes N feature fusion networks 431. The feature fusion network 431 may include a cascaded text feature fusion layer, an attention fusion layer and a feedforward layer.

[0082] The demand speech 401 is input into the first speech encoder 411 to extract the features of the demand speech 401 and obtain the initial speech feature F401. The initial speech feature F401 may include a plurality of initial speech sub-features arranged in sequence, and the plurality of initial speech sub-features are h1, h2, h3...hk. The plurality of initial speech sub-features correspond to the demand words in the demand speech 401 respectively. The initial speech feature F401 is input into the first speech decoder 412 to obtain the initial decoding feature F402 representing the recognition result of the demand word. The initial speech feature F401 and the initial decoding feature F402 are input into the speech attention fusion layer 413 for attention fusion to obtain the audio feature of the demand word corresponding to the demand word. The audio features of the demand words corresponding to the plurality of demand words are input into the second speech encoder 421 for global feature fusion to obtain the intermediate speech feature F403. The plurality of intermediate speech sub-features in the intermediate speech feature F403 are u1, u2, u3...uk.

[0083] Fusion of the intermediate speech features and multiple initial decoding features based on the attention mechanism may include inputting the initial decoding features F402 and the intermediate speech features F403 into the second speech decoder 422, and outputting speech recognition features. The second speech decoder 422 may include, for example, a forward decoder 4221 and a reverse decoder 4222. The forward decoder 4221 may, for example, decode the initial decoding features F402 and the intermediate speech features F403 based on the forward arrangement direction of the required words in the required text to obtain the forward speech recognition features. The reverse decoder 4222 may, for example, decode the initial decoding features F402 and the intermediate speech features F403 based on the reverse arrangement direction of the required words in the required text to obtain the reverse speech recognition features. The forward speech recognition features and the reverse speech recognition features are feature-fused based on the attention weights to obtain speech recognition features.

[0084] like Figure 4 As shown, the speech recognition features can be input into the attention fusion layer in multiple feature fusion networks in the large model 430, so that the text feature fusion layer and the attention fusion layer can perform multi-level deep fusion based on the initial associated reply word features and speech recognition features to obtain the reply text 402.

[0085] According to the embodiments of the present disclosure, in the process of multimodal interaction, the demand speech is processed by a large speech recognition model to obtain speech recognition features, and the attention network layer of the large model is used to deeply fuse the associated reply word features of different modalities with the speech recognition features, thereby avoiding the large computational overhead caused by the large model repeatedly aligning the speech-text embedding features and the associated reply word features in the text feature fusion layer constructed based on the self-attention mechanism. This reduces the device computing cost of the interactive behavior and improves the computing efficiency of the device in the process of multimodal information interaction.

[0086] Figure 5A A flowchart of a large model training method according to an embodiment of the present disclosure is schematically shown.

[0087] like Figure 5A As shown, the training method of the large model includes operations S510~S540.

[0088] In operation S510, a sample demand voice and a label reply text are obtained.

[0089] In operation S520, speech recognition is performed on the sample demand speech to obtain a sample speech recognition feature.

[0090] In operation S530, the sample speech recognition features are processed using the initial large model to obtain a sample reply text.

[0091] In operation S540, an initial large model is trained based on the reply text difference between the sample reply text and the label reply text to obtain a trained large model.

[0092] According to an embodiment of the present disclosure, the sample demand speech may be audio information representing the sample demand text. The sample demand speech may have any type of speech attribute, such as a specified speech rate attribute, emotion attribute, gender attribute, etc. The embodiment of the present disclosure does not limit the specific speech attribute of the sample demand speech. The label reply text may be text information used to reply to the sample demand speech.

[0093] According to an embodiment of the present disclosure, performing speech recognition on the sample demand speech may include processing the sample demand speech based on any type of speech recognition algorithm. The embodiment of the present disclosure does not limit the specific algorithm type for performing speech recognition.

[0094] According to an embodiment of the present disclosure, the sample reply text includes a plurality of sample reply words arranged in sequence, and a sample target reply word among the plurality of sample reply words is determined based on the initial attention fusion layer of the initial large model processing sample speech recognition features and sample associated reply word features, and the sample associated reply word features are related to the associated reply words arranged before the sample target reply word.

[0095] The trained big model in the training method provided in the embodiment of the present disclosure can be applied to the big model-based interaction method provided in the embodiment of the present disclosure. For example, the speech recognition features corresponding to the demand voice can be processed based on the trained big model to obtain a reply text. The technical terms involved in the big model-based interaction method provided in the embodiment of the present disclosure, including but not limited to demand voice, speech recognition features, etc., have the same or corresponding meanings as the technical terms involved in the big model training method provided in the embodiment of the present disclosure, including but not limited to sample demand voice, sample speech recognition features, etc., and the embodiments of the present disclosure will not be repeated here.

[0096] According to an embodiment of the present disclosure, the reply text difference can characterize the difference information between the sample reply text and the label reply text. Training the initial large model based on the reply text difference between the sample reply text and the label reply text can include determining the reply text loss value based on the reply text difference between the sample reply text and the label reply text. The reply text loss value is used to adjust the model parameters of the initial large model until the reply text loss value converges to obtain the trained large model.

[0097] According to an embodiment of the present disclosure, the sample speech recognition feature is determined by processing the sample demand speech using a speech recognition large model. The speech recognition large model can be a speech modality large model constructed based on an attention network algorithm.

[0098] According to an embodiment of the present disclosure, training an initial large model based on the reply text difference between a sample reply text and a labeled reply text, and obtaining the trained large model may include: determining a reply text loss value based on the reply text difference between the sample reply text and the labeled reply text; and adjusting the model parameters of the speech recognition large model and the initial large model based on the reply text loss value and the demand feature loss value, to obtain the trained large model.

[0099] According to an embodiment of the present disclosure, the demand feature loss value characterizes the difference between the sample speech recognition feature and the preset label demand text feature. The label demand text feature can be obtained by processing the sample demand text using a preset large model. Using the preset large model as a text encoder to process the sample demand text, the obtained label demand text feature can be integrated with the text features between the sample demand words of the sample demand text, and the demand intention semantics of the sample demand text can be extracted. In this way, the demand feature loss value is determined by using the difference between the label demand text feature and the sample speech recognition feature, so that the demand feature loss value can more accurately represent the demand intention understanding ability of the speech recognition large model for the sample demand speech of the speech modality, so that the model parameters of the speech recognition large model and the initial large model can be adjusted based on the reply text loss value and the demand feature loss value, so that the expression ability of the speech recognition large model for the sample demand speech of different speech attributes can be more accurately adjusted during the training process, and the understanding ability of the large model for the text semantic attributes and sample speech attributes expressed by the sample demand speech can be improved, and the understanding of the demand intention of the demand speech can be improved, and a more accurate reply text can be output.

[0100] In one example, based on the reply text loss value and the required feature loss value, adjusting the model parameters of the speech recognition large model and the initial large model respectively may include determining a joint loss value based on the reply text loss value and the required feature loss value, and adjusting the model parameters of the speech recognition large model and the initial large model respectively based on the joint loss value.

[0101] In one example, adjusting the model parameters of the speech recognition large model and the initial large model based on the reply text loss value and the demand feature loss value may include: adjusting the model parameters of the speech recognition large model based on the demand feature loss value in the first stage until the demand feature loss value converges to obtain the trained speech recognition large model. In the second stage, adjusting the model parameters of the initial large model based on the reply text loss value until the reply text loss value converges to obtain the trained large model.

[0102] According to an embodiment of the present disclosure, the sample demand speech includes sample demand text audio information representing the sample demand text, and sample environment audio information representing the speech environment sound.

[0103] According to an embodiment of the present disclosure, the sample demand text audio information can be understood as the sound audio corresponding to the sample demand words of the sample demand text, such as the sound audio clips corresponding to multiple sample demand words. The sample environment audio information can be understood as the background audio that generates the sample demand text audio information.

[0104] In one example, the speech recognition big model can be used to process the sample demand speech to more accurately capture the sample demand text audio information from the sample demand speech with the sample environment audio information, so as to avoid the interference of the sample environment audio information with the text semantics in the sample demand text. In addition, the speech recognition big model can also integrate the semantics related to the environment attribute by identifying the environment attribute represented by the sample environment audio information, so as to improve the semantic richness of the speech recognition feature expression.

[0105] According to an embodiment of the present disclosure, the sample demand speech includes a plurality of sample demand speech, and the plurality of sample demand speech have different sample speech attributes.

[0106] According to an embodiment of the present disclosure, the sample speech attribute includes at least one of the following: timbre attribute, speaking speed attribute, gender attribute, and accent attribute.

[0107] According to an embodiment of the present disclosure, the timbre attribute can represent speech audio with different timbres. The speech rate attribute can represent the number of sample demand words represented by the sample demand speech in a preset time. The gender attribute can be the gender of the target object expressing the sample demand speech. The accent attribute can be understood as the pronunciation style of the sample demand speech related to the accent.

[0108] According to the embodiments of the present disclosure, by utilizing a large speech recognition model to process sample demand speech with different sample speech attributes, the sample speech recognition features can be made to include diverse sample speech attribute semantics, so that the trained large model can learn sample speech recognition features with different sample speech attribute semantics, and combine the diverse sample speech attribute semantics and sample demand text semantics to perform text reply tasks, so that the large model can accurately capture the demand intent represented by the sample demand speech, so as to improve the matching degree between the sample reply text and the sample demand intent, and thereby improve the accuracy of the sample reply text.

[0109] According to an embodiment of the present disclosure, the label requirement text feature is determined by processing the sample requirement text using a preset large model. The preset large model can be a universal base large model, or it can also be a special large model determined after fine-tuning for a specified application scenario. The embodiment of the present disclosure does not limit the specific type of the preset large model.

[0110] According to an embodiment of the present disclosure, the sample demand speech is determined based on the sample demand text. The sample demand text can be processed based on a speech synthesis algorithm such as a hidden Markov model (HMM) to obtain the sample demand speech. The sample demand speech can have one or more sample speech attributes.

[0111] According to the embodiments of the present disclosure, the text semantics between the initial reply word features are fused based on the text feature fusion layer of the large model, and the associated reply word features and speech recognition features of different modalities are deeply aligned through the attention fusion layer, which can improve the performance of the large model for multimodal information and reduce computing pressure. In addition, by deeply aligning the feature information of different modalities, the large model can capture more accurate semantics, emotions, and contextual relationships between multimodal feature information, thereby fully utilizing the advantages of large-scale data training.

[0112] According to an embodiment of the present disclosure, the initial large model is determined by training the extended large model, and the extended large model is obtained by updating the network structure of the pre-trained basic large model based on the extended attention fusion layer.

[0113] According to the embodiments of the present disclosure, the basic big model may include a general base big model, or may also include a special big model that matches a specified scene or task type. The embodiments of the present disclosure do not limit the specific type of the basic big model, as long as it is a big model that has undergone a pre-training process. By updating the extended big model obtained by the basic big model based on the extended attention fusion layer, the basic model parameters and model structure of the basic big model can be reused, reducing the difficulty and operational complexity of the subsequent construction of the initial big model. By training the extended big model to obtain the initial big model, the initial big model can have a model structure that deeply aligns feature information of different modalities, while reducing the time required to perform model parameter adjustments on the initial big model during training, thereby improving training efficiency.

[0114] It should be noted that the large model or initial large model involved in the embodiments of the present disclosure may be a large model for processing text modal information, for example, the large model and the initial large model may be a large language model and an initial large language model. The speech recognition large model involved in the embodiments of the present disclosure may be a large model for processing audio modal information, and the embodiments of the present disclosure will not be repeated.

[0115] According to an embodiment of the present disclosure, the basic large model includes a multi-level basic feature fusion network, and the basic feature fusion network includes a cascaded basic text feature fusion layer and a basic feedforward layer.

[0116] According to an embodiment of the present disclosure, the extended large model includes a multi-stage extended feature fusion network, and the extended feature fusion network includes a cascaded basic text feature fusion layer, an extended attention fusion layer, and a basic feedforward layer. The extended large model can be constructed by adding an extended attention fusion layer between the basic text feature fusion layer and the basic feedforward layer of the basic large model.

[0117] According to an embodiment of the present disclosure, the initial big model can be determined based on the following training operations: obtaining an extended big model and a pre-trained basic big model, as well as sample extended demand text and labeled extended reply text; using the basic big model to process the sample extended demand text to obtain sample extended demand text features; using the extended big model to process the sample extended demand text features to obtain sample extended reply text; and based on the difference between the sample extended reply text and the labeled extended reply text, training the extended big model to obtain the initial big model.

[0118] According to an embodiment of the present disclosure, the sample extended demand text and the label extended reply text may include question-answer pairs extracted from real user interaction scenarios, so that the label extended reply text can more accurately reply to the demand intent of the sample extended demand text.

[0119] According to an embodiment of the present disclosure, the difference between the sample extended reply text and the label extended reply text can be characterized by a loss value obtained by processing the sample extended reply text and the label extended reply text based on a loss function. One or more model parameters of the extended large model are adjusted by the loss value between the loss values ​​obtained by the sample extended reply text and the label extended reply text until the loss value converges, and the trained initial large model can be obtained.

[0120] Figure 5B The schematic diagram shows a principle diagram of determining an initial large model according to an embodiment of the present disclosure.

[0121] like Figure 5B As shown, the basic large model 510 may include N basic feature fusion networks 511, and the basic feature fusion network 511 includes a cascaded basic text feature fusion layer 5111 and a basic feedforward layer 5112. The extended large model 520 may include N extended feature fusion networks 521, and the extended feature fusion network 521 includes a cascaded basic text feature fusion layer 5111, an extended attention fusion layer 5211, and a basic feedforward layer 5112. The initial large model is trained based on the following operations.

[0122] The sample extended requirement text 501 is input into the basic large model 510, and the sample extended requirement text feature F501 is output. The N basic feature fusion networks 511 of the basic large model 510 can be used as a text encoder to perform feature extraction and text semantic fusion on the sample extended requirement text 501, so that the sample extended requirement text feature F501 can more accurately represent the text semantics of each word of the sample extended requirement text 501 and the overall text semantics of the sample extended requirement text 501.

[0123] Inputting the sample extended demand text feature F501 into the extended attention fusion layer 5211 of each of the N extended feature fusion networks 521 of the extended large model 520 can enable the N extended feature fusion networks 521 to learn the understanding ability of the text semantics of the sample extended demand text 501 and the text semantics expressed by the sample extended reply text 502. The model parameters of the extended large model 520 are adjusted by the loss value between the label extended reply text and the sample extended reply text until the loss value converges, and the trained initial large model is obtained, so as to realize the pre-training of the initial large model by using the text modality information.

[0124] According to the embodiments of the present disclosure, after preliminary pre-training of the extended big model with sample extended demand text and label extended reply text in pure text mode, the early pre-training can be performed on the basis of the preliminary text understanding ability of the basic big model, so that the dialogue interaction ability of the trained initial big model can be consistent with the general basic big model ability, thereby improving the generalization ability of the initial attention fusion layer for different types of text understanding, reducing the training cycle of the initial big model, improving the training efficiency, and at the same time improving the text generation task execution ability of the initial big model obtained after training to reply to the demand intent of the sample extended demand text.

[0125] Figure 6 The schematic diagram shows a principle diagram of a large model training method according to an embodiment of the present disclosure.

[0126] like Figure 6 As shown, the sample demand text 601 is input into the speech synthesis model 610, and the sample demand speech 602 having the sample speech attribute is output. The sample demand speech 602 is input into the speech recognition large model 620, and the sample speech recognition feature F602 is output. The speech synthesis model 610 can be constructed based on the speech synthesis algorithm.

[0127] The sample requirement text 601 is input into the preset large model 630, and the label requirement text feature F601 is output. The preset large model 630 may include N cascaded preset feature fusion networks 631, and the preset feature fusion network 631 may include a cascaded preset text feature fusion layer and a preset feedforward layer. Using the preset large model 630 as a text encoder can make the label requirement text feature F601 more accurately represent the text semantics of the sample requirement text 601.

[0128] The initial large model 640 includes N cascaded initial feature fusion networks 641. The initial feature fusion network 641 may have a cascaded initial text feature fusion layer, an initial attention fusion layer, and an initial feedforward layer. Inputting the sample speech recognition feature F602 into the initial attention fusion layer of each of the N initial feature fusion networks 641 can enable the initial large model to deeply align the sample speech recognition features of different modalities and the sample associated reply word features in the sample reply text 603 until all the sample target reply words are generated to obtain the sample reply text 603.

[0129] The sample reply text 603 and the label reply text 604 are processed based on the loss function to obtain the reply text loss value. The sample speech recognition feature F602 and the label demand text feature F601 are processed based on the loss function to obtain the demand feature loss value. The joint loss value is determined based on the reply text loss value and the demand feature loss value, and the model parameters of the speech recognition large model and the initial large model are adjusted based on the joint loss value. Until the joint loss value converges, the trained large model can be obtained.

[0130] According to the embodiments of the present disclosure, the initial large model is trained based on the reply text loss value and the demand feature loss value, which can realize the multimodal loss value generated based on different modal information to ensure the feature alignment capability of the large model in the deep integration of different modal information, and realize the ability to understand information of different modalities, so that the large model can reach a convergence state more stably.

[0131] According to the embodiments of the present disclosure, by performing speech synthesis on the sample demand text to generate the sample demand speech, the sample demand speech can represent the demand intention expressed by the user in different scenarios, so as to improve the authenticity of the sample demand speech and improve the accuracy of the sample speech recognition feature in representing the demand intention. By using the initial attention fusion layer to deeply fuse the sample speech recognition features with the sample associated reply word features, the trained large model can be adapted to diverse speech environments or user expressions such as noise and noisy environments, thereby improving the robustness and generalization ability of the large model during the interaction process. By generating the sample demand speech based on speech synthesis and combining the real sample demand speech to construct the training data, the convergence speed of the model parameters for the large model can be accelerated, and redundant computing tasks can be reduced, thereby reducing the overall training cost.

[0132] Figure 7 A block diagram of an interaction device based on a large model according to an embodiment of the present disclosure is schematically shown.

[0133] like Figure 7 As shown, the large model-based interaction device 700 includes: a first acquisition module 710, a first speech recognition module 720 and a reply text acquisition module 730.

[0134] The first acquisition module 710 is used to acquire the required voice.

[0135] The first speech recognition module 720 is used to perform speech recognition on the demand speech to obtain speech recognition features representing the demand semantics.

[0136] The reply text acquisition module 730 is used to use the large model to process the speech recognition features to obtain the reply text, wherein the reply text includes multiple reply words arranged in sequence, and the target reply word among the multiple reply words is determined based on the speech recognition features and the associated reply word features processed by the attention fusion layer of the large model, and the associated reply word features are related to the associated reply words arranged before the target reply word.

[0137] According to an embodiment of the present disclosure, the reply text obtaining module 730 includes: an associated reply word feature obtaining submodule, a target reply word feature obtaining submodule and a target reply word obtaining submodule.

[0138] The associated reply word feature acquisition submodule is used to use the text feature fusion layer to process the initial associated reply word features representing the associated reply words to obtain the associated reply word features. The large model also includes a text feature fusion layer.

[0139] The target reply word feature acquisition submodule is used to use the attention fusion layer to process the speech recognition features and the associated reply word features to obtain the target reply word features.

[0140] The target reply word obtaining submodule is used to determine the target reply word based on the target reply word features.

[0141] According to an embodiment of the present disclosure, the text feature fusion layer and the attention fusion layer each include N layers, where N is an integer greater than 1.

[0142] According to an embodiment of the present disclosure, the target reply word feature obtaining submodule includes: a first obtaining unit and a target reply word feature determining unit.

[0143] The first acquisition unit is used to use the nth attention fusion layer to process the speech recognition features and the nth associated reply word features to obtain the nth intermediate fusion features, N≥n>1, the nth associated reply word feature is determined based on the nth text feature fusion layer processing the n-1th intermediate fusion feature, and the 1st associated reply word feature is determined by using the 1st text feature fusion layer to process the initial associated reply word features.

[0144] The target reply word feature determination unit is used to determine the target reply word feature based on the Nth intermediate fusion feature when n=N.

[0145] According to an embodiment of the present disclosure, the first speech recognition module 720 includes: an initial speech feature acquisition submodule, a first decoding submodule and a speech recognition feature acquisition submodule.

[0146] The initial speech feature acquisition submodule is used to extract features from the required speech to obtain initial speech features.

[0147] The first decoding submodule is used to decode the initial speech features to obtain a plurality of initial decoding features, wherein the initial decoding features represent the required words in the required speech.

[0148] The speech recognition feature acquisition submodule is used to fuse multiple initial decoding features and initial speech features based on the attention mechanism to obtain speech recognition features.

[0149] According to an embodiment of the present disclosure, the speech recognition feature obtaining submodule includes: a demand word audio feature obtaining unit, an intermediate speech feature obtaining unit and a speech recognition feature obtaining unit.

[0150] The required word audio feature obtaining unit is used to fuse the initial decoding feature and the initial speech feature based on the attention mechanism to obtain the required word audio feature corresponding to the required word.

[0151] The intermediate speech feature acquisition unit is used to perform global feature fusion on multiple required word audio features to obtain the intermediate speech feature.

[0152] The speech recognition feature acquisition unit is used to fuse the intermediate speech features and multiple initial decoding features based on the attention mechanism to obtain the speech recognition features.

[0153] Figure 8 A block diagram of a large model training device according to an embodiment of the present disclosure is schematically shown.

[0154] like Figure 8 As shown, the large model training device 800 includes: a second acquisition module 810, a second speech recognition module 820, a sample reply text acquisition module 830 and a training module 840.

[0155] The second acquisition module 810 is used to acquire sample demand voice and label reply text.

[0156] The second speech recognition module 820 is used to perform speech recognition on the sample demand speech to obtain the sample speech recognition features.

[0157] The sample reply text acquisition module 830 is used to use the initial large model to process the sample speech recognition features to obtain the sample reply text, wherein the sample reply text includes a plurality of sample reply words arranged in sequence, and the sample target reply word among the plurality of sample reply words is determined based on the initial attention fusion layer of the initial large model processing the sample speech recognition features and the sample associated reply word features, and the sample associated reply word features are related to the associated reply words arranged before the sample target reply word.

[0158] The training module 840 is used to train the initial large model based on the reply text difference between the sample reply text and the label reply text to obtain the trained large model.

[0159] According to an embodiment of the present disclosure, the initial large model is determined by training the extended large model, and the extended large model is obtained by updating the network structure of the pre-trained basic large model based on the extended attention fusion layer.

[0160] According to an embodiment of the present disclosure, the basic large model includes a multi-level basic feature fusion network, which includes a cascaded basic text feature fusion layer and a basic feedforward layer; the extended large model includes a multi-level extended feature fusion network, which includes a cascaded basic text feature fusion layer, an extended attention fusion layer and a basic feedforward layer.

[0161] According to an embodiment of the present disclosure, the initial big model is determined based on the following training operations: obtaining an extended big model and a pre-trained basic big model, as well as a sample extended demand text and a labeled extended reply text; using the basic big model to process the sample extended demand text to obtain sample extended demand text features; using the extended big model to process the sample extended demand text features to obtain sample extended reply text; and based on the difference between the sample extended reply text and the labeled extended reply text, training the extended big model to obtain the initial big model.

[0162] According to an embodiment of the present disclosure, the sample speech recognition features are determined by processing the sample demand speech using a large speech recognition model.

[0163] The training module includes: a reply text loss value determination submodule and a training submodule.

[0164] The reply text loss value determination submodule is used to determine the reply text loss value based on the reply text difference between the sample reply text and the label reply text.

[0165] The training submodule is used to adjust the model parameters of the speech recognition large model and the initial large model based on the reply text loss value and the demand feature loss value to obtain the trained large model, wherein the demand feature loss value represents the difference between the sample speech recognition feature and the preset label demand text feature.

[0166] According to an embodiment of the present disclosure, the label requirement text feature is determined by processing the sample requirement text using a preset large model, and the sample requirement speech is determined based on the sample requirement text.

[0167] According to an embodiment of the present disclosure, the sample demand speech includes sample demand text audio information representing the sample demand text, and sample environment audio information representing the speech environment sound.

[0168] According to an embodiment of the present disclosure, the sample demand speech includes multiple sample demand speech, and the multiple sample demand speech has different sample speech attributes, and the sample speech attribute includes at least one of the following: timbre attribute, speech speed attribute, gender attribute, and accent attribute.

[0169] Fig. 9 The structural block diagram of an intelligent agent according to an embodiment of the present disclosure is schematically shown.

[0170] In the embodiments of the present disclosure, Fig. 9 As shown, the AI ​​agent 900 may include an input module 910 , a processing module 920 , and an output module 930 .

[0171] An input module 910, used to receive input information;

[0172] A processing module 920 is used to determine a target task based on input information received by the input module, determine a large model based on the target task, and obtain output information by calling the large model to execute an interaction method based on the large model provided according to an embodiment of the present disclosure, or by calling the large model to execute a training method for the large model provided according to an embodiment of the present disclosure;

[0173] The output module 930 is used to output the output information obtained by the processing module.

[0174] According to an embodiment of the present disclosure, the input module 910 is responsible for receiving or sensing information such as queries, requests, instructions, signals or data from the outside world (e.g., users or external environment), and converting them into a format that can be understood and processed by the AI ​​agent 900. The input module 910 is the primary link for the AI ​​agent 900 to interact with the outside world, which enables the AI ​​agent 900 to efficiently and accurately obtain the necessary "sensory" information from the outside world and respond to this information.

[0175] In an example, the input module 910 may input the demand speech or sample demand speech, sample demand speech features, demand speech features, etc. described above.

[0176] In the example, the processing module 920 is the core support for the AI ​​agent 900 to process complex tasks. The processing module 920 can execute the large model-based interaction method and the large model training method described above.

[0177] In the example, the performance of the processing module 920 may be closely related to the large model on which the AI ​​agent 900 is based. In order to give full play to the capabilities of the large model, the internal structure of the processing module 920 may be designed to be highly configurable and extensible so as to cope with various types of tasks and requirements in real scenarios.

[0178] In the example, after the AI ​​agent 900 obtains the required voice, the processing module 920 can use the speech recognition large model to process the required voice to obtain speech recognition features. The large model processes the speech recognition features to obtain a reply text, and passes the reply text to the output module 930.

[0179] It is understandable that although the large model has excellent language understanding and generation capabilities, it is just like a human being, without the help of any tools, and the tasks it can solve are very limited. When AI agent 900 is given the ability to call tools, it can perform tasks such as using a calculator to complete mathematical operations, using Python to complete data analysis, and using a search engine to complete weather forecasts.

[0180] In an example, the output module 930 may output the reply text or the trained large model described above.

[0181] The AI ​​agent 900 according to the embodiment of the present disclosure can simply and effectively improve the level of intelligence, and improve flexibility and versatility.

[0182] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0183] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0184] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0185] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method as described above.

[0186] Fig.10 A schematic block diagram of an example electronic device that can be used to implement the interactive method of a large model and the training method of a large model of an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0187] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0188] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0189] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as the interaction method based on the large model and the training method of the large model. For example, in some embodiments, the interaction method based on the large model and the training method of the large model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the interaction method based on the large model and the training method of the large model described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute a large model-based interaction method or a large model training method in any other appropriate manner (eg, by means of firmware).

[0190] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0191] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0192] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0193] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0194] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0195] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0196] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0197] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An interactive method based on a large model, comprising: Get demand voice; Performing speech recognition on the demand speech to obtain speech recognition features representing the demand semantics; The speech recognition features are processed using a large model to obtain a reply text, wherein the reply text includes a plurality of reply words arranged in sequence, a target reply word among the plurality of reply words is determined based on the speech recognition features and associated reply word features processed by the attention fusion layer of the large model, and the associated reply word features are related to the associated reply words arranged before the target reply word.

2. The method according to claim 1, wherein: The method of using the large model to process the speech recognition features to obtain the reply text includes: Using a text feature fusion layer to process initial associated reply word features representing the associated reply word to obtain the associated reply word features, the large model including the text feature fusion layer; Processing the speech recognition features and the associated reply word features using the attention fusion layer to obtain target reply word features; and The target reply word is determined based on the target reply word feature.

3. The method according to claim 2, wherein: The text feature fusion layer and the attention fusion layer each include N layers, where N is an integer greater than 1; Wherein, the using the attention fusion layer to process the speech recognition feature and the associated reply word feature to obtain the target reply word feature includes: Processing the speech recognition feature and the nth associated reply word feature using the nth attention fusion layer to obtain an nth intermediate fusion feature, N≥n>1, the nth associated reply word feature is determined based on the nth text feature fusion layer processing the n-1th intermediate fusion feature, and the first associated reply word feature is determined by processing the initial associated reply word feature using the first text feature fusion layer; and When n=N, the target reply word feature is determined based on the Nth intermediate fusion feature.

4. The method according to claim 1, wherein: The performing speech recognition on the demand speech to obtain speech recognition features representing the demand semantics includes: Extracting features of the required speech to obtain initial speech features; Decoding the initial speech features to obtain a plurality of initial decoding features, wherein the initial decoding features represent the required words in the required speech; The speech recognition feature is obtained by fusing a plurality of the initial decoding features and the initial speech features based on an attention mechanism.

5. The method according to claim 2, wherein: The speech recognition feature obtained by fusing the plurality of the initial decoding features and the initial speech features based on the attention mechanism includes: Based on the attention mechanism, the initial decoding feature and the initial speech feature are fused to obtain the audio feature of the required word corresponding to the required word; Performing global feature fusion on the audio features of the plurality of required words to obtain an intermediate speech feature; The intermediate speech feature and the plurality of initial decoding features are fused based on an attention mechanism to obtain the speech recognition feature.

6. A large model training method, comprising: Get sample demand voice and label reply text; Performing speech recognition on the sample demand speech to obtain sample speech recognition features; Processing the sample speech recognition features using the initial large model to obtain a sample reply text, wherein the sample reply text includes a plurality of sample reply words arranged in sequence, a sample target reply word among the plurality of sample reply words is determined based on the initial attention fusion layer of the initial large model processing the sample speech recognition features and sample associated reply word features, and the sample associated reply word features are related to associated reply words arranged before the sample target reply word; The initial large model is trained based on the reply text difference between the sample reply text and the label reply text to obtain a trained large model.

7. The method according to claim 6, wherein: The initial large model is determined by training the extended large model, and the extended large model is obtained by updating the network structure of the pre-trained basic large model based on the extended attention fusion layer.

8. The method according to claim 7, wherein: The basic large model includes a multi-level basic feature fusion network, and the basic feature fusion network includes a cascaded basic text feature fusion layer and a basic feedforward layer; The extended large model includes a multi-stage extended feature fusion network, and the extended feature fusion network includes a cascaded basic text feature fusion layer, an extended attention fusion layer and a basic feedforward layer.

9. The method according to claim 7 or 8, wherein: The initial large model is determined based on the following training operations: Obtain the extended large model and the pre-trained basic large model, as well as the sample extended demand text and the labeled extended response text; Processing the sample extended demand text using the basic large model to obtain features of the sample extended demand text; Processing the sample extended demand text features using the extended large model to obtain a sample extended reply text; and Based on the difference between the sample extended reply text and the label extended reply text, the extended large model is trained to obtain the initial large model.

10. The method according to claim 6, wherein: The sample speech recognition feature is determined by processing the sample demand speech using a speech recognition large model; The training of the initial large model based on the reply text difference between the sample reply text and the label reply text to obtain the trained large model comprises: Determining a reply text loss value based on a reply text difference between the sample reply text and the label reply text; and Based on the reply text loss value and the requirement feature loss value, the model parameters of the speech recognition large model and the initial large model are adjusted to obtain a trained large model, wherein the requirement feature loss value represents the difference between the sample speech recognition feature and the preset label requirement text feature.

11. The method according to claim 10, wherein: The label requirement text feature is determined by processing the sample requirement text using a preset large model, and the sample requirement speech is determined based on the sample requirement text.

12. The method according to claim 11, wherein: The sample demand speech includes sample demand text audio information representing the sample demand text and sample environment audio information representing speech environment sound.

13. The method according to claim 11, wherein: The sample demand speech includes a plurality of sample demand speech, and the plurality of sample demand speech have different sample speech attributes, and the sample speech attribute includes at least one of the following: Tone attributes, speaking speed attributes, gender attributes, and accent attributes.

14. An interactive device based on a large model, comprising: The first acquisition module is used to acquire the required voice; A first speech recognition module is used to perform speech recognition on the demand speech to obtain speech recognition features representing the demand semantics; A reply text acquisition module is used to use a large model to process the speech recognition features to obtain a reply text, wherein the reply text includes a plurality of reply words arranged in sequence, and a target reply word among the plurality of reply words is determined based on the attention fusion layer of the large model processing the speech recognition features and associated reply word features, and the associated reply word features are related to the associated reply words arranged before the target reply word.

15. The device according to claim 14, wherein: The reply text obtaining module comprises: A submodule for obtaining associated reply word features, used for processing initial associated reply word features representing the associated reply word using a text feature fusion layer to obtain the associated reply word features, wherein the large model further includes the text feature fusion layer; a target reply word feature acquisition submodule, configured to process the speech recognition feature and the associated reply word feature using the attention fusion layer to obtain a target reply word feature; and The target reply word obtaining submodule is used to determine the target reply word based on the target reply word feature.

16. The device according to claim 15, wherein: The text feature fusion layer and the attention fusion layer each include N layers, where N is an integer greater than 1; Wherein, the target reply word feature acquisition submodule includes: a first obtaining unit, configured to process the speech recognition feature and the nth associated reply word feature by using an nth attention fusion layer to obtain an nth intermediate fusion feature, N≥n>1, the nth associated reply word feature is determined based on the nth text feature fusion layer processing the n-1th intermediate fusion feature, and the first associated reply word feature is determined by using the first text feature fusion layer processing the initial associated reply word feature; and The target reply word feature determination unit is used to determine the target reply word feature based on the Nth intermediate fusion feature when n=N.

17. The device according to claim 14, wherein: The first speech recognition module comprises: An initial speech feature acquisition submodule is used to extract features from the required speech to obtain initial speech features; A first decoding submodule, configured to decode the initial speech feature to obtain a plurality of initial decoding features, wherein the initial decoding features represent the required words in the required speech; The speech recognition feature acquisition submodule is used to fuse the multiple initial decoding features and the initial speech features based on the attention mechanism to obtain the speech recognition feature.

18. The device according to claim 15, wherein: The speech recognition feature acquisition submodule includes: A demand word audio feature obtaining unit, used for fusing the initial decoding feature and the initial speech feature based on an attention mechanism to obtain a demand word audio feature corresponding to the demand word; An intermediate speech feature obtaining unit, used for performing global feature fusion on the audio features of the plurality of demand words to obtain an intermediate speech feature; The speech recognition feature acquisition unit is used to fuse the intermediate speech feature and the multiple initial decoding features based on the attention mechanism to obtain the speech recognition feature.

19. A large model training device, comprising: The second acquisition module is used to obtain the sample demand voice and label reply text; A second speech recognition module is used to perform speech recognition on the sample demand speech to obtain sample speech recognition features; A sample reply text acquisition module is used to process the sample speech recognition features using the initial large model to obtain a sample reply text, wherein the sample reply text includes a plurality of sample reply words arranged in sequence, a sample target reply word among the plurality of sample reply words is determined based on the initial attention fusion layer of the initial large model processing the sample speech recognition features and sample associated reply word features, and the sample associated reply word features are related to the associated reply words arranged before the sample target reply word; The training module is used to train the initial large model based on the reply text difference between the sample reply text and the label reply text to obtain a trained large model.

20. The device according to claim 19, wherein The initial large model is determined by training the extended large model, and the extended large model is obtained by updating the network structure of the pre-trained basic large model based on the extended attention fusion layer.

21. The device according to claim 20, wherein: The basic large model includes a multi-level basic feature fusion network, and the basic feature fusion network includes a cascaded basic text feature fusion layer and a basic feedforward layer; The extended large model includes a multi-stage extended feature fusion network, and the extended feature fusion network includes a cascaded basic text feature fusion layer, an extended attention fusion layer and a basic feedforward layer.

22. The device according to claim 20 or 21, wherein: The initial large model is determined based on the following training operations: Obtain the extended large model and the pre-trained basic large model, as well as the sample extended demand text and the labeled extended response text; Processing the sample extended demand text using the basic large model to obtain features of the sample extended demand text; Processing the sample extended demand text features using the extended large model to obtain a sample extended reply text; and Based on the difference between the sample extended reply text and the label extended reply text, the extended large model is trained to obtain the initial large model.

23. The device according to claim 19, wherein: The sample speech recognition feature is determined by processing the sample demand speech using a speech recognition large model; The training module includes: a reply text loss value determination submodule, configured to determine a reply text loss value based on a reply text difference between the sample reply text and the label reply text; and The training submodule is used to adjust the model parameters of the speech recognition large model and the initial large model based on the reply text loss value and the requirement feature loss value to obtain the trained large model, wherein the requirement feature loss value represents the difference between the sample speech recognition feature and the preset label requirement text feature.

24. The device according to claim 23, wherein: The label requirement text feature is determined by processing the sample requirement text using a preset large model, and the sample requirement speech is determined based on the sample requirement text.

25. The device according to claim 24, wherein: The sample demand speech includes sample demand text audio information representing the sample demand text and sample environment audio information representing speech environment sound.

26. The device according to claim 24, wherein: The sample demand speech includes a plurality of sample demand speech, and the plurality of sample demand speech have different sample speech attributes, and the sample speech attribute includes at least one of the following: Tone attributes, speaking speed attributes, gender attributes, and accent attributes.

27. An intelligent agent, comprising: An input module, used for receiving input information; A processing module, configured to determine a target task based on the input information received by the input module, determine a large model based on the target task, and obtain output information by calling the large model to execute the method of any one of claims 1 to 5, or by calling the initial large model to execute the method of any one of claims 6 to 13; An output module is used to output the output information obtained by the processing module.

28. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

29. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 13.

30. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images