Method and apparatus for training multi-modal speech language large-scale model, device and medium

Abstract

The present disclosure provides a method and apparatus for training a multi-modal speech language large scale model, a device and a medium.SOLUTION: A solution is to input first question speech data into a multi-modal large speech language model to obtain first answer speech data generated by the multi-modal large speech language model, determine a question text corresponding to the first question speech data and an answer text corresponding to the first answer speech data, and determine a first score according to the question text and the answer text. A second score is determined according to a speech feature of the first question speech data and a speech feature of the first answer speech data, where the speech feature includes at least one of a speech clarity, a speech rate feature, a timbre feature, an intonation feature, and an emotion feature.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to the field of artificial intelligence, in particular to the field of speech data processing and data generation, and specifically to a method for training a multimodal speech language large-scale model, a method and device for generating speech data, an electronic device, a computer-readable storage medium, and a computer program. [Background technology]

[0002] Artificial intelligence is a field that studies how computers can imitate some human thought processes and intelligent behaviors (e.g., learning, reasoning, thinking, planning, etc.), and includes both hardware and software technologies. AI hardware technologies generally include sensors, AI-specific chips, cloud computing, distributed storage, and big data processing, while AI software technologies mainly include several major areas such as computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, and knowledge graph technology.

[0003] With the development of computer technology, generative models based on artificial intelligence can be applied to various forms of natural language processing tasks, including processing natural language text and natural language speech, and in particular can generate answers based on user questions and realize dialogue with users.

[0004] The approaches described in this section are not necessarily approaches that have been previously conceived or adopted. Unless otherwise noted, any approach described in this section should not be considered prior art merely because it is included in this section. Likewise, unless otherwise noted, the subject matter addressed in this section should not be considered an admission of any prior art. Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure provides a method and apparatus for training a multimodal spoken language large-scale model, an electronic device, a computer-readable storage medium, and a computer program. [Means for solving the problem]

[0006] According to one aspect of the present disclosure, there is provided a method for training a multimodal speech language large-scale model, the method including: inputting first question speech data into the multimodal speech language large-scale model to obtain first answer speech data generated by the multimodal speech language large-scale model; determining a question text corresponding to the first question speech data and an answer text corresponding to the first answer speech data; determining a first score based on the question text and the answer text; determining a second score based on speech features of the first question speech data and speech features of the first answer speech data, wherein the speech features include at least one of speech clarity, speaking rate features, timbre features, intonation features, and emotion features; and adjusting parameters of the multimodal speech language large-scale model based on the first score and the second score.

[0007] According to one aspect of the present disclosure, there is provided a method for generating speech data, comprising: acquiring question speech data from a user; inputting the question speech data into a multimodal speech language large-scale model obtained by training using the above-described training method for a multimodal speech language large-scale model, thereby acquiring answer speech data generated by the multimodal speech language large-scale model; and returning the answer speech data to the user.

[0008] According to one aspect of the present disclosure, there is provided a training device for a multimodal speech language large-scale model, the device including: a first acquisition unit configured to acquire first answer speech data generated by the multimodal speech language large-scale model by inputting first question speech data into the multimodal speech language large-scale model; a first determination unit configured to determine a question text corresponding to the first question speech data and an answer text corresponding to the first answer speech data; a second determination unit configured to determine a first score based on the question text and the answer text; a third determination unit configured to determine a second score based on speech features of the first question speech data and speech features of the first answer speech data, wherein the speech features include at least one of speech clarity, speaking rate features, timbre features, intonation features, and emotion features; and an adjustment unit configured to adjust parameters of the multimodal speech language large-scale model based on the first score and the second score.

[0009] According to one aspect of the present disclosure, there is provided a speech data generation device, the device including: a multimodal speech language large-scale model obtained by training using the multimodal speech language large-scale model training device; a second acquisition unit configured to acquire question speech data of a user; a third acquisition unit configured to acquire answer speech data generated by the multimodal speech language large-scale model by inputting the question speech data into the multimodal speech language large-scale model; and a return unit configured to return the answer speech data to the user.

[0010] According to one aspect of the present disclosure, there is provided an electronic device including at least one processor and a memory communicatively coupled to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform at least one of the method for training a multimodal speech language large-scale model and the method for generating speech data.

[0011] According to one aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon computer instructions used to cause the computer to perform at least one of the method for training a multimodal speech language large-scale model and the method for generating speech data.

[0012] According to one aspect of the present disclosure, there is provided a computer program product including a computer program that, when executed by a processor, can implement at least one of the above-described methods for training a multimodal speech language large-scale model and the above-described method for generating speech data. [Effects of the Invention]

[0013] According to one or more embodiments of the present disclosure, the quality of generated voice data can be improved, and more accurate human-machine voice question-answering dialogue can be realized.

[0014] It should be understood that the material described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily apparent from the following specification. [Brief explanation of the drawings]

[0015] The drawings illustratively illustrate examples, constitute a part of the specification, and together with the written description serve to explain exemplary embodiments of the examples. The illustrated examples are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar, but not necessarily identical, elements. [Figure 1] 1 is a schematic diagram of an example system in which various methods described herein may be implemented, according to an example embodiment of the present disclosure. [Figure 2] 1 is a flowchart illustrating a method for training a multimodal spoken language large-scale model according to an exemplary embodiment of the present disclosure. [Figure 3] FIG. 1 is a block diagram illustrating a multimodal speech language large-scale model according to an exemplary embodiment of the present disclosure. [Figure 4] FIG. 1 is a block diagram illustrating a question-and-answer evaluation model according to an exemplary embodiment of the present disclosure. [Figure 5] FIG. 1 is a schematic diagram illustrating the training process of a multimodal speech language large-scale model according to an exemplary embodiment of the present disclosure. [Figure 6] 1 is a flowchart illustrating a method for generating audio data according to an exemplary embodiment of the present disclosure. [Figure 7] FIG. 2 is a schematic diagram illustrating a process for generating audio data according to an exemplary embodiment of the present disclosure. [Figure 8] FIG. 1 is a block diagram illustrating a training device for a multimodal spoken language large-scale model according to an exemplary embodiment of the present disclosure. [Figure 9] 1 is a block diagram illustrating a configuration of an audio data generating device according to an exemplary embodiment of the present disclosure. [Figure 10] FIG. 1 is a block diagram illustrating an exemplary electronic device capable of implementing an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0016]

[0023] The following describes exemplary embodiments of the present disclosure in conjunction with the drawings. For ease of understanding, various details of the embodiments of the present disclosure are included therein, but they should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, the following description omits descriptions of known functions and structures.

[0017] In this disclosure, unless otherwise specified, the use of terms such as "first," "second," etc. to describe various elements is not intended to limit the location, timing, or importance of these elements. Such terms are used only to distinguish one element from another. In some instances, a first element and a second element may refer to the same instance of the element, or in some cases, may refer to different instances based on the context.

[0018] The terms used in the description of various examples of the present disclosure are intended only to describe particular examples and are not intended to be limiting. Unless the context clearly indicates otherwise, and unless the number of elements is specifically limited, the element may be one or more. Furthermore, as used in this disclosure, the term "and / or" covers any and all possible combinations of the listed items.

[0019] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0020] 1 illustrates a schematic diagram of an exemplary system 100 in which various methods and apparatus described herein may be implemented, according to embodiments of the present disclosure. Referring to FIG. 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to run one or more applications.

[0021] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable the implementation of methods for training large-scale models of multimodal spoken language.

[0022] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based or cloud services, for example, provided to users of client devices 101, 102, 103, 104, 105, and / or 106 in a Software as a Service (SaaS) model.

[0023] In the configuration shown in FIG. 1 , server 120 may include one or more assemblies that implement the functionality performed by server 120. These assemblies may include software assemblies, hardware assemblies, or a combination thereof that are executable by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may, in turn, utilize one or more client applications to interact with server 120 to access services provided by these assemblies. It should be understood that a variety of different system configurations are possible and may differ from system 100. Accordingly, FIG. 1 is intended to be illustrative of an example system for implementing various methods described herein and is not intended to be limiting.

[0024] A user can submit query audio data using client devices 101, 102, 103, 104, 105, and / or 106. The client devices can provide an interface through which a user of the client device interacts with the client device. The client devices can also output information to the user through the interface. Although only six client devices are shown in FIG. 1, one skilled in the art will understand that the present disclosure can support any number of client devices.

[0025] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computing devices, such as portable handheld devices, general-purpose computers (e.g., personal computers or laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors and other sensing devices, etc. These computing devices may run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (e.g., Google Chrome OS), and may include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, Android, etc. Portable handheld devices may include mobile phones, intelligent phones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (e.g., smart glasses) and other devices. Gaming systems may include various handheld gaming devices, Internet-enabled gaming devices, etc. The client device may run a variety of applications, such as Internet-related applications, communication applications (eg, email applications), short message service (SMS) applications, and may use a variety of communication protocols.

[0026] Network 110 may be any type of network known to those skilled in the art that is capable of supporting data communications using any of a variety of available protocols (including, but not limited to, TCP / IP, SNA, IPX, etc.) By way of example, one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0027] Server 120 may include one or more general-purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframes, server clusters, or any other suitable arrangement and / or combination. Server 120 may also include one or more virtual machines running virtual operating systems or other computing architectures involving virtualization (e.g., one or more flexible pools of virtualized logical storage devices to maintain the server's virtual storage devices). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0028] The computing units in server 120 may run one or more operating systems, including any of the operating systems listed above and any commercial server operating system. Server 120 may also run any one of a variety of additional server and / or middle-tier applications, such as an HTTP server, an FTP server, a CGI server, a JAVA server, a database server, etc.

[0029] In some embodiments, server 120 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0030] In some embodiments, server 120 may be a server in a distributed system or a server incorporating blockchain. Server 120 may be a cloud server, or a smart cloud computing server or smart cloud host equipped with artificial intelligence technology. A cloud server is a host product in a cloud computing service system that solves the drawbacks of traditional physical hosts and virtual private server (VPS) services, such as high management difficulty and poor business scalability.

[0031] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data or other information. For example, one or more of databases 130 may be used to store information such as audio files or video files. Databases 130 may be located in a variety of locations. For example, a database used by server 120 may be local to server 120 or may be remote from server 120 and in communication with server 120 over a network or dedicated connection. Databases 130 may be of different types. In some embodiments, a database used by server 120 may be a relational database. One or more of these databases may store, update, and retrieve data from the databases in response to instructions.

[0032] In some embodiments, one or more of databases 130 may be used by an application to store data for the application. The databases used by the application may be different types of databases, such as key-value repositories, object repositories, general-purpose repositories backed by a file system, etc.

[0033] The system 100 of FIG. 1 can be configured and operated in a variety of ways to accommodate the various methods and apparatus described in accordance with this disclosure.

[0034] In related technologies, when a data generation model based on artificial intelligence is used to conduct human-machine dialogue with a user, the data generation model is typically a large-scale language model capable of processing and generating natural language text. In such cases, when a user needs to realize a voice dialogue between a user and a computer, response voice data for informing the user can be generated based on a cascade system of a speech recognition module, a large-scale language model, and a speech synthesis module. Specifically, the user's input question voice data is converted into question text using speech recognition technology, and a large-scale language model generates answer text based on the question text. Finally, response voice data corresponding to the answer text is generated using speech synthesis technology. While this method utilizes the thinking capabilities of conventional large-scale language models, cascade errors in the system affect the overall voice question-answering performance, resulting in insufficient accuracy of the voice responses. Furthermore, this method cannot use an artificial intelligence network to generate voice responses with diversity in terms of speaking speed, tone, emotion, etc., and therefore cannot fully meet user needs. The quality of the response voice data still needs to be improved.

[0035] Based on this, the present disclosure provides a training method for a multimodal speech language large-scale model, and by setting reward scores for both text content and speech features for the answer speech data directly generated by the multimodal speech language large-scale model, the accuracy of the answer content and the answer speech quality (e.g., higher clarity, matching of speech emotion with question data, appropriate speaking rate, etc.) can be optimized based on the text score and the speech score, respectively, and more accurate speech answers can be achieved by using the optimized model.

[0036] 2 is a flowchart illustrating a method 200 for training a multimodal speech language large-scale model according to an exemplary embodiment of the present disclosure. As shown in FIG. 2, the method 200 includes: Step S201 of inputting first question speech data into the multimodal speech language large-scale model to obtain first answer speech data generated by the multimodal speech language large-scale model; a step S202 of determining a question text corresponding to the first question voice data and an answer text corresponding to the first answer voice data; a step S203 of determining a first score based on the question text and the answer text; Step S204: determining a second score based on the voice features of the first question voice data and the first answer voice data, the voice features including at least one of voice clarity, speech rate, timbre, intonation, and emotion; and a step S205 of adjusting parameters of the multimodal speech language large-scale model based on the first score and the second score.

[0037] By applying the above method 200, reward scores for both text content and speech features are set for the answer speech data directly generated by the model during the learning process of a multimodal speech language large-scale model, thereby making it possible to more accurately optimize the accuracy of the answer content and the quality of the answer speech based on the text content score and speech feature score. For example, the clarity of the answer speech can be improved, the speaking rate can be made more appropriate, or the emotional expression of the answer speech can be made more consistent with the emotional expression of the question speech. Furthermore, by using the optimized model, more accurate and high-quality speech answers can be achieved.

[0038] In some examples, the multimodal speech language large-scale model may be a speech language large-scale model pre-trained using a large-scale corpus, and the multimodal speech language large-scale model can perform intelligent understanding and content generation for multiple modalities (e.g., speech and text information). In some examples, the multimodal speech language large-scale model uses a cross-modal encoder-decoder architecture and an attention mechanism to achieve semantic alignment and integrated modeling of multimodal information. By performing optimized training on the pre-trained large-scale model using the above method 200, the semantic understanding and speech generation capabilities of the pre-trained large-scale model can be fully utilized, and the diversity of speech features in the speech data can be further optimized. A multimodal speech language large-scale model that can represent rich and accurate speech features can be more efficiently obtained, thereby achieving more accurate human-machine voice question and answering.

[0039] In some examples, determining the question text corresponding to the first question audio data and the answer text corresponding to the first answer audio data in step S202 can be achieved by using automatic speech recognition (ASR) technology, i.e., performing character recognition on the first question audio data and the first answer audio data, and determining the question text and the answer text based on the character recognition results, thereby more accurately evaluating the accuracy of the answer content based on the text content and improving the answer quality of the model.

[0040] In some examples, the question text and the answer text are input into a scoring model in step S203 to obtain a first score output from the scoring model. The first score may be used to indicate whether the facts described in the answer text are accurate and / or whether the content of the answer text matches the intent of the question text. In some examples, the scoring model may be obtained by supervised learning using question-answer text pairs labeled with reference scores, and the output of the scoring model may be used to evaluate the content quality of the answer text. In some examples, the scoring model may be built based on a large language model (LLM) obtained by training using a large corpus. For example, by fine-tuning the pre-trained large language model using the labeling data, the evaluation model can comprehensively and accurately evaluate the quality of the answer content.

[0041] According to some embodiments, determining the second score based on the speech features of the first question speech data and the speech features of the first answer speech data in step S204 includes, in response to determining that the first question speech data includes descriptive information of speech features related to the first answer speech data, determining the second score based on the descriptive information, the speech features of the first question speech data, and the speech features of the first answer speech data. This makes it possible to obtain explicit instructions (e.g., clear requirements regarding the tone, intonation, and speaking rate of the answer) contained in the question data during the scoring process, thereby enabling the optimized model to output answer data that better matches the requirements of the question.

[0042] According to some embodiments, determining a second score based on the voice features of the first question voice data and the voice features of the first answer voice data in step S204 includes determining a speaker's identity based on the voice features of the first question voice data, and determining the second score based on the identity features, the answer text, and the voice features of the first answer voice data. By acquiring the voice features of the question data in the scoring process, it is possible to avoid inconsistencies between the answer data and the question data. For example, when the question data is provided in a female voice, the answer content should not include terms for men, which can further improve the accuracy of the answer.

[0043] According to some embodiments, determining a second score based on the speech features of the first question speech data and the first answer speech data in step S204 includes inputting the first question speech data and the first answer speech data into a question-and-answer evaluation model to determine the second score output from the question-and-answer evaluation model, where the question-and-answer evaluation model is obtained by training using the first sample question speech data, the first sample answer speech data, and a reference score. By training the question-and-answer evaluation model using labeling data and further outputting a reward score for the speech features of the speech data using the question-and-answer evaluation model in the training process of a multimodal speech language large-scale model, it is possible to improve the efficiency and accuracy of scoring.

[0044] In some examples, the question-and-answer evaluation model can be trained using multiple question-and-answer audio data pairs labeled with reference scores. To improve the accuracy of the question-and-answer evaluation model in evaluating the audio features of the answer audio data, the training data can include question-and-answer audio data pairs with the same text content but different audio features. For example, the training data can simultaneously include audio data 1 corresponding to content A and timbre B and audio data 2 corresponding to content A and timbre C. The training data can also include audio data 3 corresponding to content A and emotion D and audio data 4 corresponding to content A and emotion E. By constructing question-and-answer audio data pairs with the same text content but different audio features, the influence of differences in text content on the audio data features can be eliminated, allowing the evaluation model to more accurately learn different audio features (e.g., speech clarity, speaking rate, timbre, intonation, and emotion) and more accurately evaluate the quality of audio data with different audio features.

[0045] In some examples, the first score and the second score in steps S203 and S204 may also be determined based on other methods, such as predefined scoring rules. In this example, the predefined scoring rules may include multiple dimensions, such as conciseness of the content, clarity of the speech, and whether the speech data expresses positive and optimistic emotions. By predefining the scoring rules and guiding the optimised training of the multimodal speech language large-scale model based on these rules, the output of the multimodal speech language large-scale model can be more tailored to the needs of practical application scenarios, and higher quality human-machine voice question and answering can be achieved.

[0046] According to some embodiments, adjusting parameters of the multimodal speech language large-scale model based on the first score and the second score in step S205 includes determining reward information based on the first score and the second score, and adjusting parameters of the multimodal speech language large-scale model based on a reinforcement learning strategy according to the reward information, thereby performing reinforcement learning training based on score information to obtain a multimodal speech language large-scale model with better performance.

[0047] In some examples, the model may be adjusted and optimized using a Reinforcement Learning with Human Feedback (RLHF) method in step S205, and model parameters may be adjusted based on various types of reinforcement learning strategies, such as proximal strategy optimization, group relative strategy optimization, and direct preference optimization. By adjusting the parameters of the multimodal spoken language large-scale model using the score reward information and a strategy optimization algorithm, the output results of the multimodal spoken language large-scale model can be as close as possible to the results expected by the score reward information, i.e., obtaining higher-quality response speech data and improving the accuracy of human-machine speech dialogue.

[0048] According to some embodiments, the method 200 further includes inputting second question speech data into the multimodal speech language large-scale model to obtain second answer speech data generated by the multimodal speech language large-scale model, obtaining reference answer speech data corresponding to the second question speech data, and adjusting parameters of the multimodal speech language large-scale model based on the second answer speech data and the reference answer speech data, thereby further improving the quality of the answer speech data output by the model in accordance with a model optimization method that performs supervised learning using labeling data on top of learning based on reward scores.

[0049] In some examples, the first question speech data and the second question speech data may be the same, i.e., after the question speech data is input into the multimodal speech language large-scale model, quality scoring is performed on the answer speech data generated by the model, and at the same time, a loss value may be calculated based on the answer speech data generated by the model and reference answer speech data corresponding to the question speech data, or the quality score result may be influenced based on the difference between the model output result and the reference result, thereby adjusting and optimizing model training according to the reference labeling information and quality score information of the sample data. In some examples, the first question speech data and the second question speech data may be different, i.e., different training methods are applied to the multimodal speech language large-scale model. In one example, the training process of the multimodal speech language large-scale model includes the following two stages: In the first stage, supervised fine-tuning (SFT) can be performed on the pre-trained large-scale speech model. Specifically, sample question speech data labeled with reference answer speech data is input into the pre-trained model. A loss value is calculated based on the model output and the labeling content. Model parameters are then adjusted based on the loss value. This corresponds to a technical means of training using the second question speech data and the corresponding reference answer speech data. This training can be performed, for example, using autoregressive training with a maximum likelihood goal, thereby improving the efficiency and effectiveness of training. In the second stage, unlabeled first question speech data is input into the multimodal large-scale speech model. Content quality scoring and audio quality scoring are performed on the model output, and reinforcement learning training is performed based on the two-dimensional reward scores. This can more accurately optimize the accuracy of the answer content and the model's ability to express a wide variety of speech features, thereby improving model performance.

[0050] According to some embodiments, the multimodal large-scale speech model generates the second answer speech data as follows, and generates predictive thought information based on the second question speech data, where the predictive thought information includes description information of speech features of the second answer speech data, and generates the second answer speech data based on the predictive thought information, where method 200 further includes obtaining reference thought information corresponding to the second question speech data, and adjusting parameters of the multimodal large-scale speech model based on the predictive thought information and the reference thought information. This allows the answer data generation process to incorporate a thought chain-based thinking process, i.e., the large-scale model decomposes the generation task and outputs based on a logical chain of thinking first and then generating, and uses thought information to indicate the speech features to be output, thereby making the generated answer speech data more accurate.

[0051] In some examples, the estimation mechanism of a large-scale multimodal spoken language model is built using chain-of-thought (CoT) technology, which mimics the human cognitive model of "step-by-step thinking and step-by-step reasoning." After receiving input question speech data, the model first outputs thought information and then generates answer speech data. By having the model first consider the speech features of the answer speech data that needs to be generated, the quality of the speech data generated by the model can be improved. In this example, if sample data during the model's training process is labeled with reference thought information, the model's thought process can be highly precisely optimized based on the reference thought information and the predicted thought information output by the model, further improving the accuracy of model inference and improving the quality of data generation.

[0052] According to some embodiments, the reference thought information is text information, and generating predictive thought information based on the second question speech data comprises encoding the second question speech data into a question semantic vector in a semantic vector space and generating a thought semantic vector in the semantic vector space based on the question semantic vector, wherein generating the second answer speech data based on the predictive thought information comprises generating an answer semantic vector in the semantic vector space based on the thought semantic vector and decoding the answer semantic vector into the second answer speech data, and wherein adjusting parameters of the multimodal spoken language large-scale model based on the predictive thought information and the reference thought information comprises decoding the thought semantic vector into predictive thought text and adjusting parameters of the multimodal spoken language large-scale model based on the predictive thought text and the reference thought information. This allows for more convenient and accurate labeling of the thought information of the model using text information, thereby improving the accuracy of the thoughts of the model. In such cases, the link between question-thinking-answering in voice question and answering using the model includes both voice modal data and text modal data, and by converting the voice modal data and the text modal data into the same semantic vector space, we can achieve unified multi-modal intelligent content understanding, avoid cascading losses caused by modal conversion, optimize model performance, and improve the accuracy of data generation.

[0053] In some implementations, the encoding of speech modal data and text modal data into semantic vectors in speech vector space is achieved through tokenizer technology. Specifically, the tokenizer divides the original speech modal data and text modal data into discrete text tokens and audio tokens, which are then mapped to computer-processable numeric sequences, i.e., encoded into semantic vectors. In this implementation, the text token mapping phrase table and the audio token mapping phrase table are concatenated to form the overall phrase table space of the multimodal speech language large-scale model, i.e., to map multimodal information into the same semantic vector space, enabling the model to understand and process multimodal information and improving the quality of data generation.

[0054] According to some embodiments, the reference answer speech data is labeled with speech breakpoints, and the second answer speech data is composed of a first speech fragment and a second speech fragment. Adjusting parameters of the multimodal speech language large-scale model based on the second answer speech data and the reference answer speech data includes dividing the reference answer speech data into a first reference fragment and a second reference fragment based on the speech breakpoints, adjusting parameters of the multimodal speech language large-scale model based on the first speech fragment and the first reference fragment, and adjusting parameters of the multimodal speech language large-scale model based on the second speech fragment and the second reference fragment. In some examples, the multimodal speech language large-scale model outputs a portion of the answer speech in stages during the data generation process, and is configured to output and notify the user of the already generated portion of the answer speech first, while continuing to generate the remaining portion of the answer speech. In this case, the multimodal speech language large-scale model can notify the user of the speech answer more quickly during the human-machine speech question-answer process, thereby reducing the user's waiting time. By adjusting and optimizing the model parameters using training data labeled with audio breakpoints, the model can learn the timing to divide the answer audio data, and can also determine the timing to output some of the answer audio first during the data generation process.By outputting the answer audio data in the form of an audio fragment sequence, there is no need to perform additional audio announcements after all the answer data has been generated, improving the fluency of the model's answers in the human-machine voice dialogue process and optimizing the user experience.

[0055] FIG. 3 is a block diagram illustrating a multimodal speech language large-scale model according to an exemplary embodiment of the present disclosure. As shown in FIG. 3, the multimodal speech language large-scale model includes a speech encoder 301, a speech decoder 302, and a thought and generation network 303. In this example, the speech encoder 301 is used to encode question speech data into a question semantic vector, so that the thought and generation network 303 can generate thoughts and answers based on the question semantic vector, and then output a thought semantic vector and an answer semantic vector in sequence. In one example, the thought and generation network 303 first determines a thought semantic vector based on the question semantic vector, indicates the content and speech features to be generated based on the thought content, and then performs further thought and generation based on the question semantic vector and the thought semantic vector to obtain a final answer semantic vector. In one example, the thought semantic vector is decoded and output as thought text, thereby displaying explicit thought information to the user. After viewing the thought information, the user can provide feedback to the model as needed. The speech decoder 302 is used to decode the answer semantic vector into answer speech data and output it. In this example, the question semantic vector, thought semantic vector and answer semantic vector are in the same semantic vector space, and by mapping multimodal information into this semantic vector space, the thought and generation network 303 can unify intelligent understanding and processing of multimodal information, thereby improving the accuracy of data generation.

[0056] FIG. 4 is a block diagram illustrating a question-and-answer evaluation model according to an exemplary embodiment of the present disclosure. As shown in FIG. 4, the question-and-answer evaluation model includes a speech encoder 401 and an evaluation network 402. The speech encoder 401 is used to encode the question audio data and answer audio data to be evaluated into a question semantic vector and an answer semantic vector, so that the evaluation network 402 can obtain a second score based on the question semantic vector and the answer semantic vector. In some examples, the speech encoder 401 in the question-and-answer evaluation model and the speech encoder 301 in the aforementioned thinking and generation network 303 may be the same unit, i.e., the aforementioned thinking and generation network 303 and the evaluation network 402 in the question-and-answer evaluation model can perform intelligent understanding and information processing based on the same semantic vector space.

[0057] FIG. 5 is a schematic diagram illustrating the training process of a multimodal speech language large-scale model according to an exemplary embodiment of the present disclosure. As shown in FIG. 5, after question speech data is input into a multimodal speech language large-scale model 501 to obtain answer speech data, a speech recognition system 502 is used to determine a question text corresponding to the question speech data and an answer text corresponding to the answer speech data. A first evaluation model 503 is then used to output a first score based on the question text and the answer text, indicating whether the answer to the question is accurate and comprehensive. A second evaluation model 504 is used to output a second score for speech features based on the question speech data and the answer speech data, thereby indicating whether the answer speech data is clear or whether its speech rate, timbre, and emotion match the requirements of the question data. By training, adjusting, and optimizing the multimodal speech language large-scale model 503 based on the first score and the second score, the accuracy and richness of the speech expression of the answer output by the multimodal speech language large-scale model 503 can be highly accurately optimized, and the trained model can be used to improve the accuracy of human-machine voice question and answering.

[0058] In one example, before the learning stage, the multimodal speech language large-scale model 503 uses labeling sample data to perform fine-tuning learning, which can simultaneously label reference answer speech data and reference thought information, and further utilize the labeling information compatibility to accurately optimize the model's thought process and data generation process, thereby improving learning efficiency and optimizing model performance.

[0059] In one example, the multimodal speech language large-scale model 503 outputs answer speech data in the form of a speech fragment sequence. In this case, after the multimodal speech language large-scale model 503 finishes outputting all answer speech fragments, it can concatenate them into complete answer speech data, which can then be converted into answer text and scored. When the speech breakpoints of the reference answer speech data are labeled in the labeling sample data, it can indicate whether the division timing of the answer speech fragments output from the model is accurate based on the speech breakpoint information. Adjustment and optimization based on the labeling information allows the model to learn accurate speech answer division timing and improve the fluency of the speech answer. According to one aspect of the present disclosure, there is provided a method for generating audio data. Figure 6 is a flowchart illustrating a method 600 for generating audio data according to an exemplary embodiment of the present disclosure. As shown in Figure 6, the method 600 includes: Step S601: acquiring user question voice data; Step S602: inputting the question speech data into a large-scale multimodal speech model obtained by training using method 200, thereby obtaining answer speech data generated by the large-scale multimodal speech model; and step S603 of returning the answer voice data to the user.

[0060] By using the above-mentioned multimodal speech language large-scale model to perform human-machine voice question and answering, it is possible to return to the user more accurate answer speech data with richer speech characteristics such as timbre, emotion, and speaking rate, thereby making the answer content more suited to the user's needs and improving the user experience.

[0061] FIG. 7 is a schematic diagram illustrating a speech data generation process according to an exemplary embodiment of the present disclosure. As shown in FIG. 7 , after receiving a question speech 01, the multimodal speech language large-scale model 700 first thinks and outputs step-by-step thought information A, and then outputs answer speech a obtained based on thought information A. While answer speech a is being played, the model continues thinking and outputs the next step-by-step thought information B, and then outputs answer speech b obtained based on thought information B. When the model determines that it has already output all the answers to the question speech 01, it can add a code indicating that the answer is complete to the end of answer speech b, thereby indicating that the question-answer training round for the question speech 01 is finished. After the previous training round is finished, the user can continue to input a question speech 02, and the multimodal speech language large-scale model 700 can think and generate data based on the new question speech 02, thereby outputting thought information C and answer speech c. During the data generation process, the model outputs answer voice data in the form of a voice fragment sequence, thereby realizing sequenced voice output during the voice dialogue process, eliminating the need to wait until all answer data has been generated before making a voice announcement, thereby reducing user waiting time, improving the fluency of voice responses, and further improving the user experience.

[0062] According to one aspect of the present disclosure, there is provided an apparatus for training a large-scale model of multimodal spoken language. Fig. 8 is a block diagram illustrating an apparatus 800 for training a large-scale model of multimodal spoken language according to an exemplary embodiment of the present disclosure. As shown in Fig. 8, the apparatus 800 includes: a first acquiring unit 801 configured to input first question speech data into the multimodal speech language large-scale model to acquire first answer speech data generated by the multimodal speech language large-scale model; a first determining unit 802 configured to determine a question text corresponding to the first question audio data and an answer text corresponding to the first answer audio data; a second determining unit 803 configured to determine a first score based on the question text and the answer text; a third determining unit 804 configured to determine a second score based on audio features of the first question audio data and audio features of the first answer audio data, wherein the audio features include at least one of audio clarity, speech rate, timbre, intonation, and emotion; an adjusting unit 805 configured to adjust parameters of the multimodal speech language large-scale model based on the first score and the second score.

[0063] According to some embodiments, the third determination unit 804 is configured, in response to determining that the first question audio data includes descriptive information of audio features related to the first answer audio data, to determine the second score based on the descriptive information, the audio features of the first question audio data and the audio features of the first answer audio data.

[0064] According to some embodiments, the third determination unit 804 includes a first determination subunit configured to determine a speaker's identity characteristics based on the audio characteristics of the first question audio data, and a second determination subunit configured to determine the second score based on the identity characteristics, the answer text and the audio characteristics of the first answer audio data.

[0065] According to some embodiments, the third determination unit 804 is configured to determine the second score output from the question-and-answer evaluation model by inputting the first question audio data and the first answer audio data into a question-and-answer evaluation model, where the question-and-answer evaluation model is obtained by training using first sample question audio data, first sample answer audio data, and a reference score.

[0066] According to some embodiments, the adjustment unit 805 includes a third determination subunit configured to determine reward information based on the first score and the second score, and a first adjustment subunit configured to adjust parameters of the multimodal speech language large-scale model based on a reinforcement learning strategy responsive to the reward information.

[0067] According to some embodiments, the first acquisition unit 801 is further configured to acquire second answer speech data generated by the multimodal speech language large-scale model by inputting second question speech data into the multimodal speech language large-scale model, and the device 800 further includes a fourth acquisition unit that acquires reference answer speech data corresponding to the second question speech data, wherein the adjustment unit 805 is further configured to adjust parameters of the multimodal speech language large-scale model based on the second answer speech data and the reference answer speech data.

[0068] According to some embodiments, the multimodal speech language large-scale model is configured to generate predictive thought information based on the second question speech data, where the predictive thought information includes descriptive information of speech features of the second answer speech data, and to generate the second answer speech data based on the predictive thought information, where the apparatus 800 further includes a fifth acquisition unit that acquires reference thought information corresponding to the second question speech data, and the adjustment unit 805 is configured to adjust parameters of the multimodal speech language large-scale model based on the predictive thought information and the reference thought information.

[0069] According to some embodiments, the reference thought information is text information, and the multimodal speech language large-scale model is configured to encode the second question speech data into a question semantic vector in a semantic vector space, generate a thought semantic vector in the semantic vector space based on the question semantic vector, generate an answer semantic vector in the semantic vector space based on the thought semantic vector, and decode the answer semantic vector into the second answer speech data, wherein the adjustment unit 805 includes a decoding subunit configured to decode the thought semantic vector into predicted thought text, and a second adjustment subunit configured to adjust parameters of the multimodal speech language large-scale model based on the predicted thought text and the reference thought information.

[0070] According to some embodiments, the reference answer audio data is labeled with audio breakpoints, the second answer audio data consists of a first audio fragment and a second audio fragment, and the adjustment unit 805 includes a splitting subunit configured to split the reference answer audio data into a first reference fragment and a second reference fragment based on the audio breakpoints, and a third adjustment subunit configured to adjust parameters of the multimodal speech language large-scale model based on the first audio fragment and the first reference fragment, and to adjust parameters of the multimodal speech language large-scale model based on the second audio fragment and the second reference fragment.

[0071] According to one embodiment of the present disclosure, there is provided an audio data generating device. Fig. 9 is a block diagram illustrating an audio data generating device 900 according to an exemplary embodiment of the present disclosure. As shown in Fig. 9, the device 900 includes: A multimodal speech language large-scale model 901 obtained by training using the training device 800 for a multimodal speech language large-scale model as described above; a second acquiring unit 902 configured to acquire user question voice data; a third acquiring unit 903 configured to input the question speech data into the multimodal speech language large-scale model to acquire answer speech data generated by the multimodal speech language large-scale model; a returning unit 904 configured to return the answer voice data to the user.

[0072] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of relevant user personal information shall all comply with the provisions of relevant laws and regulations and shall not violate public order and morals.

[0073] According to one aspect of the present disclosure, there is further provided an electronic device, the electronic device including at least one processor and a memory communicatively coupled to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform at least one of the method for training a multimodal speech language large-scale model and the method for generating speech data.

[0074] According to one aspect of the present disclosure, there is further provided a non-transitory computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to cause the computer to perform at least one of the above-described method for training a multimodal speech language large-scale model and the method for generating speech data.

[0075] According to one aspect of the present disclosure, there is further provided a computer program product including a computer program, wherein the computer program, when executed by a processor, further implements at least one of the above-mentioned method for training a multimodal speech language large-scale model and the method for generating speech data.

[0076] As shown in FIG. 10 , a block diagram of an electronic device 1000 that can be used as a server or client of the present disclosure will be described below as an example of a hardware device applicable to various aspects of the present disclosure. The electronic device may represent various types of digital electronic computers, such as laptop computers, desktop computers, stage computers, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various types of mobile devices, such as personal digital processing devices, mobile phones, intelligent phones, wearable devices, and other similar computing devices. The components, their connections, and their functions shown herein are merely exemplary and do not limit the implementation of the present disclosure as described and / or claimed herein.

[0077] 10, the device 1000 includes a computing unit 1001 and can perform various appropriate operations and processes by means of a computer program stored in a read-only memory (ROM) 1002 or loaded from a storage unit 1008 into a random access memory (RAM) 1003. The RAM 1003 can further store various programs and data necessary for operating the device 1000. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0078] The components of device 1000 are connected to I / O interface 1005 and include input unit 1006, output unit 1007, storage unit 1008, and communication unit 1009. Input unit 1006 may be any type of device capable of inputting information into device 1000. Input unit 1006 can generate input numeric or character information and key signal input for user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, keyboard, touch screen, trackpad, trackball, joystick, microphone, and / or remote controller. Output unit 1007 may be any type of device capable of presenting information, and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1008 may include, but is not limited to, a magnetic disk, an optical disk, etc. The communication unit 1009 enables the device 1000 to exchange information / data with other devices via a computer network, e.g., the Internet, and / or various telecommunications networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, e.g., a Bluetooth device, an 802.11 device, a Wi-Fi device, a WiMAX device, a cellular communication device, and / or the like.

[0079] The computing unit 1001 may be any of a variety of general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 1001 may include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 executes the methods and processes described above, such as the method for training a multimodal speech language large-scale model or the method for generating speech data. For example, in some embodiments, the method for training a multimodal speech language large-scale model or the method for generating speech data may be implemented as a computer software program and tangibly included in a machine-readable medium, such as the storage unit 1008. In some embodiments, some or all of the computer program may be loaded and / or installed into the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, it can perform one or more steps of the method for training a multimodal speech language large scale model or the method for generating speech data described above. Alternatively, in another embodiment, the computing unit 1001 may be configured to perform the method for training a multimodal speech language large scale model or the method for generating speech data in any other suitable way (e.g., by firmware).

[0080] Various embodiments of the systems and techniques described herein may be realized in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being embodied in one or more computer programs that may be executed and / or interpreted by a programmable system including at least one programmable processor, which may be a special purpose or general purpose programmable processor, and which may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device;

[0081] Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus such that, when executed by the processor or controller, the program code performs the functions / operations specified in the flowcharts and / or block diagrams. The program code may be entirely executed on a machine, partially executed on a machine, partially executed on a machine and partially executed on a remote machine as a separate software package, or entirely executed on a remote machine or server.

[0082] In the context of this disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples of machine-readable storage media include an electrical connection with one or more leads, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0083] To provide for interaction with a user, the systems and techniques described herein may be implemented in a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) by which a user may provide input to the computer. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and may receive input from the user in any form (including audio input, voice input, and tactile input).

[0084] The systems and techniques described herein may be implemented in a computing system including backstage components (e.g., as a data server), middleware components (e.g., as an application server), front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with the system or technique implementation), or any combination of backstage components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0085] The computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship is created by running computer programs on the corresponding computers that have a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0086] It should be understood that the various forms of flow described above may be used to rearrange the order, add or remove steps, etc. For example, the steps described in this disclosure may be performed in parallel, sequentially, or in a different order, as long as the technical solutions disclosed in this disclosure can achieve the desired results, and the present disclosure is not limited thereto.

[0087] Although embodiments or examples of the present disclosure have been described with reference to the drawings, it should be understood that the above-described methods, systems, and devices are merely exemplary embodiments or examples, and that the scope of the present disclosure is not limited by these embodiments or examples, but only by the appended claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by equivalent elements. Furthermore, steps may be performed in a different order from that described in this disclosure. Furthermore, various elements in the embodiments or examples may be combined in various ways. Importantly, as technology evolves, many elements described herein may be replaced by equivalent elements that appear later in this disclosure.

Claims

1. 1. A method for training a large-scale model of multimodal spoken language, comprising: inputting first question speech data into the multimodal speech language large-scale model to obtain first answer speech data generated by the multimodal speech language large-scale model; determining a question text corresponding to the first question voice data and an answer text corresponding to the first answer voice data; determining a first score based on the question text and the answer text; determining a second score based on voice features of the first question voice data and the first answer voice data, the voice features including at least one of voice clarity, speech rate feature, timbre feature, intonation feature, and emotion feature; and adjusting parameters of the multimodal speech language large scale model based on the first score and the second score.

2. Determining the second score based on the voice features of the first question voice data and the voice features of the first answer voice data includes:

2. The method of claim 1, further comprising, in response to determining that the first question audio data includes descriptive information of audio features related to the first answer audio data, determining the second score based on the descriptive information, the audio features of the first question audio data, and the audio features of the first answer audio data.

3. Determining the second score based on the voice features of the first question voice data and the voice features of the first answer voice data includes: determining a speaker's identity characteristics based on the speech characteristics of the first question speech data; and determining the second score based on the identity characteristics, the answer text, and audio characteristics of the first answer audio data.

4. Determining the second score based on the voice features of the first question voice data and the voice features of the first answer voice data includes: inputting the first question voice data and the first answer voice data into a question-and-answer evaluation model to determine the second score output from the question-and-answer evaluation model; The method according to claim 1 , wherein the question-and-answer evaluation model is obtained by training using first sample question speech data, first sample answer speech data, and a reference score.

5. adjusting parameters of the multimodal speech language large scale model based on the first score and the second score includes: determining reward information based on the first score and the second score; and adjusting parameters of the multimodal speech language large-scale model based on a reinforcement learning strategy responsive to the reward information.

6. inputting second question speech data into the multimodal speech language large-scale model to obtain second answer speech data generated by the multimodal speech language large-scale model; acquiring reference answer voice data corresponding to the second question voice data; The method of claim 1 , further comprising: adjusting parameters of the multimodal speech linguistic large-scale model based on the second answer speech data and the reference answer speech data.

7. The multimodal speech language large scale model generating predictive thought information including descriptive information about a voice feature of the second answer voice data based on the second question voice data; generating the second response voice data by generating the second response voice data based on the predictive thought information; wherein the method comprises: acquiring reference thought information corresponding to the second question voice data; The method of claim 6 , further comprising: adjusting parameters of the multimodal speech language large-scale model based on the predictive thought information and the reference thought information.

8. The reference thought information is text information, and generating predictive thought information based on the second question voice data includes: encoding the second question speech data into a question semantic vector in a semantic vector space; generating a thought semantic vector in the semantic vector space based on the question semantic vector; Here, generating the second response voice data based on the predictive thought information includes: generating an answer semantic vector in the semantic vector space based on the thought semantic vector; decoding the answer semantic vector into the second answer speech data; Here, adjusting the parameters of the multimodal speech language large-scale model based on the predictive thought information and the reference thought information includes: decoding the thought semantic vectors into predictive thought text; and adjusting parameters of the multimodal speech linguistic large-scale model based on the predicted thought text and the reference thought information.

9. The reference answer speech data is labeled with speech breakpoints, and the second answer speech data is composed of a first speech fragment and a second speech fragment; Here, adjusting the parameters of the multimodal speech language large-scale model based on the second answer speech data and the reference answer speech data includes: Dividing the reference answer audio data into a first reference fragment and a second reference fragment based on the audio breakpoints; adjusting parameters of the multimodal speech language large-scale model based on the first speech fragment and the first reference fragment; and adjusting parameters of the multimodal speech language large-scale model based on the second speech fragment and the second reference fragment.

10. 1. A method for generating audio data, comprising: Acquiring user question voice data; inputting the question speech data into a large-scale multimodal speech language model obtained by training using the method according to any one of claims 1 to 9, thereby obtaining answer speech data generated by the large-scale multimodal speech language model; and returning the answer voice data to the user.

11. A training device for a large-scale model of multimodal spoken language, comprising: a first acquiring unit configured to input first question speech data into the multimodal speech language large-scale model to acquire first answer speech data generated by the multimodal speech language large-scale model; a first determination unit configured to determine a question text corresponding to the first question audio data and an answer text corresponding to the first answer audio data; a second determination unit configured to determine a first score based on the question text and the answer text; a third determination unit configured to determine a second score based on speech features of the first question speech data and speech features of the first answer speech data, wherein the speech features include at least one of speech clarity, speech rate, timbre, intonation, and emotion; an adjustment unit configured to adjust parameters of the multimodal large-scale model of spoken language based on the first score and the second score.

12. A voice data generating device, a multimodal speech language large-scale model obtained by training using the device according to claim 11; a second acquisition unit configured to acquire user question voice data; a third acquisition unit configured to input the question speech data into the multimodal speech language large-scale model to acquire answer speech data generated by the multimodal speech language large-scale model; a feedback unit configured to feed back the answer voice data to the user.

13. An electronic device, at least one processor; a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the method of any one of claims 1 to 10.

14. A non-transitory computer-readable storage medium having stored thereon computer instructions used to cause a computer to carry out the method of any one of claims 1 to 10.

15. A computer program product which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images