A method and apparatus for processing multi-modal data

Abstract

The application provides a multi-modal data processing method and device, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring input data, the input data comprising at least one of text modal input data and voice modal input data; encoding the input data to obtain an encoding vector; taking the encoding vector as an input of a large language model to output a voice-text joint representation vector, the voice-text joint representation vector comprising features of voice data to be generated and features of text data to be generated; and generating voice data and text data based on the voice-text joint representation vector, the voice data and the text data having the same semantic content. The application utilizes a large language model to process input data and output a voice-text joint representation vector, and simultaneously represents text and voice through the voice-text joint representation vector, thereby realizing that the large language model has good voice understanding and generation capabilities.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method and apparatus for processing multimodal data. Background Technology

[0002] Large language models (LLMs) have developed rapidly, reaching unprecedented heights in many tasks. The most representative type of LLM is the GPT-type model. The main characteristic of this type of model is that it uses Transformers for sequence modeling, and the input and output formats are sequences of discrete codes (e.g., words, characters, letters).

[0003] The success of large language models has driven the development of multimodal large models. It is widely believed in the industry that large language models will gradually integrate data from other modalities, thus evolving into multimodal large models. Among these, multimodal large models that combine speech and text modalities are relatively common. Such models can simultaneously receive speech and text input and integrate many capabilities of text-based large language models, directly outputting speech or simultaneously outputting speech and text.

[0004] However, existing speech-text multimodal solutions have some shortcomings, such as having only speech understanding capabilities but not speech generation capabilities, or only having speech generation capabilities but not speech understanding capabilities, or having insufficient speech understanding capabilities. Summary of the Invention

[0005] The embodiments of this application provide a method and apparatus for processing multimodal data, enabling large language models to simultaneously possess good speech understanding and generation capabilities.

[0006] In a first aspect, this application provides a method for processing multimodal data, including acquiring input data, which includes input data of at least one modality, such as text modal input data and speech modal input data; encoding the input data to obtain an encoding vector; using the encoding vector as input to a large language model to output a speech-text joint representation vector, which includes features of the speech data to be generated and features of the text data to be generated; and generating speech data and text data based on the speech-text joint representation vector, wherein the content of the speech data and text data is the same.

[0007] This application utilizes a large language model to process input data and output a joint speech-text representation vector. This vector simultaneously represents both text and speech. Thus, on the one hand, it leverages the reasoning capabilities of the large language model to understand the speech input data; on the other hand, it uses the joint speech-text representation vector to generate both text and speech data. This gives the large language model both strong speech comprehension and generation capabilities, and allows it to output the same content using both speech and text modalities, making it easier for users to receive and improving the user experience. For example, based on user input data, the large language model generates a response through reasoning, which is then fed back to the user via voice broadcast and text display. Compared to a single-modal output response, users do not need to constantly stare at the screen or listen to the voice broadcast to receive the response content, making it more convenient and improving the user experience.

[0008] In one possible implementation, the joint speech-text representation vector is a continuous representation vector. Compared with discrete representation vectors, continuous representation vectors represent richer information, which is beneficial to improving the quality of subsequent speech generation.

[0009] In another possible implementation, the input data includes speech modality input data. A specific implementation of encoding the input data to obtain an encoding vector is as follows: the speech modality input data is used as input to a speech encoder, outputting a first speech encoding vector; the speech encoding vector is used as input to an adapter, outputting a second speech encoding vector. The dimension of the second speech encoding vector is the same as the input dimension of the large language model. The second speech encoding vector is a continuous representation vector. Based on the second speech encoding vector, an encoding vector is obtained. That is, when the input data includes speech modality data, the encoding vector obtained by encoding the input data only includes the second speech encoding vector input by the adapter.

[0010] The speech encoding vector output by the speech encoder is mapped to the input dimension of the large language model (i.e., the same dimension as the text embedding) through the adapter, so that the large language model can process the speech encoding vector. The speech encoding vector is represented by a continuous representation vector, which retains more information of the speech input data and is beneficial to subsequent speech understanding and generation.

[0011] In another possible implementation, the input data also includes the input data of the text modality; the encoding vector also includes the embedding representation vector corresponding to the input data of the text modality.

[0012] In other words, this application also supports simultaneous input of text and speech modal data, inferring a joint speech-text representation vector based on the text and speech modal input data, and then outputting the speech and text corresponding to the input data based on the joint speech-text representation vector. For example, the input data could be: (speech) Look, (text) How blue the sky is. The output could be: (text) Yes, the weather is so nice! (and the corresponding speech, i.e., the speech is broadcast: Yes, the weather is so nice!).

[0013] In another possible implementation, the output of the large language model is connected to the input of both the speech output module and the text output module. A specific implementation for generating speech data and text data based on the joint speech-text representation vector is as follows: the joint speech-text representation vector is used as input to the speech output module to output speech data; the joint speech-text representation vector is used as input to the text output module to output text data. By setting speech and text output headers, both speech and text generation can be supported.

[0014] In another possible implementation, the text output module includes a text filtering layer and an output layer. A specific implementation of taking the joint speech-text representation vector as input to the speech output module and outputting speech data is as follows: taking the joint speech-text representation vector as input to the text filtering layer and outputting a text representation vector; taking the text representation vector as input to the output layer and outputting a probability distribution, which indicates the probability of each word in the vocabulary being output; and sampling from the vocabulary based on the probability distribution to obtain text data.

[0015] It should be noted that the text filtering layer in this application embodiment can be implemented in a variety of ways, such as neural network structures such as multilayer perceptron (MLP) and convolutional neural networks (CNN).

[0016] By setting a text filtering layer in the text output module to filter out the speech representation part in the speech-text joint representation vector, a text representation vector is obtained. Then, the text representation vector is processed by the output layer to output a probability distribution. Based on the probability distribution, sampling is performed to obtain the token to be output, thereby realizing the generation of text data.

[0017] In another possible implementation, the speech output module includes a speech filtering layer and a mapping layer; a specific implementation of taking the speech-text joint representation vector as input to the text output module and outputting text data is as follows: taking the speech-text joint representation vector as input to the speech filtering layer and outputting a speech representation vector; taking the speech representation vector as input to the mapping layer and outputting speech data.

[0018] Understandably, speech filtering layers can be implemented in various ways, such as using neural network architectures like multilayer perceptrons (MLPs) and convolutional neural networks (CNNs). Alternatively, mapping layers can be implemented using MLPs, fully connected layers (FCs), or similar methods.

[0019] By setting a speech filtering layer in the speech output module to filter out the text representation part in the speech-text joint representation vector, a speech representation vector is obtained. Since the speech representation vector has a low sampling rate, while the real speech data has a high sampling rate, a mapping layer is needed to map the speech representation vector from the lower sampling rate to the real speech data space with a higher sampling rate. For example, the sampling rate of the speech representation vector is represented as 50 vectors per second, indicating a 50 Hz audio sampling rate. However, real speech data typically has a 16 kHz sampling rate, meaning it includes 16 KB of speech data per second. The mapping layer maps the speech representation vector to the real speech data space according to a certain ratio, thus obtaining the speech data. In another possible implementation, the large language model is an autoregressive type large language model. The multimodal data processing method provided in this application also includes: inputting the speech-text joint representation vector into the speech input module, outputting the processed speech-text joint representation vector from the speech input module, where the dimension of the processed speech-text joint representation vector is the same as the input dimension of the large language model; and using the processed speech-text joint representation vector as the input to the large language model.

[0020] Using autoregressive large language models, such as the GPT model, it is possible to support the understanding and generation of text and speech of arbitrary length.

[0021] In another possible implementation, the output steps for speech and text data are the same. That is, speech and text are played and stopped synchronously. In this way, the end of the text controls the end of the speech data output, that is, the end of the sentence in the text output is used to determine whether to stop the output of text and speech.

[0022] In another possible implementation, the multimodal data processing method provided in this application further includes: inputting the joint speech-text representation vector into the speech stop determination module; and determining whether the speech data output has stopped based on the signal output by the speech stop determination module. For example, the speech stop determination module can use a binary separator to determine whether the speech data output has stopped, decoupling it from the text output, enabling asynchronous output with the text, supporting only speech output, and automatically determining whether to stop outputting speech.

[0023] In another possible implementation, the voice stop detection module includes a binary classifier. The signal output by the voice stop detection module includes a first value or a second value. The first value indicates the end of voice data output, and the second value indicates that the voice data output has not ended. A specific implementation for determining whether the voice data output has stopped based on the signal output by the voice stop detection module is as follows: if the voice stop detection module outputs the first value, then the voice data output is determined to have stopped; if the voice stop detection module outputs the second value, then the voice data output is determined to have not stopped. In this way, the output of voice data and the output of text data are decoupled, allowing for the separate output of voice data or the separate output of text data.

[0024] In another possible implementation, the large language model is trained based on training data; the training data includes multiple training sample pairs, each training sample pair including at least one of the following: text sample and text-speech pair corresponding to the text sample, speech sample and text-speech pair corresponding to the speech sample, and text-speech pair corresponding to the text-speech sample, wherein the text and speech in the text-speech pair have the same semantics.

[0025] By designing the training data, a large language model can be made capable of processing multimodal data including text and speech.

[0026] Secondly, this application also provides a multimodal data processing apparatus, including an acquisition module, an encoding module, a joint representation module, and a generation module. The acquisition module acquires input data, which includes input data for at least one modality, such as text modality input data and speech modality input data. The encoding module encodes the input data to obtain an encoding vector. The joint representation module uses the encoding vector as input to a large language model and outputs a speech-text joint representation vector, which includes features of the speech data to be generated and features of the text data to be generated. The generation module generates speech data and text data based on the speech-text joint representation vector, wherein the speech data and text data have the same semantic content.

[0027] In one possible implementation, the joint speech-text representation vector is a continuous representation vector.

[0028] In another possible implementation, the input data includes input data for the speech modality; the encoding module is specifically used to: take the input data of the speech modality as input to the speech encoder and output a first speech encoding vector; take the speech encoding vector as input to the adapter and output a second speech encoding vector, the second speech encoding vector having the same dimension as the input dimension of the large language model, the second speech encoding vector being a continuous representation vector, and the encoding vector being the second speech encoding vector.

[0029] In another possible implementation, the input data also includes the input data of the text modality; the encoding vector also includes the embedding representation vector corresponding to the input data of the text modality.

[0030] In another possible implementation, the output of the large language model is connected to the input of the speech output module and the input of the text output module, respectively; the generation module is specifically used to: take the joint speech-text representation vector as the input of the speech output module and output speech data; take the joint speech-text representation vector as the input of the text output module and output text data.

[0031] In another possible implementation, the text output module includes a text filtering layer and an output layer. A specific implementation of taking the joint speech-text representation vector as input to the speech output module and outputting speech data is as follows: taking the joint speech-text representation vector as input to the text filtering layer and outputting a text representation vector; taking the text representation vector as input to the output layer and outputting a probability distribution, which indicates the probability of each word in the vocabulary being output; and sampling from the vocabulary based on the probability distribution to obtain text data.

[0032] In another possible implementation, the speech output module includes a speech filtering layer and a mapping layer; a specific implementation of taking the speech-text joint representation vector as input to the text output module and outputting text data is as follows: taking the speech-text joint representation vector as input to the speech filtering layer and outputting a speech representation vector; taking the speech representation vector as input to the mapping layer and outputting speech data.

[0033] In another possible implementation, the large language model is an autoregressive large language model; the multimodal data processing apparatus provided in this application also includes an input module, which is used to input the speech-text joint representation vector to the speech input module and output the speech-text joint representation vector processed by the speech input module. The dimension of the processed speech-text joint representation vector is the same as the input dimension of the large language model; the processed speech-text joint representation vector is used as the input of the large language model.

[0034] In another possible implementation, the number of output steps for voice data and text data is the same.

[0035] In another possible implementation, the multimodal data processing apparatus provided in this application further includes a stop determination module, which is used to input the joint speech-text representation vector to the speech stop determination module; and determine whether the output of speech data has stopped based on the signal output by the speech stop determination module.

[0036] In another possible implementation, the voice stop determination module includes a binary classifier. The signal output by the voice stop determination module includes a first value or a second value. The first value indicates that the voice data output has ended, and the second value indicates that the voice data output has not ended. A specific implementation for determining whether the voice data output has stopped based on the signal output by the voice stop determination module is as follows: when the voice stop determination module outputs the first value, it is determined that the voice data output has stopped; when the voice stop determination module outputs the second value, it is determined that the voice data output has not stopped.

[0037] In another possible implementation, the large language model is trained based on training data; the training data includes multiple training sample pairs, each training sample pair including at least one of the following: text sample and text-speech pair corresponding to the text sample, speech sample and text-speech pair corresponding to the speech sample, and text-speech pair corresponding to the text-speech sample, wherein the text and speech in the text-speech pair have the same semantics.

[0038] Thirdly, embodiments of this application provide a computing device, including a memory and a processor, wherein the memory stores instructions that, when executed by the processor, cause the method described in the first aspect or any possible implementation of the first aspect to be implemented.

[0039] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the method described in the first aspect or any possible implementation thereof to be implemented.

[0040] Fifthly, embodiments of this application also provide a computer program or computer program product, which includes instructions that, when executed, cause a computer to perform the method described in the first aspect or any possible implementation thereof.

[0041] In a sixth aspect, embodiments of this application also provide a chip including at least one processor and a communication interface, the processor being configured to perform the method described in the first aspect or any possible implementation thereof.

[0042] It is understandable that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here for the sake of brevity. Attached Figure Description

[0043] Figure 1 This diagram illustrates a basic framework for artificial intelligence.

[0044] Figure 2 A system architecture diagram of the sample processing system provided in this application embodiment;

[0045] Figure 3 A schematic diagram of the hardware structure of a chip provided in an embodiment of this application;

[0046] Figure 4 A flowchart illustrating the multimodal data processing method provided in this application embodiment;

[0047] Figure 5 A schematic diagram illustrating the input and output of a large language model according to an embodiment of this application is shown;

[0048] Figure 6 This paper illustrates an architectural diagram of a multimodal model provided in an embodiment of this application.

[0049] Figure 7 This illustration shows another architectural diagram of the multimodal model provided in an embodiment of this application;

[0050] Figure 8 This illustration shows another architectural diagram of the multimodal model provided in an embodiment of this application;

[0051] Figure 9 A schematic diagram of the structure of a multimodal data processing device provided in an embodiment of this application;

[0052] Figure 10 A schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0053] The term "and / or" used in this article describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0054] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same properties in the description of embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such processes, methods, systems, products, or apparatus.

[0055] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0056] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0057] First, the overall workflow of the artificial intelligence system will be described. Please refer to [link / reference]. Figure 1 , Figure 1 A schematic diagram of an artificial intelligence framework is shown, which describes the overall workflow of an artificial intelligence system and is applicable to general artificial intelligence domain needs.

[0058] The above-mentioned artificial intelligence framework is elaborated below from two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information to the industrial ecosystem of systems. From the perspective of the IT value chain, the main framework of artificial intelligence includes an infrastructure layer, a data layer, a data processing layer, a general capability layer (achieving general capabilities through data processing results, such as translation, text analysis, computer vision processing, speech recognition, image recognition, image generation, etc.), and the topmost application layer, which encapsulates the overall artificial intelligence solution, productizing intelligent information decision-making and realizing its practical application.

[0059] (1) Infrastructure:

[0060] The infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. Communication with the external world is achieved through sensors; computing power is provided by intelligent chips, including but not limited to hardware acceleration chips such as central processing units (CPUs), neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs). The basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the external world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0061] (2) Data

[0062] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0063] (3) Data processing

[0064] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0065] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0066] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0067] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0068] (4) General ability

[0069] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0070] (5) Smart Products and Industry Applications

[0071] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, safe city, and intelligent terminals.

[0072] This application can be applied to the field of natural language processing in the field of artificial intelligence. The following will introduce several application scenarios that have been implemented in products, taking natural language processing as an example.

[0073] Multimodal processing applications

[0074] The product form of this application embodiment can be a multimodal processing application. Multimodal applications can run on a terminal or a cloud-based server.

[0075] Multimodal processing applications can perform processing tasks on multimodal data, such as text or speech data from one or more modalities, and obtain processing results. These results can include data from one or more modalities of text or speech. In other words, a single processing model can process data from multiple modalities and generate data from multiple modalities.

[0076] For example, for a translation task (e.g., Chinese to English), a user can input data in any modality, such as text (e.g., text input: It's so nice to meet you) or speech (e.g., speech input: It's so nice to meet you), or input data in both text and speech models (e.g., text input: It's so nice to meet you, speech input: It's so nice). A multimodal processing application can then use the method provided in this application to process the data and obtain the translation result, such as "nice to meet you." The translation result is output in text or speech form, or simultaneously in both text and speech form.

[0077] For example, in the case of intelligent question answering tasks, a multimodal processing application can receive user input data in one modality (text or speech), or in both text and speech modalities, and process it using the methods provided in this application to obtain a reasonable response to the input data. The generated response can be returned to the user in the form of voice broadcast, text, or both text and speech. It is understood that intelligent question answering refers to a human-computer interaction question-and-answer method that receives user questions and answers them using accurate and concise natural language.

[0078] For example, a user can open a multimodal processing application installed on a terminal device and input data in at least one modality of text and speech. The multimodal processing application executes the method provided in the embodiments of this application to generate data in at least one modality of text and speech for the input data, and presents the result to the user, such as displaying or broadcasting it via voice.

[0079] For example, a user can open a multimodal processing application installed on a terminal device and input data in at least one modality, such as text or speech. The multimodal processing application sends the input data to a cloud-based server. The cloud-based server deploys a multimodal model trained using the training method provided in this application. The cloud-based server calls the multimodal model to process the input data and obtain the processing result. The cloud-based server then sends the processing result back to the terminal device, and the terminal device presents the processing result (including data in at least one modality, such as text or speech) to the user.

[0080] In some other examples, the product form of this application embodiment may also be a webpage or a mini-program that provides multimodal data (text and voice modal) processing functions.

[0081] To facilitate understanding of this solution, the embodiments of this application first combine... Figure 2 The system architecture 200 provided in the embodiments of this application will be described below, see appendix. Figure 2 The data acquisition device 260 is used to collect training data, which is the data used to train and generate the model in this embodiment of the application.

[0082] After collecting the training data, the data acquisition device 260 stores the training data in the database 230. The training device 220 trains the multimodal model based on the training data maintained in the database 230 and uses the training method provided in this application embodiment to obtain the target model / rule 201.

[0083] In practical applications, the training data maintained in database 230 may not all come from the data acquisition device 260; it may also be received from other devices. Furthermore, it should be noted that training device 220 may not necessarily train the target model / rule 201 entirely based on the training data maintained in database 230; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0084] The target model / rule 201 trained using training device 220 can be applied to different systems or devices, such as... Figure 2 In the execution device 210, the execution device 210 can be a terminal, such as a mobile phone, tablet computer, laptop computer, augmented reality (AR), virtual reality (VR), wearable device, smart robot, vehicle terminal, etc., or it can be a server or cloud device, etc.

[0085] Specifically, the training device 220 can deploy the trained generative model to the execution device 210.

[0086] exist Figure 2 In this embodiment, the execution device 210 configures the input / output (I / O) interface 212 for exchanging data with external devices. The user can input data to the I / O interface 212 through the client device 240. In this embodiment, the data may include control condition data input by the user.

[0087] Preprocessing modules 213 and 214 are used to preprocess the input data (such as text and / or voice modal input data) received from I / O interface 212. It should be understood that preprocessing modules 213 and 214 may be absent, or only one preprocessing module may be used. When preprocessing modules 213 and 214 are absent, the calculation module 211 can directly process the input data.

[0088] The calculation module 211 is used to perform calculations and other related processing on the data input from the preprocessing module 213 and the preprocessing module 214 according to the target model / rule 201 mentioned above.

[0089] During the preprocessing of input data by the execution device 210, or during the calculation module 211 of the execution device 210 performing calculations and other related processes, the execution device 210 can call the data, code, etc. of the database storage system 250 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 250.

[0090] Finally, I / O interface 212 returns the processing result (e.g., response data generated from the input data, including text and / or voice modal data, etc.) to client device 240 for use by the user. It should be understood that the target model / rule 201 differs for different data generation tasks (including translation, intelligent question answering, and chat tasks, etc.), and the processing results will also differ accordingly.

[0091] It is worth noting that the training device 220 can generate a target model / rule 201 corresponding to different downstream systems. This target model / rule 201 can achieve the above-mentioned goals or complete the above-mentioned tasks, thereby providing the user with the required results. It should also be noted that the training device 220 can generate corresponding preprocessing models for the target model / rule 201 corresponding to different downstream systems, such as the preprocessing models corresponding to preprocessing modules 213 and / or 214.

[0092] exist Figure 2 In the illustrated scenario, the user can manually specify data to be input into execution device 210 (e.g., text and / or voice modal data), for example, by operating within the interface provided by I / O interface 212. Alternatively, client device 240 can automatically input data (e.g., text and / or voice modal data) into I / O interface 212 and obtain results. If automatic data input by client device 240 requires user authorization, the user can set appropriate permissions within client device 240. The user can view the results output by execution device 210 on client device 240 (e.g., response data generated for the input data, including text and / or voice modal data, etc.), specifically through display, playback, or other methods.

[0093] It is worth noting that Figure 2 This is merely a schematic diagram of a system architecture provided in an embodiment of this application, and the positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 2 In this context, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 may also be placed within the execution device 210.

[0094] Figure 3 This is a schematic diagram of the hardware structure of a chip provided in an embodiment of this application. The chip includes a neural network processor (NPU) 30. This chip can be configured as follows: Figure 2 In the execution device 210 shown, the chip is used to perform the calculations of the calculation module 211. This chip can also be located in, for example... Figure 2The training device 220 shown is used to complete the training work of the training device 220 and output the target model / rule 201.

[0095] The multimodal data processing method of this application embodiment can be executed in the arithmetic circuit 303 and / or vector calculation unit 307 in the neural network processor 30 to obtain the processing result of the multimodal data.

[0096] The following is a brief introduction to the various modules and units in the neural network processor 30.

[0097] The neural network processor 30 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core of the neural network processor 30 is the arithmetic circuit 303. When the neural network processor 30 is working, the controller 304 in the neural network processor 30 can control the arithmetic circuit 303 to retrieve matrix data from the memory and perform multiplication operations.

[0098] In some implementations, the arithmetic circuit 303 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 303 is a two-dimensional pulsating array. The arithmetic circuit 303 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 303 is a general-purpose matrix processor.

[0099] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 302 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 301 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 308.

[0100] The vector computation unit 307 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit 307 can be used for network computation in non-convolutional / non-fully connected layers (FC) of neural networks, such as pooling, batch normalization, and local response normalization.

[0101] In some implementations, vector computation unit 307 can store the processed output vector into unified buffer 306. For example, vector computation unit 307 can apply a nonlinear function to the output of arithmetic circuit 303, such as a vector of accumulated values, to generate activation values. In some implementations, vector computation unit 307 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to arithmetic circuit 303, for example, for use in subsequent layers of a neural network.

[0102] The unified memory 306 is used to store input data and output data.

[0103] The weight data is directly transferred from the external memory to the input memory 301 and / or the unified memory 306 through the Direct Memory Access Controller (DMAC) 305, the weight data in the external memory is stored in the weight memory 302, and the data in the unified memory 306 is stored in the external memory.

[0104] BIU stands for Bus Interface Unit 310, which is used to enable interaction between the main CPU, DMAC and instruction fetch memory 309 via a bus.

[0105] The instruction fetch buffer 309, which is connected to the controller 304, is used to store the instructions used by the controller 304.

[0106] The controller 304 is used to call the instructions cached in the instruction memory 309 to control the operation of the computing accelerator.

[0107] Generally, the unified memory 306, input memory 301, weighted memory 302, and instruction fetch memory 309 can all be on-chip memories. The external memory of the NPU can be memory located outside the NPU itself, such as double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory. It should be understood that... Figure 3 The chip hardware structure shown is merely illustrative and is not intended to limit the scope of this application.

[0108] To facilitate understanding of the solutions in the embodiments of this application, the technical terms involved in the solutions provided in this application will be briefly introduced below.

[0109] Data: A general term for various multimedia data, which can include text, audio, video, images, etc.

[0110] Content refers to a portion of information within data that is crucial for humans or machines to understand the core semantics conveyed by the data. For example, to represent the content of a speech, the speech can be transcribed into text. To represent the content of a video, the video can be transcribed into a script.

[0111] Style: A concept relative to content definition. It refers to the information remaining in data besides the content information. For example, if a speech is transcribed into text to represent the content of the speech, then the style of the speech includes the speaker's timbre, ambient noise, the speaker's emotions, rhythm, etc. The content of an image can be represented as a sentence describing the objects in the image; then the corresponding style of the image includes information such as its tone and texture.

[0112] Large language models (LLMs) are a type of artificial intelligence model designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks.

[0113] Transformer: is a neural network architecture.

[0114] GPT stands for Generative Pre-trained Transformer, a pre-trained language model developed by OpenAI based on the Transformer architecture. Through pre-training on large amounts of text data, GPT models learn to generate coherent and logical text, and can be used for various natural language processing tasks, such as text generation, machine translation, text summarization, and question answering systems. Common GPT models include GPT-3.5 and GPT-4.

[0115] Language: In this article, language generally refers to natural language, that is, the language used by humans to communicate.

[0116] The following describes in detail, with reference to the accompanying drawings and embodiments, the specific implementation of the multimodal data processing method and apparatus provided in this application.

[0117] Figure 4This is a flowchart illustrating a multimodal data processing method provided in an embodiment of this application. This method can be executed by any device, equipment, platform, or cluster of devices with computing capabilities. This application does not specifically limit the specific computing device executing the method; a suitable computing device can be selected as needed. For example, it can be implemented by a server, meaning the multimodal data processing method can be implemented by a server to provide multimodal data processing services. Alternatively, it can be implemented on a cloud device to provide multimodal data processing services to cloud users in the form of a cloud service. For ease of description, the form of the executing entity will not be distinguished in the following text; all will be described as executing devices, i.e. Figure 2 The execution device 210 in the middle. For example Figure 4 As shown, the multimodal data processing method provided in this application embodiment includes at least steps S401 to S405.

[0118] In step S401, input data is acquired.

[0119] The input data in this application embodiment can be any one or more modal data, including text modal data and speech modal data. Taking the multimodal data processing method provided in this application embodiment applied to an intelligent question-and-answer / chat scenario as an example, a user can open a corresponding program, webpage, or mini-program on a smart terminal and input text modal data into an input box, such as typing "Look how blue the sky is." After obtaining the input data, the smart terminal sends it to a server (an example where the execution device is a server, which can be a cloud server) that has deployed the multimodal model trained according to this application embodiment. The server receives the input data sent by the smart terminal, obtains the input data, and then inputs the input data into the multimodal model, outputting one or more modal data, including speech modal data and text modal data. For example, outputting the text: "Yes, the weather is so nice!", and the corresponding voice broadcast: "Yes, the weather is so nice."

[0120] For example, users can wake up the response program on the smart terminal, such as a smartphone assistant, by saying "Xiaoyi, Xiaoyi," and then start a conversation with the smartphone assistant, inputting data by voice, such as "Look how blue the sky is." The smart terminal picks up the user's voice data through the microphone array and then sends the voice data to the server. The server receives the input data sent by the smart terminal and obtains the input data.

[0121] For example, users can input data using a combination of voice and text modalities. For instance, a user might type "Look" in a text box and then say, "How blue the sky is." The terminal device receives the input data, which is a combination of text and voice modalities, and then sends it to the server. The server receives the input data from the smart terminal and obtains the input data.

[0122] It is understood that the above-described method of obtaining input data is only a feasible example, intended to facilitate understanding of this solution, and does not constitute a limitation on this solution. Input data can also be obtained in various other ways. For example, if the multimodal model provided in this application embodiment is deployed on a terminal device, there is no need to send input data to the server; the terminal device can obtain the input data through interaction with the user.

[0123] In step S402, the input data is encoded to obtain an encoded vector.

[0124] In one example, for input data in the speech modality, a corresponding encoder can be deployed, such as a speech encoder. For input data in the text modality, an embedding layer can be deployed. This embedding layer transforms the text modality input data into vector form using an embedding table, making it easier to input into a multimodal large language model (MLM) for processing.

[0125] After acquiring the input data, the text modality input data is fed into the embedding layer, which converts the original text data into a dense vector representation, namely the text embedding representation vector. The speech modality input data is fed into the speech encoder to obtain the speech encoding vector. Then, an adapter maps the speech encoding vector to the input dimension of the multimodal large language model (i.e., the same dimension as the text embedding representation vector), so that the large language model can process the speech encoding vector. The speech encoding vector processed by the adapter and the text embedding representation vector are then input into the multimodal large language model for further processing. In other words, when the input data includes both text and speech modality data, the encoding vector includes the speech encoding vector output by the adapter and the text embedding representation vector. It can be understood that when the input data only includes text modality data, the encoding vector only includes the text embedding representation vector; when the input data only includes speech modality data, the encoding vector only includes the speech encoding vector output by the adapter.

[0126] For example, a user enters the text "Please translate what I said into English" into a text box, and then says "What is the average salary in the United States?". The terminal device picks up the user's voice input through a microphone. The terminal device obtains input data including text data and voice data, and then uses the text data in the text box as input to the embedding layer. The embedding layer converts the text data into a text embedding representation vector, uses the voice data picked up by the microphone as input to the speech encoder, and outputs a speech encoding vector. Then, through an adapter, the speech encoding vector is mapped to the input dimension of the multimodal large language model to obtain an encoding vector that includes the text embedding representation vector and the speech encoding vector.

[0127] In another example, to ensure the model's ability to understand and generate speech, the speech encoding vector is a continuous representation vector. Using a continuous representation vector preserves more information from the speech input data, which is beneficial for subsequent speech understanding and generation. For example, the speech encoder is a transformer with an encoder structure.

[0128] Optionally, the voice data can be in WAV format.

[0129] In step S403, the encoded vector is used as the input to the large language model, and the joint speech-text representation vector is output.

[0130] After obtaining the encoding vector, the encoding vector is input into the large language model. After the large language model performs inference, it outputs a joint speech-text representation vector. This joint speech-text representation vector represents the features of the speech data to be generated and the features of the text data to be generated. In other words, a representation vector represents both speech features and text features.

[0131] It should be noted that the large language model is a large language model obtained after special training and parameter tuning (the training and parameter tuning of the large language model will be introduced below). In other words, through the design of training data, the embodiments of this application enable the large language model to perform reasoning processing on input data of text modality, input data of speech modality, or input data of both text modality and speech modality, and output a joint speech-text representation vector.

[0132] In other words, regardless of whether the input encoding vector includes only text embedding representation vectors, only speech encoding vectors, or both text embedding representation vectors and speech encoding vectors, the speech-text joint representation vector output by the large language model includes vector representations of the features of the speech data to be generated and the features of the text data to be generated. The large language model uses the encoding vectors of the input data for inference and prediction, and outputs the speech-text joint representation vector.

[0133] In one example, the joint speech-text representation vector output by the large language model is also a continuous representation vector. This means that the text and speech features to be output are fused into a single continuous representation vector. A continuous representation vector can represent richer information, which is beneficial for improving subsequent speech generation and comprehension capabilities. Compared to methods using multi-codebook discrete speech representation, the scheme in this application generates higher-quality, more concise, and more efficient speech.

[0134] Figure 5 A schematic diagram illustrating the input and output of a large language model according to an embodiment of this application is shown. Figure 5 As shown, the input speech encoding vector of the large language model is a continuous representation vector, and the output speech representation vector is also a continuous representation vector. That is, this application adopts a continuous speech representation scheme in both speech understanding and generation. It should be noted that... Figure 5 The purpose of this statement is to emphasize that the input and output of a large language model are continuous representation vectors for language data. In practice, the output of a large language model is still a joint representation vector of speech and text.

[0135] In step S404, speech data and text data are generated based on the joint speech-text representation vector.

[0136] After obtaining the language-text joint representation vector through the above steps, the language-text joint representation vector is input to the speech output module to output speech data; the speech-text joint representation vector is used as the input to the text output module to output text data. In this way, by setting the speech output head (i.e., the speech output module) and the text output head (i.e., the text output module), speech generation and text generation can be supported.

[0137] Figure 6 This illustration shows a schematic diagram of an architecture of a multimodal model provided in an embodiment of this application. For example... Figure 6As shown, the multimodal model includes a speech encoder, an adapter, a multimodal LLM, a text output module, and a speech output module. The speech encoder encodes the input data for the speech modality to obtain a speech encoding vector. The adapter maps the speech encoding vector to the input dimension of the multimodal large language model, facilitating its processing. It should be noted that since text embedding layers are a relatively conventional technique, the embedding layer used to process text modality input data is not shown. In practice, the text is processed by the embedding layer to be converted into a text embedding vector before being input to the multimodal LLM for processing. Alternatively, the embedding layer can be used as a network structure within the input layer of the multimodal LLM, with the text input being processed at the input layer. Multimodal LLM (Locally-Led Minecraft) infers the encoded vectors of the input data (understandably, when the input data only includes speech modal data, the encoded vectors only include speech encoded vectors; when the input data includes text modal data, the encoded vectors only include text embedding vectors; when the input data includes both speech and text modal data, the encoded vectors include both speech encoded vectors and text embedding vectors) and outputs a continuous speech-text joint representation vector. This continuous speech-text joint representation vector represents the speech features and text features predicted by the inference. The speech output module outputs the generated speech data based on the speech features in the continuous speech-text joint representation vector. The text output module outputs the generated text data based on the text features in the continuous speech-text joint representation vector.

[0138] The text output module can be implemented in various ways, such as through a multilayer perceptron (MLP) and a classifier, or through convolutional neural networks (CNNs) and a classifier.

[0139] The voice output module can be implemented in various ways, such as through MLP, CNN and other methods.

[0140] It should be noted that the speech output module can convert the speech-text joint representation vector into the speech spectrum or directly into WAV output.

[0141] Multimodal LLMs can be autoregressive large language models (AR), such as GPT-type LLMs trained from scratch. Due to the characteristics of autoregressive LLMs, embodiments of this application can support the understanding and output of speech of arbitrary length.

[0142] Figure 7This illustration shows another architectural diagram of the multimodal model provided in an embodiment of this application. For example... Figure 7 As shown, the continuous speech-text joint representation output by the multimodal LLM is also fed back to the multimodal LLM so that it can understand contextual information, such as the generated content, which is beneficial for more accurate reasoning. Considering the difference between the speech encoding vector and the continuous speech-text joint representation input to the multimodal LLM in this embodiment, a speech input module is set up to process the continuous speech-text joint representation and align its dimensions with the input dimensions of the model. This is beneficial for the processing of the multimodal LLM, making it easier for the multimodal LLM to consider contextual information during reasoning, and generating more logical content.

[0143] Alternatively, the voice input module can be implemented in various ways, such as MLP, CNN, and transformer.

[0144] Considering that speech data doesn't have explicit end markers like text, such as periods or exclamation marks, the logic for stopping speech data output is more complex. In one example, the output steps for speech and text data are the same. That is, speech and text play and stop synchronously. Thus, the end of the text output controls the end of the speech data output; that is, the end-of-sentence marker in the text output determines whether to stop outputting both text and speech. For example, if the semantic content to be output by the multimodal model is "Yes, the weather is nice!", then when the text output module outputs an exclamation mark, it determines that output should stop, and the speech module's output should also stop.

[0145] In another example, voice data and text data can also be output asynchronously, that is, output independently. In this case, the output of voice data needs to be stopped by its own judgment. In this embodiment of the application, the output of voice data can be stopped by setting a voice output stop judgment module.

[0146] In another example, the multimodal model can be configured to output only text data, only speech data, or both text and speech data by setting configuration parameters. This means the multimodal model can be configured to output appropriate data for different scenarios. For example, in a smart speaker scenario, the multimodal model can be configured to output only speech data. In a quiet question-and-answer scenario, such as in a library, the multimodal model can be configured to output only text data. This saves computational resources.

[0147] Figure 8 This illustration shows another architectural diagram of the multimodal model provided in an embodiment of this application. For example... Figure 8As shown, a speech stop detection module is added to the output of the multimodal LLM. This module determines whether the speech data output has stopped by analyzing the continuous speech-text joint representation. For example, the speech stop detection module can be implemented using a binary classifier. The binary classifier determines whether the current speech output is the last step when the speech-text joint representation is output at each time step. If it is the last step, the speech output module stops subsequent processing.

[0148] Optionally, the binary classifier can be trained together with the multimodal model.

[0149] The training process of the multimodal model provided in the embodiments of this application is described below. The training of the multimodal model can be carried out in... Figure 2 The training device 220 in the middle is used for implementation.

[0150] The architecture of the multimodal model provided in this application embodiment can be as follows: Figures 6 to 8 Any of the architectures shown. Figure 7 The architecture shown is used as an example to illustrate the training process of the multimodal model provided in this application.

[0151] First, let's introduce the training dataset. The training dataset includes multiple sets of training data. Each set includes at least one of the following: a text sample and its corresponding text-to-speech pair, a speech sample and its corresponding text-to-speech pair, or a text-to-speech sample and its corresponding text-to-speech pair. In the text-to-speech pair, the text and speech have the same semantic meaning. In another example, the training dataset also includes text samples and their training data.

[0152] For example, taking the application of a multimodal model in a dialogue scenario: a set of training data in the training dataset can have the following form:

[0153] Txet-paired(speech, text): This means that the training data set includes text samples and the corresponding text-speech pairs. During training, the text samples are used as the input to the multimodal model, and the text sample pairs are used as labels.

[0154] For example, the text sample is: (text) "Look how blue the sky is!"; the text-to-speech pair is: (text) "Yes, the weather is so nice!" (speech) "Yes, the weather is so nice!".

[0155] Speech-paired(speech, text): This means that the training data includes speech samples and the corresponding text-speech pairs. During training, the speech samples are used as input to the multimodal model, and the text-speech pairs are used as labels.

[0156] For example, the speech sample is: (speech) "Look how blue the sky is!"; the text-speech pair is: (text) "Yes, the weather is so nice!" (speech) "Yes, the weather is so nice!".

[0157] Speech, text-paired(speech, text): This means that the training data includes text speech samples and the corresponding text speech pairs. During training, the text speech samples are used as the input to the multimodal model, and the text speech pairs are used as labels.

[0158] For example, the speech sample is: (speech) “Look”; (text) “The sky is so blue!”; the text-speech pair is: (text) “Yes, the weather is so nice!”; (speech) “Yes, the weather is so nice!”.

[0159] Text-Text: This means that the training data set includes text samples and their corresponding text responses. During training, the text samples serve as the input to the multimodal model, and the text responses serve as the labels.

[0160] For example, the text sample is: (text) "Look how blue the sky is!"; the text response is: (text) "Yes, the weather is so nice!".

[0161] Next, the training tasks are introduced. This application embodiment sets up a task to predict the next word in the text and a task to synthesize speech. In one example, these two tasks can be trained jointly. A loss function can be designed for training the task to predict the next word in the text; this loss function can be called the first loss function. A loss function can be designed for training the task to synthesize speech; this loss function can be called the second loss function.

[0162] Optionally, the first loss function can be a loss function such as cross-entropy loss or negative log-likelihood loss. The second loss function can be a loss function such as mean squared error loss or root mean squared error loss.

[0163] The first loss function calculates the loss by substituting the output of the text output module and the text labels into the first loss function; the second loss function calculates the loss by substituting the output of the speech output module and the speech labels into the second loss function. Then, the total loss is obtained by weighted summing of the first and second loss values. The total loss is minimized, and the parameters of the multimodal model are adjusted, including the weight parameters of the speech encoder, the adapter, the multimodal LLM, the text output module, the speech output module, and the speech input module, until the training termination condition is met (e.g., reaching the preset training rounds or model convergence).

[0164] Based on the same concept as the aforementioned embodiment of a multimodal data processing method, this application also provides a multimodal data processing apparatus 900. This apparatus 900 can be deployed on a server or terminal device to achieve text and speech multimodal data processing capabilities, while ensuring good speech understanding and generation capabilities. The multimodal data processing apparatus 900 includes components for implementing... Figure 4-8 The units or modules of each step in the multimodal data processing method shown.

[0165] Figure 9 This is a schematic diagram of the structure of a multimodal data processing device provided in an embodiment of this application. Figure 9 As shown, the multimodal data processing device 900 includes an acquisition module 901, an encoding module 902, a joint representation module 903, and a generation module 904. The acquisition module 901 acquires input data, which includes input data for at least one modality, such as text modality input data or speech modality input data. The encoding module 902 encodes the input data to obtain an encoding vector. The joint representation module 903 uses the encoding vector as input to a large language model and outputs a speech-text joint representation vector, which includes features of the speech data to be generated and features of the text data to be generated. The generation module 904 generates speech data and text data based on the speech-text joint representation vector, wherein the speech data and text data have the same semantic content.

[0166] In one possible implementation, the joint speech-text representation vector is a continuous representation vector.

[0167] In another possible implementation, the input data includes input data for the speech modality; the encoding module 902 is specifically used to: take the input data of the speech modality as input to the speech encoder and output a first speech encoding vector; take the speech encoding vector as input to the adapter and output a second speech encoding vector, the second speech encoding vector having the same dimension as the input dimension of the large language model, the second speech encoding vector being a continuous representation vector, and the encoding vector being the second speech encoding vector.

[0168] In another possible implementation, the input data also includes the input data of the text modality; the encoding vector also includes the embedding representation vector corresponding to the input data of the text modality.

[0169] In another possible implementation, the output of the large language model is connected to the input of the speech output module and the input of the text output module, respectively; the generation module 904 is specifically used to: take the speech-text joint representation vector as the input of the speech output module and output speech data; take the speech-text joint representation vector as the input of the text output module and output text data.

[0170] In another possible implementation, the text output module includes a text filtering layer and an output layer. A specific implementation of taking the joint speech-text representation vector as input to the speech output module and outputting speech data is as follows: taking the joint speech-text representation vector as input to the text filtering layer and outputting a text representation vector; taking the text representation vector as input to the output layer and outputting a probability distribution, which indicates the probability of each word in the vocabulary being output; and sampling from the vocabulary based on the probability distribution to obtain text data.

[0171] In another possible implementation, the speech output module includes a speech filtering layer and a mapping layer; a specific implementation of taking the speech-text joint representation vector as input to the text output module and outputting text data is as follows: taking the speech-text joint representation vector as input to the speech filtering layer and outputting a speech representation vector; taking the speech representation vector as input to the mapping layer and outputting speech data.

[0172] In another possible implementation, the large language model is an autoregressive large language model; the multimodal data processing apparatus 900 provided in this application also includes an input module 905, which is used to input the speech-text joint representation vector to the speech input module and output the processed speech-text joint representation vector. The dimension of the processed speech-text joint representation vector is the same as the input dimension of the large language model; the processed speech-text joint representation vector is used as the input of the large language model.

[0173] In another possible implementation, the number of output steps for voice data and text data is the same.

[0174] In another possible implementation, the multimodal data processing apparatus 900 provided in this application further includes a stop determination module 906, which is used to input the speech-text joint representation vector to the speech stop determination module; and determine whether the output of speech data has stopped based on the signal output by the speech stop determination module.

[0175] In another possible implementation, the voice stop determination module includes a binary classifier. The signal output by the voice stop determination module includes a first value or a second value. The first value indicates that the voice data output has ended, and the second value indicates that the voice data output has not ended. A specific implementation for determining whether the voice data output has stopped based on the signal output by the voice stop determination module is as follows: when the voice stop determination module outputs the first value, it is determined that the voice data output has stopped; when the voice stop determination module outputs the second value, it is determined that the voice data output has not stopped.

[0176] In another possible implementation, the large language model is trained based on training data; the training data includes multiple training sample pairs, each training sample pair including at least one of the following: text sample and text-speech pair corresponding to the text sample, speech sample and text-speech pair corresponding to the speech sample, and text-speech pair corresponding to the text-speech sample, wherein the text and speech in the text-speech pair have the same semantics.

[0177] The multimodal data processing apparatus 900 according to the embodiments of this application can correspond to executing the method described in the embodiments of this application, and the above and other operations and / or functions of each module in the multimodal data processing apparatus 900 are respectively for implementing Figure 4-8 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0178] This application embodiment also provides a computing device, including at least one processor, a memory, and a communication interface, wherein the processor is used to execute... Figure 4-8 The method described.

[0179] Figure 10 A schematic diagram of the structure of a computing device provided in an embodiment of this application.

[0180] like Figure 10 As shown, the computing device 1000 includes at least one processor 1001, a memory 1002, and a communication interface 1003. The processor 1001, memory 1002, and communication interface 1003 are communicatively connected, which can be achieved via a wired (e.g., bus) or wireless connection. The communication interface 1003 is used to send and / or receive data from other devices. The memory 1002 stores computer instructions, which the processor 1001 executes to perform the methods described in the foregoing method embodiments, thereby achieving the ability to process multimodal text and speech data and ensuring good speech understanding and generation capabilities.

[0181] It should be understood that, in the embodiments of this application, the processor 1001 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0182] The memory 1002 may include read-only memory and random access memory, and provides instructions and data to the processor 1001. The memory 1002 may also include non-volatile random access memory. Optionally, the random access memory may be, for example, high bandwidth memory (HBM).

[0183] The memory 1002 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0184] It should be understood that the computing device 1000 according to the embodiments of this application can execute the implementation of the embodiments of this application. Figure 4-8 The method shown is described in detail above, and will not be repeated here for the sake of brevity.

[0185] Embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer instructions are executed by a processor, the aforementioned method is implemented.

[0186] An embodiment of this application provides a chip including at least one processor and an interface, wherein the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to implement the method mentioned above.

[0187] Embodiments of this application provide a computer program or computer program product that includes instructions that, when executed, cause a computer to perform the methods mentioned above.

[0188] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0189] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented using hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0190] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for processing multimodal data, characterized in that, include: Acquire input data, which includes input data of at least one modality, namely text modal input data and speech modal input data; The input data is encoded to obtain an encoding vector; The encoded vector is used as the input of a large language model to output a joint speech-text representation vector, which includes the features of the speech data to be generated and the features of the text data to be generated. Based on the joint speech-text representation vector, the speech data and the text data are generated, and the speech data and the text data have the same semantic content.

2. The method according to claim 1, characterized in that, The joint speech-text representation vector is a continuous representation vector.

3. The method according to claim 1 or 2, characterized in that, The input data includes voice modality input data; The process of encoding the input data to obtain an encoding vector includes: The input data of the speech modality is used as the input of the speech encoder to output the first speech coding vector; The speech encoding vector is used as the input of the adapter to output a second speech encoding vector. The dimension of the second speech encoding vector is the same as the input dimension of the large language model. The second speech encoding vector is a continuous representation vector. The encoding vector is the second speech encoding vector.

4. The method according to claim 3, characterized in that, The input data also includes the input data for the text modality; The encoding vector also includes the embedding representation vector corresponding to the input data of the text modality.

5. The method according to any one of claims 1-4, characterized in that, The output of the large language model is connected to the input of the speech output module and the input of the text output module, respectively. The process of generating the speech data and the text data based on the joint speech-text representation vector includes: The speech-text joint representation vector is used as the input to the speech output module to output the speech data; The joint speech-text representation vector is used as input to the text output module to output the text data.

6. The method according to claim 5, characterized in that, The text output module includes a text filtering layer and an output layer; The step of using the joint speech-text representation vector as input to the speech output module and outputting the text data includes: The joint speech-text representation vector is used as the input to the text filtering layer, and the output is the text representation vector. The text representation vector is used as the input to the output layer, and a probability distribution is output, which indicates the probability that each word in the vocabulary will be output. Based on the probability distribution, the text data is obtained by sampling from the vocabulary.

7. The method according to claim 5, characterized in that, The voice output module includes a voice filtering layer and a mapping layer; The step of using the joint speech-text representation vector as input to the text output module and outputting the text data includes: The joint speech-text representation vector is used as the input to the speech filtering layer, and the output is the speech representation vector. The speech representation vector is used as the input to the mapping layer to output the speech data.

8. The method according to any one of claims 1-7, characterized in that, The large language model is an autoregressive type of large language model; The method further includes: The speech-text joint representation vector is input to the speech input module, and the speech-text joint representation vector processed by the speech input module is output. The dimension of the processed speech-text joint representation vector is the same as the input dimension of the large language model. The processed speech-text joint representation vector is used as the input to the large language model.

9. The method according to any one of claims 1-8, characterized in that, The number of output steps for the voice data and the text data is the same.

10. The method according to any one of claims 1-8, characterized in that, Also includes: The joint speech-text representation vector is input into the speech stop determination module; Based on the signal output by the voice stop determination module, it is determined whether the output of the voice data has stopped.

11. The method according to claim 10, characterized in that, The voice stop judgment module includes a binary classifier. The signal output by the voice stop judgment module includes a first value or a second value. The first value indicates that the output of the voice data has ended, and the second value indicates that the output of the voice data has not ended. The step of determining whether the output of the voice data has stopped based on the signal output by the voice stop determination module includes: When the voice stop determination module detects that it outputs the first value, it is determined that the output of the voice data has stopped. If the voice stop determination module outputs the second value, it is determined that the output of the voice data has not stopped.

12. The method according to any one of claims 1-11, characterized in that, The large language model is trained based on training data; the training data includes multiple training sample pairs, each training sample pair including at least one of the following: a text sample and the text-speech pair corresponding to the text sample, a speech sample and the text-speech pair corresponding to the speech sample, and a text-speech pair corresponding to the text-speech sample, wherein the text and speech in the text-speech pair have the same semantics.

13. A multimodal data processing device, characterized in that, include: An acquisition module is used to acquire input data, which includes input data of at least one modality, such as text modal input data and speech modal input data. An encoding module is used to encode the input data to obtain an encoding vector; The joint representation module is used to take the encoded vector as input to the large language model and output a joint speech-text representation vector, which includes the features of the speech data to be generated and the features of the text data to be generated. The generation module is used to generate the speech data and the text data based on the joint speech-text representation vector, wherein the speech data and the text data have the same semantic content.

14. The apparatus according to claim 13, characterized in that, The joint speech-text representation vector is a continuous representation vector.

15. The apparatus according to claim 13 or 14, characterized in that, The input data includes voice modality input data; The encoding module is specifically used for: The input data of the speech modality is used as the input of the speech encoder to output the first speech coding vector; The speech encoding vector is used as the input of the adapter to output a second speech encoding vector. The dimension of the second speech encoding vector is the same as the input dimension of the large language model. The second speech encoding vector is a continuous representation vector. The encoding vector is the second speech encoding vector.

16. The apparatus according to claim 15, characterized in that, The input data also includes the input data for the text modality; The encoding vector also includes the embedding representation vector corresponding to the input data of the text modality.

17. The apparatus according to any one of claims 13-16, characterized in that, The output of the large language model is connected to the input of the speech output module and the input of the text output module, respectively. The generation module is specifically used for: The speech-text joint representation vector is used as the input to the speech output module to output the speech data; The joint speech-text representation vector is used as input to the text output module to output the text data.

18. The apparatus according to claim 17, characterized in that, The text output module includes a text filtering layer and an output layer; The step of using the joint speech-text representation vector as input to the speech output module and outputting the text data includes: The joint speech-text representation vector is used as the input to the text filtering layer, and the output is the text representation vector. The text representation vector is used as the input to the output layer, and a probability distribution is output, which indicates the probability that each word in the vocabulary will be output. Based on the probability distribution, the text data is obtained by sampling from the vocabulary.

19. The apparatus according to claim 17, characterized in that, The voice output module includes a voice filtering layer and a mapping layer; The step of using the joint speech-text representation vector as input to the text output module and outputting the text data includes: The joint speech-text representation vector is used as the input to the speech filtering layer, and the output is the speech representation vector. The speech representation vector is used as the input to the mapping layer to output the speech data.

20. The apparatus according to any one of claims 13-19, characterized in that, The large language model is an autoregressive type of large language model; The device further includes: The input module is used to input the speech-text joint representation vector into the speech input module and output the speech-text joint representation vector processed by the speech input module. The dimension of the processed speech-text joint representation vector is the same as the input dimension of the large language model. The processed speech-text joint representation vector is used as the input to the large language model.

21. The apparatus according to any one of claims 13-20, characterized in that, The number of output steps for the voice data and the text data is the same.

22. The apparatus according to any one of claims 13-21, characterized in that, Also includes: A stop determination module is used to input the joint speech-text representation vector into the speech stop determination module; Based on the signal output by the voice stop determination module, it is determined whether the output of the voice data has stopped.

23. The apparatus according to any one of claims 13-21, characterized in that, The voice stop judgment module includes a binary classifier. The signal output by the voice stop judgment module includes a first value or a second value. The first value indicates that the output of the voice data has ended, and the second value indicates that the output of the voice data has not ended. The step of determining whether the output of the voice data has stopped based on the signal output by the voice stop determination module includes: When the voice stop determination module detects that it outputs the first value, it is determined that the output of the voice data has stopped. If the voice stop determination module outputs the second value, it is determined that the output of the voice data has not stopped.

24. The apparatus according to any one of claims 13-23, characterized in that, The large language model is trained based on training data; the training data includes multiple training sample pairs, each training sample pair including at least one of the following: a text sample and the text-speech pair corresponding to the text sample, a speech sample and the text-speech pair corresponding to the speech sample, and a text-speech pair corresponding to the text-speech sample, wherein the text and speech in the text-speech pair have the same semantics.

25. A computing device, comprising a memory and a processor, characterized in that, The memory stores instructions that, when executed by a processor, cause the method described in any one of claims 1-12 to be implemented.

26. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it causes the method as described in any one of claims 1-12 to be implemented.

27. A chip comprising at least one processor and a communication interface, characterized in that, The processor is used to perform the method as described in any one of claims 1-12.

28. A computer program or computer program product comprising instructions, characterized in that, When the instruction is executed, the computer performs the method as described in any one of claims 1-12.

Images