Data processing method, data processing device, electronic equipment, storage medium and program product

Abstract

The invention provides a data processing method, a data processing device, electronic equipment, a storage medium and a program product, belongs to the technical field of computers, and particularly relates to an artificial intelligence technology. According to the implementation scheme, the method comprises the steps that input data are received, and the input data comprise input audios and input texts; determining spectrum features of the input audio; mapping the spectrum features by using a mapping layer to obtain an embedded representation of the input audio for a large language model; determining an embedded representation of the input text; and processing the embedded representation of the input audio and the embedded representation of the input text by using the large language model to obtain a prediction result for the input data.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to artificial intelligence technology, and specifically to a data processing method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] In recent years, large-scale pre-training models have achieved success in the field of natural language processing. With the increase in the number of parameters or data, the performance of the model has been continuously improved.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The present disclosure provides a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product.

[0006] According to one aspect of the present disclosure, a data processing method is provided, comprising: receiving input data, wherein the input data comprises input audio and input text; determining spectral features of the input audio; mapping the spectral features using a mapping layer to obtain an embedded representation of the input audio for a large language model; determining an embedded representation of the input text; and processing the embedded representation of the input audio and the embedded representation of the input text using the large language model to obtain a prediction result for the input data.

[0007] According to another aspect of the present disclosure, a data processing device is provided, comprising: an input unit configured to receive input data, wherein the input data comprises input audio and input text; an audio feature generation unit configured to determine spectral features of the input audio; a mapping unit configured to map the spectral features using a mapping layer to obtain an embedded representation of the input audio for a large language model; a text processing unit configured to determine an embedded representation of the input text; and an output unit configured to process the embedded representation of the input audio and the embedded representation of the input text using the large language model to obtain a prediction result for the input data.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method according to an embodiment of the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute a method according to an embodiment of the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the method according to the embodiment of the present disclosure when executed by a processor.

[0011] According to one or more embodiments of the present disclosure, it is possible to simplify the training process of the audio-text joint processing model, improve the robustness and consistency of the model output, and enhance the cross-modal learning ability of the model.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0015] Figure 2 An exemplary flow chart of a data processing method according to an embodiment of the present disclosure is shown;

[0016] Figure 3 An exemplary principle of a data processing method according to an embodiment of the present disclosure is shown;

[0017] Figure 4 An exemplary block diagram of a data processing device according to an embodiment of the present disclosure is shown;

[0018] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0020] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0021] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0022] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0023] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0024] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable execution of the data processing method according to an embodiment of the present disclosure.

[0025] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0026] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0027] The user may use client devices 101, 102, 103, 104, 105 and / or 106 to obtain digital input data and output data processing results. The client device may provide an interface that enables the user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0028] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0029] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0030] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0031] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0032] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.

[0033] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0034] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

[0035] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0036] Figure 1The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0037] The autoregressive text-audio large model can combine text and audio data to achieve end-to-end text-audio modeling through autoregression, thereby significantly improving audio understanding tasks. At the same time, it is committed to solving the problem of large-scale model training expansion, and can flexibly expand the scale of pre-training data and model scale to achieve continuous improvement of model performance.

[0038] In the related art, the pre-training of text-audio large models usually adopts discretized speech coding in audio feature modeling, and representative works include SpeechGPT and AudioPaLM. When processing audio data, this type of work obtains audio discretized tokens (Token) through the trained audio tokenization module (Tokenizer), and allocates a separate vocabulary space for each audio token, so as to process audio features like the language features of text.

[0039] The above processing method in the related art has the following disadvantages:

[0040] a) The multi-stage training process for audio features is complex and difficult to expand. In the related art, a Tokenizer for audio discretization needs to be trained first. After completing this stage, all audio data is discretized through this Tokenizer and then input into the model for a second training. This multi-stage training method is cumbersome and limits the scalability of the model. For example, when processing new data, it is often necessary to start the entire training process from scratch, which is time-consuming and inefficient.

[0041] b) The final model effect depends on the performance of the audio tokenizer. Since the discretization of audio data is completely dependent on the pre-trained audio tokenizer, the performance of the tokenizer directly affects the effect of the final model. If the performance of the audio tokenizer is insufficient, the generated audio tokens may not be accurate enough, thus affecting the performance of the entire model.

[0042] In order to improve the operation effect of the large audio-text joint model, the present disclosure provides a new data processing method.

[0043] Figure 2 An exemplary flow chart of a data processing method according to an embodiment of the present disclosure is shown.

[0044] In step S202, input data is received, wherein the input data includes input audio and input text.

[0045] In step S204, the frequency spectrum characteristics of the input audio are determined.

[0046] In step S206, the spectral features are mapped using a mapping layer to obtain an embedded representation of the input audio for the large language model.

[0047] In step S208, an embedded representation of the input text is determined.

[0048] In step S210, the embedded representation of the input audio and the embedded representation of the input text are processed using the large language model to obtain a prediction result for the input data.

[0049] By utilizing the data processing method provided by the embodiments of the present disclosure, it is possible to simplify the training process of the audio-text joint processing model, improve the robustness and consistency of the model output, and enhance the cross-modal learning capability of the model.

[0050] The principles of the present disclosure will be described in detail below.

[0051] In step S202, input data may be received, wherein the input data includes input audio and input text. Step S202 may be implemented using an input device capable of receiving audio and text input. For example, a microphone may be used to receive and collect input audio, and a keyboard may be used to receive input text. For another example, input audio and input text may also be received via a network or read from a memory. The specific form of the input device, nor the specific form of the input audio and input text, is not limited herein.

[0052] The input audio and input text in the input data can have a contextual relationship. For example, a user can input an audio clip and simultaneously input the text "what does this audio clip say". The input data including the audio and text can be passed to the machine learning model together, and the model outputs the corresponding prediction results. Ideally, the model can output a response and interact with the user.

[0053] In step S204, the frequency spectrum characteristics of the input audio may be determined.

[0054] For each audio frame in the input audio, the audio frame can be Fourier transformed to obtain the transformation features of the audio frame. For example, the audio data in the audio frame can be short-time Fourier transformed to obtain the features of the audio frame. For the features obtained by Fourier transform, the transformation features of the audio frame can be further converted using a Mel filter to obtain the Mel spectrum of the audio frame. The Mel filter can enhance the low-frequency features in the audio and suppress high-frequency noise. Since humans are more sensitive to low-frequency changes than high-frequency changes, the features of the audio data can be converted into audio features that are more in line with human hearing by applying a Mel filter. The spectral features of the input audio can be generated based on the Mel spectrum of multiple audio frames in the input audio.

[0055] The frequency spectrum feature of the input audio obtained in step S204 can be represented in a two-dimensional form, and its shape is [number of frames, number of Mel filters]. The number of frames is determined by the length of the processed input audio, and the number of Mel filters used can be determined by the user according to actual conditions.

[0056] In step S206, the spectral features can be mapped using a mapping layer to obtain an embedded representation of the input audio for the large language model. In step S208, the embedded representation of the input text is determined. The embedded representation of the input audio and the embedded representation of the input text have the same size. Thus, the input audio and the input text can be jointly processed using the large language model in subsequent steps, so that the large model can realize joint learning of knowledge from massive text and audio data, and can perform end-to-end flexible modeling of audio-text, pure audio, and pure text data.

[0057] In the example, the mapping layer in step S206 can be implemented using a linear layer. The embedded representation of the input audio is obtained by linearly mapping the spectral features. Without departing from the principles of the present disclosure, any other network structure can be used to implement the mapping, as long as the spectral features of the input audio can be converted into the same form as the embedded representation of the text.

[0058] For the input text, the embedded representation of the input text can be obtained by performing text segmentation and word embedding representation on the input text. Various word segmentation methods and word embedding methods used in natural language processing can be used to obtain the embedded representation of the input text.

[0059] In step S210, the embedded representation of the input audio and the embedded representation of the input text are processed using the large language model to obtain a prediction result for the input data.

[0060] By executing steps S202-S208, the data processing method provided according to the embodiment of the present disclosure can convert the audio and text data in the input data into the same embedding representation, so that the large language model can jointly process the input audio and input text.

[0061] In step S210, before inputting the large language model, the embedded representation of the input audio and the embedded representation of the input text can be spliced ​​as the input of the large language model. When splicing, an audio start identifier can be added before the embedded representation of the input audio, and an audio end identifier can be added after the embedded representation of the input audio to identify the start and end of the audio data. In the data processing process before inputting the large language model, the input audio and the input text are processed into embedded representations of the same size. Therefore, it is possible to distinguish whether the audio data or the text data currently input into the large language model is audio data or text data by adding special symbols indicating the start and end of the audio.

[0062] Exemplary input data is shown below:

[0063] <BOS_AUDIO> Audio<EOS_AUDIO> Text

[0064] in,<BOS_AUDIO> is an exemplary audio start identifier,<EOS_AUDIO> is an exemplary audio end identifier, Audio represents input audio or an embedded representation of input audio, and Text represents input text or an embedded representation of input text. Those skilled in the art can set the specific forms of the audio start identifier and the audio end identifier according to actual conditions.

[0065] The concatenated features can be input into the large language model. In some embodiments, the large language model can be an autoregressive model. For example, the large language model can be implemented as a Transformer architecture using a causal attention mechanism. The network parameters of the large language model can be flexibly expanded according to actual conditions.

[0066] In the case where the large language model is an autoregressive model, the model can generate future prediction results based on past prediction results. When the large language model of the related art is output, the model will output a sequence, and each item in the sequence corresponds to a word embedding representation. Through the inverse operation of the word embedding, the output result of the model can be restored to the form of natural language. In the process of outputting the sequence, the model will output each item in the sequence in turn, and after outputting the current item, it will add the output current item to the input sequence to perform the prediction of the next item.

[0067] When the prediction result is outputted using the large language model in step S210, in response to determining that the current input item of the large language model is an audio start identifier or the current input item corresponds to audio data, the current output item of the large language model is converted into audio data. In response to the current input item of the large language model being an audio end identifier or the current input item corresponds to text data, the current output item of the large language model is converted into text data.

[0068] Using the above method, the large model can adaptively output the prediction result in audio form, the prediction result in text form, or the prediction result of the combination of audio and text according to the content of the input data. In the example, when the large model predicts that the audio should be output currently according to the input data, the audio start identifier will be output as the current output item. As mentioned above, for the autoregressive type model, when predicting the next output item, the currently output audio start identifier will be passed to the input end of the model as the current input. Therefore, when the model recognizes that the current input item is the audio start identifier during the prediction process, it will understand that the audio data should be output next until the model outputs the audio end identifier. Based on the above content, it can be determined that the data output between the audio start identifier and the audio end identifier are all audio data. Accordingly, if the current input item is neither the audio start identifier nor the audio end identifier, nor the data output between the audio start identifier and the audio end identifier, it can be determined that the current input item corresponds to text data. Therefore, when it is determined that the current input item is the audio start identifier or the current input item is the data located after the audio start identifier and before the audio end identifier, the model will convert the output result into audio data. For example, the large language model can pass the predicted output features to the inverse mapping layer, thereby converting the output features into spectral features. The inverse mapping layer can perform the inverse operation of the mapping layer in step S206. Further, the corresponding output audio can be generated according to the spectral features of the output features. Accordingly, when the current input item is an audio end identifier or the current input item corresponds to text data, the predicted output features can be converted into text data. For example, the large language model can pass the predicted output features to the output layer (LLM Head) of the large language model, and convert the output features into corresponding output text.

[0069] In some embodiments, the large language model is trained based on the following method: receiving sample input data, wherein the sample input data includes sample input audio and sample input text, real output audio and real output text corresponding to the sample input data; determining spectral features of the sample input audio; mapping the spectral features of the sample input audio using a mapping layer to obtain an embedded representation of the sample input audio for the large language model; determining an embedded representation of the sample input text; processing the embedded representation of the sample input audio and the embedded representation of the sample input text using the large language model to obtain a sample prediction result for the input data, wherein the sample prediction result includes a sample audio output and a sample text output; determining a first difference between the sample audio output and the real audio output and a second difference between the sample text output and the real text output; and adjusting parameters of the large language model according to a weighted sum of the first difference and the second difference.

[0070] In the example, the loss function used when training a large language model can be expressed as follows:

[0071] Loss = w a *Loss audio +w t *Loss text (1)

[0072] Among them, Loss audio Represents the first difference between the sample audio output and the real audio output, Loss text represents the second difference between the sample text output and the real text output, w a represents the weight parameter for the first difference, w t represents the weight parameter for the second difference. w can be set according to the actual situation a and w t In some examples, by setting w a and w t Make the value range of the first difference and the second difference close. In other examples, by setting w a and w t To conduct targeted training on audio or text data.

[0073] In the example, Loss audio It can be mean square error loss, Loss text It can be a cross entropy loss. Other methods can also be used to calculate the first difference and the second difference according to actual conditions.

[0074] Using the data processing method provided by the embodiment of the present disclosure, when processing audio-text using a large language model, there is no need to separately train a coding model for discretizing audio data, but extract the spectral features of the audio and convert the spectral features into a form that can be processed by the large language model, thereby achieving continuous processing of text and audio. This method greatly simplifies the entire training process and avoids the cumbersome steps of first training the audio Tokenizer and then discretizing the audio. At the same time, this unified training architecture has stronger scalability. When new data, especially audio data, is added, there is no need to train the entire model from scratch, and only needs to simply continue training, which enables the model to process dynamically changing data sets more efficiently and significantly shortens the training time. Further, the output result of the data processing method according to the embodiment of the present disclosure no longer depends on a separate audio Tokenizer, and the performance of the model will no longer be limited by the performance of the audio Tokenizer. By directly modeling the original audio features, the continuity and richness characteristics of the audio can be more completely retained. In this way, the model can obtain more information from the audio data, making the audio features generated by it more accurate and consistent, and ultimately improving the overall effect of the model. Furthermore, by jointly modeling text and audio features in the same network, the large language model can more naturally learn and capture the cross-modal relationship between text and audio. This approach not only improves the performance of the model when processing multimodal data, but also provides a better foundation for the model to be expanded to other modalities in the future, making it applicable to a wider range of scenarios.

[0075] Figure 3 An exemplary principle of a data processing method according to an embodiment of the present disclosure is shown.

[0076] like Figure 3 As shown, the input to LLM 310 may include an audio start identifier<BOS_AUDIO> , Audio Audio, Audio end identifier<EOS_AUDIO> and textText.

[0077] The audio Audio is converted into a spectral feature 301. The spectral feature 301 is a two-dimensional representation in which the number of rows and the number of columns are the number of Mel filters and the number of audio frames, respectively. Then, the spectral feature 301 is mapped using a mapping layer 320 to obtain an embedded representation of the input audio. Audio start identifier<BOS_AUDIO> , audio end identifier<EOS_AUDIO> The text Text can be processed using the embedding layer 330 to obtain a corresponding embedded representation.

[0078] LLM 310 can process the embedded representation of the input data and output the prediction result. Figure 2In the method of step S210 in the described data processing method, LLM 310 can adaptively pass the output data to the inverse mapping layer 340 to obtain the spectral features 302 of the output audio, or pass the output data to LLM Head 350 to obtain the output text according to the data type corresponding to the current input item.

[0079] Figure 4 An exemplary block diagram of a data processing device according to an embodiment of the present disclosure is shown. Figure 4 The data processing apparatus shown in FIG. 1 is used to perform the combined Figure 2 A data processing method 200 is described.

[0080] like Figure 4 As shown, the data processing device 400 includes an input unit 410 , an audio feature generating unit 420 , a mapping unit 430 , a text processing unit 440 , and an output unit 450 .

[0081] The input unit 410 may be configured to receive input data, wherein the input data includes input audio and input text.

[0082] The audio feature generation unit 420 may be configured to determine spectral features of the input audio.

[0083] The mapping unit 430 may be configured to map the spectral features using a mapping layer to obtain an embedding representation 440 of the input audio for a large language model.

[0084] The text processing unit 440 may be configured to determine an embedded representation of the input text.

[0085] The output unit 450 may be configured to process the embedded representation of the input audio and the embedded representation of the input text using a large language model to obtain a prediction result for the input data.

[0086] By using the data processing device provided by the embodiments of the present disclosure, the training process of the audio-text joint processing model can be simplified, the robustness and consistency of the model output can be improved, and the cross-modal learning ability of the model can be enhanced.

[0087] In some embodiments, the large language model is an autoregressive model, and processing the embedded representation of the input audio and the embedded representation of the input text using the large language model includes: concatenating the embedded representation of the input audio and the embedded representation of the input text as inputs of the large language model; in response to determining that the current input item of the large language model is an audio start identifier or the current input item corresponds to audio data, converting the current output item of the large language model to audio data; in response to determining that the current input item of the large language model is an audio end identifier or the current input item corresponds to text data, converting the current output item of the large language model to text data.

[0088] In some embodiments, concatenating the embedded representation of the input audio and the embedded representation of the input text includes adding an audio start identifier before the embedded representation of the input audio and adding an audio end identifier after the embedded representation of the input audio to identify the start and end of the audio data.

[0089] In some embodiments, determining the spectral features of the input audio includes: performing Fourier transform on each audio frame in the input audio to obtain transformation features of the audio frame; converting the transformation features of the audio frame using a Mel filter to obtain a Mel spectrum of the audio frame; and generating spectral features based on the Mel spectrum of multiple audio frames in the input audio.

[0090] In some embodiments, the embedded representation of the input audio and the embedded representation of the input text have the same size.

[0091] In some embodiments, the large language model is trained based on the following method: receiving sample input data, wherein the sample input data includes sample input audio and sample input text, real output audio and real output text corresponding to the sample input data; determining spectral features of the sample input audio; mapping the spectral features of the sample input audio using a mapping layer to obtain an embedded representation of the sample input audio for the large language model; determining an embedded representation of the sample input text; processing the embedded representation of the sample input audio and the embedded representation of the sample input text using the large language model to obtain a sample prediction result for the input data, wherein the sample prediction result includes a sample audio output and a sample text output; determining a first difference between the sample audio output and the real audio output and a second difference between the sample text output and the real text output; and adjusting parameters of the large language model according to a weighted sum of the first difference and the second difference.

[0092] It should be understood that Figure 4 The various modules or units of the apparatus 400 shown in FIG. 4 can be used in conjunction with the reference Figure 2 The above-described method 200 corresponds to each step in the method 200. Therefore, the operations, features and advantages described above for the method 200 are also applicable to the device 400 and the modules and units included therein. Therefore, the operations, features and advantages described above for the method 200 are also applicable to the device 400 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0093] Although specific functionality is discussed above with reference to specific modules, it should be noted that the functionality of the various units discussed herein may be separated into multiple units, and / or at least some functionality of multiple units may be combined into a single unit.

[0094] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 4 The various units described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program codes / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of units 410 to 450 can be implemented together in a system on chip (System on Chip, SoC). SoC may include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and may optionally execute the received program code and / or include embedded firmware to perform functions.

[0095] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0096] According to another aspect of the present disclosure, an electronic device is also provided, including: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method according to an embodiment of the present disclosure.

[0097] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, where the computer instructions are used to enable the computer to execute the data processing method according to the embodiment of the present disclosure.

[0098] According to another aspect of the present disclosure, a computer program product is further provided, including a computer program, wherein the computer program implements the data processing method according to the embodiment of the present disclosure when executed by a processor.

[0099] refer to Figure 5, a block diagram of an electronic device 500 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0100] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0101] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 can be any type of device that can input information to the electronic device 500. The input unit 506 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 507 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 508 can include but is not limited to a disk, an optical disk. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0102] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as method 200. For example, in some embodiments, the method 200 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method 200 described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the method 200 in any other appropriate manner (e.g., by means of firmware).

[0103] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0105] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0107] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0108] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0109] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0110] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the authorized claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. In addition, the steps may be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear after the present disclosure. The system and the implementation method of the technology interact), or in a computing system including any combination of such backend components, middleware components, or front-end components. The components of the system may be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include: local area network (LAN), wide area network (WAN), Internet, and blockchain network.

[0111] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0112] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0113] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A data processing method, comprising: Receiving input data, wherein the input data includes input audio and input text; Determining frequency spectrum characteristics of the input audio; Mapping the spectral features using a mapping layer to obtain an embedded representation of the input audio for a large language model; determining an embedded representation of the input text; The embedded representation of the input audio and the embedded representation of the input text are processed using the large language model to obtain a prediction result for the input data.

2. The data processing method according to claim 1, wherein: The large language model is an autoregressive model, Processing the embedded representation of the input audio and the embedded representation of the input text using the large language model includes: Concatenating the embedding representation of the input audio and the embedding representation of the input text as input of the large language model; In response to determining that a current input item of the large language model is an audio start identifier or that the current input item corresponds to audio data, converting a current output item of the large language model into audio data; In response to the current input item of the large language model being an audio end identifier or the current input item corresponding to text data, the current output item of the large language model is converted into text data.

3. The data processing method according to claim 2, wherein: Concatenating the embedded representation of the input audio and the embedded representation of the input text includes: The audio start identifier is added before the embedded representation of the input audio, and the audio end identifier is added after the embedded representation of the input audio to identify the start and end of audio data.

4. The data processing method according to any one of claims 1 to 3, wherein: Determining the frequency spectrum characteristics of the input audio includes: For each audio frame in the input audio, Performing Fourier transform on the audio frame to obtain a transformation feature of the audio frame; Using a Mel filter to transform the transformation feature of the audio frame to obtain a Mel spectrum of the audio frame; and The frequency spectrum feature is generated according to Mel frequency spectra of a plurality of audio frames in the input audio.

5. The data processing method according to any one of claims 1 to 3, wherein: The embedded representation of the input audio and the embedded representation of the input text have the same size.

6. The data processing method according to any one of claims 1 to 3, wherein: The large language model is trained based on the following method: Receiving sample input data, wherein the sample input data includes sample input audio and sample input text, and real output audio and real output text corresponding to the sample input data; Determining spectral characteristics of the sample input audio; Mapping the spectral features of the sample input audio using a mapping layer to obtain an embedded representation of the sample input audio for a large language model; Determining an embedding representation of the sample input text; Processing the embedded representation of the sample input audio and the embedded representation of the sample input text using the large language model to obtain a sample prediction result for the input data, wherein the sample prediction result includes a sample audio output and a sample text output; determining a first difference between the sample audio output and the actual audio output and a second difference between the sample text output and the actual text output; and A parameter of the large language model is adjusted according to a weighted sum of the first difference and the second difference.

7. A data processing device, comprising: An input unit configured to receive input data, wherein the input data includes input audio and input text; An audio feature generation unit, configured to determine a frequency spectrum feature of the input audio; A mapping unit configured to map the spectral features using a mapping layer to obtain an embedded representation of the input audio for a large language model; a text processing unit configured to determine an embedded representation of the input text; An output unit is configured to process the embedded representation of the input audio and the embedded representation of the input text using the large language model to obtain a prediction result for the input data.

8. The data processing apparatus according to claim 7, wherein: The large language model is an autoregressive model, Processing the embedded representation of the input audio and the embedded representation of the input text using the large language model includes: Concatenating the embedding representation of the input audio and the embedding representation of the input text as input of the large language model; In response to determining that a current input item of the large language model is an audio start identifier or that the current input item corresponds to audio data, converting a current output item of the large language model into audio data; In response to the current input item of the large language model being an audio end identifier or the current input item corresponding to text data, the current output item of the large language model is converted into text data.

9. The data processing apparatus according to claim 8, wherein: Concatenating the embedded representation of the input audio and the embedded representation of the input text includes: The audio start identifier is added before the embedded representation of the input audio, and the audio end identifier is added after the embedded representation of the input audio to identify the start and end of audio data.

10. The data processing device according to any one of claims 7 to 9, wherein: Determining the frequency spectrum characteristics of the input audio includes: For each audio frame in the input audio, Performing Fourier transform on the audio frame to obtain a transformation feature of the audio frame; Using a Mel filter to transform the transformation feature of the audio frame to obtain a Mel spectrum of the audio frame; and The frequency spectrum feature is generated according to Mel frequency spectra of a plurality of audio frames in the input audio.

11. The data processing device according to any one of claims 7 to 9, wherein: The embedded representation of the input audio and the embedded representation of the input text have the same size.

12. The data processing device according to any one of claims 7 to 9, wherein: The large language model is trained based on the following method: Receiving sample input data, wherein the sample input data includes sample input audio and sample input text, and real output audio and real output text corresponding to the sample input data; Determining spectral characteristics of the sample input audio; Mapping the spectral features of the sample input audio using a mapping layer to obtain an embedded representation of the sample input audio for a large language model; Determining an embedding representation of the sample input text; Processing the embedded representation of the sample input audio and the embedded representation of the sample input text using the large language model to obtain a sample prediction result for the input data, wherein the sample prediction result includes a sample audio output and a sample text output; determining a first difference between the sample audio output and the actual audio output and a second difference between the sample text output and the actual text output; and A parameter of the large language model is adjusted according to a weighted sum of the first difference and the second difference.

13. An electronic device, comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

15. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images