Data processing method, apparatus, device, and medium

Abstract

The present disclosure provides a data processing method and device, equipment and medium, relates to the field of artificial intelligence, in particular to natural language processing, image processing, audio processing, multi-modal understanding, deep learning and large model technology. The implementation scheme is: obtaining a plurality of original token vectors, wherein each of the plurality of original token vectors represents one of text, image and audio; dividing the plurality of original token vectors into a plurality of groups, and calculating a first comprehensive vector corresponding to each group using the plurality of original token vectors included in the group; inputting the plurality of first comprehensive vectors corresponding to the plurality of groups into a large model to obtain a second comprehensive vector output by the large model; and reconstructing the second comprehensive vector to obtain a plurality of target token vectors, wherein each of the plurality of target token vectors represents one of text, image and audio.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, in particular to natural language processing, image processing, audio processing, multi-modal understanding, deep learning and large model technology, and in particular to a data processing method, a data processing apparatus, an electronic device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] Artificial intelligence is a discipline that studies enabling computers to simulate some thinking processes and intelligent behaviors of humans (such as learning, reasoning, thinking, planning, etc.), which has both hardware and software technologies. Artificial intelligence hardware technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, etc.; artificial intelligence software technology mainly includes computer vision technology, speech recognition technology, natural language processing technology, and machine learning / deep learning, big data processing technology, knowledge graph technology, etc. several major directions.

[0003] In recent years, multi-modal large models have achieved great success in various understanding and generation tasks. These models can process different modal inputs including text, images, and audio, and generate corresponding multi-modal replies according to user questions.

[0004] The methods described in this section do not necessarily have to be the methods previously conceived or employed. Unless otherwise indicated, nothing in this section should be assumed to be prior art merely because of its inclusion in this section. Similarly, unless otherwise indicated, nothing contained in this section should be assumed to be known in the art. SUMMARY

[0005] The present disclosure provides a data processing method, a data processing apparatus, an electronic device, a computer readable storage medium and a computer program product.

[0006] According to an aspect of the present disclosure, a data processing method is provided, comprising: obtaining a plurality of original token vectors, wherein each of the plurality of original token vectors represents one of text, image and audio; dividing the plurality of original token vectors into a plurality of groups, and calculating a first comprehensive vector corresponding to each group from the plurality of original token vectors included in the group; inputting a plurality of first comprehensive vectors corresponding to the plurality of groups into a large model to obtain a second comprehensive vector output by the large model; and reconstructing a plurality of target token vectors from the second comprehensive vector, wherein each of the plurality of target token vectors represents one of text, image and audio.

[0007] According to another aspect of this disclosure, a data processing apparatus is provided, comprising: an acquisition unit configured to acquire a plurality of original word vectors, wherein each of the plurality of original word vectors represents one of text, image, and audio; a grouping unit configured to divide the plurality of original word vectors into a plurality of groups, and to calculate a first composite vector corresponding to each group using the plurality of original word vectors included in each of the plurality of groups; a large model configured to receive the plurality of first composite vectors corresponding to the plurality of groups, and to output a second composite vector; and a reconstruction unit configured to reconstruct a plurality of target word vectors using the second composite vector, wherein each of the plurality of target word vectors represents one of text, image, and audio.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods provided in this disclosure.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the methods provided in this disclosure.

[0010] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program wherein, when executed by a processor, the computer program implements the methods provided in this disclosure.

[0011] According to one or more embodiments of this disclosure, after converting lexical units into their corresponding vector representations, multiple lexical units are grouped together and a total vector representation is calculated for each group. This achieves sequence shortening at the model input level while preserving the original multimodal lexical information. At the output level, the complete output lexical sequence is obtained by reconstructing the second comprehensive vector output by the large model, achieving a dual improvement in both effectiveness and efficiency.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1 A schematic diagram of an exemplary system in which the various methods described herein may be implemented according to embodiments of the present disclosure is shown;

[0015] Figure 2 An exemplary process of a data processing method according to an embodiment of the present disclosure is shown;

[0016] Figure 3 An exemplary process for obtaining multiple raw word vectors according to embodiments of the present disclosure is illustrated;

[0017] Figure 4 An exemplary process for obtaining multiple raw word vectors according to embodiments of the present disclosure is illustrated;

[0018] Figure 5 A schematic diagram illustrating word vector folding, processing, and restoration according to embodiments of the present disclosure is shown;

[0019] Figure 6 An exemplary block diagram of a data processing apparatus according to embodiments of the present disclosure is shown;

[0020] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0023] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0024] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0025] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0026] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of methods according to embodiments of this disclosure.

[0027] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105, and / or 106 under a Software as a Service (SaaS) model.

[0028] exist Figure 1 In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0029] Users can use client devices 101, 102, 103, 104, 105, and / or 106 to obtain data processed by the method according to embodiments of the present disclosure. The client devices can provide an interface enabling users to interact with the client devices. The client devices can also output information to the user via this interface. Although Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0030] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0031] Network 110 can be any type of network well known to those skilled in the art, and can use any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.) to support data communication. By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0032] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0033] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0034] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105 and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105 and / or 106.

[0035] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0036] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0037] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0038] Figure 1The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0039] Figure 2 An exemplary process 200 of a data processing method according to an embodiment of the present disclosure is shown.

[0040] like Figure 2 As shown, in step S202, multiple original word vectors can be obtained, each of which represents one of the text, image, and audio.

[0041] In step S204, multiple original word vectors can be divided into multiple groups, and the first comprehensive vector corresponding to the group can be calculated using the multiple original word vectors included in each group.

[0042] In step S206, multiple first comprehensive vectors corresponding to multiple groups can be input into the large model to obtain the second comprehensive vector output by the large model. In this disclosure, the large model is also referred to as the backbone large model.

[0043] In step S208, multiple target word vectors can be reconstructed using the second comprehensive vector, where each of the multiple target word vectors represents one of the text, image, and audio.

[0044] By utilizing the embodiments of this disclosure, after converting lexical units into their corresponding vector representations, multiple lexical units are grouped together, and a total vector representation is calculated for each group. This achieves sequence shortening at the model input level while preserving the original multimodal lexical information. At the output level, the complete output lexical sequence is obtained by reconstructing the second comprehensive vector output by the large model, achieving a dual improvement in both performance and efficiency.

[0045] The principles of this disclosure will now be described in detail.

[0046] In the encoding and decoding of multimodal data such as images and audio, residual quantization strategies, such as Residual Quantizer-Variational Autoencoder (RQ-VAE) and Residual Vector Quantization (RVQ), are widely used due to their excellent reconstruction results. These methods first divide the original multimodal data into multiple segments, then calculate a token sequence for each segment, and finally output a multimodal token matrix. However, the multimodal token matrix obtained by this method usually faces the problem of a large number of tokens, which brings a significant computational burden to the training and inference of large models.

[0047] During the training and inference phases, large models (especially Transformer-based multimodal models) typically receive a sequence of terms and learn to predict the next term through autoregressive decoding. Current large models generally process the raw data in two ways: one is to directly expand the term matrix into a one-dimensional term sequence to preserve as much of the original information as possible; the other is to selectively use some terms to shorten the sequence length, thereby improving the model's data processing efficiency.

[0048] In this context, both directly expanding the lexical matrix into a one-dimensional lexical sequence and selecting some lexical units to shorten the sequence length face their own limitations:

[0049] 1. Directly expanding the lexical units into a one-dimensional lexical sequence can preserve the original information of the data, but it will face the problem of a large number of lexical units after transformation, which is more serious in long sequence tasks.

[0050] 2. While selective use of lexical units can shorten sequence length, it may result in the loss of significant original data details, making it difficult to answer questions strongly correlated with these details. In particular, for multimodal generation tasks, the lack of detail has a substantial impact on the generation of multimodal data. Therefore, selectively using some multimodal lexical units is not a universal solution suitable for multimodal understanding and generation tasks.

[0051] The data processing method disclosed herein not only preserves the original input information but also shortens the data length processed by the main large model, thereby achieving a dual improvement in effectiveness and efficiency.

[0052] In step S202, the multiple original word vectors can be obtained by processing raw data of text, images, audio, and / or other modalities. In some examples, the raw data can be encoded to obtain a word sequence. Each word in the word sequence can be a single numerical value. Then, each word can be converted into a corresponding vector representation to obtain the original word vectors.

[0053] Different methods can be used to obtain the corresponding original word vectors for different modalities.

[0054] Figure 3 An exemplary process 300 for obtaining a plurality of raw word vectors according to embodiments of the present disclosure is shown. Process 300 may be used to at least partially implement step S202 in process 200.

[0055] In step S302, the target image can be divided into multiple image blocks. The target image can be any image data that needs to be processed. This disclosure does not limit the specific method of dividing the target image.

[0056] In step S304, multiple image blocks can be encoded to obtain a first preset number of image words corresponding to each image block. This disclosure does not limit the image encoding method or the specific value of the first preset number; those skilled in the art can select and set it according to their needs.

[0057] In step S306, each image term can be mapped to an original image term vector, where multiple original term vectors include a first preset number of original image term vectors corresponding to each of multiple image patches. In some examples, a pre-trained image coding model (such as RQ-VAE or Vision Transformer) can be used to generate image term embeddings. This mapping process maps high-dimensional image patch feature vectors to corresponding low-dimensional original term vectors by querying a term embedding table or codebook. Alternatively, the mapping can be accomplished through linear transformations, neural networks, or other methods, which are not limited here.

[0058] In an exemplary embodiment, the RQ-VAE model can be used to segment and encode the target image. Taking a target image with a resolution of 1024×1024 as an example, the RQ-VAE model can compress it into a 32×32×4 word matrix. That is, the target image is divided into 32×32 image blocks, and each image block is encoded into 4 image words. Furthermore, each image word can be mapped to its corresponding original image word vector.

[0059] In some examples, in step S204, among the multiple original word vectors, a first preset number of original image word vectors corresponding to the same image patch can be grouped into the same group. This ensures that the grouping of these word vectors aligns with the semantic information of the target image, thereby improving the rationality of the grouping. This method ensures that the first synthesized vector representing the image modality used as input to the large model can be effectively associated with an actual region in the target image, enhancing semantic expressiveness and thus improving the overall performance and accuracy of the large model when processing multimodal data.

[0060] Figure 4 An exemplary process 400 for obtaining a plurality of raw word vectors according to embodiments of the present disclosure is shown. Process 400 may be used to at least partially implement step S202 in process 200.

[0061] In step S402, the target audio can be divided into multiple audio segments. The target audio can be any audio data that needs to be processed. This disclosure does not limit the specific method of dividing the target audio.

[0062] In step S404, multiple audio segments can be encoded to obtain a second preset number of audio words corresponding to each audio segment. This disclosure does not limit the audio encoding method or the specific value of the second preset number; those skilled in the art can select and set it according to their needs.

[0063] In step S406, each audio word can be mapped to an original audio word vector. Multiple original word vectors include a second preset number of original audio word vectors corresponding to each audio segment. In some examples, the audio words obtained after encoding the audio segments can be feature representations obtained after time-frequency transformation (such as Short-Time Fourier Transform (STFT) or Mel-frequency spectrum). Alternatively, pre-trained audio coding models such as Wav2Vec and Hubert can be used. To map the word features of audio segments to original audio word vectors, neural network structures such as linear transforms or multilayer perceptrons (MLPs) can be used to implement the above mapping process.

[0064] In one exemplary embodiment, each second of audio can be divided into 75 audio segments, and each audio segment can be encoded into 8 audio words.

[0065] In some examples, in step S204, among the multiple original word vectors, a second preset number of original audio word vectors corresponding to the same audio segment can be grouped into the same group. This ensures that the grouping of these word vectors aligns with the semantic information of the target audio, thereby improving the rationality of the grouping. This method ensures that the first synthesized vector representing the audio modality used as input to the large model can be effectively associated with an actual segment in the target audio, enhancing semantic expressiveness and thus improving the overall performance and accuracy of the large model when processing multimodal data.

[0066] In some examples, multiple original word vectors can be divided into multiple groups in other ways, which are not limited here.

[0067] In addition to the methods provided in the above embodiments, other methods can also be used to process the original content of modalities such as images, audio, and text to obtain the corresponding original word vectors, which are not limited here.

[0068] In step S204, calculating the first comprehensive vector corresponding to each group using the multiple original word vectors included in each of the multiple groups can include calculating a total vector representation, i.e., the first comprehensive vector, for each group of original word vectors through methods such as addition, concatenation, or neural networks (e.g., encoding networks based on the Transformer structure). This process can also be referred to as word folding.

[0069] By folding multi-lexical information, sequence shortening is achieved at the input level of the large model, thereby significantly reducing the sequence length that the main large model needs to process, while preserving the lexical information of the original data (especially the original information of multimodal data).

[0070] In step S206, the large model can receive multiple first comprehensive vectors corresponding to multiple groups and encode these first comprehensive vectors to output second comprehensive vectors. The second comprehensive vector can represent the next vector representation predicted by the large model from the multiple first comprehensive vectors. This vector representation can contain multiple lexical information, and therefore can be used for reconstruction in step S208 to restore multiple target lexical vectors.

[0071] In some examples, the large model can be based on a Transformer structure. Step S206, inputting multiple first comprehensive vectors corresponding to multiple groups into the large model to obtain the second comprehensive vector output by the large model, may include: using the large model to perform autoregressive decoding based on multiple first comprehensive vectors to obtain one or more second comprehensive vectors.

[0072] The Transformer model is a neural network model based on an attention mechanism, primarily used for Natural Language Processing (NLP) tasks such as language translation and text classification. The Transformer architecture is widely applied in the field of NLP. A Transformer model mainly consists of two parts: an encoder and a decoder. The encoder transforms the input sequence into a fixed-length vector representation, while the decoder transforms that vector into an output sequence. Between the encoder and decoder, there are one or more attention mechanism layers used to capture the correlation between the input and output sequences.

[0073] The input and output sequences of the Transformer model are both composed of tokens or their corresponding token vectors. Each token can correspond to a single character, word, or other text unit in the text modality; a pixel, image patch, or other image unit in the image modality; an audio frame, audio segment, or other audio unit in the audio modality; or a segment with specific semantics in other modalities. Furthermore, tokens can also correspond to special symbols, such as classification markers [CLS], separators [SEP], mask markers [MASK], etc.

[0074] The Transformer model primarily consists of multiple TransformerLayers, each containing two main parts: a Self-Attention layer and a Feed-Forward Network (FFN). Self-Attention is a method that calculates and synthesizes the correlations between each word and other words. This method captures the relationships between individual words (or word vectors) in the input sequence, regardless of their distance within the sequence. In the Transformer model, Self-Attention is used to calculate the influence of each word (or word vector) in the input or output sequence on other words (or word vectors). The Feed-Forward Network is a fully connected neural network used to perform non-linear transformations on the input sequence. In the Transformer model, the FFN can consist of two fully connected layers.

[0075] By using autoregressive decoding, a large model can iteratively predict subsequent multimodal information outputs until the model has completed its output. Furthermore, by having the large model output one or more second comprehensive vectors, and then reconstructing these second comprehensive vectors to obtain the target word vectors, the data length processed by the large model is shortened, achieving a dual improvement in both performance and efficiency.

[0076] In some examples, step S208, reconstructing multiple target word vectors using the second comprehensive vector, may include: using a word reconstruction unit based on a Transformer structure to perform autoregressive decoding based on the second comprehensive vector to obtain multiple target word vectors.

[0077] The lexical reconstruction unit based on the Transformer structure possesses powerful semantic learning capabilities. Through autoregressive decoding, the model can progressively enhance semantic expression during the generation process, resulting in generated target lexical vectors with high semantic consistency and coherence. This significantly improves the performance and understanding capabilities of multimodal models.

[0078] In one example, the model size of a lexical reconstruction unit can be smaller than that of a large model.

[0079] Figure 5 A schematic diagram illustrating the lexical vector folding, processing, and restoration according to embodiments of the present disclosure is shown.

[0080] During the word folding process 502, operations such as segmentation, encoding, and / or mapping can be performed on the original data of a certain modality (e.g., image, audio, text, etc.) to obtain four original word vectors A, B, C, and D corresponding to the same semantic segment (e.g., image block, audio segment, text segment, etc.). These four word vectors can be folded into a first comprehensive vector 508 by means of addition, concatenation, or neural networks. The first comprehensive vector 508 can be processed using a backbone network 504 to obtain a second comprehensive vector 510. Word decomposition 506 can be performed on the second comprehensive vector 510 to obtain the target word vector E.

[0081] The second composite vector 510 can be considered as obtained by performing word folding 502 on the target word vector E. Therefore, if autoregressive decoding is required using the backbone network 504, the first composite vector 508 (obtained by performing word folding on the original word vectors A, B, C, and D) and the second composite vector 510 (corresponding to the target word vector E) can be re-input into the backbone network 504 to obtain the next second composite vector 512. Word restoration 506 can be performed on the second composite vector 512 to obtain the target word vectors F, G, and H in sequence.

[0082] In one example, the first composite vector 508, the second composite vector 510, and the second composite vector 512 can be re-input into the backbone network 504. After completing two more rounds of autoregressive decoding, the second composite vector 514 and its corresponding target word vectors I and J, and the second composite vector 516 and its corresponding target word vectors K, L, M, and N are obtained.

[0083] In some examples, multiple original word vectors can have a preset vector length. Step S208, reconstructing multiple target word vectors using the second comprehensive vector, includes: segmenting the second comprehensive vector based on the preset vector length to obtain multiple target word vectors.

[0084] By segmenting the vector using a preset length, the complexity of processing the composite vector is reduced, avoiding the high computational overhead of gradual generation or reconstruction. Segmentation can be used as a direct operation to quickly divide the second composite vector into multiple target word vectors, thereby improving the overall processing speed.

[0085] Figure 6 An exemplary block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown. As shown in Figure 6, the data processing apparatus 600 may include an acquisition unit 610, a grouping unit 620, a large model 630, and a reconstruction unit 640. This can be utilized... Figure 6 The data processing device described in the text performs the combination Figure 2 Describe the data processing methods.

[0086] The acquisition unit 610 can be configured to acquire multiple raw word vectors, each of which represents one of the following: text, image, and audio.

[0087] Grouping unit 620 can be configured to divide multiple original word vectors into multiple groups, and use the multiple original word vectors included in each of the multiple groups to calculate a first comprehensive vector corresponding to that group.

[0088] The large model 630 can be configured to receive multiple first synthesis vectors corresponding to multiple groups, and output a second synthesis vector.

[0089] The reconstruction unit 640 can be configured to reconstruct multiple target word vectors using the second comprehensive vector, wherein each of the multiple target word vectors represents one of the text, image, and audio.

[0090] In some embodiments, the acquisition unit may include: a segmentation subunit configured to divide the target image into multiple image blocks; a first encoding subunit configured to encode the multiple image blocks to obtain a first preset number of image words corresponding to each image block; and a first mapping subunit, which maps each image word to an original image word vector in the step, wherein the multiple original word vectors include the first preset number of original image word vectors corresponding to each of the multiple image blocks.

[0091] In some embodiments, among a plurality of original word vectors, a first preset number of original image word vectors corresponding to the same image block can be grouped into the same group.

[0092] In some embodiments, the acquisition unit may include: a segmentation subunit configured to divide the target audio into multiple audio segments; a second encoding subunit configured to encode the multiple audio segments to obtain a second preset number of audio words corresponding to each audio segment; and a second mapping subunit configured to map each audio word to an original audio word vector, wherein the multiple original word vectors include the second preset number of original audio word vectors corresponding to each of the multiple audio segments.

[0093] In some embodiments, among a plurality of original word vectors, a second preset number of original audio word vectors corresponding to the same audio segment can be grouped into the same group.

[0094] In some embodiments, the reconstruction unit can be configured to use a Transformer-based lexical reconstruction unit to perform autoregressive decoding based on a second synthetic vector to obtain multiple target lexical vectors.

[0095] In some embodiments, the multiple original word vectors may have a preset vector length, and the reconstruction unit may be configured to segment the second comprehensive vector based on the preset vector length to obtain multiple target word vectors.

[0096] In some embodiments, the large model can be based on a Transformer structure and can be configured to perform autoregressive decoding based on multiple first synthetic vectors to obtain one or more second synthetic vectors.

[0097] It should be understood that Figure 6 The various modules or units of the device 600 shown can be connected to the reference. Figure 2 The steps in method 200 described correspond to each other. Therefore, the operations, features, and advantages described above for method 200 also apply to apparatus 600 and its included modules and units. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0098] Although specific functions have been discussed with reference to specific modules above, it should be noted that the functions of the various units discussed in this article can be divided into multiple units, and / or at least some functions of multiple units can be combined into a single unit.

[0099] It should also be understood that this article can describe various technologies in the general context of software and hardware components or program modules. The above regarding... Figure 6 The described units can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuit. For example, in some embodiments, one or more of units 610 to 640 can be implemented together in a System on Chip (SoC). The SoC may include an integrated circuit chip (which includes a processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or one or more components of other circuitry) and may optionally execute received program code and / or include embedded firmware to perform functions.

[0100] According to another aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory storing instructions executable by the at least one processor to enable the at least one processor to perform a method according to an embodiment of this disclosure.

[0101] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform a method according to an embodiment of this disclosure is also provided.

[0102] According to another aspect of this disclosure, a computer program product is also provided, including a computer program, wherein the computer program, when executed by a processor, implements the method according to an embodiment of this disclosure.

[0103] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0104] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0105] refer to Figure 7 The present invention describes a structural block diagram of an electronic device 700 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0106] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the electronic device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0107] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, output unit 707, storage unit 708, and communication unit 709. Input unit 706 can be any type of device capable of inputting information to electronic device 700. Input unit 706 can receive input digital or character information and generate key signal input related to user settings and / or function control of electronic device, and may include, but is not limited to, a mouse, keyboard, touch screen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 707 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 708 may include, but is not limited to, disk and optical disk. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth. TM Devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices and / or the like.

[0108] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as processes 200, 300, and 400. For example, in some embodiments, processes 200, 300, and 400 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of processes 200, 300, and 400 described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute processes 200, 300, 400 by any other suitable means (e.g., by means of firmware).

[0109] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0110] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0111] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0113] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0114] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0115] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0116] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A large model-based data processing method, comprising: obtaining a plurality of original token vectors, wherein each of the plurality of original token vectors represents one of text, image and audio; dividing the plurality of original token vectors into a plurality of groups, and calculating a first comprehensive vector corresponding to each of the plurality of groups by using a plurality of original token vectors included in the group, comprising: calculating the original token vectors included in each group by using an encoding network based on a Transformer structure to obtain a first comprehensive vector corresponding to the group, wherein the number of first comprehensive vectors corresponding to each group is less than the number of original token vectors included in the group; inputting a plurality of first comprehensive vectors corresponding to the plurality of groups into a large model based on a Transformer model to obtain a second comprehensive vector output by the large model, comprising: performing autoregressive decoding on the plurality of first comprehensive vectors based on the large model to obtain a plurality of second comprehensive vectors; and reconstructing a plurality of target token vectors by using the second comprehensive vector, comprising: performing autoregressive decoding on each of the plurality of second comprehensive vectors by using a token reconstruction unit based on a Transformer structure to obtain one or more target token vectors corresponding to the second comprehensive vector, wherein each of the plurality of target token vectors represents one of text, image and audio, and at least one of the plurality of second comprehensive vectors is reconstructed into a plurality of target token vectors.

2. The method of claim 1, wherein, Obtaining a plurality of original token vectors comprises: dividing a target image into a plurality of image blocks; encoding the plurality of image blocks to obtain a first predetermined number of image tokens corresponding to each image block; and mapping each image token to an original image token vector, wherein the plurality of original token vectors include the first predetermined number of original image token vectors corresponding to each of the plurality of image blocks.

3. The method of claim 2, wherein, The first predetermined number of original image token vectors corresponding to the same image block in the plurality of original token vectors are divided into the same group.

4. The method of claim 1, wherein, Obtaining a plurality of original token vectors comprises: dividing a target audio into a plurality of audio segments; encoding the plurality of audio segments to obtain a second predetermined number of audio tokens corresponding to each audio segment; and mapping each audio token to an original audio token vector, wherein the plurality of original token vectors include the second predetermined number of original audio token vectors corresponding to each of the plurality of audio segments.

5. The method of claim 4, wherein, The second predetermined number of original audio token vectors corresponding to the same audio segment in the plurality of original token vectors are divided into the same group.

6. The method of any one of claims 1-5, wherein, The plurality of original token vectors have a predetermined vector length, and reconstructing a plurality of target token vectors by using the second comprehensive vector comprises: splitting the second comprehensive vector based on the predetermined vector length to obtain the plurality of target token vectors.

7. A large model-based data processing apparatus, comprising: An obtaining unit configured to obtain a plurality of original token vectors, wherein each of the plurality of original token vectors represents one of a text, an image, and an audio; A grouping unit configured to divide the plurality of original token vectors into a plurality of groups, and calculate a first comprehensive vector corresponding to each of the plurality of groups by using a plurality of original token vectors included in the group, comprising: calculating, by using an encoding network based on a Transformer structure, the original token vectors included in each group to obtain a first comprehensive vector corresponding to the group, wherein the number of first comprehensive vectors corresponding to each group is less than the number of original token vectors included in the group; a large model based on a Transformer model configured to receive a plurality of first comprehensive vectors corresponding to the plurality of groups to output a second comprehensive vector, comprising: performing autoregressive decoding by using the large model based on the plurality of first comprehensive vectors to obtain a plurality of second comprehensive vectors; and A reconstruction unit configured to reconstruct a plurality of target token vectors by using the second comprehensive vector, comprising: performing autoregressive decoding on each of the plurality of second comprehensive vectors by using a token reconstruction unit based on a Transformer structure to obtain one or more target token vectors corresponding to the second comprehensive vector, wherein each of the plurality of target token vectors represents one of a text, an image, and an audio, and at least one of the plurality of second comprehensive vectors is reconstructed into a plurality of target token vectors.

8. The apparatus of claim 7, wherein, The obtaining unit comprises: a blocking subunit configured to divide a target image into a plurality of image blocks; a first encoding subunit configured to encode the plurality of image blocks to obtain a first preset number of image tokens corresponding to each image block; and a first mapping subunit configured to map each image token to an original image token vector, wherein the plurality of original token vectors include the first preset number of original image token vectors corresponding to each of the plurality of image blocks.

9. The apparatus of claim 8, wherein, The first preset number of original image token vectors corresponding to the same image block in the plurality of original token vectors are divided into the same group.

10. The apparatus of claim 7, wherein, The obtaining unit comprises: a segmenting subunit configured to divide a target audio into a plurality of audio segments; a second encoding subunit configured to encode the plurality of audio segments to obtain a second preset number of audio tokens corresponding to each audio segment; and a second mapping subunit configured to map each audio token to an original audio token vector, wherein the plurality of original token vectors include the second preset number of original audio token vectors corresponding to each of the plurality of audio segments.

11. The apparatus of claim 10, wherein, The second preset number of original audio token vectors corresponding to the same audio segment in the plurality of original token vectors are divided into the same group.

12. The apparatus of any of claims 7-11, wherein, The plurality of original token vectors have a preset vector length, and the reconstruction unit is configured to split the second comprehensive vector based on the preset vector length to obtain the plurality of target token vectors.

13. An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are for causing a computer to perform the method of any one of claims 1-6.

15. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-6.

Images