Coding method and related device

Abstract

The coding method comprises the following steps: acquiring to-be-coded data; the to-be-coded data serve as input of the content coding branch network, a content coding sequence is output, and the content coding sequence comprises a first discrete coding sequence obtained by coding content information of the to-be-coded data; the to-be-coded data serve as input of the style coding branch network, a style coding sequence is output, and the style coding sequence comprises a second discrete coding sequence obtained by coding style information of the to-be-coded data. In the application, the content encoding branch network and the style encoding branch network can independently encode the content and the style in the to-be-encoded data, so that the content and the style in the to-be-encoded data are mutually opposite and separated, 'de-entanglement 'of the content and the style in the encoding sequence is realized, and work such as content understanding and style migration of downstream tasks is facilitated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a coding method and related device. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, a multimodal large language model (MLLM) is developing rapidly. Relying on the powerful understanding and reasoning capability of the large language model, the multimodal large language model can input text, pictures, videos, voice and other modalities, uniformly understand and reason them, and generate reasonable reply content accordingly. How to make the multimodal large language model better understand the input content of multiple modalities has always been a difficulty in the development of the multimodal large language model. SUMMARY

[0003] In order to solve the above problems, the embodiments of the present application provide a coding method and related device, which can obtain a content coding sequence and a style coding sequence of the to-be-coded data by coding the to-be-coded data, so that the downstream task can independently analyze the content and style of the to-be-coded data. For example, when the downstream task is a multimodal large language model, the multimodal large language model can independently analyze the to-be-coded data from the content and style of the to-be-coded data, thereby better understanding the to-be-coded data.

[0004] To this end, the embodiments of the present application adopt the following technical solutions:

[0005] In a first aspect, the embodiments of the present application provide a coding method, mainly comprising the following steps: obtaining to-be-coded data; taking the to-be-coded data as the input of a content coding branch network, outputting a content coding sequence, the content coding sequence comprising a first discrete coding sequence obtained by coding the content information of the to-be-coded data; taking the to-be-coded data as the input of a style coding branch network, outputting a style coding sequence, the style coding sequence comprising a second discrete coding sequence obtained by coding the style information of the to-be-coded data.

[0006] In the technical solutions of the present application, the encoding system includes a content encoding branch network and a style encoding branch network. The content encoding branch network is configured to encode the content of the to-be-encoded data, and the style encoding branch network is configured to encode the style (information other than the content) of the to-be-encoded data. In some technical solutions, the encoding sequence obtained by encoding the to-be-encoded data includes information of both the content and the style. It can be understood that the content and the style in the encoding sequence are intertwined to form an integral whole, and it is difficult for a downstream task to decouple the content and the style in the to-be-encoded data, and thus it is difficult to complete the task from two perspectives of the content and the style. The downstream task can be: migrating and converting the style of the to-be-encoded data; understanding the content of the to-be-encoded data through the style of the to-be-encoded data, and the like. In the technical solutions of the present application, the content encoding branch network and the style encoding branch network can independently encode the content and the style in the to-be-encoded data, so that the content and the style in the to-be-encoded data are opposite and separated from each other. It can be understood that the content and the style in the encoding sequence are two independent parts after being disentangled, and the downstream task can use the encoding sequence of the decoupled content and style in the to-be-encoded data, and thus complete the task from two perspectives of the content and the style. The downstream task is, for example: the style of the to-be-encoded data can be migrated and converted according to the style encoding sequence; the content of the to-be-encoded data can be understood through the style of the to-be-encoded data, for example, in a telephone customer service scenario, the style encoding sequence is independent of the content, and it can be determined whether the user is angry by analyzing only the style encoding sequence of the user's voice; the content encoding sequence does not include style information, which facilitates the content understanding work of the downstream task.

[0007] In one possible implementation, the to-be-encoded data includes any one or more of audio data, image data, and video data.

[0008] In this implementation, the encoding system or the encoding method of the present application can encode data of multiple modalities, and the encoded content encoding sequence and the style encoding sequence can be used as input of a downstream task (such as a multi-modal large language model). This implementation is exemplified for data of multiple modalities, but is not limited to the few examples exemplified.

[0009] In one possible implementation, the above-mentioned inputting the to-be-encoded data into the content encoding branch network to output the content encoding sequence includes: inputting the to-be-encoded data into a content encoder to output a content representation vector; and inputting the content representation vector into a content quantizer to output the content encoding sequence.

[0010] In this implementation, the content encoding branch network comprises a content encoder and a content quantizer. The content encoder can encode the data to be encoded to obtain a content representation vector; and the content quantizer can quantize the content representation vector, such as clustering processing, to obtain a discrete content encoding sequence. Generally, when the data to be encoded is speech, image, or video, the content encoding sequence generated by the content encoder is usually continuous, for example, the representation vectors of multiple frames of speech data correspond to one encoding, and therefore, a discretization process is needed, such as a common discretization method: clustering processing.

[0011] In one possible implementation, the above-mentioned inputting the data to be encoded into the style encoding branch network and outputting a style encoding sequence comprises: inputting the data to be encoded into a style encoder and outputting a style representation vector; and inputting the style representation vector into a style quantizer and outputting a style encoding sequence.

[0012] In this implementation, the style encoding branch network comprises a style encoder and a style quantizer. The style encoder can encode the data to be encoded to obtain a style representation vector; and the style quantizer can quantize the style representation vector, such as clustering processing, to obtain a discrete style encoding sequence. Generally, when the data to be encoded is speech, image, or video, the style encoding sequence generated by the style encoder is usually continuous, for example, the representation vectors of multiple frames of speech data correspond to one encoding, and therefore, a discretization process is needed, such as a common discretization method: clustering processing.

[0013] In one possible implementation, before the data to be encoded is inputted into the content encoding branch network and the style encoding branch network, the method further comprises: inputting the data to be encoded into a base encoder to output a feature representation vector, and the feature representation vector is used as the input of the content encoding branch network and the style encoding branch network.

[0014] In this implementation, a method for pre-processing the data to be encoded is specifically limited. Before the content encoding branch network and / or the style encoding branch network encodes the data to be encoded, the original data to be encoded is pre-processed using a base encoder. The original data to be encoded refers to the data to be encoded in the initial state. The input of the base encoder is the original data to be encoded, and the output is the pre-processed data to be encoded, which reduces the workload of the content encoding branch network and the style encoding branch network, thereby improving the work efficiency of the content encoding branch network and the style encoding branch network. In other implementations, the original data to be encoded can be directly inputted into the content encoding branch network and the style encoding branch network, and the content encoding branch network and the style encoding branch network themselves perform the pre-processing and encoding work.

[0015] In a possible implementation, the content encoding branch network is trained by a training method of a first training stage, and the training method of the first training stage comprises: obtaining a first training sample and a first content label corresponding to the first training sample; taking the first training sample as an input of the content encoding branch network to be trained, and outputting a content encoding sequence of the first training sample; taking the content encoding sequence of the first training sample as an input of the first decoder, and outputting first content information corresponding to the first training sample; calculating a loss value of the first content information and the first content label to obtain a first loss value; and adjusting parameters of the content encoding branch network to minimize the first loss value.

[0016] In this implementation, in the training process of the content encoding branch network, the content encoding branch network is trained with the content label as a label, which can make the content encoding sequence more accurate and ensure that the content encoding branch network only encodes the content in the to-be-encoded data, which is equivalent to independently separating the content encoding sequence from the state of the content and style related "entanglement" in the to-be-encoded data. That is, the encoding and decoding process is trained with the content label as a label, so that the encoding and decoding process is performed on the content of the to-be-encoded data, and the trained content encoding branch network only encodes the content of the to-be-encoded data, thereby realizing independent separation of the content encoding sequence from the state of the content and style related "entanglement" and achieving the purpose of content "disentanglement".

[0017] In a possible implementation, the first content label of the first training sample is obtained based on user annotation of the content of the first training sample.

[0018] In this implementation, the determination method of the first content label is specifically defined. The content-related information in the target data includes a lot of information, such as text content, context content, and the like. However, not all content-related information is necessary or of interest to the user. By setting the content label by the user, the content information in the content encoding sequence can be made to be consistent with the user's intention Figure 1 , that is, the information irrelevant to the user's intention is excluded, so that the content encoding sequence encoded by the trained first encoding unit only retains the content-related information of interest to the user, reduces or avoids irrelevant information, ensures the specific direction of the content information in the content encoding sequence, and improves the user experience.

[0019] In a possible implementation, the style encoding branch network is trained by a training method of a second training stage, and the training method of the second training stage comprises the following steps: obtaining second training samples and second content labels of the second training samples; taking the second training samples as inputs of the content encoding branch network, outputting content encoding sequences of the second training samples, and the content encoding branch network being in a state of being trained and having parameters frozen by the first training stage; taking the second training samples as inputs of the style encoding branch network, outputting style encoding sequences of the second training samples; taking the content encoding sequences and the style encoding sequences of the second training samples as inputs of the second decoder, outputting reconstructed data; calculating a loss value of the second training samples and the reconstructed data, obtaining a second loss value; taking the style encoding sequences of the second training samples as inputs of the third decoder, outputting second content information corresponding to the styles of the second training samples; calculating a loss value of the second content labels and the second content information, obtaining a third loss value; and adjusting parameters of the style encoding branch network to minimize the second loss value and maximize the third loss value.

[0020] In this implementation, in the training process of the style encoding branch network, the second loss value is used to ensure the accuracy of the style encoding branch network for the style encoding sequence, and the third loss value is used to ensure that the style encoding sequence and the content encoding sequence in the to-be-encoded data are far away from each other, that is, the independence of the style and the content in the encoding process in the to-be-encoded data is achieved, thereby achieving the purpose of "disentanglement". This is equivalent to independently separating the style encoding sequence from the content and the style encoding sequence in the to-be-encoded data. In the process of training the style encoding branch network, the content encoding branch network is in a state of being trained and having parameters frozen, which ensures that the content encoding branch network is not trained in the process of training the style encoding branch network, thereby enabling the content encoding branch network to only encode the content in the to-be-encoded data, and ensuring the function of the content encoding branch network to independently encode the content.

[0021] In a possible implementation, the training method of the style encoding branch network further comprises the following steps: obtaining style labels of the second training samples; taking the style encoding sequences of the second training samples as inputs of the fourth decoder, outputting style information of the second training samples; and calculating a loss value of the style labels and the style information, obtaining a fourth loss value. The step of adjusting the parameters of the style encoding branch network to minimize the second loss value and maximize the third loss value comprises the following step: adjusting the parameters of the style encoding branch network to minimize the second loss value, maximize the third loss value, and minimize the fourth loss value.

[0022] In this implementation, another training objective of the style encoding branch network is specifically defined during its training process: minimizing the loss value of style annotations and style information in the second training sample, thus ensuring the specificity and directionality of style information in the style encoding sequence. In other words, style-related information in the data to be encoded includes a lot of information, such as timbre and noise. However, not all style-related information is necessary or of interest to the user. By setting the label function of style annotations, the style information in the style encoding sequence can be made related to the style annotations, thus excluding information unrelated to the style annotations. This ensures that the style encoding sequence encoded by the trained style encoding branch network retains only the necessary style-related information, reducing or avoiding the mixing of irrelevant information and guaranteeing the specificity and directionality of style information in the style encoding sequence.

[0023] In one possible implementation, the style annotation of the second training sample is obtained based on the user's style annotation of the second training sample.

[0024] This implementation specifies the method for determining style annotations. Style-related information in the target data includes a wide range of data, such as timbre and noise levels. However, not all style-related information is of interest to the user. By allowing the user to set style annotations, the style information in the style encoding sequence can be aligned with the user's intended style. Figure 1 This means that information irrelevant to the user's intent is excluded, so that the style encoding sequence encoded by the trained second encoding unit retains only the style-related information that the user cares about, reducing or avoiding the mixing of irrelevant information and ensuring the specific orientation of style information in the style encoding sequence.

[0025] In one possible implementation, the second content annotation of the second training sample is obtained based on the user's annotation of the content of the second training sample.

[0026] This implementation specifically defines the method for determining the second content annotation. The content-related information in the data to be encoded includes a variety of information, such as text content and contextual information. However, not all content-related information is necessary or of concern to the user. By allowing the user to set content annotations, the content information in the content encoding sequence can be aligned with the user's intentions. Figure 1 This means that information irrelevant to the user's intent is excluded, so that the content encoding sequence of the trained content encoding branch network retains only the content-related information that the user cares about, reducing or avoiding the mixing of irrelevant information, ensuring the specific orientation of the content information in the content encoding sequence, and improving the user experience.

[0027] Secondly, embodiments of this application provide an encoding apparatus, comprising: an acquisition module for acquiring data to be encoded; a content encoding module for taking the data to be encoded as input to a content encoding branch network and outputting a content encoding sequence, the content encoding sequence including a first discrete encoding sequence obtained by encoding content information of the data to be encoded; and a style encoding module for taking the data to be encoded as input to a style encoding branch network and outputting a style encoding sequence, the style encoding sequence including a second discrete encoding sequence obtained by encoding style information of the data to be encoded.

[0028] In one possible implementation, the data to be encoded includes any one or more of audio data, image data, and video data.

[0029] In one possible implementation, the content encoding module is specifically used to: take the data to be encoded as input to the content encoder and output a content representation vector; take the content representation vector as input to the content quantizer and output a content encoded sequence.

[0030] In one possible implementation, the style encoding module is specifically used to: take the data to be encoded as input to the style encoder and output a style representation vector; take the style representation vector as input to the style quantizer and output a style encoding sequence.

[0031] In one possible implementation, the device further includes: a basic encoding module for taking the data to be encoded as input to a basic encoder and outputting a feature representation vector, which is used as input to a content encoding branch network and a style encoding branch network, respectively.

[0032] In one possible implementation, the device further includes: a content training module, used to train a content encoding branch network using the training method of the first training stage; the content training module is specifically used to: acquire a first training sample and a first content annotation corresponding to the first training sample; use the first training sample as input to the content encoding branch network to be trained, and output the content encoding sequence of the first training sample; use the content encoding sequence of the first training sample as input to a first decoder, and output the first content information corresponding to the first training sample; calculate the loss value of the first content information and the first content annotation to obtain a first loss value; and adjust the parameters of the content encoding branch network with the goal of minimizing the first loss value.

[0033] In one possible implementation, the first content annotation of the first training sample is obtained based on the user's annotation of the content of the first training sample.

[0034] In one possible implementation, the device further includes: a style training module, used to train a style encoding branch network using the training method of the second training phase. Specifically, the style training module is used to: acquire second training samples and second content annotations of the second training samples; use the second training samples as input to the content encoding branch network and output the content encoding sequence of the second training samples, wherein the content encoding branch network is in a state where it has been trained in the first training phase and its parameters are frozen; use the second training samples as input to the style encoding branch network and output the style encoding sequence of the second training samples; use the content encoding sequence and style encoding sequence of the second training samples as input to the second decoder and output reconstructed data; calculate the loss value of the second training samples and the reconstructed data to obtain a second loss value; use the style encoding sequence of the second training samples as input to the third decoder and output the second content information corresponding to the style of the second training samples; calculate the loss value of the second content annotation and the second content information to obtain a third loss value; and adjust the parameters of the style encoding branch network with the goal of minimizing the second loss value and maximizing the third loss value.

[0035] In one possible implementation, the style training module is further configured to: obtain style annotations for the second training sample; use the style encoding sequence of the second training sample as input to the fourth decoder and output style information of the second training sample; calculate the loss value of the style annotations and style information to obtain the fourth loss value; and adjust the parameters of the style encoding branch network with the goal of minimizing the second loss value, maximizing the third loss value, and minimizing the fourth loss value.

[0036] In one possible implementation, the style annotation of the second training sample is obtained based on the user's style annotation of the second training sample.

[0037] In one possible implementation, the second content annotation of the second training sample is obtained based on the user's annotation of the content of the second training sample.

[0038] Thirdly, embodiments of this application provide an encoding system, including: a content encoding branch network, used to take data to be encoded as input and output a content encoding sequence, the content encoding sequence including a first discrete encoding sequence obtained by encoding content information of the data to be encoded; and a style encoding branch network, used to take data to be encoded as input and output a style encoding sequence, the style encoding sequence including a second discrete encoding sequence obtained by encoding style information of the data to be encoded.

[0039] Fourthly, this application provides a chip system including a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, and the processor being used to execute the method described in any possible implementation of the first aspect and / or the algorithmic functions embodied in the apparatus described in any possible implementation of the second aspect.

[0040] Fifthly, this application provides a computing device including a processor and a memory. The processor is configured to execute instructions stored in the memory, causing the computing device to perform the methods described in any possible implementation of the first aspect and / or the algorithmic functions embodied in the apparatus described in any possible implementation of the second aspect.

[0041] In a sixth aspect, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the method described in any possible implementation of the first aspect and / or the algorithmic function embodied by the apparatus described in any possible implementation of the second aspect.

[0042] In a seventh aspect, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in any possible implementation of the first aspect and / or the algorithmic function embodied by the apparatus described in any possible implementation of the second aspect, wherein the cluster of computing devices includes at least one computing device.

[0043] Eighthly, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in any possible implementation of the first aspect and / or the algorithmic function embodied by the apparatus described in any possible implementation of the second aspect, wherein the cluster of computing devices includes at least one computing device.

[0044] It is understood that the beneficial effects of aspects two through eight above can be found in the relevant descriptions in aspect one above, and will not be repeated here. Attached Figure Description

[0045] The accompanying drawings used in the embodiments or technical description are briefly introduced below.

[0046] Figure 1 A schematic diagram illustrating an application scenario of an encoding system provided in an embodiment of this application;

[0047] Figure 2 This is a schematic diagram of the composition of an encoding system provided in an embodiment of this application;

[0048] Figure 3 A flowchart illustrating an encoding method provided in an embodiment of this application;

[0049] Figure 4A flowchart illustrating a training method for an encoding system provided in an embodiment of this application;

[0050] Figure 5 A schematic diagram illustrating the composition of an example encoding system provided in this application embodiment;

[0051] Figure 6 A schematic diagram illustrating the training process of the content encoding branch network provided in this application embodiment;

[0052] Figure 7 A schematic diagram illustrating the training process of the style coding branch network provided in an embodiment of this application;

[0053] Figure 8 This is a schematic diagram illustrating the composition of an encoding device provided in an embodiment of this application;

[0054] Figure 9 A schematic diagram illustrating the composition of an encoding architecture provided in an embodiment of this application;

[0055] Figure 10 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0056] Figure 11 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0057] Figure 12 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. Detailed Implementation

[0058] In this article, the term "and / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. The symbol " / " in this article indicates that the related objects have an "or" relationship; for example, A / B means A or B.

[0059] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0060] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0061] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0062] To facilitate understanding of the solutions provided in the embodiments of this application, a brief introduction to some of the terms involved in this solution will be given first.

[0063] Data: A general term for various types of data. In this application's embodiments, the data can be text, audio, video, images, etc. In this application, during the training phase, the data input into the encoding system is also referred to as the data to be encoded; during the inference phase, the data input into the encoding system is also referred to as the data to be encoded. The terms "data to be encoded" and "data to be encoded" are used only to distinguish the different phases they belong to, and do not represent differences in the data themselves. That is to say, "data to be encoded" and "data to be encoded" can be the same or different data.

[0064] Content refers to a portion of information within the data. This information is crucial for humans or machines to understand the core semantics conveyed by the data. For example, to represent the content of a speech, the speech can be transcribed into text. To represent the content of a video, the video can be transcribed into a script.

[0065] Style: A concept relative to content definition. It refers to the information remaining in data after removing the content information. For example, if a speech is transcribed into text to represent the content of the speech, then the style of the speech includes the speaker's timbre, the ambient noise, the speaker's emotions, rhythm, etc. The content of an image can be represented as a sentence describing the objects in the image; then the corresponding style of the image includes information such as its tone and texture. For example, tone is simply different ways of pronouncing a syllable, which can be considered a kind of style information. However, in Chinese, tone is used to represent semantics; different tones represent different meanings, such as pressure or stress. In English, tone indicates mood rather than meaning; the meaning remains the same regardless of the tone of the same word.

[0066] Language: In this article, language generally refers to natural language, that is, the language used by humans to communicate.

[0067] Large Language Models (LLMs) typically refer to neural network models containing an extremely large number of parameters (usually over a billion). They possess the following characteristics: First, they have a massive parameter scale, containing billions of parameters and reaching sizes of hundreds of gigabytes or even larger. This enormous scale provides them with powerful expressive and learning capabilities. Second, they can learn multiple tasks simultaneously. Large language models often learn various NLP (Neuro-Linguistic Programming) tasks together, such as machine translation, text summarization, and question-answering systems. Multi-task learning allows the model to learn broader and more generalized language understanding abilities. Third, they require powerful computing resources. Training large language models typically requires hundreds or even thousands of GPUs (graphics processing units) and a significant amount of time, usually ranging from weeks to months. Powerful computing resources can accelerate the training process while preserving the capabilities of the large language model. Fourth, they require abundant data. Large language models require vast amounts of data for training; only with a large amount of data can the advantages of their parameter scale be fully realized. Furthermore, large language models are widely used in the field of natural language processing (NLP) and are fundamentally changing the landscape of NLP tasks, giving rise to more powerful and intelligent language technologies. Large language models are one of the important directions in the development of AI (Artificial Intelligence). At the same time, large language models also possess the ability to perform exceptionally well in various NLP tasks, such as text classification, sentiment analysis, summarization, and translation. Large language models can be used in multiple application areas, including automatic writing, chatbots, virtual assistants, voice assistants, and automatic translation.

[0068] Multimodal Large Language Model (MLLM), also known as a multimodal large model or simply a multimodal model, refers to a model based on a large language model that can receive and reason about multimodal information. One application scenario for large language models is modeling natural language sequences. The input and output data of a large language model are sequences of discrete tokens, such as discrete tokens composed of words, characters, and letters. In other words, a large language model integrates data from other modalities to develop into a multimodal large language model. The discrete tokens of other modalities can be discrete speech tokens, discrete image tokens, etc.

[0069] Because large language models are text-based and unimodal, they can only generate text from input text and cannot understand other modalities commonly used in human society, such as images, videos, and audio. To enhance the capabilities of large language models, provide users with richer interaction methods, and further align the generated results of large language models with user input, multimodal large language models (MLLMs) have emerged. Leveraging the powerful understanding and reasoning capabilities of large language models, multimodal large language models can accept multiple modalities such as text, images, videos, and audio, perform unified understanding and reasoning on them, and generate reasonable responses accordingly.

[0070] One solution involves obtaining a sequence of discrete codes for multimodal data based on speech representation clustering methods. Further, the speech signal is first processed through a neural network coding system to extract speech feature vectors; this coding system is typically a pre-trained speech model (i.e., a model pre-trained with speech data). The speech feature vectors are then converted into category codes using a quantizer, which is usually a neural network based on a clustering algorithm (e.g., K-means). This method is an unsupervised training process, requiring only input speech data without needing labels, meaning no manual annotation of the speech data is required.

[0071] In this solution, the method obtains discrete representations of speech based on unsupervised learning clustering, which cannot completely decouple content and style information in speech. For example, the discrete speech representation obtained by K-means clustering has a weak ability to distinguish Chinese tones. For instance, tones are simply different ways of pronouncing a syllable and can be considered a form of style information. However, Chinese uses tones to represent semantics; different tones convey different meanings, such as "pressure" and "stress." In English, tones indicate mood rather than meaning; the meaning remains unchanged regardless of the tone of the same word. Therefore, there is no universally accepted objective standard for classifying style and content in speech, and thus it cannot be obtained through unsupervised methods.

[0072] In another solution, an encoder-decoder-based approach is used to obtain a discretely encoded sequence of multimodal data. Further, the speech or image data is processed by a neural network coding system to extract feature data; the extracted feature data is then quantized using a quantizer to obtain a discretely encoded sequence; based on the discretely encoded sequence output by the quantizer, the input speech or image data is reconstructed using a neural network decoder; the neural network coding system and the quantizer are trained with the goal of improving reconstruction quality. For example, the quantizer can be vector quantization (VQ), residual vector quantization (RVQ), or finite scalar quantization (FSQ).

[0073] In this solution, because the goal is to reconstruct multimodal data such as speech or images, the discrete encoding of multimodal data must simultaneously represent both content and style, posing challenges to downstream tasks such as content understanding and style transfer. Content understanding tasks typically do not concern themselves with style information within multimodal data, while style transfer tasks require the separation of style information before conversion can be performed. Therefore, the discrete encoding of multimodal data, simultaneously representing both content and style, creates difficulties for downstream tasks such as content understanding and style transfer.

[0074] In view of this, this application proposes another solution. In this application embodiment, multimodal data is encoded into discrete content-encoded sequences and style-encoded sequences through two parallel processing flows: a content encoding flow and a style encoding flow. The content encoding flow includes a content encoder and a content quantizer; the style encoding flow includes a style encoder and a style quantizer. Optionally, the content and style encoding flows can share a base encoder, which is used for initial encoding of the multimodal data.

[0075] The method for generating encoded sequences in this application is applicable to data of various modalities (such as speech, image, video, etc.), realizes the discretization representation of multimodal data, and can convert various multimodal data into discrete encoded sequences. When representing multimodal data with discrete encoded sequences, the content information and style information in the multimodal data are separated and represented by their respective discrete codes, which is convenient for the different needs of downstream applications of large multimodal models.

[0076] See Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of an encoding system provided in an embodiment of this application. For example... Figure 1As shown, the encoding system of this application embodiment can encode data of various modalities to obtain encoded sequences corresponding to the data of various modalities. The encoded sequences generated by the encoding system can be used for various downstream tasks. For example... Figure 1 As shown, one example of a downstream task is to use the encoded sequence 1 generated by the encoding system of this application as input to a multimodal model and output the inferred encoded sequence 2. Another example of a downstream task is to use the encoded sequence generated by the encoding system of this application as training samples to train or fine-tune a multimodal model.

[0077] Generally, a problem exists in the process of fusing multimodal input data into a multimodal model: the data formats of various modalities differ greatly. For example, speech, images, and videos are usually in the form of continuous signals, while language (text) is usually represented by discrete codes (characters, words, letters).

[0078] In this embodiment, converting data from various modalities into a discrete encoded sequence is a feasible scheme for constructing a multimodal large language model. Similar to the discrete encoded sequence of natural language text, the encoding system in this embodiment can obtain a discrete encoded sequence of multimodal data based on multimodal data. The discrete encoded sequence of multimodal data can be well recognized and learned by the multimodal model. Therefore, the discrete encoded sequence of multimodal data can be used for any downstream task based on the multimodal model. For example, a multimodal model can be trained or fine-tuned using the discrete encoded sequence of multimodal data; inference can be performed using the trained multimodal model using the discrete encoded sequence of multimodal data.

[0079] Furthermore, the encoding sequence involved in the embodiments of this application will be described by way of example. A segment of speech is represented at a frequency of 50 Hz as a series of discrete codes. One second of speech is converted into 50 discrete codes. Arranging these symbols in chronological order forms a sequence of discrete codes. If each 16*16 block in a 256*256 resolution image is represented as a discrete code, then the 256*256 resolution image is converted into 16*16 = 256 discrete codes. Arranging these discrete codes in raster order (e.g., from top left to bottom right) yields a discrete code sequence of length 256.

[0080] Based on such Figure 1 The application scenario of the coding system shown in this application embodiment provides a coding system including a content coding branch network and a style coding branch network. The training process and inference process of the coding system will be described below.

[0081] SeeFigure 2 , Figure 2 This is a schematic diagram illustrating the composition of an encoding system provided in an embodiment of this application. Figure 2 As shown, the encoding system includes: a content encoding branch network, which takes the data to be encoded as input to the content encoding branch network and outputs the content encoding sequence corresponding to the data to be encoded; and a style encoding branch network, which takes the data to be encoded as input to the style encoding branch network and outputs the style encoding sequence corresponding to the data to be encoded.

[0082] See Figure 3 , Figure 3 This is a flowchart illustrating an encoding method provided in an embodiment of this application. Figure 3 As shown in the illustration, this application provides an encoding method applied to an encoding system, which includes a content encoding branch network and a style encoding branch network. This encoding method can be understood as a reasoning method used by the encoding system during the inference phase.

[0083] Furthermore, Figure 3 The reasoning process of the encoding system of this application embodiment is described, in which multimodal signals are encoded into content encoding sequences and style encoding sequences through two encoding processes.

[0084] like Figure 3 As shown, this encoding method mainly includes the following steps:

[0085] Step S310: Obtain the data to be encoded.

[0086] Step S320: The data to be encoded is used as input to the content encoding branch network, and the content encoding sequence corresponding to the content information of the data to be encoded is output. The content encoding sequence is also called the content encoding sequence, content discrete encoding sequence, etc.

[0087] Step S330: The data to be encoded is used as the input to the style coding branch network, and the style coding sequence corresponding to the style information of the data to be encoded is output. The style coding sequence is also called the style coding sequence, style discrete coding sequence, etc.

[0088] In the technical solution of this application, the encoding system includes a content encoding branch network and a style encoding branch network. The content encoding branch network is used to encode the content of the data to be encoded, and the style encoding branch network is used to encode the style (information other than content) of the data to be encoded. In some technical solutions, the encoded sequence obtained by encoding the data to be encoded includes both content and style information. This can be understood as the content and style in the encoded sequence being an entangled whole, making it difficult for downstream tasks to decouple the content and style in the data to be encoded, thus making it difficult to complete the task independently from both content and style perspectives. Downstream tasks may include: converting the style of the data to be encoded; understanding the content of the data to be encoded through its style, etc. However, in the technical solution of this application, the content encoding branch network and the style encoding branch network of the encoding system can independently encode the content and style in the data to be encoded, thereby making the content and style in the data to be encoded mutually exclusive and separate. This can be understood as the content and style in the encoded sequence being two independent parts formed after "de-entanglement," allowing downstream tasks to use the decoupled encoding sequences of the content and style in the data to be encoded separately, thus completing the task independently from both content and style perspectives. Downstream tasks include: converting the style of the data to be encoded, which can be achieved by changing the encoding sequence corresponding to the style; and understanding the content of the data to be encoded based on its style. For example, in a telephone customer service scenario, the style encoding sequence is independent of the content, so we can determine whether the user is angry by analyzing only the style encoding sequence of the user's voice; the independent content encoding sequence does not contain style information, which facilitates downstream tasks related to content understanding.

[0089] Optionally, the data to be encoded includes any one or more of audio data, image data, and video data. The encoding system or method of this application can encode data of multiple modalities, and the encoded content encoding sequence and style encoding sequence can be used as input to downstream tasks (such as multimodal large language models). This implementation provides examples of data of multiple modalities, but is not limited to the examples given.

[0090] Optionally, the above-mentioned method of using the data to be encoded as the input of the content encoding branch network and outputting a content encoding sequence includes: using the data to be encoded as the input of the content encoder and outputting a content representation vector; and using the content representation vector as the input of the content quantizer and outputting a content encoding sequence.

[0091] In other words, the content coding branch network includes a content encoder and a content quantizer. The content encoder encodes the data to be encoded, obtaining a content representation vector; the content quantizer performs quantization processing on the content representation vector, such as clustering, to obtain a discrete content coding sequence. Generally, when the data to be encoded is speech, image, or video, the content coding sequence generated by the content encoder is often continuous. For example, multiple frames of speech data correspond to one code. Therefore, discretization processing is required, meaning that a content quantizer is necessary.

[0092] Optionally, the above-mentioned method of using the data to be encoded as the input of the style coding branch network and outputting a style coding sequence includes: using the data to be encoded as the input of the style encoder and outputting a style representation vector; and using the style representation vector as the input of the style quantizer and outputting a style coding sequence.

[0093] In other words, the style coding branch network includes a style encoder and a style quantizer. The style encoder encodes the data to be encoded, obtaining a style representation vector; the style quantizer performs quantization processing on the style representation vector, such as clustering, to obtain a discrete style coding sequence. Generally, when the data to be encoded is speech, image, or video, the style coding sequence generated by the style encoder is often continuous. For example, multiple frames of speech data correspond to one code. Therefore, discretization processing is required, meaning that a style quantizer is necessary.

[0094] Optionally, before inputting the data to be encoded into the content encoding branch network and the style encoding branch network respectively, the method further includes: using the data to be encoded as the input of the basic encoder and outputting a feature representation vector, which is used as the input of the content encoding branch network and the style encoding branch network respectively.

[0095] In other words, a specific method for preprocessing the data to be encoded is defined. Before the content coding branch network and / or style coding branch network encode the data, the base encoder preprocesses the raw data to be encoded. The raw data to be encoded refers to the data in its initial state. The input to the base encoder is the raw data to be encoded, and the output is the preprocessed data to be encoded, which reduces the workload of the content coding branch network and style coding branch network, thereby improving their efficiency. In other implementations, the raw data to be encoded can also be directly input into the content coding branch network and style coding branch network, which then perform the preprocessing and encoding themselves.

[0096] In order to achieve such Figure 2 The coding system shown and as Figure 3The encoding method shown in this application embodiment provides a training method for an encoding system. This training method can be understood as a training method for the encoding system during the training phase.

[0097] See Figure 4 , Figure 4 This is a flowchart illustrating a training method for an encoding system provided in an embodiment of this application. Figure 4 As shown in the embodiment of this application, a training method for an encoding system mainly includes the following steps:

[0098] Step S410: Obtain the data to be encoded and the corresponding content annotations.

[0099] In one possible implementation, content annotation is obtained based on user-annotated content of the data to be encoded. Content-related information in the data to be encoded includes a variety of information, such as text content and contextual information. However, not all content-related information is necessary or of interest to the user. By allowing users to set content annotations, the content information in the content encoding sequence can be aligned with the user's intentions. Figure 1 This means excluding information irrelevant to the user's intent, thus improving the user experience.

[0100] In one possible implementation, style annotations also need to be obtained, based on the style annotations of the data to be encoded by the user. Style-related information in the data to be encoded includes a lot of information, such as timbre and noise. However, not all style-related information is of interest to the user. By allowing the user to set style annotations, the style information in the style encoding sequence can be aligned with the user's intentions. Figure 1 This means excluding information irrelevant to the user's intent, thus improving the user experience.

[0101] In other words, in this embodiment of the application, all information contained in a certain multimodal raw data to be encoded is divided into two parts: content and style. How to divide it is determined by the user. For example, the user can provide content-annotated data to specify which information is classified as content, and the information other than content is classified as style. Alternatively, the user can provide both content-annotated and style-annotated data to specify which information is classified as content and which information is classified as style.

[0102] For example, in a speech dataset, someone says the sentence, "A small apple is not a big pear." If we assign a content label to this text: "A small apple is not a big pear," and the label is in text format, then the information beyond the information carried by the text, including the speaker's gender, tone, volume, speech rate, background noise, etc., in the speech data, constitutes the style.

[0103] For example, in an image dataset, there is a black cat sitting on a white sofa. If a user provides a text annotation for this image dataset: "A black cat is sitting on a white sofa," then the information contained in this text within the image dataset constitutes the content of the image dataset. Other information in the image dataset, such as the cat's breed, eye color, fur length, sofa type, sofa upholstery material, and the surrounding room environment, constitutes the style information. Alternatively, the user can provide another text annotation for the image dataset, in script form. This script generates a black and white image showing the outlines of the cat and sofa. The information represented by this outline image includes the relative positions and sizes of the cat and sofa, which are part of the content of the image dataset. Other information in the image dataset constitutes the style, which can include the cat's fur color, eye color, sofa color, sofa upholstery material, and the environment in which the sofa is located.

[0104] Step S420: Using the data to be encoded as training samples and the content annotations as labels for the training samples, train the content encoding branch network. This is also known as the first training phase.

[0105] In one possible implementation, the content encoding branch network is trained using the data to be encoded as training samples and content annotations as labels. This includes: inputting the data to be encoded into the content encoding branch network and outputting a content encoding sequence, which represents the content of the data to be encoded; decoding the content encoding sequence to obtain content prediction information; and training the content encoding branch network with the objective of minimizing the loss between the content prediction information and the content annotations. In other words, by performing encoding and decoding once and training the encoding and decoding process using content annotations as labels, the encoding and decoding process is made to target the content of the data to be encoded. This ensures that the trained content encoding branch network only encodes the content of the data to be encoded, thereby independently separating the content encoding sequence from the state of content and style-related "entanglement," achieving the goal of content "de-entanglement."

[0106] In one possible implementation, decoding the content-encoded sequence to obtain content prediction information includes: using the content-encoded sequence as input to a first decoder and outputting content prediction information; training the content-encoded branch network includes: adjusting the parameters of the content-encoded branch network and the first decoder. Therefore, during the training of the content-encoded branch network, the first decoder is also trained, thereby optimizing its decoding function and making its decoding function more closely match the encoding function of the content-encoded branch network. In other implementations, this decoding work can also be performed by a decoder that has been trained and whose parameters are frozen.

[0107] Step S430: Using the data to be encoded as training samples, train the style encoding branch network with the goal of minimizing the second loss value and maximizing the third loss value. This is also known as the second training phase.

[0108] Specifically, the reconstructed data is obtained based on the data to be encoded and decoded. The encoding process in the encoding and decoding is implemented through the encoding system. The loss value of the reconstructed data and the data to be encoded is the second loss value. When the input of the style coding branch network is the data to be encoded, the output is a style coding sequence. The content information predicted based on the style coding sequence is obtained by decoding the style coding sequence. The loss value of the content information predicted based on the style coding sequence and the content annotation is the third loss value.

[0109] It should be noted that the data to be encoded in the first training phase is also referred to as the first training sample, and the data to be encoded in the second training phase is also referred to as the second training sample. Since the first training phase is used to train the content encoding branch network, and the second training phase is used to train the style encoding process, the first and second training phases can be asynchronous. For example, in one possible implementation, during the training of the style encoding branch network, the content encoding branch network is in a state where training is complete and parameters are frozen. Therefore, the first and second training samples can be the same data to be encoded, or they can be different data. That is, the same training sample set can be used to train both the content encoding and style encoding branch networks, or two separate training sample sets can be used to train the content encoding and style encoding branch networks respectively; this application does not impose any restrictions on this.

[0110] In one possible implementation, the calculation of the second loss value includes: taking the data to be encoded as input to the content encoding branch network and outputting a content encoding sequence, which is an encoding sequence representing the content of the data to be encoded; taking the data to be encoded as input to the style encoding branch network and outputting a style encoding sequence, which is an encoding sequence representing the style of the data to be encoded; decoding the content encoding sequence and the style encoding sequence to obtain reconstructed data; and using the loss value of the data to be encoded and the reconstructed data as the second loss value. Theoretically, combining content and style can reflect the complete information of the data to be encoded. Therefore, encoding the content encoding sequence and the style encoding sequence yields the reconstructed data. The reconstructed data can also be understood as the predicted value of the data to be encoded. The loss value of the data to be encoded and the reconstructed data is used as the second loss value. Therefore, the second loss value reflects whether the style encoding sequence encoded by the style encoding branch network contains style-related information, ensuring that the style encoding sequence encoded by the trained style encoding branch network contains style-related information.

[0111] In one possible implementation, decoding the content-encoded sequence and style-encoded sequence to obtain reconstructed data includes: using the content-encoded sequence and style-encoded sequence as input to a second decoder and outputting the reconstructed data; the training process of the style-encoded branch network includes: adjusting the parameters of the style-encoded branch network and the second decoder with the objective of minimizing the second loss value. During the training of the style-encoded branch network, the second decoder is also trained to optimize its decoding function, making its decoding function more closely match the encoding function of the style-encoded branch network. In other implementations, this decoding work can also be performed by a decoder that has been trained and whose parameters are frozen.

[0112] For example, this application provides several possible examples of how to calculate the loss value of the reconstructed data and the data to be encoded. However, these examples do not constitute a limitation on the calculation of the loss value. Other methods besides those provided in this application can also be used to calculate the loss value.

[0113] The first example involves calculating the distance between the reconstructed data and the original data to be encoded using an algorithm, and then obtaining the loss value based on this distance. For instance, when the data to be encoded is speech data, the values ​​of each sampling point can be compared; when the data to be encoded is image data, the RGB (Red, Green, Blue) values ​​of each pixel can be compared. The loss value obtained based on the distance can be implemented using a specific evaluation metric, such as PESQ (Perceptual Evaluation of Speech Quality) or SDR (Source to Distortion Ratio).

[0114] The second example uses a trained feature extractor to extract features from the original data to be encoded and the reconstructed data, and calculates the distance between the extracted features.

[0115] The third example uses the idea of ​​generative adversarial networks to train a discriminator to determine whether the current data is reconstructed data or the original data to be encoded, thereby judging the degree of distance between the reconstructed data and the data to be encoded.

[0116] In one possible implementation, the calculation of the third loss value includes: taking the data to be encoded as input to the style encoding branch network and outputting a style encoding sequence, which is used to characterize the style of the data to be encoded; decoding the style encoding sequence to obtain the content information predicted based on the style encoding sequence; and using the loss value of the content information predicted based on the style encoding sequence and the content annotation as the third loss value. Since the second loss value reflects whether the style encoding sequence after encoding by the style encoding branch network contains style-related information, it ensures that the style encoding sequence after encoding by the trained style encoding branch network contains style-related information. However, whether the style encoding sequence contains content-related information cannot be guaranteed. Therefore, during the training of the style encoding branch network, it is also necessary to maximize the third loss value as the training objective. The third loss value reflects whether the style encoding sequence after encoding by the style encoding branch network contains content-related information. Maximizing the third loss value as the training objective ensures that the style encoding sequence after encoding by the trained style encoding branch network does not contain content-related information. That is, the style encoding branch network only encodes the style in the data to be encoded, thereby achieving the goal of independently separating the style encoding sequence from the state of "entanglement" between content and style encoding sequences, and achieving the purpose of "de-entanglement" between content and style.

[0117] In one possible implementation, decoding the style-coded sequence to obtain content information predicted based on the style-coded sequence includes: using the style-coded sequence as input to a third decoder and outputting content information predicted based on the style-coded sequence; the training process of the style-coding branch network includes: adjusting the parameters of the style-coding branch network and the third decoder with the objective of maximizing the third loss value. During the training of the style-coding branch network, the third decoder is also trained to optimize its decoding function, making its decoding function more closely match the encoding function of the style-coding branch network. In other implementations, this decoding work can also be performed by a decoder that has been trained and whose parameters are frozen.

[0118] To further understand this, the purpose of the third decoder is to deconstruct the entanglement between the content and style of the data to be encoded. Specifically, it takes a style-encoded sequence as input and uses the third decoder to predict the corresponding content information based on that sequence. In other words, it decodes the content information from the style-encoded sequence. The difference between the predicted content and the content annotation is used as the third loss value. The training objective here is to maximize the error rate of the prediction (i.e., maximize the third loss value), meaning the worse the third decoder's prediction, the better. A worse prediction indicates that the style-encoded sequence does not contain the content-related information corresponding to the content annotation, resulting in better "de-entanglement" of content and style. For example, the style of speech might include timbre, speech rate, etc. Using this information as input to the third decoder, it should predict the content being spoken. Ideally, the third decoder should not predict the corresponding content information.

[0119] In one possible implementation, the training objective of the style encoding branch network also includes minimizing the fourth loss value. The fourth loss value is the loss between style information and style labels. Style information is obtained by decoding the style encoding sequence, and style labels are the style annotations of the data to be encoded. The second loss value ensures the accuracy and completeness of style-related information in the style encoding sequence; the third loss value ensures the independence of style information in the style encoding sequence, i.e., "de-entanglement"; and the fourth loss value ensures the specificity and directionality of style information in the style encoding sequence. In other words, style-related information in the data to be encoded includes a lot of information, such as timbre and noise. However, not all style-related information is necessary or of interest to the user. By setting the label function of style labels, the style information in the style encoding sequence can be correlated with the style labels, i.e., information unrelated to the style labels is excluded. This ensures that the style encoding sequence encoded by the trained style encoding branch network retains only the necessary style-related information, reducing or avoiding the mixing of irrelevant information and ensuring the specificity and directionality of style information in the style encoding sequence.

[0120] In one possible implementation, the calculation of the fourth loss value includes: taking the data to be encoded as input to the style encoding branch network, outputting a style encoding sequence, which is used to characterize the style of the data to be encoded; decoding the style encoding sequence to obtain style information; and using the loss value of the style information and style annotation as the fourth loss value. Minimizing the fourth loss value is the training objective, ensuring that the style encoding sequence after the trained style encoding branch network encodes only contains necessary style information (style annotation-related style information) and does not contain irrelevant style information, thus guaranteeing the specific directionality of the style information in the style encoding sequence.

[0121] In one possible implementation, decoding the style-encoded sequence to obtain style information includes: using the style-encoded sequence as input to a fourth decoder and outputting style information; the training process of the style-encoding branch network includes: adjusting the parameters of the style-encoding branch network and the fourth decoder with the objective of minimizing the fourth loss. During the training of the style-encoding branch network, the fourth decoder is also trained to optimize its decoding function, making its decoding function more closely match the encoding function of the style-encoding branch network. In other implementations, this decoding work can also be performed by a decoder that has been trained and whose parameters are frozen.

[0122] In one possible implementation, during the training of the style encoding branch network, the content encoding branch network is in a state where training is complete and parameters are frozen. This ensures that the content encoding branch network is not trained during the training of the style encoding branch network, thus allowing the content encoding branch network to encode only the content in the data to be encoded, guaranteeing its function of independently encoding content.

[0123] In this embodiment of the application, the encoding system includes a content encoding branch network and a style encoding branch network. The content encoding branch network is used to encode the content of the data to be encoded, and the style encoding branch network is used to encode the style (information other than content) of the data to be encoded.

[0124] In some technical solutions, the encoded sequence obtained by encoding the data to be encoded includes both content and style information. This can be understood as the content and style in the encoded sequence being an entangled whole, making it difficult for downstream tasks to decouple the content and style of the data to be encoded, thus hindering the independent completion of tasks from both content and style perspectives. Downstream tasks may include: converting the style of the data to be encoded; understanding the content of the data to be encoded through its style, etc. Unlike these technical solutions, in the embodiments of this application, the content encoding branch network and style encoding branch network of the encoding system can independently encode the content and style of the data to be encoded, thereby making the content and style of the data to be encoded mutually exclusive and separate. This can be understood as the content and style in the encoded sequence being two independent parts formed after "de-entanglement," allowing downstream tasks to use the decoupled encoded sequences of the content and style of the data to be encoded separately, thus completing tasks independently from both content and style perspectives. Downstream tasks include: converting the style of the data to be encoded, which can be achieved by changing the encoding sequence corresponding to the style; understanding the content of the data to be encoded through its style. For example, in speech, when words like "pressure" and "stress" appear, they cannot be analyzed from a content perspective alone. The tone information in the style can be used to help understand the content of the data to be encoded.

[0125] Furthermore, training the content encoding branch network with content annotations as labels during training can make the content encoding sequence more accurate, ensuring that the content encoding branch network only encodes the content in the data to be encoded. This is equivalent to independently separating the content encoding sequence from the "entangled" state of the encoding sequences of content and style in the data to be encoded.

[0126] Furthermore, during the training of the style encoding branch network, a second loss value ensures the accuracy of the style encoding sequence; a third loss value ensures that the style encoding sequence and the content encoding sequence in the data to be encoded are far apart, thus achieving the independence of style and content in the encoding process, thereby achieving the goal of "de-entanglement". This is equivalent to independently separating the style encoding sequence from the "entangled" state of the content and style encoding sequences in the data to be encoded.

[0127] To further illustrate, as Figure 2 The encoding system shown, such as Figure 3 The encoding method shown and as Figure 4 The training method of the encoding system is shown below. Using speech data as an example, and combining it with a specific example of the encoding system, the following will further exemplify the encoding system, its encoding method, and its training method in the embodiments of this application. It should be further noted that the speech data is merely exemplary data and does not constitute a limitation on the input data format of the embodiments of this application. The speech data in the embodiments of this application can also be various multimodal data such as images and videos.

[0128] See Figure 5 , Figure 5 This is a schematic diagram illustrating the composition of an example encoding system provided in an embodiment of this application. For example... Figure 5 As shown, the encoding system includes two branches: a content encoding branch network and a style encoding branch network. The content encoding branch network includes a content encoder and a content quantizer; the style encoding branch network includes a style encoder and a style quantizer.

[0129] Furthermore, the content coding branch network includes a content encoder and a content quantizer. The content encoder encodes the data to be encoded, obtaining a content-coded sequence composed of content representation vectors. The content quantizer processes the content representation vectors to obtain a content-coded sequence in discrete representation form. In other words, the content encoder can encode content-related information or content information in the data to be encoded, obtaining multiple content representation vectors; the content quantizer can quantize the content representation vectors, such as through clustering, to obtain a discrete content-coded sequence. Optionally, when the data to be encoded is text, the content coding branch network may not include a content quantizer, because text data itself is discrete.

[0130] Furthermore, the style coding branch network includes a style encoder and a style quantizer. The style encoder encodes the data to be encoded, obtaining a style coding sequence in the form of multiple style representation vectors. The style quantizer processes the style representation vectors to obtain a discrete style coding sequence. In other words, the style encoder can encode style-related information or style data in the data to be encoded, obtaining multiple style representation vectors; the style quantizer can quantize the style representation vectors, such as through clustering, to obtain a discrete style coding sequence. Optionally, when the data to be encoded is text, the style coding branch network may not include a style quantizer, because text data itself is discrete.

[0131] The content quantizer and style quantizer, in addition to de-entanglement of content and style, respectively achieve discrete representations of content and style, meaning that the content encoding sequence and style encoding sequence are discrete sequences. These discretely represented content and style encoding sequences can be directly used and analyzed by downstream tasks, such as as input to multimodal large models or for style transfer.

[0132] Optionally, such as Figure 5 As shown, the encoding system also includes: a base encoder, used to preprocess the data to be encoded before the content encoding branch network and / or style encoding branch network encode the data to be encoded.

[0133] This implementation specifically defines a method for preprocessing the data to be encoded. Before the content encoding branch network and / or style encoding branch network encode the data, a base encoder is used to preprocess the data. The input of the base encoder is the raw data to be encoded, and the output is a feature representation of the raw data to be encoded. This reduces the workload of the content encoding branch network and style encoding branch network, thereby improving their efficiency.

[0134] In one possible implementation, the method further includes: training the base encoder during the training of the content encoding branch network; and during the training of the style encoding branch network, the base encoder is in a state where training is complete and parameters are frozen.

[0135] In this implementation, the base encoder can be trained during the training of the content encoding branch network. In other implementations, the base encoder can also be trained during the training of the style encoding branch network; or the base encoder can be trained separately using the data to be encoded as training samples, while the base encoder is in a state where training is complete and parameters are frozen during the training of the content encoding branch network and the style encoding branch network.

[0136] For example, the input to the basic encoder is the raw data to be encoded, and the output is the feature representation of that raw data. For instance, when the data to be encoded is speech data, the input to the basic encoder can be a wave file, and the output can be multiple feature vectors, each representing the features of a frame in the speech. For instance, when the data to be encoded is image data, the input to the basic encoder can be an image, and the output can be multiple feature vectors, each representing the features of a slice in the image.

[0137] In other words, the base encoder is optional. When the base encoder is present, decoupling the content from the entangled state of content and style can be understood as a joint effort between the base encoder and the content encoding branch network; similarly, decoupling the style from the entangled state of content and style can be understood as a joint effort between the base encoder and the style encoding branch network. When the base encoder is absent, the data preprocessing is essentially handled by the content encoding branch network and the style encoding branch network themselves. The content encoding branch network decouples the content from the entangled state of content and style, and the style encoding branch network decouples the style from the entangled state of content and style.

[0138] The following is combined Figure 6 and Figure 7 The first training stage of the content encoding branch network and the second training stage of the style encoding branch network in the training phase of this application embodiment will be described by way of example.

[0139] See Figure 6 , Figure 6 This is a schematic diagram illustrating the training process of the content encoding branch network provided in an embodiment of this application. For example... Figure 6 As shown, the training process of the content encoding branch network mainly includes: First, defining the content of the multimodal data, such as: information that can be transcribed into text in speech data is content, musical score information is content, video script is content, etc. Users can annotate the multimodal data with content, and use this content annotation as the label for training samples to train the content encoding branch network. Second, the multimodal data is sequentially input into an optional base encoder and content encoder, outputting a content representation vector; the content representation vector is then fed into a content quantizer to obtain a content encoding sequence; the content encoding sequence is then fed into the first decoder to output content prediction information. Finally, gradient values ​​are calculated based on the content prediction information and content annotations, and the content encoding branch network is trained using the backpropagation algorithm. The process of training the content encoding branch network is also the process of training the content encoder and content quantizer. Optionally, the first decoder is also trained. Optionally, the base encoder is also trained. The training objective is to minimize the error rate of the content prediction information and content annotations.

[0140] It should be noted that after the content encoding branch network is trained, the first decoder will be discarded, and the base encoder, content encoder, and content quantizer may be retained.

[0141] See Figure 7 , Figure 7 This is a schematic diagram illustrating the training process of the style coding branch network provided in an embodiment of this application. Figure 7 As shown, the training process of the style coding branch network mainly includes: training the style coding branch network with the objectives of minimizing the second loss value, maximizing the third loss value, and optionally minimizing the fourth loss value. In each iteration step, the parameters of the style coding branch network are simultaneously adjusted with the objectives of minimizing the second loss value, maximizing the third loss value, and optionally minimizing the fourth loss value. For ease of explanation, the training process for these three training objectives in one iteration step will be illustrated below.

[0142] Furthermore, the process of training the style encoding branch network with the objective of minimizing the second loss value mainly includes: First, based on the content annotation defined above, the style or part of the style of the multimodal data can be defined. For example, the style of speech data may include the speaker's timbre, the speaker's emotion, environmental noise, etc. In practical application scenarios, it is difficult to define and exhaust the exact range included by the style, but certain parts of the style can be defined. For example, the speaker's timbre is generally a known part of the style, so the speaker's timbre can be defined, and this definition is executed through style annotation, such as annotating the training speech data to mark the speaker of each speech segment. This is also the significance of style annotation, which only includes style information that users are interested in. Second, based on the trained content encoding branch network, the output of the optional base encoder is simultaneously input into the style encoder to obtain the style representation vector (example of style representation vector); the style representation vector is input into the style quantizer to obtain the style encoding sequence; the style encoding sequence and the corresponding content encoding sequence are simultaneously input into the second decoder to output the reconstructed multimodal data (i.e., reconstructed data). Finally, gradient values ​​are calculated based on the reconstructed data and the corresponding data to be encoded, and the style quantizer and style encoder are trained via backpropagation. Optionally, a second decoder is also trained. The training objective here is to minimize the error rate of the multimodal data reconstructed by the second decoder compared to the input multimodal data (i.e., the data to be encoded).

[0143] Furthermore, the process of training the style encoding branch network with the goal of maximizing the third loss mainly includes: simultaneously inputting the style encoding sequence into the third decoder, outputting content information predicted based on the style encoding sequence; calculating gradient values ​​based on the content information predicted based on the style encoding sequence and the aforementioned content annotations, and training the style quantizer and style encoding branch network through the backpropagation algorithm; optionally, also training the third decoder. The training objective here is to maximize the error rate between the content information predicted based on the style encoding sequence and the content annotations output by the third decoder. The optimization of the third loss value aims to prevent the third decoder from predicting relevant information about the content annotations from the style encoding sequence, thereby enabling the training of the style encoding branch network and style quantizer to filter out content-related information, thus achieving the goal of deentanglement between content information and style information.

[0144] Optionally, the process of training the style encoding branch network with the objective of minimizing the fourth loss mainly includes: simultaneously inputting the style encoding sequence into the fourth decoder to output corresponding style information; calculating the gradient value based on the style information and the aforementioned style annotations; and training the style quantizer and style encoding branch network through the backpropagation algorithm. Optionally, the fourth decoder is also trained. The training objective here is to minimize the error rate of comparing the style information output by the fourth decoder with the style annotations.

[0145] During the training of the style coding branch network, the parameters of the content quantizer, content coding branch network, and base encoder are frozen, meaning their parameters are not updated. Figure 7 The asterisk (*) pattern is shown in the image.

[0146] It should be further noted that the above-mentioned encoding system, encoding method, and training method are applicable to data of various modalities. For example, based on the above encoding system, speech data can generate a discrete encoded sequence including content encoded sequences and style encoded sequences at a certain frame rate. For example, image data can be divided into blocks, generating discrete content encoded sequences and style encoded sequences for each block of image data, and then arranged in raster order from top left to bottom right to obtain the discrete encoded sequence of the image. For video data, the video is represented as a series of video frames at a certain frame rate, with each frame being an image. Each frame image is represented as a content and style encoded sequence (example of content encoded sequences and style encoded sequences) according to the image processing method described above. Then, the encoded sequences of each frame are arranged in chronological order to obtain the encoded sequence of the video.

[0147] In this embodiment, the content encoder, style encoder, basic encoder, first decoder, second decoder, third decoder, and fourth decoder can all be implemented by various neural network models. In this embodiment, the content quantizer and style quantizer include at least the implementation methods of vector quantization (VQ) and finite scalar quantization (FSQ).

[0148] In some technical solutions, the method of representation clustering is a completely unsupervised approach, lacking a clear training objective and failing to guarantee that the obtained discrete sequences fully contain content information. However, in this application's embodiment, user-annotated content labels are introduced as supervisory signals to guide the learning of the content encoding branch network, enabling a more direct and accurate acquisition of the discrete representation content encoding sequence.

[0149] In some technical solutions, the method of calculating and reconstructing the loss value of the original data to be encoded based on encoding and decoding is also a completely unsupervised or self-supervised approach. This requires reconstructing multimodal data based on the discrete encoding of the original data to be encoded, making it impossible to extract the content signal separately for discretization. However, in the embodiments of this application, two independent encoding processes—a content encoding branch network and a style encoding branch network—are used to represent the multimodal data as two different discrete encoding sequences: content and style (i.e., content encoding sequence and style encoding sequence). Furthermore, manually annotated content and style labels can be introduced to guide the learning of the discrete encoding of the content encoding branch network and the style encoding branch network.

[0150] This application proposes a method for quantizing multimodal data into discrete coding sequences. This application employs two different discrete coding sequences to represent the content and style information of multimodal data. This application uses manually annotated content data as a supervisory signal to train the content coding branch network's discrete content coding generation process. This application allows users to define content information in multimodal data using content-annotated data. This application uses manually annotated content data and manually annotated style data to train the style coding branch network's discrete style coding sequence generation process. This application allows the use of style-annotated data to define style information in multimodal data. This application reconstructs the discrete coding sequences of content and style into multimodal input data, and uses the reconstructed data (i.e., self-supervised training), content annotation, and style annotation as three training objectives to train the content coding branch network and the style coding branch network's discrete content coding and style coding generation processes.

[0151] In this embodiment, the above technical solution can separate content information and style information in multimodal data, representing them as discrete encoded sequences of content encoding sequences and style encoding sequences, respectively. The separated discrete content encoding (i.e., content encoding sequence) is suitable for training large multimodal models in conjunction with text data. The separated discrete style encoding (i.e., style encoding sequence) can be applied in multimodal style transfer, multimodal expressiveness synthesis, and other fields. The downstream application scenarios of the discrete encoded sequences of content encoding sequences and style encoding sequences are wide-ranging, and their application effect is superior to multimodal data discretization algorithms directly based on encoders and decoders.

[0152] Next, based on the methods in the above embodiments, an encoding device provided in the embodiments of this application will be introduced.

[0153] See Figure 8 , Figure 8 This is a schematic diagram illustrating the composition of an encoding device provided in an embodiment of this application. Figure 8 As shown, this application embodiment provides an encoding device 800, which mainly includes:

[0154] The acquisition module 810 is used to acquire the data to be encoded.

[0155] The content encoding module 820 is used to take the data to be encoded as the input of the content encoding branch network and output a content encoding sequence. The content encoding sequence includes a first discrete encoding sequence obtained by encoding the content information of the data to be encoded.

[0156] The style coding module 830 is used to take the data to be encoded as the input of the style coding branch network and output a style coding sequence. The style coding sequence includes a second discrete coding sequence obtained by encoding the style information of the data to be encoded.

[0157] In one possible implementation, the data to be encoded includes any one or more of audio data, image data, and video data.

[0158] In one possible implementation, the content encoding module 820 is specifically used to: take the data to be encoded as input to the content encoder and output a content representation vector; take the content representation vector as input to the content quantizer and output a content encoding sequence.

[0159] In one possible implementation, the style encoding module 830 is specifically used to: take the data to be encoded as the input of the style encoder and output a style representation vector; take the style representation vector as the input of the style quantizer and output a style encoding sequence.

[0160] In one possible implementation, the device further includes: a basic encoding module for taking the data to be encoded as input to a basic encoder and outputting a feature representation vector, which is used as input to a content encoding branch network and a style encoding branch network, respectively.

[0161] In one possible implementation, the device further includes: a content training module, used to train a content encoding branch network using the training method of the first training stage; the content training module is specifically used to: acquire a first training sample and a first content annotation corresponding to the first training sample; use the first training sample as input to the content encoding branch network to be trained, and output the content encoding sequence of the first training sample; use the content encoding sequence of the first training sample as input to a first decoder, and output the first content information corresponding to the first training sample; calculate the loss value of the first content information and the first content annotation to obtain a first loss value; and adjust the parameters of the content encoding branch network with the goal of minimizing the first loss value.

[0162] In one possible implementation, the first content annotation of the first training sample is obtained based on the user's annotation of the content of the first training sample.

[0163] In one possible implementation, the device further includes: a style training module, used to train a style encoding branch network using the training method of the second training phase. Specifically, the style training module is used to: acquire second training samples and second content annotations of the second training samples; use the second training samples as input to the content encoding branch network and output the content encoding sequence of the second training samples, wherein the content encoding branch network is in a state where it has been trained in the first training phase and its parameters are frozen; use the second training samples as input to the style encoding branch network and output the style encoding sequence of the second training samples; use the content encoding sequence and style encoding sequence of the second training samples as input to the second decoder and output reconstructed data; calculate the loss value of the second training samples and the reconstructed data to obtain a second loss value; use the style encoding sequence of the second training samples as input to the third decoder and output the second content information corresponding to the style of the second training samples; calculate the loss value of the second content annotation and the second content information to obtain a third loss value; and adjust the parameters of the style encoding branch network with the goal of minimizing the second loss value and maximizing the third loss value.

[0164] In one possible implementation, the style training module is further configured to: obtain style annotations for the second training sample; use the style encoding sequence of the second training sample as input to the fourth decoder and output style information of the second training sample; calculate the loss value of the style annotations and style information to obtain the fourth loss value; and adjust the parameters of the style encoding branch network with the goal of minimizing the second loss value, maximizing the third loss value, and minimizing the fourth loss value.

[0165] In one possible implementation, the style annotation of the second training sample is obtained based on the user's style annotation of the second training sample.

[0166] In one possible implementation, the second content annotation of the second training sample is obtained based on the user's annotation of the content of the second training sample.

[0167] See Figure 9 , Figure 9 This is a schematic diagram illustrating the composition of an encoding architecture provided in an embodiment of this application. In this embodiment, an encoding architecture 900 deployed with this embodiment may include a terminal 910 and a server 920. The server 920 may include one or more servers (…). Figure 9 (The example includes a server) The server 920 can provide the methods or apparatus provided in the embodiments of this application to one or more terminals.

[0168] Optionally, a relevant application may be installed on the terminal 910. This application or webpage can provide an interface, allowing the terminal 910 to receive data to be encoded input by the user on the interface and send this data to the server 920. The server 920 is equipped with the encoding system of this embodiment and can encode the data to be encoded according to the above encoding method to obtain a content encoding sequence and a style encoding sequence. Optionally, the content encoding sequence and style encoding sequence can also be returned to the terminal 910.

[0169] Optionally, the terminal 910 is also used to obtain information such as content annotation and style annotation input by the user.

[0170] It should be understood that in some optional implementations, the terminal 910 may also deploy its own encoding system and complete the encoding work without the cooperation of the server 920, and this application embodiment is not limited in this regard. In some optional implementations, the server 920 may also base its encoding on multimodal data to be encoded received from other devices outside the terminal or locally stored, without the cooperation of the terminal 910, and this application embodiment is not limited in this regard.

[0171] The following description Figure 9The product form of the terminal 910. In this application embodiment, the terminal 910 can be a mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0172] The following description Figure 9 The product form of Server 920. It can be further understood that Server 920 can be various types of servers, such as x89 architecture servers, specifically rack servers, blade servers, high-density servers, platform servers, or high-performance servers, etc. In other words, this application embodiment does not specifically limit the specific type of server. Furthermore, it can be understood that... Figure 1 The server structure shown does not constitute a limitation on the server structure. A server may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0173] Furthermore, the server 920 can be configured as an independent physical server, or as a server cluster or distributed system consisting of multiple physical servers. It can also be configured as a cloud server or cloud server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The cloud server cluster is deployed in several cloud data centers. The software can be an application that implements object control methods, but it is not limited to the above forms.

[0174] Next, the communication connection method between terminal 910 and server 920 is described. For example, terminal 910 and server 920 are connected via a network, enabling terminal 910 to access the cloud management platform deployed on the cloud server cluster. The network can be a wired network or a wireless network. For example, a wired network can be a cable network, fiber optic network, Digital Data Network (DDN), etc., while a wireless network can be a telecommunications network, internal network, Internet, Local Area Network (LAN), Wide Area Network (WAN), Wireless Local Area Network (WLAN), Metropolitan Area Network (MAN), Public Service Telephone Network (PSTN), Bluetooth network, ZigBee network, Global System for Mobile Communications (GSM), CDMA (Code Division Multiple Access) network, CPRS (General Packet Radio Service) network, etc., or any combination thereof.

[0175] Understandably, a network can use any known network communication protocol to enable communication between different terminal layers and gateways. These network communication protocols can be various wired or wireless communication protocols, such as Ethernet, Universal Serial Bus (USB), FireWire, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), New Radio (NR), Bluetooth, Wireless Fidelity (Wi-Fi), and other communication protocols.

[0176] In one possible scenario, server 920 can function as a cloud platform (a software platform employing application virtualization technology, integrating multiple functions such as software search, download, use, management, and backup). In practical use, server 920 can deploy a cloud management platform and a data center, with terminal 910 interacting with the cloud through the cloud management platform. Additionally, the data center can deploy nodes, which can be virtual machine instances, container instances, physical servers, etc.

[0177] In another possible scenario, the method provided in this application embodiment can be implemented by software. The software has a terminal and a server; the terminal 910 runs the software, and the server 920 runs the software's server. While the terminal 910 is running the software, it can call the server running on the server 920 to implement the method provided in this application embodiment.

[0178] In other words, the method provided in this application embodiment can be applied to the terminal 910 or the server 920. In specific implementation, it can run as software on the terminal 910 or the server 920; for example, the software can be a service or an application. This application embodiment can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application embodiment can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0179] This application also provides a computing device 1000. For example... Figure 10 As shown, the computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, memory 1006, and communication interface 1008 communicate with each other via the bus 1002. The computing device 1000 can be a server, such as a central server, an edge server, or a local server in a local data center, or it can be an electronic device such as a desktop computer, laptop computer, or smartphone. It should be understood that this application does not limit the number of processors and memories in the computing device 1000.

[0180] Bus 1002 can be a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The Unified Bus is also known as the Lingqu Bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 10 The bus 1004 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1004 may include a path for transmitting information between various components of the computing device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).

[0181] The processor 1004 may include any one or more of the following computing devices: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, SoC, offload card, accelerator card, etc.

[0182] The memory 1006 may include volatile memory, such as random access memory (RAM). The processor 1004 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, the memory 1006 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0183] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 1006, or two or more types of storage media can be configured to realize the function of memory 1006. This application does not limit this.

[0184] The memory 1006 stores executable program code, and the processor 1004 executes the executable program code to implement the functions of the encoding device 800 described above, thereby implementing the method described in the above embodiments. That is, the memory 1006 stores instructions for executing the method described in the above embodiments.

[0185] Alternatively, the memory 1006 may store executable code, which the processor 1004 executes to implement the functions of the encoding device 800 described above, thereby implementing the method described in the above embodiments. That is, the memory 1006 stores instructions for executing the method described in the above embodiments.

[0186] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.

[0187] As one possible implementation, the computing device 1000 may also include a chip system, which includes a processor and a power supply circuit. The power supply circuit supplies power to the processor, and the processor executes the operation steps corresponding to the methods in the embodiments of this application. For simplicity, further details are omitted here. The processor can be implemented using a GPU, or it can be implemented using computing devices or AI chips such as a DPU, NPU, XPU, SoC, offloading card, or accelerator card.

[0188] As one possible implementation, the computing device 1000 may include various types of processors 1004, that is, the computing device 1000 is a heterogeneous device. For example, the computing device 1000 includes a CPU and a GPU, and at least one of the processors 1004 can execute the operation steps corresponding to the method of the embodiments of this application. For the sake of brevity, further details are omitted here.

[0189] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be an electronic device such as a desktop computer, a laptop computer, or a smartphone.

[0190] like Figure 11As shown, the computing device cluster includes at least one computing device 1000. The memory 1006 of one or more computing devices 1000 in the computing device cluster may store the same instructions for performing the methods described in the above embodiments.

[0191] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the methods described in the above embodiments. In other words, a combination of one or more computing devices 1000 can jointly execute instructions for executing the methods described in the above embodiments.

[0192] It should be noted that the memory 1006 in different computing devices 1000 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the aforementioned encoding device 800. That is, the instructions stored in the memory 1006 of different computing devices 1000 can implement the functions of one or more modules of the aforementioned encoding device 800.

[0193] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 12 One possible implementation is shown. For example... Figure 12 As shown, the two computing devices 1000A and 1000B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1006 in computing device 1000A stores instructions for the functions of one or more modules of the aforementioned encoding device 800. Simultaneously, the memory 1006 in computing device 1000B stores instructions for the functions of another one or more modules of the aforementioned encoding device 800.

[0194] It should be understood that Figure 12 The functions of computing device 1000A shown can also be performed by multiple computing devices 1000. Similarly, the functions of computing device 1000B can also be performed by multiple computing devices 1000.

[0195] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 11 and Figure 12 The connection method of the computing device cluster is different in that the memory 1006 of one or more computing devices 1000 in the computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0196] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the aforementioned data processing method. In other words, a combination of one or more computing devices 1000 can jointly execute the instructions for performing the aforementioned method.

[0197] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When executed by a cluster of computing devices including at least one computing device, the computer program instructions cause the cluster of computing devices to perform the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium capable of being stored in the cluster of computing devices or a data storage device such as a data center containing one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives).

[0198] Based on the methods in the above embodiments, this application provides a computer program product containing instructions. The computer program product may be software or program products containing instructions, capable of running on a computing device or stored on any available medium. When the instructions are executed by a cluster of computing devices containing at least one computing device, at least one computing device in the cluster of computing devices performs the methods in the above embodiments.

[0199] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0200] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0201] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0202] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0203] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. An encoding method, characterized in that, include: Obtain the data to be encoded; The data to be encoded is used as the input to the content coding branch network, and the output is a content coding sequence, which includes a first discrete coding sequence obtained by encoding the content information of the data to be encoded. The data to be encoded is used as input to a style coding branch network, and a style coding sequence is output. The style coding sequence includes a second discrete coding sequence obtained by encoding the style information of the data to be encoded.

2. The method according to claim 1, characterized in that, The data to be encoded includes any one or more of audio data, image data, and video data.

3. The method according to claim 1 or 2, characterized in that, The step of using the data to be encoded as input to the content encoding branch network and outputting a content encoding sequence includes: The data to be encoded is used as input to the content encoder, and the output is a content representation vector; The content representation vector is used as the input to the content quantizer, and the output is a content encoding sequence.

4. The method according to any one of claims 1-3, characterized in that, The step of using the data to be encoded as input to the style coding branch network and outputting a style coding sequence includes: The data to be encoded is used as input to the style encoder, and the output is a style representation vector; The style representation vector is used as the input to the style quantizer, and the output is a style encoding sequence.

5. The method according to any one of claims 1-4, characterized in that, Before inputting the data to be encoded into the content encoding branch network and the style encoding branch network respectively, the process further includes: The data to be encoded is used as input to the basic encoder, and a feature representation vector is output. The feature representation vector is used as input to the content encoding branch network and the style encoding branch network, respectively.

6. The method according to any one of claims 1-5, characterized in that, The content encoding branch network is trained using the training method of the first training phase, which includes: Obtain the first training sample and the first content annotation corresponding to the first training sample; The first training sample is used as the input to the content encoding branch network to be trained, and the content encoding sequence of the first training sample is output. The content encoding sequence of the first training sample is used as the input of the first decoder, and the first content information corresponding to the first training sample is output. Calculate the loss value of the first content information and the first content annotation to obtain the first loss value; The parameters of the content encoding branch network are adjusted with the goal of minimizing the first loss value.

7. The method according to claim 6, characterized in that, The first content annotation of the first training sample is obtained based on the annotation of the content of the first training sample by the user.

8. The method according to claim 6 or 7, characterized in that, The style coding branch network is trained using a second training phase training method, which includes: Obtain the second training sample and the second content annotation of the second training sample; The second training sample is used as the input of the content coding branch network, and the content coding sequence of the second training sample is output. The content coding branch network is in a state where it has been trained in the first training stage and the parameters are frozen. The second training sample is used as the input to the style coding branch network, and the style coding sequence of the second training sample is output. The content encoding sequence and style encoding sequence of the second training sample are used as input to the second decoder, and the output is reconstructed data; Calculate the loss value of the second training sample and the reconstructed data to obtain the second loss value; The style encoding sequence of the second training sample is used as the input of the third decoder, and the second content information corresponding to the style of the second training sample is output. Calculate the loss value of the second content annotation and the second content information to obtain the third loss value; The parameters of the style coding branch network are adjusted with the goal of minimizing the second loss value and maximizing the third loss value.

9. The method according to claim 8, characterized in that, The training method for the style coding branch network also includes: Obtain the style annotations for the second training sample; The style encoding sequence of the second training sample is used as the input of the fourth decoder to output the style information of the second training sample. Calculate the loss value of the style annotation and the style information to obtain the fourth loss value; The step of adjusting the parameters of the style coding branch network with the goal of minimizing the second loss value and maximizing the third loss value includes: The parameters of the style coding branch network are adjusted with the goal of minimizing the second loss value, maximizing the third loss value, and minimizing the fourth loss value.

10. The method according to claim 9, characterized in that, The style annotation of the second training sample is obtained based on the style annotation of the second training sample by the user.

11. The method according to any one of claims 8-10, characterized in that, The second content annotation of the second training sample is obtained based on the annotation of the content of the second training sample by the user.

12. An encoding device, characterized in that, include: The acquisition module is used to acquire the data to be encoded. The content encoding module is used to take the data to be encoded as the input of the content encoding branch network and output a content encoding sequence, wherein the content encoding sequence includes a first discrete encoding sequence obtained by encoding the content information of the data to be encoded; The style coding module is used to take the data to be encoded as the input of the style coding branch network and output a style coding sequence, wherein the style coding sequence includes a second discrete coding sequence obtained by encoding the style information of the data to be encoded.

13. The apparatus according to claim 12, characterized in that, The data to be encoded includes any one or more of audio data, image data, and video data.

14. The apparatus according to claim 12 or 13, characterized in that, The content encoding module is specifically used for: The data to be encoded is used as input to the content encoder, and the output is a content representation vector; The content representation vector is used as the input to the content quantizer, and the output is a content encoding sequence.

15. The apparatus according to any one of claims 12-14, characterized in that, The style encoding module is specifically used for: The data to be encoded is used as input to the style encoder, and the output is a style representation vector; The style representation vector is used as the input to the style quantizer, and the output is a style encoding sequence.

16. The apparatus according to any one of claims 12-15, characterized in that, Also includes: The basic encoding module is used to take the data to be encoded as the input of the basic encoder and output a feature representation vector, which is used as the input of the content encoding branch network and the style encoding branch network, respectively.

17. The apparatus according to any one of claims 12-16, characterized in that, The apparatus further includes: a content training module, used to train the content encoding branch network using the training method of the first training stage; the content training module is specifically used for: Obtain the first training sample and the first content annotation corresponding to the first training sample; The first training sample is used as the input to the content encoding branch network to be trained, and the content encoding sequence of the first training sample is output. The content encoding sequence of the first training sample is used as the input of the first decoder, and the first content information corresponding to the first training sample is output. Calculate the loss value of the first content information and the first content annotation to obtain the first loss value; The parameters of the content encoding branch network are adjusted with the goal of minimizing the first loss value.

18. The apparatus according to claim 17, characterized in that, The first content annotation of the first training sample is obtained based on the annotation of the content of the first training sample by the user.

19. The apparatus according to claim 17 or 18, characterized in that, The device further includes: a style training module, used to train the style coding branch network using the training method of the second training stage, the style training module specifically used for: Obtain the second training sample and the second content annotation of the second training sample; The second training sample is used as the input of the content coding branch network, and the content coding sequence of the second training sample is output. The content coding branch network is in a state where it has been trained in the first training stage and the parameters are frozen. The second training sample is used as the input to the style coding branch network, and the style coding sequence of the second training sample is output. The content encoding sequence and style encoding sequence of the second training sample are used as input to the second decoder, and the output is reconstructed data; Calculate the loss value of the second training sample and the reconstructed data to obtain the second loss value; The style encoding sequence of the second training sample is used as the input of the third decoder, and the second content information corresponding to the style of the second training sample is output. Calculate the loss value of the second content annotation and the second content information to obtain the third loss value; The parameters of the style coding branch network are adjusted with the goal of minimizing the second loss value and maximizing the third loss value.

20. The apparatus according to claim 19, characterized in that, The style training module is also used for: Obtain the style annotations for the second training sample; The style encoding sequence of the second training sample is used as the input of the fourth decoder to output the style information of the second training sample. Calculate the loss value of the style annotation and the style information to obtain the fourth loss value; The parameters of the style coding branch network are adjusted with the goal of minimizing the second loss value, maximizing the third loss value, and minimizing the fourth loss value.

21. The apparatus according to claim 20, characterized in that, The style annotation of the second training sample is obtained based on the style annotation of the second training sample by the user.

22. The apparatus according to any one of claims 19-21, characterized in that, The second content annotation of the second training sample is obtained based on the annotation of the content of the second training sample by the user.

23. An encoding system, characterized in that, include: A content coding branch network is used to take data to be encoded as input and output a content coding sequence, wherein the content coding sequence includes a first discrete coding sequence obtained by encoding the content information of the data to be encoded. A style coding branch network is used to take the data to be encoded as input and output a style coding sequence, wherein the style coding sequence includes a second discrete coding sequence obtained by encoding the style information of the data to be encoded.

24. A chip system, characterized in that, The chip system includes a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, the processor being used to execute the algorithmic functions embodied in the method as described in any one of claims 1-11 and / or the apparatus as described in any one of claims 12-22.

25. A computing device, characterized in that, The computing device includes a processor and memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the algorithmic functions embodied by the method as described in any one of claims 1-11 and / or the apparatus as described in any one of claims 12-22.

26. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the algorithmic functions embodied by the method as described in any one of claims 1-11 and / or the apparatus as described in any one of claims 12-22.

27. A computer-readable storage medium, characterized in that, The system includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the algorithmic functions embodied by the method as described in any one of claims 1-11 and / or the apparatus as described in any one of claims 12-22, wherein the cluster of computing devices includes at least one computing device.

28. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the algorithmic functions embodied by the method as described in any one of claims 1-11 and / or the apparatus as described in any one of claims 12-22, wherein the computing device cluster includes at least one computing device.

Images