Digital content generation method, voice packet generation method and computer program product

Abstract

The invention discloses a digital content generation method, a voice packet generation method and a computer program product, and the method comprises the steps: obtaining first reference digital content; encoding the first reference digital content to obtain corresponding reference multi-codebook data; wherein at least two codebook data in the reference multi-codebook data respectively represent at least two kinds of original information of the first reference digital content; obtaining target multi-codebook data corresponding to target digital content to be generated according to the reference multi-codebook data; and decoding the target multi-codebook data to obtain a target digital content. According to the scheme provided by the invention, the detail information of the reference digital content can be captured, so that the generation quality of the target digital content is improved.

Description

Technical Field

[0001] This application relates to the field of data generation technology, specifically to a method for generating digital content, a method for generating voice packs, and a computer program product. Background Technology

[0002] With the rapid development of artificial intelligence technology, the generation of digital content has become extremely important. Digital content refers to content that can be stored and processed in digital formats, and typically includes, but is not limited to, audio, images, and video data.

[0003] Currently, the industry commonly uses encoding and modeling of source digital content to generate the desired digital content. However, encoding source digital content often focuses on only one aspect, resulting in missing details in the generated digital content. For example, encoded audio data may lack emotional rhythm, encoded image data may lack color and texture, and encoded video may lack resolution and smoothness. Therefore, improving the quality of generated digital content has become an urgent technical problem to be solved. Summary of the Invention

[0004] This application provides a method for generating digital content, a method for generating voice packets, and a computer program product to improve the quality of generated target digital content. The specific solution is as follows: In a first aspect, embodiments of this application provide a method for generating digital content, comprising: acquiring first reference digital content; obtaining corresponding reference multi-codebook data by encoding the first reference digital content; wherein at least two codebook data in the reference multi-codebook data respectively represent at least two types of original information of the first reference digital content; acquiring target multi-codebook data corresponding to target digital content to be generated based on the reference multi-codebook data; and decoding the target multi-codebook data to obtain target digital content.

[0005] Secondly, embodiments of this application provide a voice packet generation method, comprising: acquiring reference audio content, reference text content representing the semantics of the reference audio content, and target text content representing the semantics of target audio content to be generated; determining reference multi-codebook data corresponding to the reference audio content, reference text single-codebook data corresponding to the reference text content, and target text single-codebook data corresponding to the target text content; wherein, at least two codebook data in the reference multi-codebook data respectively represent at least two types of original information of the reference audio content; acquiring target multi-codebook data corresponding to the target audio content to be generated based on the reference multi-codebook data, the reference text single-codebook data, and the target text single-codebook data; and decoding the target multi-codebook data to obtain the target audio content.

[0006] Thirdly, embodiments of this application also provide a computer program product, including a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the method as described in the first or second aspect.

[0007] Compared with existing technologies, the digital content generation method provided in this application captures the original information of the first reference digital content through reference multi-codebook data. This allows for the generation of target multi-codebook data, which describes detailed information about the target digital content to be generated, based on the reference multi-codebook data encoded from the first reference digital content. Furthermore, decoding the target multi-codebook data yields the target digital content with detailed information. Therefore, this application can capture the detailed information of the reference digital content, thereby improving the generation quality of the target digital content. Attached Figure Description

[0008] Figure 1 This is a diagram of a digital content generation system provided in an embodiment of this application for implementing a digital content generation method.

[0009] Figure 2 This is a schematic diagram illustrating an application scenario of the digital content generation method provided in this application embodiment.

[0010] Figure 3 This is a flowchart of a method for generating digital content provided in an embodiment of this application.

[0011] Figure 4 This is a data processing flowchart of the digital content encoder in the digital content generation method provided in the embodiments of this application.

[0012] Figure 5 This is a schematic diagram of the training of the digital content encoder in the digital content generation method provided in the embodiments of this application.

[0013] Figure 6This is a schematic diagram illustrating an example of obtaining reference multicodebook data using residual vector quantization in the digital content generation method provided in this application embodiment.

[0014] Figure 7 This is a flowchart illustrating the process of generating target multi-codebook data corresponding to the digital content of the unit to be predicted in the digital content generation method provided in this application embodiment.

[0015] Figure 8 This is a flowchart of the voice packet generation method provided in the embodiments of this application.

[0016] Figure 9 This is a structural block diagram of an example of a digital content generation apparatus provided in an embodiment of this application.

[0017] Figure 10 This is a structural block diagram of an example of an electronic device for generating digital content or voice packets, provided in an embodiment of this application. Detailed Implementation

[0018] It should be noted that the terms "first," "second," "third," etc., in the claims, specification, and drawings of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. Such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in an order other than that shown or described herein. Furthermore, the terms "comprising," "having," and their variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. It should be understood that in the embodiments of this application, "at least one" means one or more, and "more than one" means two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "Including A, B, and / or C" means including any one, two, or three of A, B, and C. It should be understood that in the embodiments of this application, "B corresponding to A," "B corresponding to A," "A corresponds to B," or "B corresponds to A" indicates that B is associated with A, and B can be determined based on A. Determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0019] Based on the reasons mentioned in the background section, the first embodiment of this application provides a method for generating digital content to capture detailed information of reference digital content, thereby improving the quality of the generated target digital content. This method can be applied to a server; of course, in some online or offline digital content generation scenarios, this method can also be applied to a client. This application embodiment does not limit the type of the executing entity.

[0020] To more clearly demonstrate the digital content generation method provided in the embodiments of this application, the application scenarios of the digital content generation method provided in the embodiments of this application are first introduced.

[0021] Application Scenario 1: The digital content generation method provided in this application embodiment can be applied to the field of audio generation. Specifically, it can be used for voice pack generation in map applications, intelligent customer service, intelligent robots, telephone customer service, intelligent question answering in education and training, and voice responses to players during game interactions. Taking the digital content generation method provided in this application embodiment for voice pack generation in map applications as an example, when the map application is opened, a reference digital content upload control can be displayed. In response to the user's trigger operation on the digital content upload control, first reference digital content can be obtained. In some scenarios, in response to the user's trigger operation on the digital content upload control, reference text can be displayed, and the first reference digital content can be reference audio content recorded based on the reference text content. Please refer to... Figure 2The system includes interfaces (2-a) and (2-b), both of which are interface diagrams for map applications. Interface (2-a) displays a reference text content display area 201 and a recording control 202. The reference text display area 201 displays reference text content such as "Please go straight at the intersection ahead". The user presses the recording control 202 to record "Please go straight at the intersection ahead". Thus, the reference audio content is the audio content of "Please go straight at the intersection ahead" recorded by the user's own voice. The voice pack will generate the target audio content (i.e., the target numerical content) of the target text content "Please turn left at the intersection ahead" recorded by the user's own voice. When the user is navigating, as shown in interface (2-b), when a left turn is required at the intersection ahead, the audio corresponding to the target audio content of the target text content "Please turn left at the intersection ahead" in the voice pack will be played. In other scenarios, the first reference digital content can be at least a portion of a song performed by a singer that the user is recently interested in, selected and uploaded by the user in a third-party application (a third-party application refers to an application different from the application the user is currently using (such as a map application). A third-party application can utilize the interfaces (APIs), toolkits (SDKs), or other open resources provided by the application the user is currently using to integrate with the application, thereby providing the user with additional functions and services). It can also be a clip from a movie or TV show featuring an actor uploaded by the user, or an audio recording of the user or someone else (such as the user's parents, children, or spouse). For example, the first reference digital content might be a song performed by singer A uploaded by the user; another example is a clip of actor B performing in movie or TV show C uploaded by the user; yet another example is an audio recording of the user saying "The weather is so nice today, perfect for going out" uploaded by the user; and yet another example is an audio recording of the user's child saying "I like watching xx cartoon" uploaded by the user. Based on the acquired first reference digital content, it is possible to generate target digital content that matches the timbre, rhythm, pitch, and other details of the first reference digital content. For example, it is possible to generate a navigation voice package that has the timbre, rhythm, pitch, and other details of a particular singer, actor, user, or other person.

[0022] Application Scenario 2: The digital content generation method provided in this application can be applied to the field of images or videos. Specifically, it can be used for image or video generation in human-computer interaction. In the dialogue interface of human-computer interaction, users can input image content, video content, or audio content, which is the first reference digital content. When the first reference digital content is image content, target digital content with color intensity, light and shadow, and other details matching the image content can be generated. For example, based on image content of parents reading with their children, video content of parents reading with their children and interacting can be generated. When the first reference digital content is video content, audio content or image content matching the rhythm, plot development, color, and other details matching the video content can be generated. For example, based on a video of horseback riding and archery, audio content with the sound of horseshoes and arrows can be generated, or image content of horseback riding and archery can be generated. When the first reference digital content is audio content, video content matching the emotion, rhythm, and tempo matching the audio content can be generated. For example, based on a piece of soothing music, video content of waves gently lapping the beach and leaves swaying in the sea breeze can be generated.

[0023] To facilitate understanding of the digital content generation method provided in the embodiments of this application, the following is combined with... Figure 1This application provides a digital content generation system 100 for implementing a digital content generation method. The system 100 may include a terminal 101, a server 102, and a network 103. The user-held terminal 101 can connect to the server 102 via the network. The terminal is any device with computing hardware capable of supporting and executing software application tools corresponding to the digital content generation method. Terminal 101 can be an electronic device with display and data processing functions, such as a mobile phone, tablet, smartwatch, desktop computer, smart TV, VR device, in-vehicle device, wearable device, or laptop. Terminal 101 has a map application installed, or it has a browser installed that allows access to the map application. The user can upload first reference digital content to terminal 101. This first reference digital content can be reference audio recorded using the user's own voice. Terminal 101 can send the first reference digital content to server 102, so that server 102 can generate target digital content that matches the timbre, tone, rhythm, emotion, and other details of the first reference digital content. This target digital content can be a navigation voice package with the user's own timbre, tone, rhythm, and emotion, and the server can send the navigation voice package to the terminal. Terminal 101 can store the navigation voice package. When the user selects to play the navigation voice package, the data in the navigation voice package stored in digital form can be converted into an analog signal through a digital-to-analog converter (DAC) and played through a speaker. Server 102 possesses high-speed central processing unit (CPU) computing power, long-term reliable operation, powerful input / output (I / O) external data throughput capabilities, and better scalability. Server 102 can be a single server or a server cluster. Server 102 is used to generate corresponding target digital content upon receiving the first reference digital content and send the target digital content to the terminal. Server 102 can also provide other specific services to terminal 101, such as user information access, website access, application access, etc., which are not specifically limited in this application. The network can be a wireless network or a wired network, such as a wireless local area network (WLAN), local area network (LAN), cellular network, 2G network, 3G network, 4G network, 5G network, etc. Different terminals can also connect to other terminals or to the server using their own Bluetooth network or hotspot network.

[0024] Figure 3 This is a flowchart of a digital content generation method provided in an embodiment of this application, including steps S101 to S104.

[0025] Step S101: Obtain the first reference digital content.

[0026] This step is used to obtain the first reference digital content, which is the digital content used when generating the target digital content.

[0027] The digital content refers to content that can be stored and processed in a digital format, such as audio, images, and video data. In this embodiment, the first reference digital content and the target digital content to be generated can be the same type of content or different types of content. For example, if the first reference digital content is image content, the target digital content to be generated can be image content, audio content, or video content; or if the first reference digital content is audio content, the target digital content to be generated can be audio content, image content, or video content; or if the first reference digital content is video content, the target digital content to be generated can be audio content, image content, or video content.

[0028] When the first reference digital content is audio content, it can be audio content corresponding to specified text content or audio content corresponding to custom text content. For example, a map application may provide specified text content such as "Please turn left at the intersection ahead" or "Red light 10 seconds remaining," and users can read the specified text content aloud to obtain the corresponding audio content; another example is that users can upload their own or someone else's audio content to the map application.

[0029] Step S102: Obtain the corresponding reference multicodebook data by encoding the first reference digital content.

[0030] This step is used to obtain reference multi-codebook data corresponding to the first reference digital content. At least two codebooks in the reference multi-codebook data represent at least two types of original information of the first reference digital content. The reference multi-codebook data includes two or more codebooks. When some codebooks in the reference multi-codebook data represent the original information of the first reference digital content, the remaining codebooks can represent the modeling details of the first reference digital content. Since at least two codebooks in the reference multi-codebook data represent at least two types of original information of the first reference digital content, the original information of the first reference digital content can be captured based on the obtained reference multi-codebook data.

[0031] The first reference digital content has at least two kinds of raw information, which refers to unprocessed information used to reflect the detailed information of the first reference digital content.

[0032] If the first reference digital content is audio content, then the at least two types of original information include, but are not limited to, at least two of semantic information, timbre information, fundamental frequency information, energy information, prosody information, and emotional information. Semantic information refers to the actual content conveyed by the audio content; timbre information describes the sound characteristics of the audio content, determined by the resonance characteristics of the sound-producing body (such as the shape of the vocal tract), and is used to distinguish different speakers or instruments; fundamental frequency information represents the pitch of the sound; energy information represents the strength (i.e., loudness) of the sound; prosody information characterizes the rhythm, accents, and pauses of the audio content; and emotional information characterizes the emotional state contained in the audio content. Emotional states can be categorized according to different criteria. Specifically, emotional states can be divided into seven categories: anger, surprise, nausea, happiness, fear, sadness, and neutrality. Neutrality indicates the absence of any emotion. Emotional states can also be divided into four categories: joy, anger, sorrow, and happiness.

[0033] If the first reference digital content is image content, then the at least two types of original information include, but are not limited to, at least two of color information, brightness information, and texture information. Specifically, color information represents the color information (typically RGB channel values) of each spatial location in the image corresponding to the image content; brightness information represents the lightness or darkness of each spatial location in the image corresponding to the image content; and texture information describes the smoothness or roughness of each spatial location in the image corresponding to the image content.

[0034] If the first reference digital content is video content, then the at least two types of raw information include, but are not limited to, at least two of the following: color information, brightness information, texture information, semantic information, timbre information, fundamental frequency information, energy information, prosody information, emotional information, and optical flow information. Among these, optical flow information is used to describe inter-frame pixel motion and represents the movement information of objects in the video corresponding to the video content.

[0035] The reference multi-codebook data corresponding to the first reference digital content includes a preset number of codebook data, which is greater than or equal to 2. For example, the preset number may be two, three, four, or eight, etc. This preset number can be specifically set based on actual needs, and this application does not impose specific limitations on it. At least two codebook data in the reference multi-codebook data can respectively represent at least two types of original information of the first reference digital content. That is, one of the codebook data in the reference multi-codebook data can represent the first original information of the first reference digital content, and the other codebook data in the reference multi-codebook data can represent the second original information of the first reference digital content.

[0036] The preset number of codebook data included in the reference multi-codebook data may have a ranking order. At least two codebook data in the reference multi-codebook data have different position data in the reference multi-codebook data.

[0037] Optionally, when the digital content is audio content, the first codebook data in the reference multi-codebook data may be the codebook data corresponding to the semantic information of the first reference digital content, and the second codebook data in the reference multi-codebook data may be the codebook data corresponding to the timbre information of the first reference digital content.

[0038] For ease of explanation, subsequent embodiments of this application will be described using the example where both the first reference digital content and the target digital content to be generated are audio content. At least two codebooks in the reference multi-codebook data can be sets of quantization codes (tokens) corresponding to at least two types of original information obtained based on different codebooks. Taking the first reference digital content as audio content as an example, the at least two codebooks in the reference multi-codebook data are respectively the set of quantization codes corresponding to the semantic information and the set of quantization codes corresponding to the timbre information of the first reference digital content. Furthermore, each codebook in the reference multi-codebook data can be a quantization encoding set corresponding to different original information obtained based on different codebooks. Continuing with the example of the first reference digital content as audio content, the reference multi-codebook data, in addition to including codebook data for the quantization encoding set corresponding to the semantic information of the first reference digital content and codebook data for the quantization encoding set corresponding to the timbre information of the first reference digital content, can also include codebook data for the quantization encoding set corresponding to the prosody information of the first reference digital content, the quantization encoding set corresponding to the fundamental frequency information of the first reference digital content, the quantization encoding set corresponding to the energy information of the first reference digital content, the quantization encoding set corresponding to the emotional information of the first reference digital content, and so on. A codebook is a data structure used to convert continuous or complex signals into a series of discrete values. In practical applications, a codebook can be viewed as a lookup table, mapping the input signal to nearby codewords (i.e., the quantized vectors in subsequent steps) through the quantization process, thereby achieving data compression. Typically, quantization encoding refers to the position of the codeword (i.e., the quantized vector) in the codebook.

[0039] In a specific implementation, the first reference digital content can be discretized into reference multi-codebook data using a digital content encoder. For example... Figure 4 As shown, the first reference digital content can be input into the digital content encoder to obtain the reference multicodebook data corresponding to the first reference digital content output by the digital content encoder.

[0040] The digital content encoder can be trained by: acquiring reference sample digital content; inputting the reference sample digital content into the digital content encoder to be trained to obtain the predicted multi-codebook data corresponding to the reference sample digital content output by the digital content encoder to be trained; and training the digital content encoder based on the predicted multi-codebook data.

[0041] In one optional training method, training the digital content encoder based on the predicted multi-codebook data includes: acquiring multiple original label information of the reference sample digital content; acquiring the original prediction information corresponding to at least two predicted codebook data in the predicted multi-codebook data; and training the digital content encoder based on the multiple original label information and the original prediction information corresponding to the distribution of the at least two predicted codebook data.

[0042] Specifically, the digital content encoder is trained based on the various original tag information and the original prediction information corresponding to the at least two prediction codebook data distributions, including: obtaining loss data between each type of original prediction information and the original tag information of the same type in the original prediction information corresponding to the at least two prediction codebook data; and training the digital content encoder based on the loss data between each type of original prediction information and the original tag information of the same type.

[0043] The purpose of this training method is to train a digital content encoder that can accurately output multi-codebook data, including at least two codebook data representing at least two kinds of original information for a digital content.

[0044] The reference sample digital content can be one of the following: reference sample audio content, reference sample video content, and reference sample image content. For example, the reference sample digital content may include multiple audio segments from multiple users.

[0045] The following describes the training process of the digital content encoder, using the reference sample digital content as an example of the reference sample audio content: The various original label information of the reference sample digital content refers to the unprocessed information of the reference sample digital content. For the reference sample audio content, its original label information may include at least two of the following: label semantic information, label timbre information, label fundamental frequency information, label energy information, label prosody information, and label sentiment information. Here, label semantic information refers to the textual information of the reference sample audio content. For example, if the reference sample audio content is the audio content corresponding to "turn left at the intersection ahead," then the semantic information is "turn left at the intersection ahead." The label timbre information, label fundamental frequency information, label energy information, label prosody information, and label sentiment information can be extracted from the reference sample digital content using a feature extraction model. Specifically, the label timbre information can be extracted from the reference sample audio content using a timbre extraction model, the label prosody information can be extracted using a prosody prediction model, the label energy information can be extracted using an energy prediction model, the label fundamental frequency information can be extracted using a fundamental frequency prediction model, and the label sentiment information can be extracted using a sentiment prediction model. After obtaining the reference sample digital content and its various original label information, the reference sample digital content can be input into the digital content encoder to be trained. The digital content encoder to be trained can then output the predicted multi-codebook data corresponding to the reference sample digital content. For example, the reference sample audio content can be input into the audio encoder to be trained to obtain the predicted multi-codebook data corresponding to the reference sample audio content output by the audio encoder to be trained.

[0046] Subsequently, the original prediction information corresponding to at least two prediction codebooks in the predicted multi-codebook data can be obtained. For example, if the predicted multi-codebook data includes prediction codebook data corresponding to eight codebooks, the original prediction information corresponding to at least two of these prediction codebooks can be obtained. For the reference sample audio content, the original prediction information corresponding to at least two prediction codebooks in its predicted multi-codebook data can be at least two of the following: predicted semantic information, predicted timbre information, predicted fundamental frequency information, predicted energy information, predicted prosody information, and predicted emotion information.

[0047] In a specific implementation, the original prediction information can be obtained in the following way: each input data is input into a different information prediction model based on a multi-level autoregressive structure, and the original prediction information output by each information prediction model based on a multi-level autoregressive structure is obtained.

[0048] Each input data includes each of the at least two prediction codebooks and the original label information. The original information type of the original information of the reference sample digital content represented by any one of the at least two prediction codebooks is the same as the original information type of the predicted original information output by the information prediction model based on a multi-level autoregressive structure, which inputs any one of the prediction codebooks. For example, for reference sample audio content, the original information type of the original information of the reference sample audio content represented by any one of the prediction codebooks is a semantic type, and the original information type of the predicted original information output by the information prediction model based on a multi-level autoregressive structure, which inputs any one of the prediction codebooks, is also a semantic type. By using the information prediction model based on a multi-level autoregressive structure, the predicted original information can be obtained more accurately. Thus, with highly accurate predicted original information, the predicted multi-codebook data output by the trained encoder can more accurately represent the required original information.

[0049] The information prediction model based on a multilevel autoregressive (MLA) structure is used to output raw information from the input codebook data. Different MLA-based information prediction models are used to predict raw information of different types. The MLA-based information prediction model can be a trained model or a model to be trained. When the MLA-based information prediction model is a model to be trained, it can be jointly trained with the digital content encoder.

[0050] like Figure 5As shown, after inputting the reference sample digital content into the digital content encoder 01 to be trained, the digital content encoder outputs predicted multi-codebook data. This predicted multi-codebook data includes a first predicted codebook data in the first position, a second predicted codebook data in the second position, a third predicted codebook data in the third position, and so on. Then, any one of the at least two predicted codebook data can be embedded as a predicted codebook embedding vector. The predicted codebook embedding vector corresponding to this predicted codebook data, along with original label information of the same type as the original information predicted by the information prediction model 03 based on a multi-level autoregressive structure, is input into the information prediction model 03 based on a multi-level autoregressive structure. The information prediction model 03 based on a multi-level autoregressive structure then outputs the predicted original information corresponding to this predicted codebook data. Taking the multi-level autoregressive information prediction model 03 as an example for predicting semantic information, when the prediction codebook embedding vector corresponding to any prediction codebook data is input into the multi-level autoregressive information prediction model 03, the multi-level autoregressive information prediction model 03 outputs the predicted semantic information corresponding to the first word (e.g., the predicted semantic information is "early"). Then, the prediction codebook embedding vector corresponding to any prediction codebook data and the tag semantic information corresponding to the first word (e.g., the tag semantic information is "good") are input into the multi-level autoregressive information prediction model 03, and the multi-level autoregressive information prediction model 03 outputs the predicted semantic information corresponding to the first word (e.g., the predicted semantic information is "early"). The prediction model 03 outputs the predicted semantic information corresponding to the second word. Then, the prediction codebook embedding vector corresponding to any one of the prediction codebook data, the label semantic information corresponding to the first word, and the label semantic information corresponding to the second word are input into the information prediction model 03 based on a multi-level autoregressive structure. The information prediction model 03 based on a multi-level autoregressive structure outputs the predicted semantic information corresponding to the third word, and so on, until the information prediction model 03 based on a multi-level autoregressive structure outputs the predicted semantic information corresponding to the last word. Finally, the predicted semantic information corresponding to each word is combined to obtain the predicted semantic information corresponding to any one of the prediction codebook data. The original prediction information corresponding to any one of the prediction codebook data is output by the information prediction model 03 based on a multi-layer autoregressive structure.

[0051] Taking the predicted semantic information and predicted timbre information corresponding to at least two predicted codebooks in the multi-codebook data as examples, different information prediction models based on multi-level autoregressive structures can include at least two of the following: speech prediction models and timbre prediction models. The speech prediction model obtains the predicted semantic information corresponding to one of the at least two predicted codebooks, and the timbre prediction model obtains the predicted timbre information corresponding to the other of the at least two predicted codebooks. Specifically, one of the at least two predicted codebooks in the multi-codebook data is input into a speech recognition model to output predicted semantic information, and the other of the at least two predicted codebooks is input into a timbre prediction model to output predicted timbre information.

[0052] Optionally, before inputting each input data into different information prediction models based on multi-level autoregressive structures to obtain the original prediction information output by each information prediction model based on multi-level autoregressive structures, a preset correspondence can be obtained between the position data of each prediction codebook data in the prediction multi-codebook data and the original information type of at least one label original information in the multiple label original information (this preset correspondence is used to indicate the type of original information that the user expects to represent for the prediction codebook data corresponding to the position data), as a reference correspondence; and a preset correspondence can be obtained between each information prediction model based on multi-level autoregressive structures and the original information type of at least one label original information in the multiple label original information (this preset correspondence is used to indicate the original information type of the original information predicted by each information prediction model based on multi-level autoregressive structures), as a model type correspondence.Accordingly, for any one of the at least two predicted codebook data, the position data of that predicted codebook data in the predicted multi-codebook data can be obtained as arbitrary codebook position data (for example, for one of the at least two predicted codebook data, if its position data in the predicted multi-codebook data indicates that the predicted codebook data is the first position in the predicted multi-codebook data, then the first position is taken as the arbitrary codebook position data of that predicted codebook data); according to the reference correspondence, the original information type of the tag original information corresponding to the arbitrary codebook position data is determined as the information type corresponding to the arbitrary codebook (for example, if the predicted codebook data located at the first position in the reference correspondence corresponds to the semantic type, and the arbitrary codebook position data is the first position, then the information type corresponding to the arbitrary codebook is the semantic type); according to the model type correspondence, the information prediction model based on the multi-level autoregressive structure corresponding to the information type corresponding to the arbitrary codebook is determined as the information prediction model based on the multi-level autoregressive structure corresponding to the arbitrary codebook). The information prediction model (for example, if the information type corresponding to any codebook is a semantic type, and the speech prediction model in the model type correspondence corresponds to the semantic type, then the information prediction model based on the multi-level autoregressive structure corresponding to any codebook is a speech prediction model); since the original information type corresponding to the position data of any prediction codebook data in the prediction multi-codebook data in the reference correspondence is consistent with the original information type corresponding to the information prediction model based on the multi-level autoregressive structure corresponding to any codebook in the model type correspondence, the original information of any prediction codebook data and the original label information can be input into the information prediction model based on the multi-level autoregressive structure corresponding to any codebook to obtain the original prediction information output by the information prediction model based on the multi-level autoregressive structure corresponding to any codebook (for example, for the prediction codebook data that is the first position in the prediction multi-codebook data among the at least two prediction codebook data, inputting it into the speech prediction model can predict the semantic information of the prediction codebook data that is the first position).

[0053] Subsequently, the digital content encoder is trained based on the acquired original information of the various tags and the original prediction information corresponding to each of the at least two prediction codebooks. That is, the digital content encoder is trained using the acquired original information of the various tags as given tags and the original prediction information corresponding to the prediction codebooks, thereby obtaining the digital content encoder.

[0054] In practical implementation, a loss function preset for the original information can be used to calculate the loss data between each type of predicted original information and the original label information of the same type corresponding to the at least two predicted codebook data. Based on the loss data between each type of predicted original information and the original label information of the same type, the digital content encoder is trained. Specifically, when the information prediction model based on the multi-layer autoregressive structure is a trained model, if the loss data between at least one type of predicted original information and the original label information of the same type is not the preset acceptable loss data, the encoder parameters of the digital content encoder to be trained are adjusted until, in the predicted multi-codebook data output by the digital content encoder after adjusting the encoder parameters, the loss data between each type of predicted original information and the original label information of the same type corresponding to the at least two predicted codebook data are all the preset acceptable loss data. The digital content encoder after adjusting the encoder parameters at this point is then determined as the digital content encoder. When the information prediction model based on the multi-level autoregressive structure is the model to be trained, if the loss data between at least one predicted original information and the original label information of the same type is not the preset acceptable loss data, then the model parameters of the information prediction model based on the multi-level autoregressive structure to be trained and / or the encoder parameters of the digital content encoder to be trained are adjusted until, in the predicted multi-codebook data output by the digital content encoder after adjusting the parameters of the information prediction model based on the multi-level autoregressive structure and / or the encoder parameters, the loss data between each predicted original information and the original label information of the same type corresponding to at least two predicted codebook data is the preset acceptable loss data. At this time, the information prediction model based on the multi-level autoregressive structure after adjusting the parameters of the information prediction model based on the multi-level autoregressive structure and / or the digital content encoder after adjusting the encoder parameters are determined as the information prediction model based on the multi-level autoregressive structure and the digital content encoder.

[0055] It should be noted that there is no specific order in which the audio encoder is trained based on the loss data between the predicted semantic information and the label semantic information, and in the training of the audio encoder based on the loss data between the predicted timbre information and the label timbre information.

[0056] In this way, a digital content encoder capable of accurately outputting multi-codebook data including at least two codebook data representing at least two types of raw information for a digital content can be trained. When the first reference digital content is input into the trained digital content encoder, the loss data between each predicted raw information and the label raw information of the same type in the predicted raw information corresponding to at least two predicted codebook data output by the digital content encoder is a preset acceptable loss data, that is, the difference between each predicted raw information and the label raw information of the same type is small.

[0057] In an optional implementation, before training the digital content encoder based on the various original tag information and the predicted original information corresponding to the at least two predicted codebook data, the following steps may be included: obtaining a preset correspondence between the position data of each predicted codebook data in the predicted multi-codebook data and the original information type of at least one original tag information in the various original tag information, as a reference correspondence; obtaining an actual correspondence between the position data of each predicted codebook data in the predicted multi-codebook data and the original information type of at least one original predicted information in the predicted original information corresponding to the at least two predicted codebook data, as a predicted correspondence. Accordingly, the digital content encoder can be trained based on the various original tag information, the predicted original information corresponding to the at least two predicted codebook data, the reference correspondence, and the predicted correspondence. If the loss data between at least one predicted original information and the original label information of the same type is not the preset acceptable original information loss data, or if the loss data between the predicted correspondence and the reference correspondence corresponding to at least one predicted codebook data is not the preset acceptable correspondence, then the encoder parameters of the digital content encoder to be trained are adjusted until, in the predicted multi-codebook data output by the digital content encoder after adjusting the encoder parameters, the loss data between each predicted original information and the original label information of the same type in the predicted original information of at least two predicted codebook data are the preset acceptable original information loss data, and the loss data between the predicted correspondence and the reference correspondence corresponding to each predicted codebook data are the preset acceptable correspondence loss data match. The digital content encoder after adjusting the encoder parameters at this time is determined as the digital content encoder.

[0058] This step is used to constrain the position of the codebook data corresponding to at least two original information items in the multi-codebook data generated by the digital content encoder. For example, for a trained audio encoder, the first codebook data in the multi-codebook data it generates represents the semantic information in the original information, and the second codebook data represents the timbre information in the original information.

[0059] By using the above method, the difference between each prediction original information and the label original information of the same type can be small in the prediction original information corresponding to at least two prediction codebooks output by the trained digital content encoder, and the difference between the prediction correspondence and the reference correspondence corresponding to each prediction codebook can be small. For example, in the predicted multi-codebook data output by the trained audio encoder, at least two of the predicted codebook data correspond to the predicted semantic information of one of the predicted codebook data, with a small difference between the predicted semantic information and the label semantic information. The other predicted codebook data corresponds to the predicted timbre information, with a small difference between the predicted timbre information and the label timbre information. Furthermore, at least one of the predicted codebook data is the first position in the predicted multi-codebook data, and the original information type corresponding to the predicted original information of this predicted codebook data is a semantic type. The original information type corresponding to the first position in the reference correspondence is also a semantic type. At least one of the predicted codebook data is the second position in the predicted multi-codebook data, and the original information type corresponding to the predicted original information of this predicted codebook data is a timbre type. The original information type corresponding to the second position in the reference correspondence is also a timbre type. In other words, the predicted multi-codebook data output by the trained audio encoder not only includes predicted codebook data representing semantic information and predicted codebook data representing timbre information, but can also be configured according to the user's expectations so that the first predicted codebook data in the predicted multi-codebook data represents the predicted codebook data representing semantic information, and the second predicted codebook data represents the predicted codebook data representing timbre information.

[0060] Furthermore, in this embodiment, a digital content decoder can be combined to jointly train the digital content encoder and the digital content decoder to be trained. The digital content generation method provided in this embodiment further includes the following steps: inputting the predicted multi-codebook data corresponding to the reference sample digital content into the digital content decoder to be trained, to obtain the predicted sample digital content output by the digital content decoder to be trained. Accordingly, the digital content encoder and digital content decoder can be trained based on the reference sample digital content and the predicted sample digital content. The digital content decoder is used to decode the target multi-codebook data to obtain the target digital content.

[0061] This step is used to jointly train the digital content encoder and the digital content decoder to be trained.

[0062] It should be noted that, in this embodiment of the application, during the training process, the input data of the digital content decoder to be trained is the predicted multi-codebook data directly output by the digital content encoder to be trained. In subsequent use, the input data of the trained digital content decoder is the target multi-codebook data corresponding to the target digital content to be generated. The digital content decoder can accurately reconstruct the target multi-codebook data to obtain the target digital content.

[0063] Combined with appendix Figure 5 The process involves inputting the predicted multi-codebook data output by the digital content encoder 01 to be trained into the digital content decoder 02 to be trained. The digital content decoder then outputs predicted sample digital content. For example, the reference sample digital content can be reference sample audio content, and the digital content encoder 01 to be trained can be the audio encoder. The reference sample audio content is input into the audio encoder to be trained, which outputs predicted multi-codebook data. Then, the predicted multi-codebook data output by the audio encoder to be trained is input into the audio decoder to be trained, which outputs predicted sample audio content. Based on the reference sample digital content and the predicted sample digital content, the digital content encoder and digital content decoder can be trained.

[0064] Specifically, the sample digital content loss data between the predicted sample digital content and the reference sample digital content can be calculated by using a loss function preset for the sample digital content, or the reference sample digital content and the predicted sample digital content can be input into a discriminator (e.g., Figure 5 The reference sample digital content and the predicted sample digital content are input into the discriminator (04). The discriminator calculates the sample digital content loss data between the predicted sample digital content and the reference sample digital content. Based on the sample digital content loss data, the digital content encoder and digital content decoder are trained. If the sample digital content loss data is not the preset acceptable digital content loss data, the encoder parameters of the digital content encoder to be trained and / or the decoder parameters of the digital content decoder to be trained are adjusted until the sample digital content loss data obtained after adjusting the encoder parameters and / or decoder parameters is the preset acceptable digital content loss data. The digital content encoder and digital content decoder after adjusting the encoder parameters and / or decoder parameters at this time are determined as the digital content encoder and the digital content decoder.

[0065] Thus, the reference multi-codebook data corresponding to the first reference digital content is obtained through the digital content encoder. The specific method for obtaining the reference multi-codebook data corresponding to the first reference digital content through the digital content encoder is described below: In one optional implementation, the first reference digital content can be encoded by the digital content encoder to obtain the embedding vector of the first reference digital content; the embedding vector of the first reference digital content can then be vector quantized to obtain the reference multi-codebook data corresponding to the first reference digital content.

[0066] The vector quantization process refers to the process of quantizing continuous embedding vectors into discrete codebook data. Specifically, at least one unit of the first reference digital content can be encoded by the digital content encoder to obtain the embedding vector corresponding to each unit in the first reference digital content, and vector quantization processing can be performed on the embedding vector corresponding to each unit in the at least one unit to obtain the reference multi-codebook data of the digital content of each unit in the at least one unit.

[0067] Taking the first reference digital content as audio content as an example, at least one frame (approximately 40 milliseconds) of the reference audio content can be encoded by the digital content encoder to obtain the embedding vector of at least one frame in the reference audio content. The embedding vector of the at least one frame is then subjected to vector quantization processing to obtain the reference multicodebook data of the audio content of the at least one frame.

[0068] The embedding vector for each frame in at least one frame of the reference audio content refers to a fixed-length numerical vector generated for each frame, which is designed to capture the speech features of each frame. The embedding vector for at least one frame of the reference audio content can be the embedding vector of some frames in the reference audio content, or it can be the embedding vector of all frames in the reference audio content.

[0069] In one optional implementation, it is possible to detect whether there is a target frame to be removed in the reference audio content whose quality is lower than a preset quality threshold; if the target frame exists, the target frame is removed from the reference audio content to obtain processed reference audio content; all frames of the processed reference audio content are encoded to obtain the embedding vector of all frames in the processed reference audio content; if the target frame does not exist, all frames of the reference audio content are encoded to obtain the embedding vector of all frames in the reference audio content.

[0070] Table 1 shows an example of the embedding vector of at least one frame in the reference audio content described in the digital content generation method provided in this application:

[0071] In Table 1 above, at least one frame of the reference audio content includes a first frame, a second frame, a third frame, and a fourth frame. The embedding vector of the first frame is Z1, the embedding vector of the second frame is Z2, the embedding vector of the third frame is Z3, and the embedding vector of the fourth frame is Z4.

[0072] Subsequently, vector quantization can be performed on the embedding vector of the at least one frame to obtain reference multi-codebook data of the audio content of the at least one frame. Specifically, the digital content encoder can use residual vector quantization to convert the embedding vector of the at least one frame into reference multi-codebook data of the audio content of the at least one frame.

[0073] The method of converting the embedding vector of the at least one frame into reference multi-codebook data of the audio content of the at least one frame using residual vector quantization specifically includes the following steps: based on the embedding vector of the at least one frame, searching for the quantized vector corresponding to the quantized vector to be queried for the at least one frame in any one of the preset codebooks; and determining the reference multi-codebook data of the audio content of the at least one frame based on the quantized vector corresponding to the quantized vector in the at least one codebook.

[0074] The quantized vector to be queried in the at least one frame for any codebook refers to the vector corresponding to the quantized vector to be queried in any codebook for the at least one frame.

[0075] The step of searching for the quantized vector corresponding to the quantized vector to be queried for the at least one frame in any of the preset codebooks means: searching for the quantized vector in any of the preset codebooks whose similarity to the quantized vector to be queried for the at least one frame in any of the preset codebooks is greater than a preset similarity threshold. In other words, it means searching for the quantized vector in any of the preset codebooks whose distance (e.g., Euclidean distance) to the quantized vector to be queried for the at least one frame in any of the preset codebooks is less than a preset distance threshold.

[0076] It should be noted that for different frames within at least one frame of the reference audio content, the quantization vector to be queried for the same codebook may not be the same. Similarly, for the same frame within at least one frame of the reference audio content, the quantization vector to be queried for different codebooks may not be the same.

[0077] In this step, for each frame in the at least one frame, the quantized vector corresponding to the quantized vector to be queried for that frame is searched in each codebook of the preset codebook.

[0078] Taking a preset codebook comprising eight codebooks and the at least one frame comprising four frames as an example, for the embedding vector of each of the four frames, the quantized vector corresponding to the quantized vector to be queried for that frame in any one of the eight codebooks can be found in any one of the codebooks. For example, for the first frame, the quantized vector corresponding to the quantized vector to be queried for the first frame can be found in the first codebook representing semantic information; the quantized vector corresponding to ...

[0079] In a specific implementation, the step of searching for the quantized vector corresponding to the quantized vector of the at least one frame in any codebook from any codebook in the preset codebook, based on the embedding vector of the at least one frame, can be achieved through the following steps: For the first codebook in the preset codebook, the embedding vector of the at least one frame is used as the quantized vector to be queried for the at least one frame in the first codebook, and the quantized vector corresponding to the quantized vector to be queried for the at least one frame in the first codebook is searched from the first codebook. For any codebook located after the first codebook in the preset codebook, determine the residual vector between the quantized vector found in the codebook preceding the at least one frame and the quantized vector to be queried for the preceding codebook in the at least one frame, and use the residual vector as the quantized vector to be queried for the at least one frame in the at least one frame in the at least one codebook, and find the quantized vector corresponding to the quantized vector to be queried for the at least one frame in the at least one frame in the at least one codebook.

[0080] This implementation is used to determine the quantized vector to be queried for each codebook in at least one frame, so as to facilitate the query of the corresponding quantized vector in each codebook based on the quantized vector.

[0081] like Figure 6 The diagram illustrates an example of obtaining reference multi-codebook data using residual vector quantization in the digital content generation method provided in this application. The embedding vector of the first reference digital content is input into the first codebook. The quantized vector corresponding to the embedding vector of the first reference digital content is obtained through the first codebook. The residual vector between the embedding vector and the quantized vector is input into the second codebook. The quantized vector corresponding to the residual vector is obtained through the second codebook. This process is repeated until the nth codebook is obtained. Figure 6 (Taking an example with n codebooks) The output is the quantized vector. Then, the quantized vectors output by the first n-1 codebooks, the quantized vector output by the nth codebook, and the unquantized vector input to the nth codebook are superimposed to obtain the reference multi-codebook vector, and the reference multi-codebook data is obtained based on the reference multi-codebook vector.

[0082] Taking the first reference digital content as the reference audio content as an example, as shown in Table 2, it is an example table of at least one frame of the reference audio content to be quantized and the quantized vector in the digital content generation method provided in the embodiments of this application: Table 2.

[0083] As can be seen in Table 2 above, for the first codebook, the quantized vector to be queried for each frame in at least one frame of the reference audio content is the embedding vector of that frame. Specifically, if the embedding vector of the first frame is Z1, the embedding vector of the second frame is Z2, the embedding vector of the third frame is Z3, and the embedding vector of the fourth frame is Z4, then the quantized vector to be queried for the first frame in the first codebook is Z1, the quantized vector to be queried for the second frame in the first codebook is Z2, the quantized vector to be queried for the third frame in the first codebook is Z3, and the quantized vector to be queried for the fourth frame in the first codebook is Z4. Thus, when the corresponding quantized vector is found in the first codebook, the quantized vector found by Z1 in the first codebook is E. 11 The quantized vector found by Z2 in the first codebook is E. 21 The quantized vector found by Z3 in the first codebook is E. 31 The quantized vector found by Z1 in the first codebook is E. 41 .

[0084] For any codebook following the first codebook in the preset codebook (the second codebook, the third codebook, etc.), the quantized vector to be queried for any codebook in each of the at least one frame is the residual vector between the quantized vector found in the preceding codebook of the codebook in each of the at least one frame and the quantized vector to be queried for the preceding codebook in each frame. Specifically, the quantized vector to be queried in the second codebook in the first frame of the at least one frame is Z1-E. 11 Z1-E 11 For the first frame, the vector to be quantized, Z1, is to be queried in the first codebook, and the quantized vector E found by Z1 in the first codebook is... 11 The residual vector between them, so that Z1-E can be found in the second codebook. 11 The corresponding quantized vector E 12 The quantization vector to be queried in the third codebook for the first frame of at least one frame is Z1-E. 11 -E 12 Z1-E 11 -E 12 The quantization vector Z1-E to be queried in the first frame for the second codebook 11 and Z1-E 11 The quantized vector E found in the second codebook 12 The residual vector between them, so that Z1-E can be found in the third codebook. 11 -E 12 The corresponding quantized vector E 13 The quantization vector to be queried in the fourth codebook for the first frame of at least one frame is Z1-E. 11 -E 12 -E 13 Z1-E 11 -E 12 -E 13 The quantization vector Z1-E to be queried in the first frame for the third codebook 11 -E 12 and Z1-E 11 -E 12 The quantized vector E found in the third codebook 13 The residual vector between them, so that Z1-E can be found in the fourth codebook. 11 -E 12 -E 13 The corresponding quantized vector E 14Similarly, the quantized vectors of the second, third, and fourth frames are obtained from each codebook. Then, for each frame, the quantized vectors obtained from each codebook are superimposed to obtain the multi-codebook vector corresponding to each frame. Finally, reference multi-codebook data for the audio content of each frame is obtained based on the multi-codebook vector.

[0085] Table 3 shows an example of reference multicodebook data for at least one frame of audio content in the digital content generation method provided in this application embodiment: Table 3.

[0086] As can be seen, the quantized vectors corresponding to the first frame in the first to fourth codebooks are E respectively. 11 E 12 E 13 E 14 The resulting multi-codebook vector after superposition is E. 11 E 12 E 13 E 14 The reference multi-codebook data for the audio content of the first frame is (11, 12, 13, 14); the quantized vectors of the second frame in the first to fourth codebooks are E respectively. 21 E 22 E 23 E 24 The resulting multi-codebook vector after superposition is E. 21 E 22 E 23 E 24 The reference multi-codebook data for the audio content of the second frame is (21, 22, 23, 24); the quantized vectors of the third frame in the first to fourth codebooks are E respectively. 31 E 32 E 33 E 34 The resulting multi-codebook vector after superposition is E. 31 E 32 E 33 E 34 The reference multi-codebook data for the audio content of the third frame is (31, 32, 33, 34); the quantized vectors corresponding to the fourth frame in the first to fourth codebooks are E respectively. 41 E 42 E 43 E 44 The resulting multi-codebook vector after superposition is E. 41 E 42 E 43 E 44The reference multicodebook data for the audio content of the fourth frame is (41, 42, 43, 44).

[0087] Thus, by performing residual vector quantization on the embedding vector of the first reference digital content to obtain the reference multi-codebook data of the first reference digital content, the difference between the reference multi-codebook data of the first reference digital content and the embedding vector of the first reference digital content can be reduced, so that the reference multi-codebook data can express the original information of the first reference digital content more accurately.

[0088] Step S103: Obtain the target multi-codebook data corresponding to the target digital content to be generated based on the reference multi-codebook data. This step is used to obtain the target multi-codebook data corresponding to the target digital content to be generated through the reference multi-codebook data.

[0089] The target multi-codebook data is used to describe the detailed information of the target digital content to be generated. This detailed information includes semantic information, timbre information, prosody information, fundamental frequency information, energy information, and other information. At least two codebooks in the target multi-codebook data represent at least two types of target information of the target digital content to be generated. This target information reflects the detailed information of the target digital content. The at least two types of target information describe the key attributes of the target digital content to be generated. Taking the target digital content to be generated as target audio content as an example, the at least two types of target information include target semantic information, target timbre information, target prosody information, target fundamental frequency information, target energy information, and target fundamental frequency information. These target information collectively determine the key attributes of the target audio content to be generated, such as semantic content, speaker identity, pronunciation style, and emotional expression, providing a good foundation for the generation of the target audio content.

[0090] Optionally, the digital content type of the target digital content to be generated is the same as the digital content type of the first reference digital content; the target information type of each of the at least two types of target information is the same as the original information type of one of the at least two types of original information; the target codebook information type correspondence is the same as the reference codebook information type correspondence; the target codebook information type correspondence is the correspondence between the position data of each codebook data in the target multi-codebook data and the target information type of at least one of the at least two types of target information; and the reference codebook information type correspondence is the correspondence between the position data of each codebook data in the reference multi-codebook data and the original information type of at least one of the at least two types of original information.

[0091] Taking the example where the target digital content's digital content type is audio and the first reference digital content's digital content type is also audio, if one of the at least two original information types has a semantic type and the other has a timbre type, then one of the at least two target information types has a semantic type and the other has a timbre type. If one of the at least two codebooks in the reference multi-codebook data is in the first position and the other is in the second position, and the original information type corresponding to the first position is semantic and the original information type corresponding to the second position is timbre, then one of the at least two codebooks in the target multi-codebook data is in the first position, and the codebook in the first position represents the semantic information of the target audio content to be generated; the other codebook is in the second position, and the codebook in the second position represents the timbre information of the target audio content to be generated.

[0092] Optionally, the target digital content to be generated includes multiple units of digital content. For example, if the target digital content is audio or video content, then the multiple units of digital content are multiple frames of audio or video content; or if the target digital content is text content (such as a novel), then the multiple units of digital content are multiple chapters of text content.

[0093] like Figure 7 As shown, obtaining the target multi-codebook data corresponding to the target digital content to be generated based on the reference multi-codebook data may include the following steps: performing vector embedding processing on the reference multi-codebook data and the predicted content codebook data corresponding to the digital content of at least one unit of the target digital content to obtain a codebook embedding vector; inputting the codebook embedding vector into a vector representation prediction model 05 (used to predict the vector representation of the unit to be predicted through the codebook embedding vector) to obtain the vector representation of the digital content of one unit of the target digital content output by the vector representation prediction model 05, which serves as the vector representation of the unit to be predicted (each unit of the target digital content); inputting the vector representation of the unit to be predicted into a codebook data prediction model 06 (used to convert the vector representation of the unit to be predicted into the target multi-codebook data corresponding to the digital content of the unit to be predicted) to obtain the target multi-codebook data corresponding to the digital content of the unit to be predicted output by the codebook data prediction model 06.

[0094] This step is used to generate target multi-codebook data corresponding to the digital content of the unit to be predicted by predicting the vector representation of the unit to be predicted in the target digital content to be generated, thereby obtaining the target multi-codebook data corresponding to the target digital content to be generated.

[0095] Specifically, the reference multi-codebook data is input into the vector representation prediction model to obtain the vector representation of the digital content of the first unit to be predicted, output by the vector representation prediction model, which serves as the vector representation of the first unit to be predicted. The vector representation of the first unit to be predicted is then input into the codebook data prediction model to obtain the target multi-codebook data corresponding to the digital content of the first unit to be predicted, output by the codebook data prediction model. Taking the first unit to be predicted of the target digital content as the first frame of the target audio content to be generated as an example, the reference multi-codebook data can be input into the vector representation prediction model to obtain the vector representation of the audio content of the first frame, output by the vector representation prediction model, which serves as the vector representation of the first frame. The vector representation of the first frame is then input into the codebook data prediction model to obtain the target multi-codebook data corresponding to the digital content of the first frame, output by the codebook data prediction model.

[0096] For any prediction unit other than the first prediction unit, the reference multi-codebook data and the predicted content codebook data corresponding to the digital content of at least one unit of the target digital content can be input into the vector representation prediction model to obtain the vector representation of the digital content of one prediction unit of the target digital content output by the vector representation prediction model, which is used as the vector representation of the prediction unit; the vector representation of the prediction unit is input into the codebook data prediction model to obtain the target multi-codebook data corresponding to the digital content of the prediction unit output by the codebook data prediction model.

[0097] Specifically, after obtaining the target multi-codebook data corresponding to the digital content of the first unit (e.g., the first frame) of the target digital content to be generated (e.g., target audio content), the reference multi-codebook data and the target multi-codebook data corresponding to the digital content of the first unit can be input into a vector representation prediction model to obtain the vector representation of the digital content of the second unit to be predicted (e.g., the second frame) of the target digital content output by the vector representation prediction model. The vector representation of the digital content of the second unit to be predicted is then input into a codebook data prediction model to obtain the target multi-codebook data corresponding to the digital content of the second unit to be predicted, output by the codebook data prediction model. Then, the... The target multi-codebook data corresponding to the digital content of at least some units in the first and second units is input into a vector representation prediction model to obtain a vector representation of the digital content of the third unit to be predicted (e.g., the third frame) of the target digital content output by the vector representation prediction model. The vector representation of the digital content of the third unit to be predicted is then input into a codebook data prediction model to obtain the target multi-codebook data corresponding to the digital content of the third unit to be predicted output by the codebook data prediction model. This process is repeated until the target multi-codebook data corresponding to the digital content of the last unit to be predicted (e.g., the last frame) output by the codebook data prediction model is obtained.

[0098] In this way, the target multi-codebook data corresponding to the digital content of each unit to be predicted can be predicted one by one using the reference multi-codebook data and the predicted content codebook data corresponding to the digital content of at least one unit of the predicted target digital content. Since the reference multi-codebook data captures the original information of the first reference digital content, which reflects the detailed information of the first reference digital content, target multi-codebook data describing the detailed information of the target digital content to be generated can be generated based on the reference multi-codebook data.

[0099] Optionally, when obtaining the target multi-codebook data corresponding to the target digital content to be generated based on the reference multi-codebook data, a second reference digital content can be further obtained, and the reference codebook data of the second reference digital content can be determined. The target multi-codebook data corresponding to the target digital content to be generated can then be obtained based on the reference multi-codebook data and the reference codebook data. It should be noted that the reference codebook data corresponding to the second reference digital content can be single-codebook data or multi-codebook data. Then, the target multi-codebook data corresponding to the target digital content to be generated can be obtained based on the reference multi-codebook data corresponding to the first reference digital content and the reference codebook data corresponding to the second reference digital content. Specifically, the reference multi-codebook data corresponding to the first reference digital content and the reference codebook data corresponding to the second reference digital content can be concatenated to obtain concatenated multi-codebook data; the target multi-codebook data corresponding to the target digital content to be generated can then be obtained based on the concatenated multi-codebook data.

[0100] When the digital content is audio content, the first reference digital content is reference audio content, and the target digital content is target audio content. Reference text content representing the semantics of the reference audio content and target text content representing the semantics of the target audio content to be generated can be obtained. The reference text content and target text content constitute the second reference digital content. Reference text single-codebook data corresponding to the reference text content and target text single-codebook data corresponding to the target text content are determined. The reference text single-codebook data and target text single-codebook data constitute the reference codebook data. Based on the reference multi-codebook data, the reference text single-codebook data, and the target text single-codebook data, the target multi-codebook data corresponding to the target audio content to be generated is obtained. For example, for the target audio content to be generated, the first reference digital content is reference audio content obtained by recording a reference text content (e.g., going straight at the intersection ahead) using user A's voice, and the target text content is the target text content to be used in generating the target audio content (e.g., turning left at the intersection ahead).

[0101] When obtaining the target multi-codebook data corresponding to the target audio content to be generated based on the reference multi-codebook data, the reference text single-codebook data, and the target text single-codebook data, the splicing will fail because the reference text single-codebook data and the target text single-codebook data are single-codebook data, and the number of codebook data in the reference multi-codebook data and the reference text single-codebook data are inconsistent. Therefore, the splicing can be completed by first supplementing the reference text single-codebook data and the target text single-codebook data according to the number of codebook data contained in the reference multi-codebook data, to obtain the reference text multi-codebook data and the target text multi-codebook data; wherein the number of codebook data contained in the reference text multi-codebook data is consistent with the number of codebook data contained in the reference multi-codebook data, and the number of codebook data contained in the target text multi-codebook data is consistent with the number of codebook data contained in the reference multi-codebook data. For example, if the reference multi-codebook data contains 8 codebook data, then the reference text single-codebook data and the target text single-codebook data can be completed into reference text multi-codebook data and target text multi-codebook data, respectively, each containing 8 codebook data. Then, the reference multi-codebook data, the reference text multi-codebook data, and the target text multi-codebook data are concatenated to obtain concatenated multi-codebook data; based on the concatenated multi-codebook data, the target multi-codebook data corresponding to the target audio content to be generated is obtained.

[0102] It should be noted that, when obtaining the target multi-codebook data corresponding to the target digital content to be generated based on the concatenated multi-codebook data, the step of inputting the reference multi-codebook data and the predicted content codebook data corresponding to the digital content of at least one unit of the predicted target digital content into the vector representation prediction model to obtain the vector representation of the digital content of a unit to be predicted of the target digital content output by the vector representation prediction model, the vector representation of the unit to be predicted refers to: inputting the concatenated multi-codebook data and the predicted content codebook data corresponding to the digital content of at least one unit of the predicted target digital content into the vector representation prediction model to obtain the vector representation of the digital content of a unit to be predicted of the target digital content output by the vector representation prediction model, and using this as the vector representation of the unit to be predicted.

[0103] In this way, by combining the concatenated multi-codebook data with the predicted codebook data corresponding to the digital content of at least one unit of the predicted target digital content, the target multi-codebook data corresponding to the digital content of each unit to be predicted can be predicted one by one. The concatenated multi-codebook data can simultaneously capture the detailed information of the first reference digital content and the detailed information of the second reference digital content, thereby selectively generating target multi-codebook data containing partial detailed information of the first reference digital content and partial detailed information of the second reference digital content. For example, capturing the semantic information, timbre information, prosody information, emotional information, energy information, and fundamental frequency information of the reference audio content, and capturing the semantic information of the target text content, thereby generating target multi-codebook data containing the semantic information of the target text content, the timbre information, prosody information, emotional information, energy information, and fundamental frequency information of the reference audio content.

[0104] Step S104: Decode the target multicodebook data to obtain the target digital content.

[0105] This step is used to reconstruct the target digital content using the target multi-codebook data. Specifically, the target multi-codebook data can be input into the digital content decoder obtained through the aforementioned training to obtain the target digital content output by the digital content decoder. When the digital content is audio content, step S104 specifically refers to: inputting the target multi-codebook data into an audio decoder to obtain the target audio content output by the audio decoder, where the target audio content is a speech waveform.

[0106] The second embodiment of this application provides a method for generating voice packets, such as... Figure 8 The process includes steps S201 to S205. Step S201: Obtain reference audio content, reference text content representing the semantics of the reference audio content, and target text content representing the semantics of the target audio content to be generated. Step S202: Determine the reference multi-codebook data corresponding to the reference audio content, the reference text single-codebook data corresponding to the reference text content, and the target text single-codebook data corresponding to the target text content; wherein, at least two codebook data in the reference multi-codebook data respectively represent at least two types of original information of the reference audio content. Step S203: Based on the reference multi-codebook data, the reference text single-codebook data, and the target text single-codebook data, obtain the target multi-codebook data corresponding to the target audio content to be generated. Step S204: Decode the target multi-codebook data to obtain the target audio content. Step S205: Generate a speech packet based on the target audio content.

[0107] Corresponding to the digital content generation method provided in the first embodiment of this application, the third embodiment of this application also provides a digital content generation apparatus, such as... Figure 9 As shown, the digital content generation apparatus 700 includes: a first acquisition unit 701, used to acquire first reference digital content; a second acquisition unit 702, used to obtain corresponding reference multi-codebook data by encoding the first reference digital content; wherein at least two codebook data in the reference multi-codebook data respectively represent at least two kinds of original information of the first reference digital content; a third acquisition unit 703, used to acquire target multi-codebook data corresponding to the target digital content to be generated according to the reference multi-codebook data; and a decoding unit 704, used to decode the target multi-codebook data to obtain the target digital content.

[0108] Corresponding to the method provided in the second embodiment of this application, the fourth embodiment of this application also provides a speech packet generation apparatus, including: a fourth acquisition unit, configured to acquire reference audio content, reference text content representing the semantics of the reference audio content, and target text content representing the semantics of target audio content to be generated; a determination unit, configured to determine reference multi-codebook data corresponding to the reference audio content, reference text single-codebook data corresponding to the reference text content, and target text single-codebook data corresponding to the target text content; wherein, at least two codebook data in the reference multi-codebook data respectively represent at least two kinds of original information of the reference audio content; a fifth acquisition unit, configured to acquire target multi-codebook data corresponding to the target audio content to be generated based on the reference multi-codebook data, the reference text single-codebook data, and the target text single-codebook data; a processing unit, configured to decode the target multi-codebook data to obtain target audio content; and a generation unit, configured to generate a speech packet based on the target audio content.

[0109] Corresponding to the methods provided in the first or second embodiments of this application, the fifth embodiment of this application also provides an electronic device for generating digital content or voice packets. For example... Figure 10 The electronic device includes a processor 801, a memory 803, a communication bus 804, and a communication interface 802. The memory 803 is used to store computer instructions for data processing. When the computer instructions are read and executed by the processor 801, they perform the method described in the first embodiment of this application.

[0110] Corresponding to the methods provided in the first or second embodiments of this application, the sixth embodiment of this application provides a computer-readable storage medium storing a program for a method of generating digital content or a method of generating voice packets. The program is executed by a processor to perform the methods provided in the first or second embodiments of this application.

[0111] Corresponding to the methods provided in the first or second embodiments of this application, the seventh embodiment of this application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the methods provided in the first or second embodiments of this application.

[0112] It should be noted that for detailed descriptions of the methods, apparatus, electronic devices, computer-readable storage media, and computer program products provided in the second, third, fourth, fifth, sixth, and seventh embodiments of this application, please refer to the relevant descriptions of the first embodiment of this application, which will not be repeated here.

[0113] This application may involve the use of user data. In practical applications, user-specific personal data may be used within the scope permitted by applicable laws and regulations of the country of application (e.g., with explicit user consent and effective notification to the user). The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A method of generating digital content, characterized by, The method comprises: obtaining a first reference digital content; obtaining corresponding reference multi-codebook data by encoding the first reference digital content; wherein at least two codebook data in the reference multi-codebook data respectively represent at least two original information of the first reference digital content; obtaining target multi-codebook data corresponding to a target digital content to be generated according to the reference multi-codebook data; performing decoding processing on the target multi-codebook data to obtain the target digital content.

2. The method of claim 1, wherein, The method comprises: inputting the first reference digital content into a digital content encoder to obtain reference multi-codebook data corresponding to the first reference digital content output by the digital content encoder.

3. The method of claim 2, wherein, The digital content encoder is obtained by training in the following manner: obtaining reference sample digital content; inputting the reference sample digital content into a digital content encoder to be trained to obtain predicted multi-codebook data corresponding to the reference sample digital content output by the digital content encoder to be trained; training the digital content encoder according to the predicted multi-codebook data.

4. The method of claim 3, wherein, The method comprises: obtaining multiple label original information of the reference sample digital content; obtaining predicted original information corresponding to at least two predicted codebook data in the predicted multi-codebook data respectively; obtaining loss data between each predicted original information corresponding to the at least two predicted codebook data respectively and label original information of the same type; training the digital content encoder according to the loss data between each predicted original information and label original information of the same type.

5. The method of claim 4, wherein, The method comprises: inputting each input data into different information prediction models based on a multi-layer autoregressive structure to obtain predicted original information output by each information prediction model based on a multi-layer autoregressive structure, wherein each input data comprises each predicted codebook data in the at least two predicted codebook data and label original information, and the original information type of the original information of the reference sample digital content represented by any one predicted codebook data in the at least two predicted codebook data is the same as the original information type of the predicted original information output by the information prediction model based on a multi-layer autoregressive structure input by the any one predicted codebook data.

6. The method of claim 3, wherein, The method further comprises: inputting the predicted multi-codebook data corresponding to the reference sample digital content into a digital content decoder to be trained to obtain predicted sample digital content output by the digital content decoder to be trained; The method comprises: obtaining sample digital content loss data between the predicted sample digital content and the reference sample digital content; If the sample digital content loss data is not the preset acceptable digital content loss data, the encoder parameters of the to-be-trained digital content encoder and / or the decoder parameters of the to-be-trained digital content decoder are adjusted until the sample digital content loss data obtained after the adjustment of the encoder parameters and / or the decoder parameters is the preset acceptable digital content loss data, and the digital content encoder and the digital content decoder after the adjustment of the encoder parameters and / or the decoder parameters are determined as the digital content encoder and the digital content decoder; the digital content decoder is used for decoding processing on the target multi-codebook data to obtain the target digital content.

7. The method according to any one of claims 1 to 6, characterized in that, When the digital content is audio content, the first reference digital content is reference audio content, and the target digital content is target audio content. The target multi-codebook data corresponding to the target digital content to be generated is obtained according to the reference multi-codebook data, including: obtaining reference text content representing the semantics of the reference audio content and target text content representing the semantics of the target audio content to be generated; determining reference text single-codebook data corresponding to the reference text content and target text single-codebook data corresponding to the target text content; obtaining target multi-codebook data corresponding to the target audio content to be generated according to the reference multi-codebook data, the reference text single-codebook data and the target text single-codebook data.

8. The method of claim 7, wherein, The target multi-codebook data corresponding to the target audio content to be generated is obtained according to the reference multi-codebook data, the reference text single-codebook data and the target text single-codebook data, including: completing the reference text single-codebook data and the target text single-codebook data according to the number of codebook data contained in the reference multi-codebook data, to obtain reference text multi-codebook data and target text multi-codebook data; splicing the reference multi-codebook data, the reference text multi-codebook data and the target text multi-codebook data to obtain spliced multi-codebook data; obtaining target multi-codebook data corresponding to the target audio content to be generated according to the spliced multi-codebook data.

9. A method of generating a speech packet, characterized by, including: obtaining reference audio content, reference text content representing the semantics of the reference audio content and target text content representing the semantics of the target audio content to be generated; determining reference multi-codebook data corresponding to the reference audio content, reference text single-codebook data corresponding to the reference text content and target text single-codebook data corresponding to the target text content; at least two codebook data in the reference multi-codebook data represent at least two original information of the reference audio content respectively; obtaining target multi-codebook data corresponding to the target audio content to be generated according to the reference multi-codebook data, the reference text single-codebook data and the target text single-codebook data; decoding processing on the target multi-codebook data to obtain target audio content; generating a voice packet according to the target audio content.

10. A computer program product, characterised in that, A computer program product comprising computer program instructions, which when executed by a processor, implement the method steps of any of claims 1-9.

Images