Audio and video generation method, electronic device and computer readable storage medium

Abstract

The present application provides an audio and video generation method, an electronic device and a computer-readable storage medium, the method comprising: obtaining target text information; inputting the target text information into a pre-trained audio and video generation autoregressive model to obtain a target video token number and a target audio token number; decoding the target video token number to obtain a target video corresponding to the target text information; decoding the target audio token number to obtain a target audio corresponding to the target text information, since the target video token number is generated according to the target text information, a first decoding function, a predicted video token number and a predicted audio token number, and the target audio token number is generated according to the target text information, a second decoding function, a predicted video token number and a predicted audio token number, therefore, the target video token number and the target audio token number realize the association between the video information and the audio information, and the generated target video and audio have high synchronization.

Description

Technical Field

[0001] The present application relates to the technical field of audio and video conversion, and in particular to an audio and video generation method, an electronic device and a computer-readable storage medium. Background Art

[0002] Currently, audio generation and video generation mostly generate corresponding audio or video given text. There are currently two commonly used frameworks. One is to generate audio and video separately through the diffusion model framework; the other is to generate audio and video separately through the autoregressive structure.

[0003] Currently, the method of generating audio and video through autoregressive structure generally adopts an interlaced generation method, for example, generating video first and then generating audio. This makes the generated video and audio have no direct connection, which in turn causes the generated audio and video to have problems of audio and video being out of sync and low quality. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an audio and video generation method, an electronic device and a computer-readable storage medium, so as to solve the problem of audio and video being out of sync and of low quality caused by using an interlaced generation method to generate audio and video through an autoregressive structure.

[0005] In a first aspect, the present invention provides an audio and video generation method, the method comprising: obtaining target text information; inputting the target text information into a pre-trained audio and video generation autoregressive model to obtain a target number of video tokens and a target number of audio tokens output by the audio and video generation autoregressive model; decoding the target number of video tokens to obtain a target video corresponding to the target text information; decoding the target number of audio tokens to obtain a target audio corresponding to the target text information; wherein the target number of video tokens is generated by the audio and video generation autoregressive model according to the target text information, a first decoding function, a predicted number of video tokens and a predicted number of audio tokens; the target number of audio tokens is generated by the audio and video generation autoregressive model according to the target text information, a second decoding function, a predicted number of video tokens and a predicted number of audio tokens; the predicted number of video tokens is generated by the audio and video generation autoregressive model according to the target text information and the first decoding function; the predicted number of audio tokens is generated by the audio and video generation autoregressive model according to the target text information and the second decoding function.

[0006] The audio and video generation method designed above, this scheme first obtains the target text information, and then based on the pre-acquired audio and video generation autoregressive model, outputs the corresponding target video token number and target audio token number according to the target text information, and then decodes the target video token number to obtain the target video corresponding to the target text information, and decodes the target audio token number to obtain the target audio corresponding to the target text information. Since the audio and video generation autoregressive model of this scheme generates the target video token number based on the target text information, the first decoding function, the predicted video token number and the predicted audio token number, therefore, the autoregressive prediction process of the target video token number refers to the audio information; similarly, the audio and video generation autoregressive model of this scheme generates the target audio token number based on the target text information, the second decoding function, the predicted video token number and the predicted audio token number, therefore, the autoregressive prediction process of the target audio token number refers to the video information, so that the target video token number and the target audio token number realize the association between the video information and the audio information, and then realize the association between the generated target video features and the target audio features, so that the target video and audio generated by this scheme are highly synchronized, and thus the synchronization and quality of the audio and video are higher.

[0007] In an optional implementation of the first aspect, the audio and video generation autoregressive model is trained and obtained by: obtaining a training set; wherein the training set includes multiple samples, each sample includes text sample information and video sample information and audio sample information corresponding to the text sample information; compressing and quantizing each video sample information to obtain a first number of video sample tokens corresponding to each video sample information; compressing and quantizing each audio sample information to obtain a second number of audio sample tokens corresponding to each audio sample information; training a preset autoregressive model based on multiple text sample information in the training set, the first number of video sample tokens corresponding to each text sample information, and the second number of audio sample tokens to obtain a trained audio and video generation autoregressive model.

[0008] In an optional implementation of the first aspect, a preset autoregressive model is trained according to multiple text sample information in a training set, a first number of video sample tokens corresponding to each text sample information, and a second number of audio sample tokens to obtain a trained audio and video generation autoregressive model, including: determining a first predicted number of video sample tokens according to the text sample information and a first decoding function through a preset autoregressive model, and determining a first predicted number of audio sample tokens according to the text sample information and the second decoding function; calculating a first loss value of the preset autoregressive model according to the first predicted number of video sample tokens, the second predicted number of audio sample tokens, the first number, and the second number; judging whether the first loss value reaches the preset loss value; if the first loss value does not reach the preset loss value, judging whether the number of video sample tokens currently being processed reaches the first introduced number, and judging whether the number of audio sample tokens currently being processed is Whether it reaches the second introduction quantity, wherein the first introduction quantity represents the number of video sample tokens that need to introduce audio information for video decoding, and the second introduction quantity represents the number of audio sample tokens that need to introduce video information for audio decoding; if it is determined that the number of video sample tokens currently being processed does not reach the first introduction quantity, and the number of audio sample tokens currently being processed reaches the second introduction quantity, then the second predicted number of video sample tokens is determined according to the first decoding function and the first predicted number of video sample tokens, and the second predicted number of audio sample tokens is determined according to the second decoding function, the first predicted number of audio samples and the first predicted number of video samples; the second loss value is calculated according to the second predicted number of video sample tokens, the second predicted number of audio sample tokens, the first number and the second number; it is determined whether the second loss value reaches the preset loss value; if the second loss value reaches the preset loss value, a trained audio and video generation autoregressive model is obtained.

[0009] In an optional implementation of the first aspect, the method also includes: if it is determined that the number of video sample tokens currently being processed has reached the first introduction number, and the number of audio sample tokens currently being processed has not reached the second introduction number, then determining the second predicted number of video sample tokens based on the first predicted number of video sample tokens, the first decoding function, and the first predicted number of audio samples, and determining the second predicted number of audio sample tokens based on the first predicted number of audio samples and the second decoding function.

[0010] In an optional implementation of the first aspect, the method also includes: if it is determined that the number of video sample tokens currently being processed reaches the first introduction number, and the number of audio sample tokens currently being processed reaches the second introduction number, then determining the second predicted number of video sample tokens based on the first predicted number of video sample tokens, the first decoding function, and the first predicted number of audio samples, and determining the second predicted number of audio sample tokens based on the first predicted number of audio samples, the first predicted number of video samples, and the second decoding function.

[0011] In an optional implementation of the first aspect, the method also includes: if it is determined that the second loss value does not reach the preset loss value, then determining whether the number of video sample tokens currently being processed reaches the first introduction number, and determining whether the number of audio sample tokens currently being processed reaches the second introduction number; if it is determined that the number of video sample tokens currently being processed reaches the first introduction number, and the number of audio sample tokens currently being processed does not reach the second introduction number, then determining the third predicted number of video sample tokens according to the first decoding function, the second predicted number of video sample tokens, and the second predicted number of audio sample tokens, and determining the third predicted number of audio sample tokens according to the second decoding function and the second predicted number of audio samples; calculating the second loss value according to the third predicted number of video sample tokens, the third predicted number of audio sample tokens, the first number, and the second number; determining whether the third loss value reaches the preset loss value; if the third loss value reaches the preset loss value, obtaining a trained audio and video generation autoregressive model.

[0012] Among the various implementations mentioned above, this scheme introduces audio or video information to participate in the next token number prediction when the number of currently processed sample tokens of video or audio reaches the introduced number, so that the trained audio and video autoregressive generation model can refer to audio information in the process of video generation, and can refer to video information in the process of audio generation, so that the audio and video autoregressive generation model generated by this scheme can quickly generate video and audio, so that the generated video and audio have high synchronization, thereby making the synchronization and quality of audio and video higher.

[0013] In an optional implementation of the first aspect, before determining the first predicted number of video sample tokens according to the text sample information and the first decoding function through a preset autoregressive model, and determining the first predicted number of audio sample tokens according to the text sample information and the second decoding function, the method also includes: calculating the video alignment number and the audio alignment number according to the first number and the second number, respectively, wherein the video alignment number represents the number of video sample tokens required for the video when the video and audio are aligned, and the audio alignment number represents the number of audio sample tokens required for the audio when the audio and video are aligned; determining the first introduction number according to the video alignment number, and determining the second introduction number according to the audio alignment number.

[0014] In an optional implementation of the first aspect, the video alignment number and the audio alignment number are calculated respectively according to the first number and the second number, including: calculating the greatest common divisor of the first number and the second number; calculating the quotient of the greatest common divisor and the first number to obtain the video alignment number; calculating the quotient of the greatest common divisor and the second number to obtain the audio alignment number.

[0015] In a second aspect, the present application provides an audio and video generation device, which includes an acquisition module, an input module and a decoding module. The acquisition module is used to acquire target text information, and the input module is used to input the target text information into a pre-trained audio and video generation autoregressive model to obtain the target number of video tokens and the target number of audio tokens output by the audio and video generation autoregressive model, wherein the target number of video tokens is generated by the audio and video generation autoregressive model according to the target text information, a first decoding function, a predicted number of video tokens and a predicted number of audio tokens; the target number of audio tokens is generated by the audio and video generation autoregressive model according to the target text information, a second decoding function, a predicted number of video tokens and a predicted number of audio tokens; the predicted number of video tokens is generated by the audio and video generation autoregressive model according to the target text information and the first decoding function; the predicted number of audio tokens is generated by the audio and video generation autoregressive model according to the target text information and the second decoding function; the decoding module is used to decode the target number of video tokens to obtain the target video corresponding to the target text information; and decode the target number of audio tokens to obtain the target audio corresponding to the target text information.

[0016] The audio and video generation device designed above, this scheme first obtains the target text information, and then based on the pre-acquired audio and video generation autoregressive model, outputs the corresponding target video token number and target audio token number according to the target text information, and then decodes the target video token number to obtain the target video corresponding to the target text information, and decodes the target audio token number to obtain the target audio corresponding to the target text information. Since the audio and video generation autoregressive model of this scheme generates the target video token number based on the target text information, the first decoding function, the predicted video token number and the predicted audio token number, the autoregressive prediction process of the target video token number refers to the audio information; similarly, the audio and video generation autoregressive model of this scheme generates the target audio token number based on the target text information, the second decoding function, the predicted video token number and the predicted audio token number, so the autoregressive prediction process of the target audio token number refers to the video information, so that the target video token number and the target audio token number realize the association between the video information and the audio information, and then realize the association between the generated target video features and the target audio features, so that the target video and audio generated by this scheme are highly synchronized, and then the synchronization and quality of the audio and video are higher.

[0017] In an optional implementation of the second aspect, the device also includes a training module for obtaining a training set; wherein the training set includes multiple samples, each sample includes text sample information and video sample information and audio sample information corresponding to the text sample information; each video sample information is compressed and quantized to obtain a first number of video sample tokens corresponding to each video sample information; each audio sample information is compressed and quantized to obtain a second number of audio sample tokens corresponding to each audio sample information; a preset autoregressive model is trained according to multiple text sample information in the training set, the first number of video sample tokens corresponding to each text sample information, and the second number of audio sample tokens to obtain a trained audio and video generation autoregressive model.

[0018] In an optional implementation manner of the second aspect, the training module is specifically used to determine the first predicted number of video sample tokens according to the text sample information and the first decoding function through a preset autoregressive model, and determine the first predicted number of audio sample tokens according to the text sample information and the second decoding function; calculate the first loss value of the preset autoregressive model according to the first predicted number of video sample tokens, the second predicted number of audio sample tokens, the first number and the second number; determine whether the first loss value reaches the preset loss value; if the first loss value does not reach the preset loss value, determine whether the number of video sample tokens currently being processed reaches the first introduction number, and determine whether the number of audio sample tokens currently being processed reaches the second introduction number, wherein the first introduction number indicates the video sample number of audio information that needs to be introduced for video decoding. The present token number and the second introduction number represent the number of audio sample tokens that need to be introduced into the video information for audio decoding; if it is determined that the number of video sample tokens currently being processed does not reach the first introduction number, and the number of audio sample tokens currently being processed reaches the second introduction number, then the second predicted number of video sample tokens is determined according to the first decoding function and the first predicted number of video sample tokens, and the second predicted number of audio sample tokens is determined according to the second decoding function, the first predicted number of audio samples and the first predicted number of video samples; the second loss value is calculated according to the second predicted number of video sample tokens, the second predicted number of audio sample tokens, the first number and the second number; it is determined whether the second loss value reaches the preset loss value; if the second loss value reaches the preset loss value, a trained audio and video generation autoregressive model is obtained.

[0019] In an optional implementation of the second aspect, the training module is further specifically used to determine the second predicted number of video sample tokens based on the first predicted number of video sample tokens, the first decoding function and the first predicted number of audio samples if it is determined that the number of video sample tokens currently being processed has reached the first introduction number and the number of audio sample tokens currently being processed has not reached the second introduction number, and to determine the second predicted number of audio sample tokens based on the first predicted number of audio samples and the second decoding function.

[0020] In an optional implementation of the second aspect, the training module is further specifically used to determine the second predicted number of video sample tokens based on the first predicted number of video sample tokens, the first decoding function and the first predicted number of audio samples if it is determined that the number of video sample tokens currently being processed reaches the first introduced number and the number of audio sample tokens currently being processed reaches the second introduced number, and to determine the second predicted number of audio sample tokens based on the first predicted number of audio samples, the first predicted number of video samples and the second decoding function.

[0021] In an optional implementation manner of the second aspect, the training module is also specifically used to determine whether the number of video sample tokens currently being processed reaches the first introduction number, and determine whether the number of audio sample tokens currently being processed reaches the second introduction number if it is determined that the second loss value does not reach the preset loss value; if it is determined that the number of video sample tokens currently being processed reaches the first introduction number, and the number of audio sample tokens currently being processed does not reach the second introduction number, then determine the third predicted number of video sample tokens according to the first decoding function, the second predicted number of video sample tokens, and the second predicted number of audio sample tokens, and determine the third predicted number of audio sample tokens according to the second decoding function and the second predicted number of audio samples; calculate the second loss value according to the third predicted number of video sample tokens, the third predicted number of audio sample tokens, the first number, and the second number; determine whether the third loss value reaches the preset loss value; if the third loss value reaches the preset loss value, obtain a trained audio and video generation autoregressive model.

[0022] In an optional implementation of the second aspect, the device also includes a calculation module, which is used to calculate the video alignment number and the audio alignment number according to the first number and the second number, respectively, wherein the video alignment number represents the video and audio alignment, and the number of video sample tokens required for the video, and the audio alignment number represents the audio and video alignment, and the number of audio sample tokens required for the audio; a determination module is used to determine the first introduction number according to the video alignment number, and determine the second introduction number according to the audio alignment number.

[0023] In an optional implementation of the second aspect, the calculation module is specifically used to calculate the greatest common divisor of the first number and the second number; calculate the quotient of the greatest common divisor and the first number to obtain the video alignment number; calculate the quotient of the greatest common divisor and the second number to obtain the audio alignment number.

[0024] In a third aspect, the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes any optional method described in the first aspect when executing the computer program.

[0025] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs any optional method described in the first aspect.

[0026] In a fifth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, performs the steps of any optional method described in the first aspect.

[0027] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 A schematic diagram of a first process flow of the audio and video generation method provided in an embodiment of the present application;

[0030] Figure 2 A second flow chart of the audio and video generation method provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of the structure of audio and video token prediction provided in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of the structure of an audio and video generating device provided in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0034] Icons: 400 - acquisition module; 410 - input module; 420 - decoding module; 430 - training module; 440 - calculation module; 450 - determination module; 5 - electronic device; 501 - processor; 502 - memory; 503 - communication bus. DETAILED DESCRIPTION

[0035] The following embodiments of the technical solution of the present application will be described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0037] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0038] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0039] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0040] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0041] In the description of the embodiments of the present application, the technical terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the embodiments of the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the embodiments of the present application.

[0042] In the description of the embodiments of the present application, unless otherwise clearly specified and limited, technical terms such as "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0043] In the prior art, audio generation and video generation mostly generate corresponding audio or video from a given text. There are currently two commonly used frameworks: one is to generate audio and video separately through a diffusion model framework; the other is to generate audio and video separately through an autoregressive structure.

[0044] Currently, the method of generating audio and video through autoregressive structure generally adopts an interlaced generation method, for example, generating video first and then generating audio. This makes the generated video and audio have no direct connection, which in turn causes the generated audio and video to have problems of audio and video being out of sync and low quality.

[0045] Based on the above problems, the present application designs an audio and video generation method, an electronic device and a computer-readable storage medium, which process video and audio respectively through two-way autoregression. During the video processing process, when the number of processed video tokens reaches the number of tokens that require the introduction of audio information for decoding, the audio information is introduced into the video processing process, and the next number of video tokens is predicted, so that the autoregressive prediction process of the number of video tokens refers to the audio information; similarly, during the audio processing process, when the number of processed audio tokens reaches the number of tokens that require the introduction of video information for decoding, the video information is introduced into the audio processing process, and the next number of audio tokens is predicted, so that the autoregressive prediction process of the number of audio tokens refers to the video information, so that the obtained target video token number and target audio token number realize the association between video information and audio information, and then realize the correlation between the generated target video features and target audio features, so that the target video and audio generated by this scheme are highly synchronized, and thus the synchronization and quality of audio and video are higher.

[0046] Based on the above ideas, the present application first provides an audio and video generation method, which can be applied to a computing device, including but not limited to a computer, a server, a computing chip, a GPU, a CPU, etc. Figure 1 As shown, the audio and video generation method can be implemented by the following methods, including:

[0047] Step S100: Acquire target text information.

[0048] Step S110: input the target text information into a pre-trained audio and video generation autoregressive model to obtain the target video token number and the target audio token number output by the audio and video generation autoregressive model.

[0049] Step S120: Decode the target video token quantity to obtain the target video corresponding to the target text information; decode the target audio token quantity to obtain the target audio corresponding to the target text information.

[0050] In the above implementation, the target text information represents the text information to be converted into video and audio. For example, the target text may be a text description of a certain scenic spot, and the video content and audio content corresponding to the text description of the scenic spot need to be generated. The target text information may include target text features, which may be obtained by encoding the target text. Specifically, the target text features may be text encoding features obtained by LLM encoding the target text content.

[0051] In the case of obtaining the target text information by the above method, this scheme inputs the obtained target text information into the pre-trained audio and video generation autoregressive model, and outputs the target number of video tokens and the target number of audio tokens through the audio and video generation autoregressive model. Specifically, the target number of video tokens is generated by the audio and video generation autoregressive model according to the target text information, the first decoding function, the predicted number of video tokens and the predicted number of audio tokens, and the target number of audio tokens is generated by the audio and video generation autoregressive model according to the target text information, the second decoding function, the predicted number of video tokens and the predicted number of audio tokens. Among them, the predicted number of video tokens is generated by the audio and video generation autoregressive model according to the target text information and the first decoding function; the predicted number of audio tokens is generated by the audio and video generation autoregressive model according to the target text information and the second decoding function, the first decoding function and the second decoding function are pre-configured decoding functions, and the optimal parameters of the first decoding function and the second decoding function are obtained by training the autoregressive model. Among them, the number of tokens represents a representation, called a token, and one second of video and / or audio can be compressed, quantized and discretized into one or more tokens. Among them, since the video and audio adopt different compression and quantization methods, the number of tokens obtained is also different. For example, the video compression and quantization discretization can adopt the MAGVIT-V2 compression and quantization method, or the TITOK compression and quantization method, and the audio compression and quantization discretization can adopt DAC for compression and quantization. Specifically, assuming that the video adopts the TITOK compression and quantization method, in this case, 1 second of video can be compressed and quantized into 100 tokens, and 1 second of audio can be compressed and quantized into 50 tokens. It should be noted here that the specific values ​​of the above compression and quantization are only for the sake of ease of understanding as an example, and the specific values ​​shall be based on the values ​​obtained by the compression and quantization method.

[0052] Since the audio and video generation autoregressive model of the present scheme generates the target number of video tokens based on the target text information, the first decoding function, the predicted number of video tokens and the predicted number of audio tokens, the autoregressive prediction process of the target number of video tokens refers to the audio information; similarly, the audio and video generation autoregressive model of the present scheme generates the target number of audio tokens based on the target text information, the second decoding function, the predicted number of video tokens and the predicted number of audio tokens, therefore, the autoregressive prediction process of the target number of audio tokens refers to the video information, so that the target number of video tokens and the target number of audio tokens can realize the association between the video information and the audio information, and then realize the association between the generated target video features and the target audio features, so that the target video and audio generated by the present scheme are highly synchronized, and thus the synchronization and quality of the audio and video are higher.

[0053] When the target number of video tokens and the target number of audio tokens are obtained in the above manner, the present solution decodes the target number of video tokens to obtain the target video corresponding to the target text information, and decodes the target number of audio tokens to obtain the target audio corresponding to the target text information. Specifically, as a possible implementation, the present solution can respectively decode and restore the target number of video tokens through the compression and quantization modules corresponding to the video, thereby obtaining the target video, and decode and restore the target number of audio tokens through the compression and quantization models corresponding to the audio, thereby obtaining the target audio.

[0054] The audio and video generation method designed above, this scheme first obtains the target text information, and then based on the pre-acquired audio and video generation autoregressive model, outputs the corresponding target video token number and target audio token number according to the target text information, and then decodes the target video token number to obtain the target video corresponding to the target text information, and decodes the target audio token number to obtain the target audio corresponding to the target text information. Since the audio and video generation autoregressive model of this scheme generates the target video token number based on the target text information, the first decoding function, the predicted video token number and the predicted audio token number, therefore, the autoregressive prediction process of the target video token number refers to the audio information; similarly, the audio and video generation autoregressive model of this scheme generates the target audio token number based on the target text information, the second decoding function, the predicted video token number and the predicted audio token number, therefore, the autoregressive prediction process of the target audio token number refers to the video information, so that the target video token number and the target audio token number realize the association between the video information and the audio information, and then realize the association between the generated target video features and the target audio features, so that the target video and audio generated by this scheme are highly synchronized, and thus the synchronization and quality of the audio and video are higher.

[0055] In an optional implementation of this embodiment, the training process of the audio and video generation autoregressive model described above is as follows: Figure 2 As shown, including:

[0056] Step S200: obtaining a training set, where the training set includes a plurality of samples, each sample including text sample information and video sample information and audio sample information corresponding to the text sample information.

[0057] Step S210: compress and quantize each video sample information to obtain a first number of video sample tokens corresponding to each video sample information.

[0058] Step S220: compress and quantize each audio sample information to obtain a second number of audio sample tokens corresponding to each audio sample information.

[0059] Step S230: Train the preset autoregressive model according to multiple text sample information in the training set, a first number of video sample tokens corresponding to each text sample information, and a second number of audio sample tokens to obtain a trained audio and video generation autoregressive model.

[0060] In the above implementation, the training set may include multiple samples, each of which includes text sample information and video sample information and audio sample information corresponding to the text sample information. For example, the training set may include the text content of the explanation of each of the multiple scenic spots in a certain scenic spot, as well as the video explanation content and audio explanation content corresponding to each explanation text content. Among them, the text sample information may specifically be a text sample feature, and specifically may be a text sample feature obtained by LLM encoding the text sample content. The video sample information and the audio sample information may specifically be a video of a certain time and an audio of a certain time.

[0061] In the case where multiple samples are obtained in the above manner, this solution can compress and quantize each video sample information, thereby obtaining a first number of video sample tokens corresponding to each video sample information. Specifically, according to the above description, this solution can use the MAGVIT-V2 compression and quantization method or the TITOK compression and quantization method to compress and quantize each video sample information, thereby obtaining a first number of video sample tokens corresponding to each video sample information. For example, this solution can compress and quantize the video at time t into A video sample token.

[0062] Similarly, this solution can compress and quantize each audio sample information, so as to obtain a second number of audio sample tokens corresponding to each audio sample information. Specifically, according to the above description, this solution can use the DAC compression and quantization method to compress and quantize the audio sample information, so as to obtain a second number of audio sample tokens corresponding to the audio sample information. For example, this solution can compress and quantize the audio sample information at time t into An audio sample token.

[0063] On the basis of the above, this scheme trains the preset autoregressive model based on the text sample information of each sample, the first number of video sample tokens corresponding to the video sample information, and the second number of audio sample tokens corresponding to the audio sample information, so as to obtain an audio and video autoregressive generation model.

[0064] Specifically, the present scheme can first determine the first predicted number of video sample tokens according to the text sample information and the first decoding function through a preset autoregressive model, and determine the first predicted number of audio sample tokens according to the text sample information and the second decoding function; calculate the first loss value of the preset autoregressive model according to the first predicted number of video sample tokens, the second predicted number of audio sample tokens, the first number and the second number; determine whether the first loss value reaches the preset loss value; if the first loss value does not reach the preset loss value, determine whether the number of video sample tokens currently being processed reaches the first introduced number, and determine whether the number of audio sample tokens currently being processed reaches the second introduced number.

[0065] As a possible implementation, if it is determined that the number of video sample tokens currently being processed has not reached the first introduction number, and the number of audio sample tokens currently being processed has reached the second introduction number, the second predicted number of video sample tokens is determined based on the first decoding function and the first predicted number of video sample tokens, and the second predicted number of audio sample tokens is determined based on the second decoding function, the first predicted number of audio samples, and the first predicted number of video samples.

[0066] As another possible implementation, if it is determined that the number of video sample tokens currently being processed has reached the first introduction number, and the number of audio sample tokens currently being processed has not reached the second introduction number, the second predicted number of video sample tokens is determined based on the first predicted number of video sample tokens, the first decoding function, and the first predicted number of audio samples, and the second predicted number of audio sample tokens is determined based on the first predicted number of audio samples and the second decoding function.

[0067] As another possible implementation, if it is determined that the number of video sample tokens currently being processed reaches the first introduction number, and the number of audio sample tokens currently being processed reaches the second introduction number, the second predicted number of video sample tokens is determined based on the first predicted number of video sample tokens, the first decoding function, and the first predicted number of audio samples, and the second predicted number of audio sample tokens is determined based on the first predicted number of audio samples, the first predicted number of video samples, and the second decoding function.

[0068] When the second predicted video sample token number and the second predicted audio sample token number are obtained in the above manner, this scheme calculates the second loss value based on the second predicted video sample token number, the second predicted audio sample token number, the first number and the second number; determines whether the second loss value reaches the preset loss value; if the second loss value reaches the preset loss value, a trained audio and video generation autoregressive model is obtained.

[0069] If it is determined that the second loss value does not reach the preset loss value, repeat the aforementioned introduction quantity judgment step and sample token quantity judgment step to continue to obtain the third predicted video sample token quantity and the third predicted audio sample token quantity, and then calculate the second loss value based on the third predicted video sample token quantity, the third predicted audio sample token quantity, the first quantity and the second quantity; determine whether the third loss value reaches the preset loss value; if the third loss value reaches the preset loss value, obtain a trained audio and video generation autoregressive model.

[0070] The first introduction quantity described above indicates the number of video sample tokens required to introduce audio information for video decoding, and the second introduction quantity indicates the number of audio sample tokens required to introduce video information for audio decoding.

[0071] Specifically, the first introduction quantity and the second introduction quantity can be calculated in the following manner: the video alignment quantity and the audio alignment quantity are calculated according to the first quantity and the second quantity, respectively, and then the first introduction quantity is determined according to the video alignment quantity, and the second introduction quantity is determined according to the audio alignment quantity. The video alignment quantity indicates the number of video sample tokens required for the video to align with the audio, and the audio alignment quantity indicates the number of audio sample tokens required for the audio to align with the video.

[0072] Specifically, this solution can calculate the greatest common divisor of the first number and the second number, then calculate the quotient of the greatest common divisor and the first number to obtain the number of video alignments, and calculate the quotient of the greatest common divisor and the second number to obtain the number of audio alignments. For example, the above description of compressing and quantizing the video at time t is , i.e. j video sample tokens, compress and quantize the audio sample information at time t into That is, i audio sample tokens. In this case, this scheme can calculate the greatest common divisor o of i and j, and then calculate the quotient of o and i to get the number of audio alignments. , calculate the quotient of o and j to get the number of video alignments , and finally calculate j and The quotient of gets the first introduced quantity , calculate i and The quotient of the second introduced quantity is ,Right now , .

[0073] The above training process can be illustrated by the following example:

[0074] According to the above description, this solution can compress and quantize the samples so that each video sample information can obtain j video sample tokens, and each audio sample information can obtain i audio sample tokens. The first introduced quantity is calculated as , the second introduced quantity is In this case, when generating the k1th video sample token (where k1 is ) will add the audio information of the corresponding moment, so that when generating the video, there will be audio information to control it; similarly, when generating the k2th audio sample token (where k2 is ) will add the video information of the corresponding moment, so that the video information will be used to control the generation of audio sample tokens.

[0075] Specifically, for videos:

[0076] For example, when k1=0, ;

[0077] When k1>0 and k1 is Multiples of hour:

[0078] ;in, ;

[0079] Other cases: ;

[0080] Similarly, for audio:

[0081] For example, when k2=0, ;

[0082] When k2>0 and k2 is Multiples of hour:

[0083] ;in, ;

[0084] Other cases:

[0085] ;

[0086] Wherein, LLM-M() represents a decoding function, and the decoding function can adopt any existing decoding function.

[0087] The above process can be explained by the following example. is 2, is 1, in which case, for video:

[0088] When making the first prediction, that is, k1=0, k2=0:

[0089] This scheme obtains the first predicted video sample token number according to the text sample information and the first decoding function, that is, ;

[0090] This scheme obtains the first predicted audio sample token number based on the text sample information and the second decoding function, that is, .

[0091] When the second prediction is made, that is, k1=1, k2=1:

[0092] For video: Since k1=1, it is less than In this case, the scheme obtains the number of second predicted video sample tokens according to the first predicted video sample token and the first decoding function, that is, ;

[0093] For audio: Since k2=1, it is less than In this case, the present invention obtains the second predicted audio sample token quantity according to the first predicted audio sample token and the second decoding function, that is, ;

[0094] When the third prediction is made, that is, k1=2, k2=2:

[0095] For video: Since k1=2, it is less than In this case, the scheme obtains the number of third predicted video sample tokens according to the second predicted video sample token and the first decoding function, that is, ;

[0096] For audio, since k2=2, it is equal to In this case, the scheme introduces the second predicted video sample token number, and then obtains the third predicted audio sample token number according to the second predicted audio sample token number, the second predicted video sample token number and the second decoding function, that is, .

[0097] When the fourth prediction is made, that is, k1=3, k2=3:

[0098] For video, since k1=3, it is equal to A multiple of +1, in this case, the present scheme introduces a third predicted audio sample token number, and then obtains a fourth predicted video sample token number according to the third predicted video sample token number, the third predicted audio sample token number and the first decoding function;

[0099] For audio, since k2=3, it is not equal to A multiple of +1, in this case, this scheme obtains the fourth predicted audio sample token number according to the third predicted audio sample token number and the second decoding function.

[0100] The subsequent process is similar. When the k of the video or audio reaches the multiple of the corresponding greatest common divisor + 1, the number of predicted sample tokens of the audio or video information is introduced for prediction. When it does not reach the multiple of the corresponding greatest common divisor + 1, the number of tokens obtained by the previous prediction is used to continue the prediction. For example, Figure 3 As shown in the figure, the schematic diagram of the video or audio token number prediction structure, where x is the predicted number of video sample tokens or the predicted number of audio sample tokens, a / v_km represents the number of audio sample tokens or the number of video sample tokens obtained from the last interactive prediction, Cross Attention represents the cross attention mechanism, Self Attention represents the self-attention mechanism, FN represents the fully connected layer, and LN represents the regularization. Please continue to refer to Figure 3 When the k of the video or audio reaches the multiple of the corresponding greatest common divisor +1, the cross attention receives the data of a / v_km. At this time, a / v_km represents the number of audio sample tokens or video sample tokens obtained from the last prediction obtained by the interaction. If it does not reach the multiple of the corresponding greatest common divisor +1, a / v_km is 0, thus realizing the prediction.

[0101] It should be noted that after each prediction, this solution will calculate the loss value to determine whether the model parameters have been trained to meet the requirements.

[0102] Figure 4 The present application provides a schematic structural block diagram of an audio and video generating device. It should be understood that the device is applied to the electronic device described above. Figure 1 and 3The method embodiment executed in the embodiment corresponds to the method embodiment, and can execute the steps involved in the aforementioned method. The specific functions of the device can refer to the description above. To avoid repetition, the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or fixed in the operating system (OS) of the device. Specifically, the device includes: an acquisition module 400, an input module 410 and a decoding module 420, wherein the acquisition module 400 is used to acquire target text information, and the input module 410 is used to input the target text information into a pre-trained audio and video generation autoregressive model to obtain the target video token number and the target audio token number output by the audio and video generation autoregressive model, wherein the target video token number is generated by the audio and video generation autoregressive model according to the target text information, a first decoding function, a predicted video token number and a predicted audio token number; the target audio token number is generated by the audio and video generation autoregressive model according to the target text information, a second decoding function, a predicted video token number and a predicted audio token number; the predicted video token number is generated by the audio and video generation autoregressive model according to the target text information and the first decoding function; the predicted audio token number is generated by the audio and video generation autoregressive model according to the target text information and the second decoding function; the decoding module 420 is used to decode the target video token number to obtain the target video corresponding to the target text information; and decode the target audio token number to obtain the target audio corresponding to the target text information.

[0103] The audio and video generation device designed above, this scheme first obtains the target text information, and then based on the pre-acquired audio and video generation autoregressive model, outputs the corresponding target video token number and target audio token number according to the target text information, and then decodes the target video token number to obtain the target video corresponding to the target text information, and decodes the target audio token number to obtain the target audio corresponding to the target text information. Since the audio and video generation autoregressive model of this scheme generates the target video token number based on the target text information, the first decoding function, the predicted video token number and the predicted audio token number, the autoregressive prediction process of the target video token number refers to the audio information; similarly, the audio and video generation autoregressive model of this scheme generates the target audio token number based on the target text information, the second decoding function, the predicted video token number and the predicted audio token number, so the autoregressive prediction process of the target audio token number refers to the video information, so that the target video token number and the target audio token number realize the association between the video information and the audio information, and then realize the association between the generated target video features and the target audio features, so that the target video and audio generated by this scheme are highly synchronized, and then the synchronization and quality of the audio and video are higher.

[0104] According to some embodiments of the present application, the device also includes a training module 430, which is used to obtain a training set; wherein the training set includes multiple samples, each sample includes text sample information and video sample information and audio sample information corresponding to the text sample information; each video sample information is compressed and quantized to obtain a first number of video sample tokens corresponding to each video sample information; each audio sample information is compressed and quantized to obtain a second number of audio sample tokens corresponding to each audio sample information; a preset autoregressive model is trained according to multiple text sample information in the training set, the first number of video sample tokens corresponding to each text sample information, and the second number of audio sample tokens to obtain a trained audio and video generation autoregressive model.

[0105] According to some embodiments of the present application, the training module 430 is specifically used to determine the first predicted number of video sample tokens according to text sample information and a first decoding function through a preset autoregressive model, and determine the first predicted number of audio sample tokens according to text sample information and a second decoding function; calculate the first loss value of the preset autoregressive model according to the first predicted number of video sample tokens, the second predicted number of audio sample tokens, the first number and the second number; determine whether the first loss value reaches the preset loss value; if the first loss value does not reach the preset loss value, determine whether the number of video sample tokens currently being processed reaches the first introduction number, and determine whether the number of audio sample tokens currently being processed reaches the second introduction number, wherein the first introduction number indicates the video that needs to introduce audio information for video decoding. The number of sample tokens, the second introduction number represents the number of audio sample tokens that need to be introduced into the video information for audio decoding; if it is determined that the number of video sample tokens currently being processed does not reach the first introduction number, and the number of audio sample tokens currently being processed reaches the second introduction number, then the second predicted number of video sample tokens is determined according to the first decoding function and the first predicted number of video sample tokens, and the second predicted number of audio sample tokens is determined according to the second decoding function, the first predicted number of audio samples and the first predicted number of video samples; the second loss value is calculated according to the second predicted number of video sample tokens, the second predicted number of audio sample tokens, the first number and the second number; it is determined whether the second loss value reaches the preset loss value; if the second loss value reaches the preset loss value, a trained audio and video generation autoregressive model is obtained.

[0106] According to some embodiments of the present application, the training module 430 is also specifically used to determine the second predicted number of video sample tokens based on the first predicted number of video sample tokens, the first decoding function and the first predicted number of audio samples if it is determined that the number of video sample tokens currently being processed has reached the first introduction number and the number of audio sample tokens currently being processed has not reached the second introduction number, and to determine the second predicted number of audio sample tokens based on the first predicted number of audio samples and the second decoding function.

[0107] According to some embodiments of the present application, the training module 430 is also specifically used to determine the second predicted number of video sample tokens based on the first predicted number of video sample tokens, the first decoding function and the first predicted number of audio samples if it is determined that the number of video sample tokens currently being processed has reached the first introduced number and the number of audio sample tokens currently being processed has reached the second introduced number, and to determine the second predicted number of audio sample tokens based on the first predicted number of audio samples, the first predicted number of video samples and the second decoding function.

[0108] According to some embodiments of the present application, the training module 430 is also specifically used to determine whether the number of video sample tokens currently being processed reaches the first introduction number, and determine whether the number of audio sample tokens currently being processed reaches the second introduction number if it is determined that the second loss value does not reach the preset loss value; if it is determined that the number of video sample tokens currently being processed reaches the first introduction number, and the number of audio sample tokens currently being processed does not reach the second introduction number, then determine the third predicted number of video sample tokens according to the first decoding function, the second predicted number of video sample tokens, and the second predicted number of audio sample tokens, and determine the third predicted number of audio sample tokens according to the second decoding function and the second predicted number of audio samples; calculate the second loss value according to the third predicted number of video sample tokens, the third predicted number of audio sample tokens, the first number, and the second number; determine whether the third loss value reaches the preset loss value; if the third loss value reaches the preset loss value, obtain a trained audio and video generation autoregressive model.

[0109] According to some embodiments of the present application, the device also includes a calculation module 440, which is used to calculate the video alignment number and the audio alignment number according to the first number and the second number, respectively, wherein the video alignment number represents the video and audio alignment, and the number of video sample tokens required for the video, and the audio alignment number represents the audio and video alignment, and the number of audio sample tokens required for the audio; a determination module 450, which is used to determine the first introduction number according to the video alignment number, and determine the second introduction number according to the audio alignment number.

[0110] According to some embodiments of the present application, the calculation module 440 is specifically used to calculate the greatest common divisor of the first number and the second number; calculate the quotient of the greatest common divisor and the first number to obtain the video alignment number; calculate the quotient of the greatest common divisor and the second number to obtain the audio alignment number.

[0111] According to some embodiments of the present application, Figure 5 As shown, the present application provides an electronic device 5, including: a processor 501 and a memory 502, the processor 501 and the memory 502 are interconnected and communicate with each other through a communication bus 503 and / or other forms of connection mechanisms (not marked), the memory 502 stores a computer program executable by the processor 501, and when the computing device is running, the processor 501 executes the computer program to execute any optional implementation method, such as steps S100 to S120: obtaining target text information; inputting the target text information into a pre-trained audio and video generation autoregressive model to obtain a target video token number and a target audio token number output by the audio and video generation autoregressive model; decoding the target video token number to obtain a target video corresponding to the target text information; decoding the target audio token number to obtain a target audio corresponding to the target text information.

[0112] The present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method in any of the aforementioned optional implementations is executed.

[0113] Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0114] The present application provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the method in any optional implementation manner.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and they should all be included in the scope of the claims and specification of the present application. In particular, as long as there is no structural conflict, the various technical features mentioned in the various embodiments can be combined in any way. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions that fall within the scope of the claims.

Claims

1. A method for generating audio and video, characterized in that: The method comprises: Get target text information; Input the target text information into a pre-trained audio and video generation autoregressive model to obtain the target video token number and the target audio token number output by the audio and video generation autoregressive model; Decoding the target video token number to obtain the target video corresponding to the target text information; Decoding the target audio token quantity to obtain the target audio corresponding to the target text information; The target number of video tokens is generated by an audio and video generation autoregressive model according to target text information, a first decoding function, a predicted number of video tokens, and a predicted number of audio tokens; The target number of audio tokens is generated by an audio and video generation autoregressive model according to target text information, a second decoding function, a predicted number of video tokens, and a predicted number of audio tokens; The predicted number of video tokens is generated by an audio and video generation autoregressive model according to target text information and a first decoding function; The predicted number of audio tokens is generated by an audio and video generation autoregressive model according to target text information and a second decoding function; The audio and video generation autoregressive model is trained in the following way: Obtaining a training set; wherein the training set includes a plurality of samples, each sample including text sample information and video sample information and audio sample information corresponding to the text sample information; Compress and quantize each video sample information to obtain a first number of video sample tokens corresponding to each video sample information; Compress and quantize each audio sample information to obtain a second number of audio sample tokens corresponding to each audio sample information; Training a preset autoregressive model according to multiple text sample information in a training set, a first number of video sample tokens corresponding to each text sample information, and a second number of audio sample tokens to obtain a trained audio and video generation autoregressive model; The method trains a preset autoregressive model according to multiple text sample information in a training set, a first number of video sample tokens corresponding to each text sample information, and a second number of audio sample tokens to obtain a trained audio and video generation autoregressive model, including: Determine the first predicted number of video sample tokens according to the text sample information and the first decoding function through a preset autoregressive model, and determine the first predicted number of audio sample tokens according to the text sample information and the second decoding function; Calculate a first loss value of a preset autoregressive model according to the first predicted video sample token quantity, the second predicted audio sample token quantity, the first quantity, and the second quantity; Determining whether the first loss value reaches a preset loss value; If the first loss value does not reach the preset loss value, determine whether the number of video sample tokens currently being processed reaches the first introduction number, and determine whether the number of audio sample tokens currently being processed reaches the second introduction number, wherein the first introduction number represents the number of video sample tokens that need to introduce audio information for video decoding, and the second introduction number represents the number of audio sample tokens that need to introduce video information for audio decoding; If it is determined that the number of video sample tokens currently being processed has not reached the first introduction number, and the number of audio sample tokens currently being processed has reached the second introduction number, determining the second predicted number of video sample tokens according to the first decoding function and the first predicted number of video sample tokens, and determining the second predicted number of audio sample tokens according to the second decoding function, the first predicted number of audio samples, and the first predicted number of video samples; Calculate a second loss value based on the second predicted video sample token number, the second predicted audio sample token number, the first number, and the second number; Determining whether the second loss value reaches a preset loss value; If the second loss value reaches the preset loss value, a trained audio and video generation autoregressive model is obtained.

2. The method according to claim 1, characterized in that The method further comprises: If it is determined that the number of video sample tokens currently being processed has reached the first introduction number, and the number of audio sample tokens currently being processed has not reached the second introduction number, the second predicted number of video sample tokens is determined based on the first predicted number of video sample tokens, the first decoding function, and the first predicted number of audio samples, and the second predicted number of audio sample tokens is determined based on the first predicted number of audio samples and the second decoding function.

3. The method according to claim 1, characterized in that The method further comprises: If it is determined that the number of video sample tokens currently being processed reaches the first introduction number, and the number of audio sample tokens currently being processed reaches the second introduction number, the second predicted number of video sample tokens is determined based on the first predicted number of video sample tokens, the first decoding function, and the first predicted number of audio samples, and the second predicted number of audio sample tokens is determined based on the first predicted number of audio samples, the first predicted number of video samples, and the second decoding function.

4. The method according to claim 1, characterized in that: The method further comprises: If it is determined that the second loss value does not reach the preset loss value, determining whether the number of video sample tokens currently being processed reaches the first introduction number, and determining whether the number of audio sample tokens currently being processed reaches the second introduction number; If it is determined that the number of video sample tokens currently being processed has reached the first introduction number, and the number of audio sample tokens currently being processed has not reached the second introduction number, determining a third predicted number of video sample tokens according to the first decoding function, the second predicted number of video sample tokens, and the second predicted number of audio sample tokens, and determining a third predicted number of audio sample tokens according to the second decoding function and the second predicted number of audio samples; Calculate a third loss value based on the third predicted video sample token number, the third predicted audio sample token number, the first number, and the second number; Determining whether the third loss value reaches a preset loss value; If the third loss value reaches the preset loss value, a trained audio and video generation autoregressive model is obtained.

5. The method according to claim 1, characterized in that Before determining the first predicted video sample token quantity according to the text sample information and the first decoding function through the preset autoregressive model, and determining the first predicted audio sample token quantity according to the text sample information and the second decoding function, the method further includes: Calculate the number of video alignments and the number of audio alignments according to the first number and the second number, respectively, wherein the number of video alignments represents the number of video sample tokens required for the video to align the video and the audio, and the number of audio alignments represents the number of audio sample tokens required for the audio to align the audio and the video; The first introduction quantity is determined according to the video alignment number, and the second introduction quantity is determined according to the audio alignment number.

6. The method according to claim 5, characterized in that The step of respectively calculating the number of video alignments and the number of audio alignments according to the first number and the second number includes: Calculate the greatest common divisor of the first number and the second number; Calculate the quotient of the greatest common divisor and the first number to obtain the number of video alignments; The quotient of the greatest common divisor and the second number is calculated to obtain the number of audio alignments.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images