Low-resource audio subtitle generation methods, devices, electronic equipment and media

By using text data to train the language decoder in a multimodal pre-trained model and replacing it with an audio encoder, the problem of insufficient audio-subtitle data is solved, and high-precision audio subtitle generation is achieved.

CN117809654BActive Publication Date: 2024-11-12BEIJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202311593526.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-27
Publication Date
2024-11-12
Estimated Expiration
2043-11-27

AI Technical Summary

Technical Problem

In the prior art, due to insufficient sample size of the audio-subtitle data pair, the training accuracy of the audio subtitle generation model is not high, and accurate audio subtitles cannot be generated.

Method used

Using multimodal pre-training model, text data with sufficient sample size is used to train the language decoder, and the language encoder is replaced with an audio encoder, and the audio pairing data is fine-tuned to indirectly train to obtain a high-precision audio language decoder. , to generate audio subtitles.

Benefits of technology

在仅有少量音频-字幕数据的情况下,仍能训练得到较高精度的音频多模态预训练模型,实现了音频字幕生成的高效解决方案。

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117809654B_ABST
    Figure CN117809654B_ABST
Patent Text Reader

Abstract

The present application discloses a low-resource audio subtitle generation method, device, electronic device and medium. By applying the technical solution of the present application, in a multimodal pre-training model including a language encoder and an audio encoder, text data with a relatively sufficient sample size can be first used to train an existing language encoder to obtain a language decoder. The language encoder can then be replaced with an audio encoder to indirectly train a language decoder for the audio encoder. This allows a high-precision audio multimodal pre-training model to be trained using only audio pairing data with a small sample size. This achieves a technical solution that can achieve high model performance when there are only a small amount of audio-subtitle data pairs available for training.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to audio data processing technology, and in particular to a low-resource audio subtitle generation method, device, electronic device and medium. Background Art

[0002] Audio signals can provide people with rich information in daily life. Audio subtitles are a task that enables machines to accurately perceive the content of audio signals like humans. It is also a key step towards general artificial intelligence, enabling machines to have human-like auditory perception capabilities.

[0003] Traditional audio subtitle generation models usually include an audio encoder and an audio language decoder. The audio encoder encodes the input audio and sends it to the language decoder to generate a subtitle description of the corresponding audio. However, due to the fuzziness and timing characteristics of audio signals, in the process of training audio subtitle generation models in related technologies, it is usually impossible to obtain sufficient audio pairing data (i.e., audio-subtitle data) for training. This also leads to the problem that the audio subtitle generation model obtained in the prior art has low training accuracy due to low resources. As a result, it is impossible to generate accurate audio subtitles for users. Summary of the invention

[0004] The embodiments of the present application provide a low-resource audio subtitle generation method, device, electronic device and medium. The embodiments of the present application are used to solve the problem existing in the related art that a high-precision audio multimodal pre-training model cannot be trained due to the small amount of audio pairing data available for training.

[0005] According to one aspect of an embodiment of the present application, a low-resource audio subtitle generation method is provided, the method comprising:

[0006] Using a plurality of text data and the language encoder, a text language decoder is obtained;

[0007] The language encoder is replaced by the audio encoder, and the text language decoder is trained using a plurality of audio pairing data to obtain an audio language decoder, wherein each audio pairing data includes audio data and associated audio description text;

[0008] The audio encoder and the audio language decoder in the multimodal pre-trained model are used to generate corresponding audio subtitles for the audio to be recognized.

[0009] Optionally, in another embodiment of the method of the present application, the step of obtaining a text language decoder by using a plurality of text data and the language encoder includes:

[0010] Acquire multiple text data, perform word segmentation on the text data, and extract word embedding features from each text data; and,

[0011] Creating a corresponding initial text language decoder for the language encoder of the multimodal pre-trained model, wherein the initial text language decoder includes a mapping layer and a language model;

[0012] The initial text language decoder is trained by utilizing the word embedding features in each text data to obtain the text language decoder.

[0013] Optionally, in another embodiment of the method of the present application, the using of word embedding features in each text data to train the initial text language decoder to obtain the text language decoder includes:

[0014] Inputting each word embedding feature in each text data into the mapping layer in the initial text language decoder, so that the mapping layer maps the word embedding feature into a plurality of prefix vectors;

[0015] Inputting the plurality of prefix vectors into the language model in the initial text language decoder, so that the plurality of prefix vectors are reconstructed by the language model to obtain a reconstructed text;

[0016] A first cross entropy loss between the reconstructed text and the text data is calculated, and gradient back propagation is performed until the cross entropy loss is minimized, thereby determining to obtain the text language decoder.

[0017] Optionally, in another embodiment of the method of the present application, replacing the language encoder with the audio encoder includes:

[0018] The language encoder in the multimodal pre-training model is replaced by the audio encoder, and a coding-decoding relationship is established between the audio encoder and the text language decoder.

[0019] Optionally, in another embodiment of the method of the present application, after establishing the encoding / decoding relationship between the audio encoder and the text language decoder, the method further includes:

[0020] Inputting the audio data in the audio pairing data into the audio encoder to obtain audio embedding features corresponding to each audio data;

[0021] Inputting each audio embedding feature into the mapping layer in the text language decoder so that the mapping layer maps the audio embedding feature into a plurality of audio prefix vectors;

[0022] Inputting the plurality of audio prefix vectors into the language model in the text language decoder, so that the language model performs audio description on the plurality of audio prefix vectors to obtain a reconstructed audio description text;

[0023] The second cross entropy loss between the reconstructed audio description text and the audio description text is calculated, and gradient back propagation is performed until the first cross entropy loss is minimized, thereby determining to obtain the audio language decoder.

[0024] According to another aspect of the embodiment of the present application, a low-resource audio subtitle generation device is provided, the device comprising:

[0025] An acquisition module is configured to obtain a text language decoder using a plurality of text data and the language encoder;

[0026] a training module, configured to replace the language encoder with the audio encoder, and train the text language decoder using a plurality of audio pairing data to obtain an audio language decoder, wherein each audio pairing data comprises audio data and an associated audio description text;

[0027] The generation module is configured to generate corresponding audio subtitles for the audio to be recognized by using the audio encoder and the audio language decoder in the multimodal pre-training model.

[0028] According to another aspect of the embodiments of the present application, an electronic device is provided, including:

[0029] A memory for storing executable instructions; and

[0030] The display is used to execute the executable instructions with the memory to complete the operation of any of the above-mentioned low-resource audio subtitle generation methods.

[0031] According to another aspect of an embodiment of the present application, a computing device readable storage medium is provided for storing instructions readable by the computing device, wherein the instructions, when executed, perform the operations of any of the above-mentioned low-resource audio subtitle generation methods.

[0032] In the present application, a text language decoder can be obtained by using multiple text data and language encoders; the language encoder is replaced with an audio encoder, and the text language decoder is trained using multiple audio pairing data to obtain an audio language decoder, wherein each audio pairing data contains audio data and associated audio description text; and the audio language decoder in the multimodal pre-training model is used to generate corresponding audio subtitles for the audio to be recognized.

[0033] By applying the technical solution of the present application, in a multimodal pre-training model including a language encoder and an audio encoder, a language decoder can be firstly trained for an existing language encoder using text data with a relatively sufficient sample size. The language encoder can then be replaced with an audio encoder to indirectly train a language decoder for the audio encoder. This allows a high-precision audio multimodal pre-training model to be trained using only audio pairing data with a small sample size. This achieves a technical solution that can achieve high model performance when there is only a small amount of audio-subtitle data pairs available for training.

[0034] The technical solution of the present application is further described in detail below using multiple embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present application and, together with the description, serve to explain the principles of the present application.

[0036] The present application can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0037] Figure 1 A schematic diagram of a method for generating audio subtitles with low resources provided by an embodiment of the present application is shown;

[0038] Figure 2 The overall flow chart of a low-resource audio subtitle generation method provided by an embodiment of the present application is shown;

[0039] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown;

[0040] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown;

[0041] Figure 5 A schematic diagram of the structure of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0042] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present application.

[0043] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0044] The following description of at least one exemplary embodiment is merely illustrative in nature and is not intended to limit the present application, its application, or uses.

[0045] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0046] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0047] In addition, the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0048] It should be noted that all directional indications in the embodiments of the present application (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0049] Combine the following Figure 1-Figure 2 The following describes a method for generating audio subtitles with low resources according to an exemplary embodiment of the present application. It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principle of the present application, and the embodiments of the present application are not limited in this respect. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0050] Furthermore, the present application also proposes a low-resource audio subtitle generation method, device, moving vehicle and medium.

[0051] 2. Figure 1 The flowchart of a method for generating audio subtitles with low resources according to an embodiment of the present application is schematically shown. Figure 1 As shown, the method is applied to a multimodal pre-trained model including a language encoder and an audio encoder, including:

[0052] S101, using a plurality of text data and a language encoder, obtaining a text language decoder;

[0053] S102, replacing the language encoder with an audio encoder, and training a text language decoder using a plurality of audio pairing data to obtain an audio language decoder, wherein each audio pairing data includes audio data and associated audio description text;

[0054] S103, using the audio encoder and audio language decoder in the multimodal pre-training model to generate corresponding audio subtitles for the audio to be recognized.

[0055] In one embodiment, the multimodal pre-trained model in the embodiment of the present application may include a language encoder and an audio encoder, wherein the language encoder and the audio encoder may be language encoders and audio encoders already in other multimodal semantic models.

[0056] As an example, the multimodal pre-training model may be a comparative audio-text pre-training model (i.e., a CLAP model), or a guided audio-text pre-training model (i.e., a BLAT model), etc. It is understandable that the multimodal semantic model needs to include at least a language encoder and an audio encoder.

[0057] Among them, as one of the five sensory signals (vision, hearing, touch, smell, and taste), audio signals can provide users with rich information in daily life.

[0058] Furthermore, audio subtitles are a task that enables machines to accurately perceive the content of audio signals like humans, and are also a key step towards general artificial intelligence, enabling machines to have human-like auditory perception capabilities. In daily life, it has great application prospects, for example, it can help people with disabilities integrate into society more naturally, and serve as a life assistant to remind users.

[0059] At present, audio subtitle generation methods based on deep learning have made great progress. For example, a multimodal pre-trained model can encapsulate multimodal signals through an encoder deployed on the model and use a decoder to generate subtitle sentences about the signal.

[0060] In the related art, due to the fuzziness and temporal characteristics of audio signals, it is very challenging to obtain sufficient audio-subtitle data pairs for training in the audio subtitle generation task. Especially in some low-resource scenarios (such as audio subtitle generation in specific scenarios and special styles), it is often not feasible to collect enough audio-subtitle pairs.

[0061] In other words, in these scenarios, traditional methods often fail to train multimodal pre-trained models to converge due to insufficient available training data, making it difficult to generate satisfactory subtitles.

[0062] Based on the above existing problem that the multimodal pre-training model cannot be fully trained due to insufficient sample size of audio-caption data, the present application proposes a method for improving the convergence of the multimodal pre-training model by using text data with sufficient sample size.

[0063] like Figure 2 As shown, taking the multimodal pre-trained model as the existing language encoder and audio encoder in the CLAP model as an example, a low-resource audio subtitle generation method proposed in an embodiment of the present application is exemplarily described:

[0064] Step 1: Acquire multiple text data, perform word segmentation on the text data, extract word embedding features in each text data; and create a corresponding initial text language decoder for the language encoder of the multimodal pre-trained model.

[0065] The initial text language decoder includes a mapping layer and a language model.

[0066] In one approach, the CLAP pre-trained model contains two encoders: the CLAP language encoder and the CLAP audio encoder, which constructs a multimodal semantic space through contrastive learning.

[0067] In one approach, the embodiment of the present application needs to first initialize the CLAP model to obtain the language encoder and audio encoder of the CLAP model, and also needs to initialize the parameters of a language decoder in the multimodal pre-training model proposed in the embodiment of the present application.

[0068] Furthermore, the embodiment of the present application also needs to perform word segmentation operations on the text data to extract word embedding features in each text data.

[0069] Step 2: Input each word embedding feature in each text data into the mapping layer in the initial text language decoder, so that the mapping layer maps the word embedding feature into multiple prefix vectors.

[0070] It is understandable that it is relatively difficult to obtain audio pairing data (i.e., audio data and its corresponding subtitle description data), but it is relatively easy to obtain pure text data (for example, a large amount of text data can be easily obtained from blogs and e-books). Therefore, the embodiment of the present application can use a large amount of text data to train the language encoder in the multimodal pre-training model to obtain the corresponding language decoder.

[0071] In one embodiment, the present application may first initialize the CLAP model to obtain the CLAP model Enc t (•) Audio encoder Enc a (•) and the language encoder Enc t (•); and preprocess the Audio to obtain the acoustic feature A, perform word segmentation on the text data Caption and Text, and extract their word embedding features X = (x1,…x L ). Where L is the total number of words.

[0072] In one approach, the embodiment of the present application requires using the word embedding feature X to train a text language decoder (ie, a decoder corresponding to a language encoder).

[0073] As an example, the language decoder consists of two parts: a mapping layer and a language model. Through training, the language decoder can acquire the ability to reconstruct text from a multimodal semantic space.

[0074] Step 3: Input the multiple prefix vectors into the multimodal pre-trained model in the initial text language decoder, so that the multimodal pre-trained model reconstructs the multiple prefix vectors to obtain reconstructed text.

[0075] In one embodiment, the present application can extract word embedding features X of input text data and input them into the language encoder Enc of the CLAP model. t (•) to obtain text semantic embedding And initialize the language decoder parameters.

[0076] Furthermore, the embodiment of the present application can obtain an initial language decoder, wherein the initial text language decoder includes two parts: a mapping layer and a language model.

[0077] Among them, the mapping layer F is used to embed the semantics into Enc t (X) is mapped into K prefix vectors, namely:

[0078]

[0079] Among them, the prefix vector m corresponding to each text t The same dimension as word embedding, the combination of all text prefix vectors is the text prefix M t . Then each text prefix M t The language model is fed into the initial text language decoder so that the input text can be reconstructed from it.

[0080] Step 4: Calculate the first cross entropy loss between the reconstructed text and the text data, and perform gradient back propagation until the cross entropy loss is minimized, and then determine to obtain a text language decoder.

[0081] In one embodiment, the present application embodiment needs to calculate the first cross entropy loss between the reconstructed text and the text data. As an example, the expression of the first cross entropy loss is as follows:

[0082]

[0083] In one embodiment, the present application can perform gradient back propagation to minimize the loss until the model is trained to converge, thereby determining a text language decoder (i.e., a decoder obtained after training the initial text language decoder).

[0084] Step 5: Replace the language encoder in the multimodal pre-trained model with an audio encoder, and establish a codec relationship between the audio encoder and the text language decoder.

[0085] In one embodiment, after training the language encoder to obtain the text language decoder, the embodiment of the present application needs to transfer the text language decoder to the audio encoder for use. t (•) Replaced with the audio encoder Enc of the CLAP model a (·). This allows us to solve the problem of feature differences between different modalities in the multimodal semantic space by fine-tuning the mapping layer with a small learning rate and freezing the parameters of the language model during the training process.

[0086] Step 6: Input the audio data in the audio pairing data into the audio encoder to obtain audio embedding features corresponding to each audio data; and input each audio embedding feature into a mapping layer in the text language decoder so that the mapping layer maps the audio embedding features into multiple audio prefix vectors.

[0087] Step 7: Input the multiple audio prefix vectors into the multimodal pre-trained model in the text language decoder, so that the multimodal pre-trained model performs audio description on the multiple audio prefix vectors to obtain reconstructed audio description text.

[0088] In one embodiment, the present application can extract the audio data in the audio pairing data and send the acoustic feature A in the audio data to the audio encoder Enc of the CLAP model. a (·) to obtain audio embedding features

[0089] Furthermore, the embodiment of the present application can also embed the CLAP audio into the feature Enc a (A) Input to the text language decoder, so that after the mapping layer of the text language decoder, the audio prefix vector M is obtained a =F(Enc a (A)), and then feed the audio prefix vector into the language model to recover the reconstructed audio description text corresponding to the input audio data.

[0090] Step 8: Calculate the second cross entropy loss between the reconstructed audio description text and the audio description text, and perform gradient back propagation until the second cross entropy loss is minimized, thereby determining the audio language decoder in the multimodal pre-training model.

[0091] In one embodiment, the present application embodiment needs to calculate the second cross entropy loss between the reconstructed audio description text and the audio description text in the audio pairing data. As an example, the expression of the second cross entropy loss is as follows:

[0092]

[0093] Step 9: Use the audio encoder and audio language decoder in the multimodal pre-training model to generate corresponding audio subtitles for the audio to be recognized.

[0094] By applying the technical solution of the present application, in a multimodal pre-training model including a language encoder and an audio encoder, a language decoder can be firstly trained for an existing language encoder using text data with a relatively sufficient sample size. The language encoder can then be replaced with an audio encoder to indirectly train a language decoder for the audio encoder. This allows a high-precision audio multimodal pre-training model to be trained using only audio pairing data with a small sample size. This achieves a technical solution that can achieve high model performance when there is only a small amount of audio-subtitle data pairs available for training.

[0095] Optionally, in another embodiment of the method of the present application, the step of obtaining a text language decoder by using a plurality of text data and the language encoder includes:

[0096] Acquire multiple text data, perform word segmentation on the text data, and extract word embedding features from each text data; and,

[0097] Creating a corresponding initial text language decoder for the language encoder of the multimodal pre-trained model, wherein the initial text language decoder includes a mapping layer and a language model;

[0098] The initial text language decoder is trained by utilizing the word embedding features in each text data to obtain the text language decoder.

[0099] Optionally, in another embodiment of the method of the present application, the using of word embedding features in each text data to train the initial text language decoder to obtain the text language decoder includes:

[0100] Inputting each word embedding feature in each text data into the mapping layer in the initial text language decoder, so that the mapping layer maps the word embedding feature into a plurality of prefix vectors;

[0101] Inputting the plurality of prefix vectors into the language model in the initial text language decoder, so that the plurality of prefix vectors are reconstructed by the language model to obtain a reconstructed text;

[0102] A first cross entropy loss between the reconstructed text and the text data is calculated, and gradient back propagation is performed until the cross entropy loss is minimized, thereby determining to obtain the text language decoder.

[0103] Optionally, in another embodiment of the method of the present application, replacing the language encoder with the audio encoder includes:

[0104] The language encoder in the multimodal pre-training model is replaced by the audio encoder, and a coding-decoding relationship is established between the audio encoder and the text language decoder.

[0105] Optionally, in another embodiment of the method of the present application, after establishing the encoding / decoding relationship between the audio encoder and the text language decoder, the method further includes:

[0106] Inputting the audio data in the audio pairing data into the audio encoder to obtain audio embedding features corresponding to each audio data;

[0107] Inputting each audio embedding feature into the mapping layer in the text language decoder so that the mapping layer maps the audio embedding feature into a plurality of audio prefix vectors;

[0108] Inputting the plurality of audio prefix vectors into the language model in the text language decoder, so that the language model performs audio description on the plurality of audio prefix vectors to obtain a reconstructed audio description text;

[0109] The second cross entropy loss between the reconstructed audio description text and the audio description text is calculated, and gradient back propagation is performed until the first cross entropy loss is minimized, thereby determining to obtain the audio language decoder.

[0110] In another embodiment of the present application, Figure 3 As shown, the present application also provides a low-resource audio subtitle generation device. It includes:

[0111] The acquisition module 201 is configured to obtain a text language decoder using a plurality of text data and the language encoder;

[0112] The training module 202 is configured to replace the language encoder with the audio encoder and train the text language decoder using a plurality of audio pairing data to obtain an audio language decoder, wherein each audio pairing data includes audio data and associated audio description text;

[0113] The generation module 203 is configured to generate corresponding audio subtitles for the audio to be recognized by using the audio encoder and the audio language decoder in the multimodal pre-training model.

[0114] By applying the technical solution of the present application, in a multimodal pre-training model including a language encoder and an audio encoder, a language decoder can be firstly trained for an existing language encoder using text data with a relatively sufficient sample size. The language encoder can then be replaced with an audio encoder to indirectly train a language decoder for the audio encoder. This allows a high-precision audio multimodal pre-training model to be trained using only audio pairing data with a small sample size. This achieves a technical solution that can achieve high model performance when there is only a small amount of audio-subtitle data pairs available for training.

[0115] In another embodiment of the present application, the training module 202 is configured as follows:

[0116] Acquire multiple text data, perform word segmentation on the text data, and extract word embedding features from each text data; and,

[0117] Creating a corresponding initial text language decoder for the language encoder of the multimodal pre-trained model, wherein the initial text language decoder includes a mapping layer and a language model;

[0118] The initial text language decoder is trained by utilizing the word embedding features in each text data to obtain the text language decoder.

[0119] In another embodiment of the present application, the training module 202 is configured as follows:

[0120] Inputting each word embedding feature in each text data into the mapping layer in the initial text language decoder, so that the mapping layer maps the word embedding feature into a plurality of prefix vectors;

[0121] Inputting the plurality of prefix vectors into the language model in the initial text language decoder, so that the plurality of prefix vectors are reconstructed by the language model to obtain a reconstructed text;

[0122] A first cross entropy loss between the reconstructed text and the text data is calculated, and gradient back propagation is performed until the cross entropy loss is minimized, thereby determining to obtain the text language decoder.

[0123] In another embodiment of the present application, the training module 202 is configured as follows:

[0124] The language encoder in the multimodal pre-training model is replaced by the audio encoder, and a coding-decoding relationship is established between the audio encoder and the text language decoder.

[0125] In another embodiment of the present application, the training module 202 is configured as follows:

[0126] Inputting the audio data in the audio pairing data into the audio encoder to obtain audio embedding features corresponding to each audio data;

[0127] Inputting each audio embedding feature into the mapping layer in the text language decoder so that the mapping layer maps the audio embedding feature into a plurality of audio prefix vectors;

[0128] Inputting the plurality of audio prefix vectors into the language model in the text language decoder, so that the language model performs audio description on the plurality of audio prefix vectors to obtain a reconstructed audio description text;

[0129] The second cross entropy loss between the reconstructed audio description text and the audio description text is calculated, and gradient back propagation is performed until the first cross entropy loss is minimized, thereby determining to obtain the audio language decoder.

[0130] The present application also provides an electronic device to perform the above-mentioned low-resource audio subtitle generation method. Figure 4 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 4 As shown, the electronic device 3 includes: a processor 300, a memory 301, a bus 302 and a communication interface 303, wherein the processor 300, the communication interface 303 and the memory 301 are connected via the bus 302; the memory 301 stores a computer program that can be run on the processor 300, and when the processor 300 runs the computer program, the low-resource audio subtitle generation method provided in any of the aforementioned embodiments of the present application is executed.

[0131] The memory 301 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 303 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0132] The bus 302 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 301 is used to store programs, and the processor 300 executes the programs after receiving the execution instruction. The video transmission method disclosed in any implementation of the embodiment of the present application may be applied to the processor 300, or implemented by the processor 300.

[0133] The processor 300 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 300. The above processor 300 can be a general-purpose processor, including a processor (Central Processing Unit, referred to as CPU), a network processor (Network Processor, referred to as NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 301, and the processor 300 reads the information in the memory 301 and completes the steps of the above method in combination with its hardware.

[0134] The electronic device provided in the embodiment of the present application and the low-resource audio subtitle generation method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented therein.

[0135] The present application also provides a computer-readable storage medium corresponding to the low-resource audio subtitle generation method provided in the above embodiment. Figure 5 The computer-readable storage medium shown is a CD 40 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the video transmission method provided by any of the aforementioned embodiments will be executed.

[0136] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0137] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the video transmission method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0138] It should be noted that:

[0139] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0140] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0141] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0142] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A low-resource audio subtitle generation method, characterized in that: The method is applied to a multimodal pre-training model including a language encoder and an audio encoder, and includes: Using multiple text data and the language encoder, a text language decoder is obtained, including: obtaining multiple text data, performing word segmentation operation on the text data, and extracting word embedding features in each text data; and creating a corresponding initial text language decoder for the language encoder of the multimodal pre-training model, wherein the initial text language decoder includes a mapping layer and a language model; using the word embedding features in each text data, the initial text language decoder is trained to obtain the text language decoder, including: inputting the word embedding features in each text data into the mapping layer in the initial text language decoder, so that the mapping layer maps the word embedding features into multiple prefix vectors; inputting the multiple prefix vectors into the language model in the initial text language decoder, so that the language model reconstructs the multiple prefix vectors to obtain reconstructed text; calculating the first cross entropy loss between the reconstructed text and the text data, and performing gradient back propagation until the first cross entropy loss is minimized, and then determining to obtain the text language decoder; Replacing the language encoder with the audio encoder comprises: replacing the language encoder in the multimodal pre-trained model with the audio encoder, and establishing a codec relationship between the audio encoder and the text language decoder; and training the text language decoder with a plurality of audio pairing data to obtain an audio language decoder, wherein each audio pairing data comprises audio data and an associated audio description text; wherein training the text language decoder with a plurality of audio pairing data to obtain an audio language decoder comprises: inputting the audio data in the audio pairing data into the audio encoder to obtain audio embedding features corresponding to each audio data; inputting each audio embedding feature into the mapping layer in the text language decoder so that the mapping layer maps the audio embedding feature into a plurality of audio prefix vectors; inputting the plurality of audio prefix vectors into the language model in the text language decoder so that the language model performs audio description on the plurality of audio prefix vectors to obtain a reconstructed audio description text; calculating a second cross entropy loss between the reconstructed audio description text and the audio description text, and performing gradient back propagation until the second cross entropy loss is minimized, and then determining that the audio language decoder is obtained; The audio encoder and the audio language decoder in the multimodal pre-trained model are used to generate corresponding audio subtitles for the audio to be recognized.

2. A low-resource audio subtitle generation device, characterized in that: The device is applied to a multimodal pre-training model including a language encoder and an audio encoder, and includes: The acquisition module is configured to obtain a text language decoder using a plurality of text data and the language encoder; the acquisition module is specifically used to perform the following steps: obtain a plurality of text data, perform word segmentation on the text data, and extract word embedding features in each text data; and create a corresponding initial text language decoder for the language encoder of the multimodal pre-training model, wherein the initial text language decoder includes a mapping layer and a language model; train the initial text language decoder using the word embedding features in each text data to obtain the text language decoder, including: inputting the word embedding features in each text data into the mapping layer in the initial text language decoder, so that the mapping layer maps the word embedding features into a plurality of prefix vectors; inputting the plurality of prefix vectors into the language model in the initial text language decoder, so that the language model reconstructs the plurality of prefix vectors to obtain a reconstructed text; calculating a first cross entropy loss between the reconstructed text and the text data, and performing gradient back propagation until the first cross entropy loss is minimized, and then determining to obtain the text language decoder; A training module is configured to replace the language encoder with the audio encoder, and train the text language decoder using a plurality of audio pairing data to obtain an audio language decoder, wherein each audio pairing data includes audio data and an associated audio description text; wherein the audio language decoder is trained using a plurality of audio pairing data to obtain the audio language decoder, including: inputting the audio data in the audio pairing data into the audio encoder to obtain audio embedding features corresponding to each audio data; inputting each audio embedding feature into the mapping layer in the text language decoder, so that the mapping layer maps the audio embedding feature into a plurality of audio prefix vectors; inputting the plurality of audio prefix vectors into the language model in the text language decoder, so that the language model performs audio description on the plurality of audio prefix vectors to obtain a reconstructed audio description text; calculating a second cross entropy loss between the reconstructed audio description text and the audio description text, and performing gradient back propagation until the second cross entropy loss is minimized, and then determining to obtain the audio language decoder; The generation module is configured to generate corresponding audio subtitles for the audio to be recognized by using the audio encoder and the audio language decoder in the multimodal pre-training model.

3. An electronic device, characterized in that: include: A memory for storing executable instructions; as well as, A processor is used to execute the executable instructions with the memory to complete the operation of the low-resource audio subtitle generation method of claim 1.

4. A computer-readable storage medium for storing computer-readable instructions, characterized in that: When the instructions are executed, the operations of the low-resource audio subtitle generation method of claim 1 are performed.

Citation Information

Cited By

  • Subtitle matching display method and system based on audio file processing

    CN119922372A

  • Subtitle matching and display method and system based on audio file processing

    CN119922372B