Audio and video core content processing method and related device

Abstract

The application discloses a core content processing method and related device of audio and video. The method comprises the following steps: inputting a first visual feature sequence of a key frame sequence in a sample audio and video and a first audio feature sequence of an audio frame sequence of the sample audio and video into a cross-modal projection layer in a preset generation model, and outputting a second visual feature sequence and a second audio feature sequence conforming to a text embedding dimension of the preset generation model; inputting the second visual feature sequence, the second audio feature sequence and a prompt word feature of a core content generation prompt word into an attention layer and a low-rank adapter in the preset generation model, and outputting a first fusion feature sequence; inputting the first fusion feature sequence into a generation layer in the preset generation model, and outputting a predicted core content of the sample audio and video; and based on a difference between the predicted core content and a sample core content corresponding to the sample audio and video, fine-tuning and training the cross-modal projection layer and the low-rank adapter to obtain a core content generation model.

Description

Technical Field

[0001] This application relates to the field of audio and video processing technology, and in particular to a method and related apparatus for processing core audio and video content. Background Technology

[0002] We are currently in an era of information explosion, with a massive increase in audio and video content. On audio and video platforms, users typically filter out audio and video content that interests them based on its core content, such as the title. Therefore, showcasing good core content in audio and video can significantly improve their exposure and click-through rates.

[0003] In related technologies, the core content processing method for audio and video is to simply fuse visual and audio features in the audio and video to obtain fused features, and then generate the core content of the audio and video under the guidance of core content generation prompts through a pre-trained generation model.

[0004] However, the fused features obtained by simply fusing visual and audio features in audio and video are difficult to accurately represent the relationship between visual and audio information in audio and video. This makes it difficult for the core content generated for audio and video to accurately reflect the core of audio and video, and the training cost of the pre-trained generation model is high. Summary of the Invention

[0005] To address the aforementioned technical issues, this application provides a core content processing method and related apparatus for audio and video. Through feature projection of a cross-modal projection layer, the feature dimensions of both visual and audio features are aligned to the text embedding dimension of a pre-defined generation model, eliminating feature dimension differences between different modal features and laying a structurally compatible foundation for multimodal feature fusion. Feature fusion based on an attention layer achieves deep fusion of visual and audio features to fully explore the deep correlations between different modal features. A low-rank adapter is introduced to provide a technical basis for reducing the number of training parameters during model training. Fine-tuning the training of the cross-modal projection layer and the low-rank adapter eliminates the need to train the attention layer and generation layer, significantly reducing the number of training parameters and lowering training costs. Furthermore, this enables the core content generation model to generate core content efficiently and accurately for audio and video.

[0006] The embodiments of this application disclose the following technical solutions: On the one hand, embodiments of this application provide a method for processing core audio and video content, the method comprising: By using the cross-modal projection layer in the preset generation model, feature projection is performed on the first visual feature sequence of the keyframe sequence in the sample audio and video, and the first audio feature sequence of the audio frame sequence of the sample audio and video, to obtain a second visual feature sequence and a second audio feature sequence that conform to the text embedding dimension of the preset generation model. The second visual feature sequence, the second audio feature sequence, and the prompt word features of the core content generation prompt words are fused through the attention layer and low-rank adapter in the preset generation model to obtain the first fused feature sequence. The core content of the sample audio and video is generated by generating the first fused feature sequence through the generation layer in the preset generation model. Based on the difference between the predicted core content and the sample core content corresponding to the sample audio and video, the cross-modal projection layer and the low-rank adapter are fine-tuned and trained to obtain the core content generation model.

[0007] On the other hand, embodiments of this application provide an audio and video core content processing device, the device comprising: a feature projection unit, a feature fusion unit, a core content generation unit, and a fine-tuning training unit; The feature projection unit is used to project the first visual feature sequence of the keyframe sequence in the sample audio and video and the first audio feature sequence of the audio frame sequence in the sample audio and video through the cross-modal projection layer in the preset generation model, so as to obtain a second visual feature sequence and a second audio feature sequence that conform to the text embedding dimension of the preset generation model. The feature fusion unit is used to perform feature fusion on the second visual feature sequence, the second audio feature sequence, and the prompt word features of the core content prompt words through the attention layer and low-rank adapter in the preset generation model to obtain a first fused feature sequence; The core content generation unit is used to generate core content from the first fused feature sequence through the generation layer in the preset generation model, so as to obtain the predicted core content of the sample audio and video. The fine-tuning training unit is used to fine-tune the cross-modal projection layer and the low-rank adapter based on the difference between the predicted core content and the sample core content corresponding to the sample audio and video, so as to obtain the core content generation model.

[0008] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory: The memory is used to store computer programs and to transfer the computer programs to the processor; The processor is configured to execute the method described in any of the foregoing aspects according to instructions in the computer program.

[0009] On the other hand, embodiments of this application provide a computer-readable storage medium for storing a computer program that, when run on a computer device, causes the computer device to perform the methods described in any of the foregoing aspects.

[0010] On the other hand, embodiments of this application provide a computer program product, including a computer program that, when run on a computer device, causes the computer device to perform the method described in any of the foregoing aspects.

[0011] As can be seen from the above technical solution, the first visual feature sequence of the keyframe sequence in the sample audio and video, and the first audio feature sequence of the audio frame sequence in the sample audio and video are input into the cross-modal projection layer in the preset generation model for feature projection, and the output is a second visual feature sequence and a second audio feature sequence that conform to the text embedding dimension of the preset generation model. Through the feature projection of the cross-modal projection layer, the feature dimensions of the visual features and the feature dimensions of the audio features are aligned to the text embedding dimension of the preset generation model, eliminating the feature dimension differences between different modal features, and laying a structurally compatible foundation for multimodal feature fusion.

[0012] The second visual feature sequence, the second audio feature sequence, and the prompt word features generated from the core content are input into the attention layer and low-rank adapter in the preset generation model for feature fusion, and the first fused feature sequence is output. The feature fusion based on the attention layer realizes the deep fusion of visual and audio features to fully explore the deep correlation between features of different modalities. The introduction of the low-rank adapter provides a technical basis for reducing the number of training parameters when training the model.

[0013] The first fused feature sequence is input into the generation layer of the preset generation model. Based on the powerful generation capability of the preset generation model, the core content generation task is defined, and the predicted core content of the sample audio and video is output, so that the predicted core content accurately reflects the core of the sample audio and video. Based on the difference between the predicted core content and the sample core content corresponding to the sample audio and video, the cross-modal projection layer and low-rank adapter are fine-tuned and trained, so that the preset generation model learns to output the sample core content for the sample audio and video, thus obtaining the core content generation model. There is no need to train the attention layer and the generation layer, which greatly reduces the number of training parameters, thereby reducing the training cost, and enables the core content generation model to generate core content for audio and video efficiently and accurately. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1A schematic diagram of a computer system for processing core audio and video content according to an embodiment of this application; Figure 2 A schematic diagram illustrating an application scenario of a core audio and video content processing method provided in an embodiment of this application; Figure 3 A flowchart illustrating a core content processing method for audio and video provided in this application embodiment; Figure 4 A schematic diagram illustrating a method for training a preset generation model to obtain a core content generation model, as provided in an embodiment of this application; Figure 5 A schematic diagram illustrating how a first fused feature sequence is obtained by fusing a second visual feature sequence, a second audio feature sequence, and cue word features, as provided in an embodiment of this application; Figure 6 A schematic diagram illustrating another method for obtaining a first fused feature sequence by fusing a second visual feature sequence, a second audio feature sequence, and cue word features, as provided in an embodiment of this application; Figure 7 This is a schematic diagram illustrating how a core content generation model generates multiple candidate core contents for a target audio / video file, as provided in an embodiment of this application. Figure 8 A schematic diagram illustrating a method for obtaining a second fused feature sequence by fusing a third visual feature sequence, a third audio feature sequence, and cue word features, as provided in an embodiment of this application; Figure 9 A schematic diagram illustrating another method for obtaining a second fused feature sequence by fusing a third visual feature sequence, a third audio feature sequence, and cue word features, as provided in an embodiment of this application; Figure 10 This is a schematic diagram illustrating how to determine target core content from multiple candidate core content for target audio and video, as provided in an embodiment of this application. Figure 11 This is a schematic diagram illustrating another method for determining target core content from multiple candidate core content for target audio and video, provided as an embodiment of this application. Figure 12 A structural diagram of an audio and video core content processing device provided in an embodiment of this application; Figure 13 A structural diagram of a server provided in an embodiment of this application; Figure 14 This is a structural diagram of a terminal provided in an embodiment of this application. Detailed Implementation

[0016] The embodiments of this application will now be described with reference to the accompanying drawings.

[0017] Currently, effectively showcasing core content in audio and video can significantly improve their exposure and click-through rates. This core content is typically generated by simply fusing visual and audio features from the audio and video to create a fused feature. This fused feature is then generated by a pre-trained generative model guided by prompts related to the core content. However, this simple fusion of visual and audio features fails to accurately represent the relationship between visual and audio information within the audio and video. Consequently, the generated core content may not accurately reflect the core essence of the audio and video, and the pre-trained generative model has a high training cost.

[0018] This application provides a method for processing core content in audio and video. The method involves inputting the first visual feature sequence of keyframes and the first audio feature sequence of audio frames from a sample audio / video sample into a cross-modal projection layer of a preset generation model for feature projection. The output is a second visual feature sequence and a second audio feature sequence that conform to the text embedding dimension of the preset generation model, aligning the feature dimensions of both the visual and audio features to the text embedding dimension of the preset generation model. The second visual feature sequence, the second audio feature sequence, and the prompt word features of the core content generation prompt words are then input into the attention layer and low-rank adapter of the preset generation model for feature fusion. The first fused feature sequence is output to fully explore the deep correlation between features of different modalities. A low-rank adapter is introduced to reduce the number of training parameters. The first fused feature sequence is input into the generation layer of the preset generation model to generate core content and output the predicted core content of the sample audio and video, so that the predicted core content accurately reflects the core of the sample audio and video. Based on the difference between the predicted core content and the sample core content corresponding to the sample audio and video, the cross-modal projection layer and the low-rank adapter are fine-tuned to obtain the core content generation model. There is no need to train the attention layer and the generation layer, which greatly reduces the number of training parameters, thereby reducing the training cost. The core content generation model can generate core content for audio and video efficiently and accurately.

[0019] To facilitate understanding of the core audio and video content processing method provided in the embodiments of this application, the computer system of the core audio and video content processing method will be described first below.

[0020] See Figure 1 This figure is a schematic diagram of a computer system for a core audio and video content processing method provided in an embodiment of this application. The computer system 100 includes multiple devices, such as multiple terminals 1400 and multiple servers 1300. The terminals 1400 and the servers 1300 can communicate with each other through a communication network, and the servers 1300 obtain the data they need from the database 1500.

[0021] The communication network uses standard communication technologies and / or protocols, typically the Internet, but can also be any network, including but not limited to Bluetooth, local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, private network, or any combination of virtual private network. In some embodiments, custom or dedicated data communication technologies may be used to replace or supplement the aforementioned data communication technologies.

[0022] The terminal can be an electronic device such as a smartphone, wearable device, personal computer (PC), intelligent voice interaction device, smart home appliance, vehicle terminal, aircraft, unmanned vending terminal, extended reality (XR) device, etc. XR devices can include virtual reality (VR) devices, augmented reality (AR) devices, and mixed reality (MR) devices. A client application for the target application can be installed and run on the terminal. This target application can be an application that supports core audio and video content processing, or it can be other applications that support audio and video processing, etc., and this application does not limit its form. Furthermore, this application does not limit the form of the target application, including but not limited to applications (Apps), mini-programs, etc., installed on the terminal, and it can also be in the form of a webpage.

[0023] A server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services such as cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data. A server can also be a backend server for the target application, providing backend services to the client of the target application, such as core audio / video content processing services or audio / video processing services.

[0024] To facilitate understanding of the core audio and video content processing method provided in this application embodiment, the following example uses a server as the execution subject of the core audio and video content processing method to illustrate the application scenarios of the core audio and video content processing method.

[0025] See Figure 2 This figure is a schematic diagram illustrating an application scenario of an audio / video core content processing method provided in an embodiment of this application. Figure 2 In this application scenario, the terminal 1400, server 1300, and database 1500 are included.

[0026] The aforementioned target application is installed on terminal 1400 to acquire audio and video data and provide it to server 1300, enabling server 1300 to perform core audio and video content processing. Server 1300 provides support for the operation of the target application. Database 1500 stores the data required by server 1300. The following sections (A1-A8) provide a detailed explanation.

[0027] In step A1, terminal 1400 acquires sample audio and video based on a preset generation model. For example, terminal 1400 acquires sample audio and video X based on the preset generation model. s .

[0028] In step A2, terminal 1400 sends sample audio and video to server 1300. For example, terminal 1400 sends sample audio and video X to server 1300. s .

[0029] In step A3, server 1300 uses the cross-modal projection layer in the preset generation model to project the first visual feature sequence of the keyframe sequence in the sample audio / video and the first audio feature sequence of the audio frame sequence in the sample audio / video, thereby obtaining a second visual feature sequence and a second audio feature sequence that conform to the text embedding dimension of the preset generation model. For example, server 1300 projects the sample audio / video X... s The first visual feature sequence of the keyframe sequence (i.e., x) s-v The sequence), and the first audio feature sequence (i.e., x) of the audio frame sequence of the sample audio and video. s-a The input sequence is a cross-modal projection layer in a pre-defined generative model, which projects features onto the output sequence. The output sequence is a second visual feature sequence (i.e., y) that conforms to the text embedding dimension (4096 dimensions) of the pre-defined generative model. s-v (sequence) and second audio feature sequence (i.e., y) s-a sequence).

[0030] In step A4, server 1300 uses the attention layer and low-rank adapter in the preset generation model to fuse the features of the second visual feature sequence, the second audio feature sequence, and the prompt word features of the core content generation prompt words to obtain a first fused feature sequence. For example, server 1300 will fuse the second visual feature sequence (i.e., y...) s-v (sequence), second audio feature sequence (i.e., y) s-a Sequence), and the prompt word features of the core content generation prompt word y promptThe attention layer and low-rank adapter in the pre-defined generative model are input for feature fusion, and the first fused feature sequence is output (i.e., y). s sequence).

[0031] In step A5, server 1300 generates core content from the first fused feature sequence using the generation layer in a preset generation model, thereby obtaining the predicted core content of the sample audio and video. For example, server 1300 will generate the core content from the first fused feature sequence (i.e., y... s (Sequence) The core content is generated by the generation layer in the preset generation model, and the output is sample audio and video X. s The core content of the prediction C s '.

[0032] In step A6, server 1300 fine-tunes the cross-modal projection layer and low-rank adapter based on the difference between the predicted core content and the sample core content corresponding to the sample audio and video, thereby obtaining the core content generation model. For example, server 1300 based on the predicted core content C s 'and X s The corresponding core content of the sample C s By fine-tuning the training of the cross-modal projection layer and low-rank adapter to identify the differences between them, a core content generation model is obtained.

[0033] In step A7, server 1300 sends model data of the core content generation model to terminal 1400.

[0034] In step A8, the terminal 1400 updates the preset generation model to the core content generation model based on the model data of the core content generation model.

[0035] Therefore, the first visual feature sequence of the keyframe sequence in the sample audio and video, and the first audio feature sequence of the audio frame sequence in the sample audio and video are input into the cross-modal projection layer in the preset generation model for feature projection, and the output is a second visual feature sequence and a second audio feature sequence that conform to the text embedding dimension of the preset generation model. Through the feature projection of the cross-modal projection layer, the feature dimensions of the visual features and the feature dimensions of the audio features are aligned to the text embedding dimension of the preset generation model, eliminating the feature dimension differences between different modal features, and laying a structurally compatible foundation for multimodal feature fusion.

[0036] The second visual feature sequence, the second audio feature sequence, and the prompt word features generated from the core content are input into the attention layer and low-rank adapter in the preset generation model for feature fusion, and the first fused feature sequence is output. The feature fusion based on the attention layer realizes the deep fusion of visual and audio features to fully explore the deep correlation between features of different modalities. The introduction of the low-rank adapter provides a technical basis for reducing the number of training parameters when training the model.

[0037] The first fused feature sequence is input into the generation layer of the preset generation model. Based on the powerful generation capability of the preset generation model, the core content generation task is defined, and the predicted core content of the sample audio and video is output, so that the predicted core content accurately reflects the core of the sample audio and video. Based on the difference between the predicted core content and the sample core content corresponding to the sample audio and video, the cross-modal projection layer and low-rank adapter are fine-tuned and trained, so that the preset generation model learns to output the sample core content for the sample audio and video, thus obtaining the core content generation model. There is no need to train the attention layer and the generation layer, which greatly reduces the number of training parameters, thereby reducing the training cost, and enables the core content generation model to generate core content for audio and video efficiently and accurately.

[0038] The core audio and video content processing method provided in this application embodiment can be executed by a server. However, in other embodiments of this application, the terminal may also have similar functions to the server to execute the core audio and video content processing method provided in this application embodiment, or the terminal and the server may jointly execute the core audio and video content processing method provided in this application embodiment. This embodiment does not limit this.

[0039] The following describes in detail a method for processing core audio and video content provided in this application through method embodiments.

[0040] See Figure 3 ,Should Figure 3 The flowchart below illustrates a core content processing method for audio and video provided in this application. For ease of description, the following embodiment uses the aforementioned server as the execution subject of the core content processing method for audio and video. The core content processing method for audio and video includes the following steps S301-S304.

[0041] S301: By using the cross-modal projection layer in the preset generation model, feature projection is performed on the first visual feature sequence of the keyframe sequence in the sample audio and video, and the first audio feature sequence of the audio frame sequence in the sample audio and video, to obtain the second visual feature sequence and the second audio feature sequence that conform to the text embedding dimension of the preset generation model.

[0042] S302: By using the attention layer and low-rank adapter in the preset generation model, feature fusion is performed on the second visual feature sequence, the second audio feature sequence, and the prompt word features of the core content generation prompt words to obtain the first fused feature sequence.

[0043] In related technologies, the core content processing method for audio and video typically involves simply fusing visual and audio features to obtain a fused feature. This fused feature is then used to generate the core content of the audio and video by a pre-trained generative model guided by core content generation prompts. However, the fused feature obtained by simply fusing visual and audio features in audio and video is difficult to accurately represent the relationship between visual and audio information in the audio and video. This results in the generated core content failing to accurately reflect the core of the audio and video, and the training cost of the pre-trained generative model is also high.

[0044] In this embodiment, to address the aforementioned issues, considering that attention-based feature fusion can achieve deep fusion between different modal features, a low-rank adapter is added after the attention layer applying the attention mechanism. The parameters of the attention layer are frozen while the parameters of the low-rank adapter are trained, reducing the number of training parameters. Furthermore, the different modal features of audio and video, namely visual and audio features, need to be aligned with the feature dimensions of the text features required by the model. Therefore, when training a preset generation model to obtain a core content generation model based on sample audio and video and the corresponding sample core content, the first step is to align the visual features in the first visual feature sequence of the keyframe sequence in the sample audio and video to the text embedding dimension of the preset generation model through the cross-modal projection layer in the preset generation model, obtaining a second visual feature sequence. Similarly, the audio features in the first audio feature sequence of the audio frame sequence in the sample audio and video are aligned to the text embedding dimension of the preset generation model, obtaining a second audio feature sequence. Then, the second visual feature sequence, the second audio feature sequence, and the prompt word features of the core content generation prompt words are deeply fused through the attention layer and low-rank adapter in the preset generation model to obtain a first fused feature sequence.

[0045] Based on this, the first visual feature sequence of the keyframe sequence in the sample audio and video, and the first audio feature sequence of the audio frame sequence in the sample audio and video are first input into the cross-modal projection layer in the preset generation model for feature projection, and the output is a second visual feature sequence and a second audio feature sequence that conform to the text embedding dimension of the preset generation model; then the second visual feature sequence, the second audio feature sequence, and the prompt word features of the core content generation prompt words are input into the attention layer and low-rank adapter in the preset generation model for feature fusion, and the output is a first fused feature sequence.

[0046] The preset generation model is a model to be trained for generating core content from audio and video. The cross-modal projection layer is a feature processing layer that projects visual and audio features onto the feature dimensions of the text features required by the model. The sample audio and video is the audio and video of the core content generation model obtained by training the preset generation model. The keyframe sequence in the sample audio and video is a sequence formed by extracting multiple keyframe images from the sample audio and video. The first visual feature sequence is a sequence formed by the representation features of the keyframe images in the keyframe sequence of the sample audio and video. The audio frame sequence of the sample audio and video is a sequence formed by dividing the sample audio and video into multiple audio frames. The first audio feature sequence is a sequence formed by the representation features of the spectral features of the audio frames in the audio frame sequence of the sample audio and video. The text embedding dimension of the preset generation model is the feature dimension of the text features required by the model. The second visual feature sequence is a sequence formed by projecting the visual features in the first visual feature sequence onto visual features that conform to the text embedding dimension, where the feature dimension of the visual features in the first visual feature sequence is the text embedding dimension. The second audio feature sequence is a sequence formed by projecting the audio features in the first audio feature sequence onto audio features that conform to the text embedding dimension, where the feature dimension of the audio features in the first audio feature sequence is the text embedding dimension.

[0047] Among them, the attention layer in the preset generation model is a processing layer that deeply fuses different features; the low-rank adapter in the preset generation model is a processing layer that first reduces the dimensionality of the output features of the attention layer and then increases it; the core content generation prompt words are prompt words that guide the preset generation model to generate core content for sample audio and video; the prompt word features are the representation features of the core content generation prompt words; the first fusion feature sequence is a sequence of features formed by first reducing the dimensionality of the deep fusion features of the visual features in the second visual feature sequence, the audio features in the second audio feature sequence, and the prompt word features, and the feature dimension of the first fusion feature in the first fusion feature sequence is the text embedding dimension.

[0048] The step of obtaining the first visual feature sequence of the keyframe sequence in the sample audio and video may include: extracting keyframes from the sample audio and video according to a preset time period to obtain a keyframe sequence in the sample audio and video; and extracting features from the keyframe sequence in the sample audio and video to obtain a first visual feature sequence. Specifically, extracting features from the keyframe sequence in the sample audio and video to obtain the first visual feature sequence may include: adjusting the size of the keyframe images in the keyframe sequence in the sample audio and video according to the image size required by the visual encoder to obtain adjusted keyframe images in the keyframe sequence in the sample audio and video, wherein the adjusted keyframe images conform to the image size required by the visual encoder; and extracting features from the adjusted keyframe images in the keyframe sequence in the sample audio and video using the visual encoder to obtain the first visual feature sequence.

[0049] The step of obtaining the first audio feature sequence of the audio frame sequence of the sample audio and video may include: performing audio frame segmentation on the sample audio and video according to a preset audio frame length and a preset frame segmentation step length to obtain the audio frame sequence of the sample audio and video; performing spectrum calculation on the audio frame sequence of the sample audio and video to obtain the spectrum feature sequence of the audio frame sequence of the sample audio and video; and performing feature extraction on the spectrum feature sequence of the audio frame sequence of the sample audio and video to obtain the first audio feature sequence. Specifically, performing feature extraction on the spectrum feature sequence of the audio frame sequence of the sample audio and video to obtain the first audio feature sequence may be: performing feature extraction on the spectrum feature sequence of the audio frame sequence of the sample audio and video using an audio encoder to obtain the first audio feature sequence.

[0050] Furthermore, based on the timestamps of the keyframe sequences and the audio frame sequences in the sample audio and video, the first visual feature sequence and the first audio feature sequence are time-aligned to obtain the visual-audio feature sequence.

[0051] This approach aligns the feature dimensions of both visual and audio features in the sample audio and video samples to the text embedding dimension of the pre-defined generative model through feature projection across the cross-modal projection layer. This eliminates the feature dimension differences between different modal features and lays a structurally compatible foundation for multimodal feature fusion. Feature fusion based on the attention layer achieves deep fusion of visual and audio features to fully explore the deep correlations between different modal features. The introduction of a low-rank adapter provides a technical basis for reducing the number of training parameters when training the model.

[0052] As an example, computer devices will sample audio and video X s The first visual feature sequence of the keyframe sequence (i.e., x) s-v The sequence), and the first audio feature sequence (i.e., x) of the audio frame sequence of the sample audio and video. s-a The input sequence is a cross-modal projection layer in a pre-defined generative model, which projects features onto the output sequence. The output sequence is a second visual feature sequence (i.e., y) that conforms to the text embedding dimension (4096 dimensions) of the pre-defined generative model. s-v (sequence) and second audio feature sequence (i.e., y) s-a The computer device will use the second visual feature sequence (i.e., y) to generate the sequence. s-v (sequence), second audio feature sequence (i.e., y) s-a Sequence), and the prompt word features of the core content generation prompt word y prompt The attention layer and low-rank adapter in the pre-defined generative model are input for feature fusion, and the first fused feature sequence is output (i.e., y). s sequence).

[0053] Furthermore, the aforementioned preset time period is the keyframe extraction time period, which can be, for example, 1 second; the image size required by the visual encoder can be 224×224 pixels; the feature dimension of the visual features in the first visual feature sequence can be 768 dimensions. The preset audio frame length can be 25ms, the preset frame step size is the audio frame step size, which can be 10ms, and the feature dimension of the audio features in the first audio feature sequence can be 768 dimensions; the calculation formula for the spectral features in the spectral feature sequence can be as follows: ; in, f This indicates the frequency of audio frames in the audio frame sequence. ln (·) denotes the natural logarithm function; M ( f ) represents the spectral characteristics of audio frames in an audio frame sequence.

[0054] S303: By using the generation layer in the preset generation model, the core content of the first fusion feature sequence is generated to obtain the predicted core content of the sample audio and video.

[0055] S304: Based on the difference between the predicted core content and the sample core content corresponding to the sample audio and video, fine-tune the cross-modal projection layer and low-rank adapter to obtain the core content generation model.

[0056] In this embodiment, after performing S301-S302 to align the first visual feature sequence of the keyframe sequence in the sample audio / video and the first audio feature sequence of the audio frame sequence of the sample audio / video to the text embedding dimension of the preset generation model, a second visual feature sequence and a second audio feature sequence are obtained. Then, the second visual feature sequence, the second audio feature sequence, and the prompt word features of the core content generation prompt words are deeply fused to obtain a first fused feature sequence. Based on the fact that the first fused feature sequence accurately represents the visual and audio information of the sample audio / video and accurately represents the core content generation task, the core content of the sample audio / video needs to be predicted based on the first fused feature sequence. Combined with the sample core content corresponding to the sample audio / video, a cross-modal projection layer and a low-rank adapter are trained to obtain the core content generation model. Therefore, the core content of the sample audio / video is first predicted based on the first fused feature sequence using the generation layer in the preset generation model; then, combined with the sample core content corresponding to the sample audio / video, only the cross-modal projection layer and the low-rank adapter in the preset generation model are trained to obtain the core content generation model.

[0057] Based on this, the first fused feature sequence is first input into the generation layer of the preset generation model to generate core content, and the predicted core content of the sample audio and video is output. Then, based on the difference between the predicted core content and the sample core content corresponding to the sample audio and video, the cross-modal projection layer and low-rank adapter are fine-tuned to obtain the core content generation model.

[0058] The generation layer is a processing layer that generates core content from sequences formed by deep fusion of visual features, audio features, and cue word features of audio and video. The predicted core content of the sample audio and video is the core content generated by the sample audio and video through a preset generation model. The sample core content corresponding to the sample audio and video is the core content pre-annotated for the sample audio and video. The core content generation model is a trained model for generating core content for audio and video.

[0059] This approach generates core content through a generative layer. Based on a clear core content generation task and powerful generation capabilities, it generates predicted core content for sample audio and video, ensuring that the predicted core content accurately reflects the core of the sample audio and video. Considering the difference between the predicted core content and the corresponding sample core content of the sample audio and video, the cross-modal projection layer and low-rank adapter are fine-tuned during training. This allows the pre-defined generative model to learn to output sample core content for sample audio and video without training attention and generation layers, greatly reducing the number of training parameters and lowering training costs. Furthermore, the trained core content generation model can efficiently and accurately generate core content for audio and video.

[0060] As an example, based on the above example, the computer device will fuse the first feature sequence (i.e., y) s (Sequence) Input to the generation layer in the preset generation model, output sample audio / video X s The core content of the prediction C s Computer equipment is based on the core content of prediction, C. s 'and X s The corresponding core content of the sample C s By fine-tuning the training of the cross-modal projection layer and low-rank adapter to identify the differences between them, a core content generation model is obtained.

[0061] For example, the core content can be any of the general content that represents the core, such as the title, abstract, summary, or conclusion; that is, the core is any of the title, abstract, summary, or conclusion.

[0062] See Figure 4 ,Should Figure 4This is a schematic diagram illustrating a method for training a preset generation model to obtain a core content generation model, as provided in an embodiment of this application. Based on the above, on one hand, keyframe extraction is performed on the sample audio / video to obtain a keyframe sequence, and feature extraction is performed on this keyframe sequence to obtain a first visual feature sequence. On the other hand, audio framing is performed on the sample audio / video to obtain an audio frame sequence, and spectral calculation is performed on this audio frame sequence to obtain a spectral feature sequence, which is then used for feature extraction to obtain a first audio feature sequence. The first visual feature sequence and the first audio feature sequence are then subjected to feature projection through the cross-modal projection layer in the preset generation model to obtain a second visual feature sequence and a second audio feature sequence that conform to the text embedding dimension of the preset generation model. Based on the feature extraction of the core content generation prompt words to obtain prompt word features, the second visual feature sequence, the second audio feature sequence, and the prompt word features are then fused through the attention layer and low-rank adapter in the preset generation model to obtain a first fused feature sequence.

[0063] The first fused feature sequence is generated by the core content of the generation layer in the preset generation model to obtain the predicted core content of the sample audio and video. The difference between the predicted core content and the sample core content corresponding to the sample audio and video can be fine-tuned to train the cross-modal projection layer and low-rank adapter to obtain the core content generation model.

[0064] As can be seen from the above technical solution, the first visual feature sequence of the keyframe sequence in the sample audio and video, and the first audio feature sequence of the audio frame sequence in the sample audio and video are input into the cross-modal projection layer in the preset generation model for feature projection, and the output is a second visual feature sequence and a second audio feature sequence that conform to the text embedding dimension of the preset generation model. Through the feature projection of the cross-modal projection layer, the feature dimensions of the visual features and the feature dimensions of the audio features are aligned to the text embedding dimension of the preset generation model, eliminating the feature dimension differences between different modal features, and laying a structurally compatible foundation for multimodal feature fusion.

[0065] The second visual feature sequence, the second audio feature sequence, and the prompt word features generated from the core content are input into the attention layer and low-rank adapter in the preset generation model for feature fusion, and the first fused feature sequence is output. The feature fusion based on the attention layer realizes the deep fusion of visual and audio features to fully explore the deep correlation between features of different modalities. The introduction of the low-rank adapter provides a technical basis for reducing the number of training parameters when training the model.

[0066] The first fused feature sequence is input into the generation layer of the preset generation model. Based on the powerful generation capability of the preset generation model, the core content generation task is defined, and the predicted core content of the sample audio and video is output, so that the predicted core content accurately reflects the core of the sample audio and video. Based on the difference between the predicted core content and the sample core content corresponding to the sample audio and video, the cross-modal projection layer and low-rank adapter are fine-tuned and trained, so that the preset generation model learns to output the sample core content for the sample audio and video, thus obtaining the core content generation model. There is no need to train the attention layer and the generation layer, which greatly reduces the number of training parameters, thereby reducing training costs, saving computing resources and computing time, and enabling the core content generation model to generate core content for audio and video efficiently and accurately.

[0067] In this embodiment, when performing the above-mentioned S302, which uses the attention layer and low-rank adapter in the preset generation model to fuse the features of the second visual feature sequence, the second audio feature sequence, and the prompt word features of the core content to obtain the first fused feature sequence, it is considered that the fusion is mainly based on the deep fusion of visual features in the second visual feature sequence and audio features in the second audio feature sequence using an attention mechanism. Therefore, the second visual feature sequence and the second audio feature sequence are first input into the attention layer for feature fusion to obtain a multimodal feature sequence; then the multimodal feature sequence and prompt word features are fused to obtain the first feature sequence; next, the first feature sequence is input into the low-rank adapter, and feature dimensionality reduction is performed based on the dimensionality reduction matrix in the low-rank adapter, and feature dimensionality increase is performed based on the dimensionality increase matrix in the low-rank adapter to obtain the second feature sequence; finally, the first feature sequence and the second feature sequence are fused to obtain the first fused feature sequence. Based on this, this application provides a possible implementation method, and the above-mentioned S302 may include, for example, the following S1-S4 (not shown in the figure).

[0068] S1: Through the attention layer, feature fusion is performed on the second visual feature sequence and the second audio feature sequence to obtain a multimodal feature sequence.

[0069] S2: Perform feature fusion on the multimodal feature sequence and prompt word features to obtain the first feature sequence.

[0070] S3: Using the dimensionality reduction and dimensionality increase matrices in the low-rank adapter, perform dimensionality reduction and dimensionality increase on the first feature sequence to obtain the second feature sequence.

[0071] S4: Perform feature fusion on the first feature sequence and the second feature sequence to obtain the first fused feature sequence.

[0072] Among them, the multimodal feature sequence is a sequence formed by the deep fusion of visual features in the second visual feature sequence and audio features in the second audio feature sequence, and the feature dimension of the multimodal features in this multimodal feature sequence is the text embedding dimension; the first feature sequence is a feature concatenation sequence of the multimodal feature sequence and the cue word features, and the feature dimension of the first feature in this first feature sequence is the text embedding dimension; the dimensionality reduction matrix in the low-rank adapter is a matrix that reduces the feature dimension of the features; the dimensionality increase matrix in the low-rank adapter is a matrix that increases the feature dimension of the features; the second feature sequence is a sequence formed by the first feature in the first feature sequence after dimensionality reduction and then dimensionality increase, and the feature dimension of the second feature in this second feature sequence is the text embedding dimension.

[0073] This approach, based on the attention mechanism of the attention layer, first deeply fuses the second visual feature sequence and the second audio feature sequence to obtain a multimodal feature sequence, and then fuses it with the cue word feature to form the first feature sequence. This approach first focuses on fully exploring the deep correlation between visual and audio features in the sample audio and video, and then simply fuses the cue word feature. While ensuring deep fusion of visual and audio features in the sample audio and video, it reduces fusion complexity, further saving computational resources and time, and laying the foundation for efficient and accurate prediction of the core content of the sample audio and video. Based on the dimensionality reduction and dimensionality increase matrices in the low-rank adapter, the first feature in the first feature sequence is first reduced in dimensionality and then increased in dimensionality to obtain the second feature sequence, which is then fused with the first feature sequence to form the first fused feature sequence. The dimensionality reduction and dimensionality increase matrices have few parameters. When training the model, the parameters of the attention layer and the dimensionality reduction and dimensionality increase matrices are frozen, which facilitates reducing the number of training parameters and reducing training costs to achieve efficient training.

[0074] As an example, based on the above example, the computer device first processes the second visual feature sequence (i.e., y... s-v (sequence) and second audio feature sequence (i.e., y) s-a The sequence is input to the attention layer and the low-rank adapter for feature fusion, resulting in a multimodal feature sequence (i.e., y). s-va Sequence); then fuse multimodal feature sequences (i.e., y s-va Sequence and cue word features y prompt The first feature sequence (i.e., y) is obtained. s-1 Sequence); the first feature sequence (i.e., y) s-1 The sequence is input to a low-rank adapter. First, feature dimensionality reduction is performed based on the dimensionality reduction matrix in the low-rank adapter. Then, feature dimensionality increase is performed based on the dimensionality increase matrix in the low-rank adapter to obtain the second feature sequence (i.e., y). s-2 Sequence); fused with the first feature sequence (i.e., y s-1 (sequence) and second feature sequence (i.e., y) s-2 (sequence), to obtain the first fusion feature sequence (i.e., y)s sequence).

[0075] See Figure 5 ,Should Figure 5 This is a schematic diagram illustrating the fusion of a second visual feature sequence, a second audio feature sequence, and cue word features to obtain a first fused feature sequence, as provided in an embodiment of this application. Based on the above, the second visual feature sequence and the second audio feature sequence undergo feature fusion at an input attention layer to obtain a multimodal feature sequence; this multimodal feature sequence and the cue word features undergo feature fusion to obtain a first feature sequence; this first feature sequence undergoes feature dimensionality reduction based on a low-rank adapter and feature dimensionality increase based on an up-dimensionality matrix to obtain a second feature sequence; the second feature sequence and the first feature sequence can be fused into the first fused feature sequence.

[0076] In this embodiment, when performing the above-described S1—fusing the second visual feature sequence and the second audio feature sequence through the attention layer to obtain a multimodal feature sequence—it is considered that the second visual feature sequence and the second audio feature sequence are input into the attention layer, and the attention mechanism of this attention layer is used to determine the correlation between different modal features. Therefore, the second visual feature sequence and the second audio feature sequence are input into the attention layer for feature fusion. First, the correlation weight between the visual features in the second visual feature sequence and the audio features in the second audio feature sequence is determined. Then, the second visual feature sequence and the second audio feature sequence are weighted according to the correlation weight to obtain the multimodal feature sequence. Based on this, this application provides a possible implementation method, and the above-described S1 may include, for example, the following S1a-S1b (not shown in the figure).

[0077] S1a: Through the attention layer, feature fusion is performed on the second visual feature sequence and the second audio feature sequence to obtain the correlation weight between the visual features in the second visual feature sequence and the audio features in the second audio feature sequence.

[0078] S1b: Based on the association weights, the second visual feature sequence and the second audio feature sequence are weighted to obtain the multimodal feature sequence.

[0079] The association weight between visual features in the second visual feature sequence and audio features in the second audio feature sequence can be a quantification of the degree of attention paid by visual features in the second visual feature sequence to audio features in the second audio feature sequence.

[0080] Furthermore, the association weight between visual features in the second visual feature sequence and audio features in the second audio feature sequence can also be a quantification of the degree of attention paid by audio features in the second audio feature sequence to visual features in the second visual feature sequence.

[0081] This method targets the second visual feature sequence and the second audio feature sequence. Based on the attention mechanism of the attention layer, it first determines the association weight between the visual features in the second visual feature sequence and the audio features in the second audio feature sequence. This clarifies the degree of association between the visual and audio features, enabling full mining of the deep association between visual and audio features in the sample audio and video. Then, the second visual feature sequence and the second audio feature sequence are weighted according to the association weight, and the visual features in the second visual feature sequence and the audio features in the second audio feature sequence are deeply fused to obtain a multimodal feature sequence. This multimodal feature sequence accurately represents the visual features, audio features, and the deep association between the visual and audio features, laying the foundation for accurately predicting the core content of the sample audio and video.

[0082] As an example, based on the above example, the computer device will use the second visual feature sequence (i.e., y) s-v (sequence) and second audio feature sequence (i.e., y) s-a The sequence is input into the attention layer for feature fusion. First, the second visual feature sequence (i.e., y) is determined. s-v Visual features y in sequence s-v With the second audio feature sequence (i.e., y) s-a Audio features y in sequence s-a Association weight between Attention ( Q , K , V Then, according to the association weight Attention ( Q , K , V Weighted second visual feature sequence (i.e., y) s-v (sequence) and second audio feature sequence (i.e., y) s-a (sequence), to obtain the multimodal feature sequence (i.e., y s-va sequence).

[0083] This association weight Attention ( Q , K , V The calculation formula for ) is as follows: ; , , ; in, W Q , W K , W V It is a learnable weight matrix of 4096×4096; ys-v Indicates visual features in the second visual feature sequence; y s-a Indicates the audio features in the second audio feature sequence; d k This indicates that the feature dimension of K is 4096. softmax (·) represents the normalization function.

[0084] In this embodiment, when performing the above-described S302, which uses the attention layer and low-rank adapter in the preset generation model to fuse the second visual feature sequence, the second audio feature sequence, and the prompt word features of the core content to obtain the first fused feature sequence, considering that the attention mechanism can deeply fuse the visual features in the second visual feature sequence, the audio features in the second audio feature sequence, and the prompt word features, the second visual feature sequence, the second audio feature sequence, and the prompt word features are first input into the attention layer for feature fusion to obtain the third feature sequence; then, the third feature sequence is input into the low-rank adapter, where feature dimensionality reduction is performed first based on the dimensionality reduction matrix in the low-rank adapter, and feature dimensionality increase is performed based on the dimensionality increase matrix in the low-rank adapter to obtain the fourth feature sequence; finally, the third feature sequence and the fourth feature sequence are fused to obtain the first fused feature sequence. Based on this, this application provides a possible implementation, and the above-described S302 may include, for example, the following S5-S7 (not shown in the figure).

[0085] S5: Through the attention layer, the second visual feature sequence, the second audio feature sequence, and the prompt word features are fused to obtain the third feature sequence.

[0086] S6: Using the dimension reduction and dimension increase matrices in the low-rank adapter, perform feature dimension reduction and feature dimension increase on the third feature sequence to obtain the fourth feature sequence.

[0087] S7: Perform feature fusion on the third and fourth feature sequences to obtain the first fused feature sequence.

[0088] The third feature sequence is a sequence formed by the deep fusion of visual features from the second visual feature sequence, audio features from the second audio feature sequence, and cue word features. The feature dimension of the third feature in this third feature sequence is the text embedding dimension. The dimensionality reduction matrix in the low-rank adapter is a matrix that reduces the feature dimension of the feature. The dimensionality increase matrix in the low-rank adapter is a matrix that increases the feature dimension of the feature. The fourth feature sequence is a sequence formed by the third feature after dimensionality reduction and then dimensionality increase. The feature dimension of the fourth feature in this fourth feature sequence is the text embedding dimension.

[0089] This approach, based on the attention mechanism of the attention layer, deeply fuses the second visual feature sequence, the second audio feature sequence, and the cue word features to obtain the third feature sequence. This fully explores the deep correlations between visual features, audio features, and cue word features in the sample audio and video. It ensures that, based on the deep fusion of visual and audio features in the sample audio and video, it further accurately clarifies the correlation between visual and audio features and the core content generation task, laying the foundation for more accurate prediction of the core content of the sample audio and video. Based on the dimensionality reduction and dimensionality increase matrices in the low-rank adapter, the third feature in the third feature sequence is first reduced in dimensionality and then increased in dimensionality to obtain the fourth feature sequence, which is then fused with the third feature sequence to form the first fused feature sequence. The dimensionality reduction and dimensionality increase matrices have few parameters. When training the model, the parameters of the attention layer and the dimensionality reduction and dimensionality increase matrices are frozen, which facilitates reducing the number of training parameters and reducing training costs to achieve efficient training.

[0090] As an example, the computer device first processes the second visual feature sequence (i.e., y) s-v (sequence), second audio feature sequence (i.e., y) s-a Sequence and cue word features y prompt The input attention layer performs feature fusion to obtain the third feature sequence (i.e., y). s-3 Sequence); then the third feature sequence (i.e., y) s-3 The sequence is input to a low-rank adapter. First, feature dimensionality reduction is performed based on the dimensionality reduction matrix in the low-rank adapter. Then, feature dimensionality increase is performed based on the dimensionality increase matrix in the low-rank adapter, resulting in the fourth feature sequence (i.e., y). s-4 Sequence); Finally, the third feature sequence (i.e., y) is fused. s-3 (sequence) and the fourth feature sequence (i.e., y) s-4 (sequence), to obtain the first fusion feature sequence (i.e., y) s sequence).

[0091] See Figure 6 ,Should Figure 6 This is a schematic diagram illustrating another method for fusing a second visual feature sequence, a second audio feature sequence, and cue word features to obtain a first fused feature sequence, as provided in an embodiment of this application. Based on the above, the second visual feature sequence, the second audio feature sequence, and the cue word features are fused through an attention layer to obtain a third feature sequence; this third feature sequence is then subjected to feature dimensionality reduction based on a dimensionality reduction matrix and feature dimensionality increase based on an dimensionality increase matrix by a low-rank adapter to obtain a fourth feature sequence; this fourth feature sequence and the third feature sequence can be fused into the first fused feature sequence.

[0092] In this embodiment, based on S3 and S6 above, considering that the feature dimension of the first feature in the first feature sequence is the text embedding dimension of the preset generation model, and the feature dimension of the third feature in the third feature sequence is the text embedding dimension, the dimension reduction matrix in the low-rank adapter is based on the matrix that reduces the feature dimension of the feature. Therefore, the matrix dimension of the dimension reduction matrix can be determined by the text embedding dimension and the preset feature dimension which is smaller than the text embedding dimension. The dimension increase matrix in the low-rank adapter is based on the matrix that increases the feature dimension of the feature, specifically the matrix that increases the preset feature dimension. Considering that the feature dimension of the second feature in the second feature sequence is the text embedding dimension, and the feature dimension of the fourth feature in the fourth feature sequence is the text embedding dimension, the matrix dimension of the dimension increase matrix can be determined by the preset feature dimension and the text embedding dimension. Based on this, this application provides a possible implementation method. The steps for determining the matrix dimension of the dimension reduction matrix and the matrix dimension of the dimension increase matrix may include, for example, the following S8-S9 (not shown in the figure).

[0093] S8: Determine the matrix dimension of the dimensionality reduction matrix based on the preset feature dimension and text embedding dimension; the preset feature dimension is smaller than the text embedding dimension.

[0094] S9: Determine the matrix dimension of the upscaling matrix based on the text embedding dimension and the preset feature dimension.

[0095] Among them, the preset feature dimension is much smaller than the text embedding dimension; the dimension reduction matrix is ​​the matrix of preset feature dimension multiplied by text embedding dimension; the dimension increase matrix is ​​the matrix of text embedding dimension multiplied by preset feature dimension.

[0096] This method determines that the dimensionality reduction matrix in the low-rank adapter is a matrix multiplied by the text embedding dimension, using a preset feature dimension and a text embedding dimension smaller than the text embedding dimension. This ensures that the dimensionality reduction matrix can reduce features that conform to the text embedding dimension to features that conform to the preset feature dimension. Conversely, it determines that the dimensionality increase matrix in the low-rank adapter is a matrix multiplied by the text embedding dimension, using the text embedding dimension and the preset feature dimension. This ensures that features that conform to the preset feature dimension can be increased to features that conform to the text embedding dimension. In other words, this method ensures that the features output from the attention layer are first reduced to the preset feature dimension by the dimensionality reduction matrix in the low-rank adapter, thus reducing the number of parameters relative to the attention layer, and then increased to the text embedding dimension by the dimensionality increase matrix in the low-rank adapter, thus aligning with the feature dimension of the output features of the attention layer.

[0097] As an example, based on the above example, the dimensionality reduction matrix is ​​a matrix consisting of a preset feature dimension (8 dimensions) multiplied by the text embedding dimension (4096 dimensions); the dimensionality increase matrix is ​​a matrix consisting of the text embedding dimension (4096 dimensions) multiplied by the preset feature dimension (8 dimensions). Specifically, through the dimensionality reduction matrix A and the dimensionality increase matrix B in the low-rank adapter, the input feature a is subjected to feature dimensionality reduction and feature increase to obtain the transformed feature. The input feature a and the transformed feature are then fused to obtain the output feature b, i.e., b = Wa + αBAa, where α is a scaling factor and W is the weight matrix of the attention layer corresponding to the low-rank adapter.

[0098] In this embodiment, when executing S304 above, which fine-tunes the cross-modal projection layer and low-rank adapter based on the difference between the predicted core content and the sample core content corresponding to the sample audio / video, to obtain the specific implementation of the core content generation model, the predicted probability that the predicted core content is the sample core content is actually first determined. That is, each predicted word in the predicted core content is the cumulative value of the predicted probability of the corresponding sample word in the sample core content. Then, the cumulative value is maximized to make the predicted core content close to the sample core content, so as to fine-tune the training of the cross-modal projection layer and low-rank adapter, and the trained preset generation model is used as the core content generation model. Based on this, this application provides a possible implementation method. The above S304 may include, for example, the following S304a-S304b (not shown in the figure).

[0099] S304a: Determine the sum of the prediction probabilities of multiple predicted words in the core content of the prediction and the corresponding multiple sample words in the core content of the sample.

[0100] S304b: By maximizing the sum of prediction probabilities, the cross-modal projection layer and low-rank adapter are fine-tuned to obtain the core content generation model.

[0101] In this context, the predicted word elements in the core content are the sum of the predicted probabilities of the corresponding sample word elements in the core content. Specifically, each predicted word element in the core content is the cumulative value of the predicted probabilities of the corresponding sample word elements in the core content.

[0102] This method first determines the sum of prediction probabilities of multiple predicted words in the predicted core content being the corresponding multiple sample words in the sample core content. The smaller the sum of prediction probabilities, the greater the difference between the predicted core content and the sample core content; the larger the sum of prediction probabilities, the smaller the difference between the predicted core content and the sample core content. The cross-modal projection layer and low-rank adapter are fine-tuned by maximizing the probability sum. This allows the predicted core content to be close to the sample core content without training the attention layer and the generation layer. This enables the pre-defined generation model to learn to output multiple sample words in the sample core content for the sample audio and video, thereby enabling the trained core content generation model to generate core content efficiently and accurately for audio and video.

[0103] As an example, the prediction probabilities and corresponding loss formulas for multiple predicted words in the core content are as follows: ; in, p ij Indicates the first i The core content of the prediction of the sample audio and video is the first j Each predicted word is the probability of a corresponding word in the core content of the sample audio / video. y ij Indicates the first i The core content of the sample audio / video corresponding to the sample is the first j The true labels of each sample word element y ij =1; N This indicates the total number of sample audio and video files; M This represents the total number of predicted words in the core content of the sample audio / video, that is, the total number of sample words in the core content of the sample audio / video.

[0104] In addition, when fine-tuning the cross-modal projection layer and the low-rank adapter, the AdamW optimizer was used with a learning rate of 2e-5 and a weight decay of 0.01. The gradient was calculated using the backpropagation algorithm, and the optimizer was used to update the parameters of the cross-modal projection layer and the low-rank adapter. One training epoch was completed by traversing all sample audio data. Fine-tuning training was carried out after 5 training epochs.

[0105] In this embodiment, based on the cross-modal projection layer used to project visual features from the first visual feature sequence and audio features from the first audio feature sequence to visual features and audio features conforming to the text embedding dimension, respectively, to obtain a second visual feature sequence and a second audio feature sequence, the cross-modal projection layer needs to include a first transformation matrix for projecting the first visual feature sequence to the second visual feature sequence and a second transformation matrix for projecting the first audio feature sequence to the second audio feature sequence. Considering that the feature dimension of the visual features in the second visual feature sequence is the text embedding dimension, the matrix dimension of the first transformation matrix can be determined by the text embedding dimension and the feature dimension of the visual features in the first visual feature sequence. Considering that the feature dimension of the audio features in the second audio feature sequence is the text embedding dimension, the matrix dimension of the second transformation matrix can be determined by the text embedding dimension and the feature dimension of the audio features in the first audio feature sequence. Based on this, this application provides a possible implementation method, wherein the cross-modal projection layer includes a first transformation matrix and a second transformation matrix. The first transformation matrix is ​​used to project a first visual feature sequence into a second visual feature sequence, and the second transformation matrix is ​​used to project a first audio feature sequence into a second audio feature sequence. The steps for determining the matrix dimension of the first transformation matrix and the matrix dimension of the second transformation matrix may include, for example, the following S10-S11 (not shown in the figure).

[0106] S10: Determine the matrix dimension of the first transformation matrix based on the text embedding dimension and the feature dimension of the visual features in the first visual feature sequence.

[0107] S11: Determine the matrix dimension of the second transformation matrix based on the text embedding dimension and the feature dimension of the audio features in the first audio feature sequence.

[0108] The first transformation matrix is ​​a matrix that multiplies the text embedding dimension by the feature dimension of the visual features in the first visual feature sequence; the second transformation matrix is ​​a matrix that multiplies the text embedding dimension by the feature dimension of the audio features in the first audio feature sequence.

[0109] This method determines the first transformation matrix in the cross-modal projection layer by multiplying the text embedding dimension by the feature dimension of the visual features in the first visual feature sequence, based on the text embedding dimension and the feature dimension of the visual features in the first visual feature sequence. This allows the first transformation matrix to align the visual features in the first visual feature sequence to the text embedding dimension of the preset generation model. Similarly, by multiplying the text embedding dimension by the feature dimension of the audio features in the first audio feature sequence, the second transformation matrix in the cross-modal projection layer is determined to be multiplied by the text embedding dimension by the feature dimension of the audio features in the first audio feature sequence. This allows the second transformation matrix to align the audio features in the first audio feature sequence to the text embedding dimension of the preset generation model. This method eliminates the feature dimension differences between different modal features, thus laying a structurally compatible foundation for multimodal feature fusion.

[0110] As an example, the first transformation matrix is ​​a matrix consisting of the text embedding dimension (4096 dimensions) multiplied by the feature dimension of the visual features in the first visual feature sequence (768 dimensions); the second transformation matrix is ​​a matrix consisting of the text embedding dimension (4096 dimensions) multiplied by the feature dimension of the audio features in the first audio feature sequence (768 dimensions). Specifically, this is achieved through the first transformation matrix W in the cross-modal projection layer. v For the first visual feature sequence (i.e., x) s-v Visual features x in sequence s-v Perform feature projection to obtain the second visual feature sequence (i.e., y). s-v Visual features y in sequence s-v , that y s-v =W v ×x s-v +b v , where b v Indicates the bias feature, b v The feature dimension is 4096; through the second transformation matrix W in the cross-modal projection layer a For the first audio feature sequence (i.e., x) s-a Audio features x in sequence s-a Perform feature projection to obtain the second audio feature sequence (i.e., y). s-v Audio features y in sequence s-a , that y s-a =W a ×x s-a +b a , where b a Indicates the bias feature, b a The feature dimension is 4096.

[0111] Furthermore, in this embodiment, after the core content generation model is obtained through training of the cross-modal projection layer and the low-rank adapter, a preset performance index value representing the core content generation performance of the core content generation model can be determined by using the generated core content of multiple audio-visual videos to be tested based on the core content of the core content generation model, as well as the labeled core content of multiple audio-visual videos to be tested. This optimizes the core content generation model, resulting in an optimized core content generation model with better core content generation performance. Based on this, this application provides a possible implementation method, which may further include, for example, the following S12-S13 (not shown in the figure).

[0112] S12: Based on the generated core content of multiple audio and video videos under test using the core content generation model, and the labeled core content of multiple audio and video videos under test, determine the preset performance index value of the core content generation model; the preset performance index value is used to represent the core content generation performance of the core content generation model.

[0113] S13: Optimize the core content generation model based on preset performance index values ​​to obtain the optimized core content generation model.

[0114] Among them, the generated core content of multiple audio and video videos based on the core content generation model includes the generated core content of each audio and video video based on the core content generation model, and the generated core content of each audio and video video based on the core content generation model is the core content generated by the core content generation model for each audio and video video under test; the labeled core content of multiple audio and video videos under test includes the labeled core content of each audio and video video under test, and the labeled core content of each audio and video video under test is the core content pre-labeled for each audio and video video under test.

[0115] This method targets multiple audio and video files for testing. By analyzing the generated core content of each audio and video file based on the core content generation model and the labeled core content of that audio and video file, a preset performance index value representing the core content generation performance of the core content generation model is determined. This allows for a clear understanding of the core content generation performance of the core content generation model, specifically the accuracy with which the model generates core content for audio and video files. This approach optimizes the core content generation model, enhancing its core content generation performance to obtain an optimized model. Ultimately, the optimized model can generate core content more efficiently and accurately for audio and video files.

[0116] As an example, the preset performance metric value can be accuracy, that is, the proportion of test audio and video that are correctly tested by the core content generation model out of multiple test audio and video; the preset performance metric value can also be the macro average F1 score, that is, the harmonic mean of the precision and recall of the core content generation model.

[0117] Furthermore, in this embodiment, after training the preset generation model to obtain the core content generation model using the above S301-S304 steps, the core content generation model can efficiently and accurately generate core content for the target audio and video. This process includes: firstly, inputting the third visual feature sequence of the keyframe sequence in the target audio and video, and the third audio feature sequence of the audio frame sequence in the target audio and video, into the cross-modal projection layer in the core content generation model for feature projection, outputting a fourth visual feature sequence and a fourth audio feature sequence that conform to the text embedding dimension; then inputting the fourth visual feature sequence, the fourth audio feature sequence, and the cue word features into the attention layer and low-rank adapter in the core content generation model for feature fusion, outputting a second fused feature sequence; and finally inputting the second fused feature sequence into the generation layer in the core content generation model to output multiple candidate core contents of the target audio and video. Based on this, this application provides a possible implementation method, which may also include the following S14-S16 (not shown in the figure).

[0118] S14: Through the cross-modal projection layer in the core content generation model, feature projection is performed on the third visual feature sequence of the keyframe sequence in the target audio and video, and the third audio feature sequence of the audio frame sequence in the target audio and video, to obtain the fourth visual feature sequence and the fourth audio feature sequence that conform to the text embedding dimension.

[0119] S15: By using the attention layer and low-rank adapter in the core content generation model, feature fusion is performed on the fourth visual feature sequence, the fourth audio feature sequence, and the cue word features to obtain the second fused feature sequence.

[0120] S16: Through the generation layer in the core content generation model, the second fusion feature sequence is used to generate core content, thereby obtaining multiple candidate core contents of the target audio and video.

[0121] Among them, the target audio and video is the audio and video to be generated as the core content; the keyframe sequence in the target audio and video is a sequence formed by extracting multiple keyframe images from the target audio and video; the third visual feature sequence is a sequence formed by the representation features of the keyframe images in the keyframe sequence of the target audio and video; the audio frame sequence of the target audio and video is a sequence formed by dividing the target audio and video into multiple audio frames; the third audio feature sequence is a sequence formed by the representation features of the spectral features of the audio frames in the audio frame sequence of the target audio and video; the fourth visual feature sequence is a sequence formed by projecting the visual features in the third visual feature sequence onto visual features that conform to the text embedding dimension, where the feature dimension of the visual features in the fourth visual feature sequence is the text embedding dimension; the fourth audio feature sequence is a sequence formed by projecting the audio features in the third audio feature sequence onto audio features that conform to the text embedding dimension, where the feature dimension of the audio features in the fourth audio feature sequence is the text embedding dimension.

[0122] Among them, the second fusion feature sequence is a sequence of features formed by first reducing the dimensionality and then increasing the dimensionality of the deep fusion features of the visual features in the fourth visual feature sequence, the audio features in the fourth audio feature sequence, and the prompt word features. The feature dimension of the second fusion feature in this second fusion feature sequence is the text embedding dimension. The multiple candidate core contents of the target audio and video are multiple core contents generated by the core content generation model for the target audio and video.

[0123] The step of obtaining the third visual feature sequence of the keyframe sequence in the target audio-visual video may include: extracting keyframes from the target audio-visual video according to a preset time period to obtain a keyframe sequence in the target audio-visual video; and extracting features from the keyframe sequence in the target audio-visual video to obtain a third visual feature sequence. Specifically, extracting features from the keyframe sequence in the target audio-visual video to obtain the third visual feature sequence may include: adjusting the size of the keyframe images in the keyframe sequence in the target audio-visual video according to the image size required by the visual encoder to obtain adjusted keyframe images in the keyframe sequence in the target audio-visual video, wherein the adjusted keyframe images conform to the image size required by the visual encoder; and extracting features from the adjusted keyframe images in the keyframe sequence in the target audio-visual video using the visual encoder to obtain the third visual feature sequence.

[0124] The step of obtaining the third audio feature sequence of the audio frame sequence of the target audio and video may include: performing audio frame segmentation on the target audio and video according to a preset audio frame length and a preset frame step length to obtain the audio frame sequence of the target audio and video; performing spectrum calculation on the audio frame sequence of the target audio and video to obtain the spectrum feature sequence of the audio frame sequence of the target audio and video; and extracting features from the spectrum feature sequence of the audio frame sequence of the target audio and video to obtain the third audio feature sequence. Specifically, extracting features from the spectrum feature sequence of the audio frame sequence of the target audio and video to obtain the third audio feature sequence may be done by: extracting features from the spectrum feature sequence of the audio frame sequence of the target audio and video using an audio encoder to obtain the third audio feature sequence.

[0125] In addition, based on the timestamps of the keyframe sequences in the target audio and video and the timestamps of the audio frame sequences in the target audio and video, the third visual feature sequence and the third audio feature sequence are time-aligned to obtain the visual audio feature sequence.

[0126] This approach aligns the feature dimensions of both the visual and audio features of the target audio / video to the text embedding dimension of the core content generation model through feature projection across the cross-modal projection layer. This eliminates the feature dimension differences between different modal features and lays a structurally compatible foundation for multimodal feature fusion. Feature fusion based on the attention layer achieves deep fusion of visual and audio features to fully explore the deep correlations between different modal features. Through core content generation in the generation layer, multiple candidate core contents are generated for the target audio / video based on a clear core content generation task and powerful generation capabilities, ensuring that the candidate core contents accurately reflect the core of the target audio / video.

[0127] As an example, based on the above example, the third visual feature sequence of the keyframe sequence in the target audio / video X (i.e., x) v The sequence), and the third audio feature sequence (i.e., x) of the audio frame sequence of the target audio and video. a The input sequence is processed by the cross-modal projection layer in the core content generation model, and the output is a fourth visual feature sequence conforming to the text embedding dimension (4096 dimensions). v (sequence) and the fourth audio feature sequence (i.e., y) a The computer device will use the fourth visual feature sequence (i.e., y) to generate the sequence. v (sequence), the fourth audio feature sequence (i.e., y) a (sequence), and cue word features y promptThe attention layer and low-rank adapter in the core content generation model are used for feature fusion, outputting a second fused feature sequence (i.e., the y sequence). The computer device then inputs this second fused feature sequence (i.e., the y sequence) into the generation layer of the core content generation model, outputting multiple candidate core contents C1, C2, ..., C3 for the target audio / video X. n n is a positive integer, n≥2; for example, n=3.

[0128] Specifically, outputting multiple candidate contents uses the beam search algorithm, which is a heuristic search algorithm that retains the candidate paths with the highest probability beam size=n at each step. The calculation formula for the beam search algorithm is as follows: ; in, y Indicates the second fusion feature sequence; z 1：t-1 This indicates the first word element to the second word element in the candidate core content. t -1 word element; z t This represents the t-th word element in the candidate core content; P(z) t |z 1：t-1 , y） Indicates the first word to the second word in the second fusion feature sequence and candidate core content. t Given -1 lexical units, the probability of the t-th lexical unit in the candidate core content; log(·) represents the logarithmic function; T represents the total number of multiple words in the candidate core content; argmax represents taking the t-th word in the candidate core content corresponding to the maximum probability. z 1：T This indicates the core content of the candidate.

[0129] See Figure 7 ,Should Figure 7This illustration illustrates a method for generating multiple candidate core content for a target audio / video using a core content generation model, as provided in this application embodiment. Based on the above, on one hand, the target audio / video undergoes keyframe extraction to obtain a keyframe sequence, which is then used for feature extraction to obtain a third visual feature sequence. On the other hand, the target audio / video undergoes audio framing to obtain an audio frame sequence, which is then used for spectral calculation to obtain a spectral feature sequence. This spectral feature sequence is then used for feature extraction to obtain a third audio feature sequence. The third visual feature sequence and the third audio feature sequence are then subjected to feature projection through the cross-modal projection layer in the core content generation model to obtain a fourth visual feature sequence and a fourth audio feature sequence conforming to the text embedding dimension. The fourth visual feature sequence, the fourth audio feature sequence, and the cue word features are then fused through the attention layer and low-rank adapter in the core content generation model to obtain a second fused feature sequence. This second fused feature sequence is then used for core content generation through the generation layer in the core content generation model to obtain multiple candidate core content for the target audio / video.

[0130] This application provides a possible implementation, and the above S15 may include, for example, the following S15a-S15d (not shown in the figure).

[0131] S15a: Through the attention layer, feature fusion is performed on the fourth visual feature sequence and the fourth audio feature sequence to obtain the fused feature sequence.

[0132] S15b: Perform feature fusion on the fused feature sequence and prompt word features to obtain the fifth feature sequence.

[0133] S15c: Using the dimension reduction and dimension increase matrices in the low-rank adapter, the fifth feature sequence is subjected to feature dimension reduction and feature dimension increase to obtain the sixth feature sequence.

[0134] S15d: Perform feature fusion on the fifth and sixth feature sequences to obtain the second fused feature sequence.

[0135] Among them, the fused feature sequence is a sequence formed by the deep fusion of visual features in the fourth visual feature sequence and audio features in the fourth audio feature sequence, and the feature dimension of the fused features in the fused feature sequence is the text embedding dimension; the fifth feature sequence is a feature concatenation sequence of the fused feature sequence and the prompt word features, and the feature dimension of the fifth feature in the fifth feature sequence is the text embedding dimension; the sixth feature sequence is a sequence formed by the features of the fifth feature in the fifth feature sequence after dimensionality reduction and then dimensionality increase, and the feature dimension of the sixth feature in the sixth feature sequence is the text embedding dimension.

[0136] See Figure 8 ,Should Figure 8 This illustration shows a method for fusing a third visual feature sequence, a third audio feature sequence, and cue word features to obtain a second fused feature sequence, as provided in this application embodiment. Based on the above, the fourth visual feature sequence and the fourth audio feature sequence undergo feature fusion at the input attention layer to obtain a fused feature sequence; this fused feature sequence and the cue word features undergo feature fusion to obtain a fifth feature sequence; this fifth feature sequence undergoes feature dimensionality reduction based on a low-rank adapter and feature dimensionality increase based on an up-dimensional matrix to obtain a sixth feature sequence; this fifth feature sequence and the sixth feature sequence can be fused into the second fused feature sequence.

[0137] This application provides a possible implementation, in which S15 may include, for example, the following S15e-S15g (not shown in the figure).

[0138] S15e: Through the attention layer, the fourth visual feature sequence, the fourth audio feature sequence, and the cue word features are fused to obtain the seventh feature sequence.

[0139] S15f: Using the dimension reduction and dimension increase matrices in the low-rank adapter, the seventh feature sequence is subjected to feature dimension reduction and feature dimension increase to obtain the eighth feature sequence.

[0140] S15g: Perform feature fusion on the seventh and eighth feature sequences to obtain the second fused feature sequence.

[0141] Among them, the seventh feature sequence is a sequence formed by the deep fusion of visual features in the fourth visual feature sequence, audio features in the fourth audio feature sequence, and prompt word features. The feature dimension of the seventh feature in this seventh feature sequence is the text embedding dimension. The eighth feature sequence is a sequence formed by the features of the seventh feature in the seventh feature sequence after dimensionality reduction and then dimensionality increase. The feature dimension of the eighth feature in this eighth feature sequence is the text embedding dimension.

[0142] See Figure 9 ,Should Figure 9 This is a schematic diagram illustrating another method for fusing a third visual feature sequence, a third audio feature sequence, and cue word features to obtain a second fused feature sequence, as provided in an embodiment of this application. Based on the above, the fourth visual feature sequence, the fourth audio feature sequence, and the cue word features are fused through an attention layer to obtain a seventh feature sequence; this seventh feature sequence is then subjected to feature dimensionality reduction based on a dimensionality reduction matrix and feature dimensionality increase based on an dimensionality increase matrix by a low-rank adapter to obtain an eighth feature sequence; this seventh feature sequence and the eighth feature sequence can be fused into the second fused feature sequence.

[0143] Furthermore, in this embodiment, after generating multiple candidate core contents for the target audio and video using the core content generation model through S14-S16, considering that the more the candidate core contents match the target audio and video semantically, the more accurately the candidate core contents reflect the core of the target audio and video, and thus can be used as the target core contents of the target audio and video; therefore, the semantic matching degree between each candidate core content and the target audio and video is calculated using the semantic features of each candidate core content and the semantic features of the target audio and video; the candidate core content corresponding to the highest semantic matching degree is used as the target core contents of the target audio and video. Based on this, this application provides a possible implementation method, which may further include, for example, the following S17-S18 (not shown in the figure).

[0144] S17: Perform semantic matching calculations on the semantic features of each candidate core content and the semantic features of the target audio and video to obtain the semantic matching degree between each candidate core content and the target audio and video.

[0145] S18: The candidate core content corresponding to the highest semantic matching degree is determined as the target core content of the target audio and video.

[0146] In this context, the semantic features of each candidate core content are the representation features of the semantic information of that candidate core content; the semantic features of the target audio and video are the representation features of the semantic information of the target audio and video; and the semantic matching degree between each candidate core content and the target audio and video is the similarity metric value between the semantic information of the candidate core content and the semantic information of the target audio and video.

[0147] The steps for obtaining the semantic features of each candidate core content can be: performing semantic extraction on each candidate core content to obtain its semantic features. Similarly, the steps for obtaining the semantic features of the target audio / video can be: performing semantic extraction on the target audio / video to obtain its semantic features.

[0148] This method extracts semantic features from each candidate core content and target audio / video for multiple candidate core content and target audio / video. It calculates the semantic matching degree between each candidate core content and the target audio / video, thus determining the degree of matching between the semantic information of each candidate core content and the semantic information of the target audio / video. Based on the principle that the higher the semantic matching degree between each candidate core content and the target audio / video, the more closely the semantic information of the candidate core content matches the semantic information of the target audio / video, the candidate core content with the highest semantic matching degree is taken as the target core content of the target audio / video, so that the target core content can most accurately reflect the core of the target audio / video.

[0149] See Figure 10 ,Should Figure 10This illustration provides a method for determining the target core content from multiple candidate core contents in an embodiment of this application. Based on the above, on one hand, each candidate core content undergoes semantic extraction to obtain its semantic features; on the other hand, the target audio / video undergoes semantic extraction to obtain its semantic features; the semantic features of each candidate core content and the semantic features of the target audio / video are matched to obtain the semantic matching degree between each candidate core content and the target audio / video; for each candidate core content and the target audio / video, the candidate core content corresponding to the highest semantic matching degree can be determined, thereby determining the target core content of the target audio / video.

[0150] Furthermore, in this embodiment, after executing S14-S16 to generate multiple candidate core contents for the target audio and video using the core content generation model, it is further considered that the core content of the audio and video has a certain length limit, that is, the core content of the audio and video needs to conform to a preset length range. Therefore, candidate core contents whose content length conforms to the preset length range can be filtered out from multiple candidate core contents as multiple filtered candidate core contents. Then, in S17, the semantic features of each candidate core content and the semantic features of the target audio and video are matched and calculated to obtain the semantic matching degree between each candidate core content and the target audio and video. Specifically, the semantic matching degree between each filtered candidate core content and the target audio and video is calculated by using the semantic features of each filtered candidate core content and the semantic features of the target audio and video, so that the filtered candidate core content corresponding to the highest semantic matching degree is used as the target core content of the target audio and video.

[0151] Based on this, this application provides a possible implementation method, which may further include the following S19 (not shown in the figure): filtering multiple candidate core contents according to a preset length range to obtain multiple filtered candidate core contents; the content length of each filtered candidate core content conforms to the preset length range; correspondingly, the above S17 may include S17a (not shown in the figure): matching and calculating the semantic features of each filtered candidate core content with the semantic features of the target audio and video to obtain the semantic matching degree between each filtered candidate core content and the target audio and video.

[0152] Among them, the filtered candidate core content is candidate core content whose content length conforms to the preset length range.

[0153] This method targets multiple candidate core contents of the target audio / video. It filters out candidate core contents whose length falls within a preset range, using these as the filtered candidate core contents. This prevents candidate core contents whose length does not meet the preset range from being used as the target audio / video's core content, thus ensuring that the target audio / video's core content conforms to the preset length range. Semantic features of each filtered candidate core content are then extracted, along with the semantic features of the target audio / video. The semantic matching degree between each filtered candidate core content and the target audio / video is calculated, determining the degree of matching between their semantic information and that of the target audio / video. This lays the foundation for identifying the core content that best reflects the target audio / video and falls within the preset length range, while also reducing computational resources.

[0154] As an example, the preset length range can be 8-20 characters.

[0155] See Figure 11 ,Should Figure 11 This is another schematic diagram illustrating how to determine the target core content from multiple candidate core contents for a target audio / video, as provided in this application embodiment. Based on the above, the multiple candidate core contents are filtered according to a preset length range to obtain multiple filtered candidate core contents. On one hand, each filtered candidate core content undergoes semantic extraction to obtain semantic features. On the other hand, the target audio / video undergoes semantic extraction to obtain semantic features. The semantic features of each filtered candidate core content and the semantic features of the target audio / video are matched to obtain the semantic matching degree between each filtered candidate core content and the target audio / video. For each filtered candidate core content and the target audio / video, the filtered candidate core content corresponding to the highest semantic matching degree can be determined, thereby determining the target core content of the target audio / video.

[0156] In summary, the audio and video core content processing method provided in this application uses the attention layer in the pre-generated model (or core content generation model) to perform deeper interaction and fusion of different modal features. Specifically, the attention layer automatically learns the correlation weights between different modal features based on the attention mechanism, so that the generated core content can more accurately reflect the core of the audio and video.

[0157] Based on the original network layers of the pre-generated model, a low-rank adapter is introduced to efficiently fine-tune the parameters of the pre-generated model through low-rank decomposition technology. Specifically, the low-rank adapter decomposes the weight matrix into the product of two low-rank matrices, eliminating the need to train the attention layer corresponding to the low-rank adapter, greatly reducing the number of training parameters, lowering training costs, and significantly reducing the required computing resources and time, while maintaining the expressive power of the model, making it easy to apply even in resource-constrained scenarios.

[0158] Furthermore, this core content generation model is better adapted to specific domains. Specifically, through fine-tuning training with a small amount of domain-specific data, the model can quickly learn the language style and feature patterns of that domain, generating more targeted and personalized core content. For example, in the food audio-visual field, it can generate more attractive and professional core content for food-related audio-visual videos, drawing the attention of more food enthusiasts.

[0159] It should be noted that, based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods.

[0160] The audio and video core content processing method provided in this application is applied to audio and video core content generation scenarios, thereby efficiently and accurately generating suitable and accurate core content for audio and video, such as title content, summary content, brief description content, or summary content. Utilizing the core content generation model in this audio and video core content processing method, when self-media creators upload audio and video, it can quickly generate more attractive and accurate core content based on multimodal information such as visual and audio information in the audio and video, solving problems in related technologies such as insufficient multimodal feature fusion, large number of model training parameters and high cost, difficulty in applying to specific fields, and lack of specificity and personalization. When performing automated management on audio and video platforms, it can automatically add suitable and accurate core content to massive amounts of audio and video to improve recommendation effects, thus having wide applications in audio and video creation platforms, audio and video operation platforms, etc.

[0161] based on Figure 3 Corresponding to the audio and video core content processing method provided in the embodiments, this application also provides an audio and video core content processing apparatus, see below. Figure 12 ,Should Figure 12 This is a structural diagram of an audio and video core content processing device provided in an embodiment of the present application. The audio and video core content processing device 1200 includes: a feature projection unit 1201, a feature fusion unit 1202, a core content generation unit 1203, and a fine-tuning training unit 1204. The feature projection unit 1201 is used to project the first visual feature sequence of the keyframe sequence in the sample audio and video and the first audio feature sequence of the audio frame sequence in the sample audio and video through the cross-modal projection layer in the preset generation model, so as to obtain a second visual feature sequence and a second audio feature sequence that conform to the text embedding dimension of the preset generation model. The feature fusion unit 1202 is used to fuse the second visual feature sequence, the second audio feature sequence, and the prompt word features of the core content prompt words through the attention layer and low-rank adapter in the preset generation model to obtain the first fused feature sequence. The core content generation unit 1203 is used to generate core content from the first fusion feature sequence through the generation layer in the preset generation model, so as to obtain the predicted core content of the sample audio and video. The fine-tuning training unit 1204 is used to fine-tune the cross-modal projection layer and low-rank adapter based on the difference between the predicted core content and the sample core content corresponding to the sample audio and video, so as to obtain the core content generation model.

[0162] In one possible implementation, the feature fusion unit 1202 is specifically used for: By using an attention layer, feature fusion is performed on the second visual feature sequence and the second audio feature sequence to obtain a multimodal feature sequence; The multimodal feature sequence and the prompt word features are fused to obtain the first feature sequence; The second feature sequence is obtained by performing feature dimensionality reduction and feature dimensionality increase on the first feature sequence using the dimensionality reduction and dimensionality increase matrices in the low-rank adapter. The first feature sequence and the second feature sequence are fused to obtain the first fused feature sequence.

[0163] In one possible implementation, the feature fusion unit 1202 is specifically used for: Through the attention layer, feature fusion is performed on the second visual feature sequence and the second audio feature sequence to obtain the association weight between the visual features in the second visual feature sequence and the audio features in the second audio feature sequence; Based on the association weights, the second visual feature sequence and the second audio feature sequence are weighted to obtain the multimodal feature sequence.

[0164] In one possible implementation, the feature fusion unit 1202 is specifically used for: The third feature sequence is obtained by fusing the second visual feature sequence, the second audio feature sequence, and the cue word features through the attention layer. The fourth feature sequence is obtained by performing feature dimensionality reduction and feature dimensionality increase on the third feature sequence using the dimensionality reduction and dimensionality increase matrices in the low-rank adapter. The third and fourth feature sequences are fused to obtain the first fused feature sequence.

[0165] In one possible implementation, the device 1200 further includes: a first determining unit; The first determining unit is used for: The matrix dimension of the dimensionality reduction matrix is ​​determined based on the preset feature dimension and text embedding dimension; the preset feature dimension is smaller than the text embedding dimension. The matrix dimension of the upscaling matrix is ​​determined based on the text embedding dimension and the preset feature dimension.

[0166] In one possible implementation, fine-tuning the training unit 1204 is specifically used for: The prediction probability of multiple predicted words in the core content is the sum of the prediction probabilities of multiple sample words in the core content of the sample. By maximizing the sum of prediction probabilities, the cross-modal projection layer and low-rank adapter are fine-tuned to obtain the core content generation model.

[0167] In one possible implementation, the cross-modal projection layer includes a first transformation matrix and a second transformation matrix. The first transformation matrix is ​​used to project a first visual feature sequence into a second visual feature sequence, and the second transformation matrix is ​​used to project a first audio feature sequence into a second audio feature sequence. The device 1200 also includes a second determining unit. The second determining unit is used for: The matrix dimension of the first transformation matrix is ​​determined based on the text embedding dimension and the feature dimension of the visual features in the first visual feature sequence. The matrix dimension of the second transformation matrix is ​​determined based on the text embedding dimension and the feature dimension of the audio features in the first audio feature sequence.

[0168] In one possible implementation, the device 1200 further includes: a third determining unit and an optimization unit; The third determining unit is used for: Based on the generated core content of multiple audio and video samples under test using the core content generation model, and the labeled core content of multiple audio and video samples under test, the preset performance index value of the core content generation model is determined; the preset performance index value is used to represent the core content generation performance of the core content generation model. Optimization unit, used for: The core content generation model is optimized based on preset performance index values ​​to obtain an optimized core content generation model.

[0169] In one possible implementation, the feature projection unit 1201 is further configured to: By using the cross-modal projection layer in the core content generation model, feature projection is performed on the third visual feature sequence of the keyframe sequence in the target audio and video, and the third audio feature sequence of the audio frame sequence in the target audio and video, to obtain the fourth visual feature sequence and the fourth audio feature sequence that conform to the text embedding dimension. The feature fusion unit 1202 is also used for: By using the attention layer and low-rank adapter in the core content generation model, the fourth visual feature sequence, the fourth audio feature sequence, and the cue word features are fused to obtain the second fused feature sequence. The core content generation unit 1203 is also used for: By using the generation layer in the core content generation model, core content is generated from the second fusion feature sequence to obtain multiple candidate core contents for the target audio and video.

[0170] In one possible implementation, the device 1200 further includes: a matching calculation unit and a fourth determination unit; Matching calculation unit, used for: The semantic features of each candidate core content are matched and calculated with the semantic features of the target audio and video to obtain the semantic matching degree between each candidate core content and the target audio and video. The fourth determining unit is used for: The candidate core content corresponding to the highest semantic matching degree is determined as the target core content of the target audio and video.

[0171] In one possible implementation, the device 1200 further includes a content filtering unit; The content filtering unit is used to filter multiple candidate core contents according to a preset length range to obtain multiple filtered candidate core contents; the content length of each filtered candidate core content conforms to the preset length range. The matching calculation unit is specifically used for: The semantic features of each filtered candidate core content are matched with the semantic features of the target audio and video to obtain the semantic matching degree between each filtered candidate core content and the target audio and video.

[0172] As can be seen from the above technical solution, the first visual feature sequence of the keyframe sequence in the sample audio and video, and the first audio feature sequence of the audio frame sequence in the sample audio and video are input into the cross-modal projection layer in the preset generation model for feature projection, and the output is a second visual feature sequence and a second audio feature sequence that conform to the text embedding dimension of the preset generation model. Through the feature projection of the cross-modal projection layer, the feature dimensions of the visual features and the feature dimensions of the audio features are aligned to the text embedding dimension of the preset generation model, eliminating the feature dimension differences between different modal features, and laying a structurally compatible foundation for multimodal feature fusion.

[0173] The second visual feature sequence, the second audio feature sequence, and the prompt word features generated from the core content are input into the attention layer and low-rank adapter in the preset generation model for feature fusion, and the first fused feature sequence is output. The feature fusion based on the attention layer realizes the deep fusion of visual and audio features to fully explore the deep correlation between features of different modalities. The introduction of the low-rank adapter provides a technical basis for reducing the number of training parameters when training the model.

[0174] The first fused feature sequence is input into the generation layer of the preset generation model. Based on the powerful generation capability of the preset generation model, the core content generation task is defined, and the predicted core content of the sample audio and video is output, so that the predicted core content accurately reflects the core of the sample audio and video. Based on the difference between the predicted core content and the sample core content corresponding to the sample audio and video, the cross-modal projection layer and low-rank adapter are fine-tuned and trained, so that the preset generation model learns to output the sample core content for the sample audio and video, thus obtaining the core content generation model. There is no need to train the attention layer and the generation layer, which greatly reduces the number of training parameters, thereby reducing the training cost, and enables the core content generation model to generate core content for audio and video efficiently and accurately.

[0175] This application also provides a computer device, which may be a server, see [link to previous document]. Figure 13 ,Should Figure 13This application provides a structural diagram of a server 1300. The server 1300 can vary significantly due to different configurations or performance. It may include one or more processors, such as a central processing unit (CPU) 1322, and a memory 1332, as well as one or more storage media 1330 (e.g., one or more mass storage devices) for storing application programs 1342 or data 1344. The memory 1332 and storage media 1330 can be temporary or persistent storage. The program stored in the storage media 1330 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 1322 may be configured to communicate with the storage media 1330 and execute the series of instruction operations stored in the storage media 1330 on the server 1300.

[0176] Server 1300 may also include one or more power supplies 1326, one or more wired or wireless network interfaces 1350, one or more input / output interfaces 1358, and / or one or more operating systems 1341, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0177] In this embodiment, the central processing unit 1322 in the server 1300 can execute the methods provided in the various optional implementations of the above embodiments.

[0178] The computer device provided in this application embodiment can also be a terminal, see [link to relevant documentation]. Figure 14 ,Should Figure 14 This is a structural diagram of a terminal provided in an embodiment of this application. Taking a smartphone as an example, the smartphone includes: a radio frequency (RF) circuit 1410, a memory 1332, an input unit 1430, a display unit 1440, a sensor 1450, an audio circuit 1460, a wireless Fidelity (WiFi) module 1470, a processor 1480, and a power supply 1490, etc. The input unit 1430 may include a touch panel 1431 and other input devices 1432, the display unit 1440 may include a display panel 1441, and the audio circuit 1460 may include a speaker 1461 and a microphone 1462. Those skilled in the art will understand that... Figure 14 The smartphone structure shown does not constitute a limitation on smartphones and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0179] The memory 1332 can be used to store software programs and modules. The processor 1480 executes various functions and data processing of the smartphone by running the software programs and modules stored in the memory 1332. The memory 1332 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the smartphone (such as audio data, phonebook, etc.). In addition, the memory 1332 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0180] The processor 1480 is the control center of the smartphone, connecting various parts of the smartphone via various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 1332, and by calling data stored in the memory 1332. Optionally, the processor 1480 may include one or more processing units; preferably, the processor 1480 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 1480.

[0181] In this embodiment, the processor 1480 in the smartphone can execute the methods provided in the various optional implementations of the above embodiments.

[0182] According to one aspect of this application, a computer-readable storage medium is provided for storing a computer program that, when run on a computer device, causes the computer device to perform the methods provided in various optional implementations of the above embodiments.

[0183] According to one aspect of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the methods provided in the various optional implementations of the above embodiments.

[0184] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0185] The terms "first," "second," etc., used in this application's specification and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0186] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0187] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0189] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing computer programs, such as USB flash drives, portable hard drives, read-only memory (ROM), RAM, magnetic disks, or optical disks.

[0190] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0191] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for processing core audio and video content, characterized in that, The method includes: By using the cross-modal projection layer in the preset generation model, feature projection is performed on the first visual feature sequence of the keyframe sequence in the sample audio and video, and the first audio feature sequence of the audio frame sequence of the sample audio and video, to obtain a second visual feature sequence and a second audio feature sequence that conform to the text embedding dimension of the preset generation model. The first fused feature sequence is obtained by fusing the second visual feature sequence, the second audio feature sequence, and the prompt word features of the core content generation prompt words through the attention layer and low-rank adapter in the preset generation model. The core content of the sample audio and video is generated by generating the first fused feature sequence through the generation layer in the preset generation model. Based on the difference between the predicted core content and the sample core content corresponding to the sample audio and video, the cross-modal projection layer and the low-rank adapter are fine-tuned and trained to obtain the core content generation model.

2. The method according to claim 1, characterized in that, The first fused feature sequence is obtained by fusing the second visual feature sequence, the second audio feature sequence, and the prompt word features of the core content generation prompt words through the attention layer and low-rank adapter in the preset generation model, including: The attention layer is used to fuse the second visual feature sequence and the second audio feature sequence to obtain a multimodal feature sequence. The multimodal feature sequence and the prompt word features are fused to obtain a first feature sequence; The first feature sequence is subjected to feature dimensionality reduction and feature dimensionality increase through the dimensionality reduction matrix and dimensionality increase matrix in the low-rank adapter to obtain the second feature sequence. The first feature sequence and the second feature sequence are fused to obtain the first fused feature sequence.

3. The method according to claim 2, characterized in that, The step of fusing features from the second visual feature sequence and the second audio feature sequence through the attention layer to obtain a multimodal feature sequence includes: The attention layer is used to fuse the second visual feature sequence and the second audio feature sequence to obtain the association weights between the visual features in the second visual feature sequence and the audio features in the second audio feature sequence. The second visual feature sequence and the second audio feature sequence are weighted according to the association weights to obtain the multimodal feature sequence.

4. The method according to claim 1, characterized in that, The first fused feature sequence is obtained by fusing the second visual feature sequence, the second audio feature sequence, and the prompt word features of the core content generation prompt words through the attention layer and low-rank adapter in the preset generation model, including: Through the attention layer, the second visual feature sequence, the second audio feature sequence, and the cue word feature are fused to obtain a third feature sequence; The fourth feature sequence is obtained by performing feature dimensionality reduction and feature dimensionality increase on the third feature sequence using the dimensionality reduction matrix and dimensionality increase matrix in the low-rank adapter. The third feature sequence and the fourth feature sequence are fused to obtain the first fused feature sequence.

5. The method according to claim 2 or 4, characterized in that, The steps for determining the matrix dimensions of the reduced-dimensional matrix and the increased-dimensional matrix include: The dimension of the dimensionality reduction matrix is ​​determined based on the preset feature dimension and the text embedding dimension; the preset feature dimension is smaller than the text embedding dimension. The matrix dimension of the upsizing matrix is ​​determined based on the text embedding dimension and the preset feature dimension.

6. The method according to claim 1, characterized in that, The step of fine-tuning the cross-modal projection layer and the low-rank adapter based on the difference between the predicted core content and the sample core content corresponding to the sample audio and video to obtain the core content generation model includes: The prediction probabilities of multiple predicted words in the predicted core content being the sum of the prediction probabilities of multiple sample words in the sample core content are determined. The core content generation model is obtained by fine-tuning the cross-modal projection layer and the low-rank adapter by maximizing the sum of the predicted probabilities.

7. The method according to claim 1, characterized in that, The cross-modal projection layer includes a first transformation matrix and a second transformation matrix. The first transformation matrix is ​​used to project the first visual feature sequence into the second visual feature sequence, and the second transformation matrix is ​​used to project the first audio feature sequence into the second audio feature sequence. The steps for determining the matrix dimensions of the first transformation matrix and the second transformation matrix include: The matrix dimension of the first transformation matrix is ​​determined based on the text embedding dimension and the feature dimension of the visual features in the first visual feature sequence. The matrix dimension of the second transformation matrix is ​​determined based on the text embedding dimension and the feature dimension of the audio features in the first audio feature sequence.

8. The method according to claim 1, characterized in that, The method further includes: Based on the generated core content of multiple audio and video samples under test using the core content generation model, and the labeled core content of the multiple audio and video samples under test, a preset performance index value for the core content generation model is determined; the preset performance index value is used to represent the core content generation performance of the core content generation model. The core content generation model is optimized based on the preset performance index value to obtain the optimized core content generation model.

9. The method according to claim 1, characterized in that, The method further includes: Through the cross-modal projection layer in the core content generation model, feature projection is performed on the third visual feature sequence of the keyframe sequence in the target audio and video, and the third audio feature sequence of the audio frame sequence of the target audio and video, to obtain the fourth visual feature sequence and the fourth audio feature sequence that conform to the text embedding dimension. The attention layer and low-rank adapter in the core content generation model are used to fuse the fourth visual feature sequence, the fourth audio feature sequence, and the cue word features to obtain a second fused feature sequence. The core content generation model generates core content from the second fused feature sequence through the generation layer, thereby obtaining multiple candidate core contents for the target audio and video.

10. The method according to claim 9, characterized in that, The method further includes: The semantic features of each candidate core content are matched and calculated with the semantic features of the target audio and video to obtain the semantic matching degree between each candidate core content and the target audio and video; The candidate core content corresponding to the highest semantic matching degree is determined as the target core content of the target audio and video.

11. The method according to claim 10, characterized in that, The method further includes: The multiple candidate core contents are filtered according to a preset length range to obtain multiple filtered candidate core contents; the content length of each filtered candidate core content conforms to the preset length range. The step of matching and calculating the semantic features of each candidate core content with the semantic features of the target audio and video to obtain the semantic matching degree between each candidate core content and the target audio and video includes: The semantic features of each filtered candidate core content are matched with the semantic features of the target audio and video to obtain the semantic matching degree between each filtered candidate core content and the target audio and video.

12. A core content processing device for audio and video, characterized in that, The device includes: a feature projection unit, a feature fusion unit, a core content generation unit, and a fine-tuning training unit; The feature projection unit is used to project the first visual feature sequence of the keyframe sequence in the sample audio and video and the first audio feature sequence of the audio frame sequence in the sample audio and video through the cross-modal projection layer in the preset generation model, so as to obtain a second visual feature sequence and a second audio feature sequence that conform to the text embedding dimension of the preset generation model. The feature fusion unit is used to perform feature fusion on the second visual feature sequence, the second audio feature sequence, and the prompt word features of the core content prompt words through the attention layer and low-rank adapter in the preset generation model to obtain a first fused feature sequence; The core content generation unit is used to generate core content from the first fused feature sequence through the generation layer in the preset generation model, so as to obtain the predicted core content of the sample audio and video. The fine-tuning training unit is used to fine-tune the cross-modal projection layer and the low-rank adapter based on the difference between the predicted core content and the sample core content corresponding to the sample audio and video, so as to obtain the core content generation model.

13. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store computer programs and to transfer the computer programs to the processor; The processor is configured to execute the method according to any one of claims 1-11 according to instructions in the computer program.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when run on a computer device, causes the computer device to perform the method according to any one of claims 1-11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is run on a computer device, it causes the computer device to perform the method according to any one of claims 1-11.

Images