Audio file generation method and device, equipment, storage medium and program product

Abstract

The invention relates to an audio file generation method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining target multimedia content in response to a multimedia content input event; performing multi-modal feature analysis on the target multimedia content based on the feature analysis model to obtain multi-modal features of the target multimedia content; determining an audio control parameter of the target multimedia content based on the multi-modal feature; and generating an audio file of the target multimedia content based on the audio control parameter. By adopting the method, the matching degree of the dubbling music of the multimedia content can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an audio file generation method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Technology

[0002] When sharing multimedia content such as videos and images, users can add background music to enrich their expression.

[0003] In existing technologies, most users manually select music from a preset music library to add background music to multimedia content. However, existing methods suffer from low matching accuracy. Summary of the Invention

[0004] Therefore, it is necessary to provide an audio file generation method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve the matching accuracy in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a method for generating audio files, including:

[0006] Responding to a multimedia content input event, the target multimedia content is retrieved;

[0007] Multimodal feature analysis of the target multimedia content is performed based on the feature analysis model to obtain the multimodal features of the target multimedia content;

[0008] Determine the audio control parameters of the target multimedia content based on multimodal features;

[0009] Based on audio control parameters, an audio file for the target multimedia content is generated.

[0010] In one embodiment, generating an audio file of target multimedia content based on audio control parameters includes: obtaining audio text information of the target multimedia content based on the target multimedia content; and generating an audio file based on the audio control parameters and the audio text information.

[0011] In one embodiment, obtaining audio text information of the target multimedia content based on the target multimedia content includes: inputting the target multimedia content into a picture description model to obtain timeline-based picture description information of the target multimedia content output by the picture description model; and determining audio text information based on the picture description information.

[0012] In one embodiment, the multimodal features include a visual emotion vector, a scene theme vector, and a motion rhythm vector; the feature analysis model includes a first analysis unit, a second analysis unit, and a third analysis unit; multimodal feature analysis is performed on the target multimedia content to obtain the multimodal features of the target multimedia content, including: using the first analysis unit to determine the visual emotion vector based on a convolutional neural network and the target multimedia content; using the second analysis unit to determine the scene theme vector based on a hybrid model and the target multimedia content; and using the third analysis unit to determine the motion rhythm vector based on optical flow and the target multimedia content.

[0013] In one embodiment, the first analysis unit is used to determine the visual emotion vector based on the convolutional neural network and the target multimedia content, including: using the first analysis unit to input the target multimedia content into the convolutional neural network to obtain at least one of the color distribution information, facial expression information, and body movement intensity information of the target multimedia content output by the convolutional neural network; generating multidimensional emotion coordinates based on at least one of the color distribution information, facial expression information, and body movement intensity information; and determining the visual emotion vector based on the multidimensional emotion coordinates.

[0014] In one embodiment, the second analysis unit is used to determine the scene topic vector based on the hybrid model and the target multimedia content, including: using the second analysis unit to input the target multimedia content into the hybrid model to obtain the scene semantic tag of the target multimedia content output by the hybrid model; and determining the scene topic vector based on the scene semantic tag.

[0015] In one embodiment, the motion rhythm vector is determined using a third analysis unit based on optical flow and target multimedia content, including: determining the inter-frame motion energy of the target multimedia content based on optical flow and target multimedia content using the third analysis unit; generating a rhythm intensity curve and BPM recommendation information based on the inter-frame motion energy; and determining the motion rhythm vector based on the rhythm intensity curve and BPM recommendation information.

[0016] In one embodiment, generating an audio file of target multimedia content based on audio control parameters includes: generating at least one of melody information, harmony information, rhythm information, and bass information based on the audio control parameters; and generating an audio file based on at least one of the melody information, harmony information, rhythm information, and bass information.

[0017] Secondly, this application also provides an audio file generation apparatus, comprising:

[0018] The acquisition module is used to acquire target multimedia content in response to multimedia content input events;

[0019] The execution module is used to perform multimodal feature analysis on the target multimedia content based on the feature analysis model, and to obtain the multimodal features of the target multimedia content.

[0020] The determination module is used to determine the audio control parameters of the target multimedia content based on multimodal features;

[0021] The generation module is used to generate audio files for the target multimedia content based on audio control parameters.

[0022] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the embodiments of the first aspect above.

[0023] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the embodiments of the first aspect above.

[0024] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the embodiments of the first aspect above.

[0025] The aforementioned audio file generation method, apparatus, computer device, computer-readable storage medium, and computer program product, in response to a multimedia content input event, acquire target multimedia content, then perform multimodal feature analysis on the target multimedia content based on a feature analysis model to obtain multimodal features of the target multimedia content, then determine audio control parameters of the target multimedia content based on the multimodal features, and finally generate an audio file of the target multimedia content based on the audio control parameters. The audio file generation method provided in this application, because it generates audio files of multimedia content based on multimedia content, achieves a higher degree of matching with the multimedia content compared to audio files obtained from a preset music library in the prior art, thus avoiding the problem of low matching accuracy. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a diagram illustrating the application environment of an audio file generation method in one embodiment.

[0028] Figure 2 This is a flowchart illustrating an audio file generation method in one embodiment;

[0029] Figure 3 This is a flowchart illustrating a method for generating an audio file in one embodiment;

[0030] Figure 4 This is a flowchart illustrating a method for obtaining audio text information in one embodiment;

[0031] Figure 5 This is a flowchart illustrating a method for obtaining multimodal features in one embodiment;

[0032] Figure 6 This is a flowchart illustrating a method for determining visual emotion vectors in one embodiment;

[0033] Figure 7 This is a flowchart illustrating a method for determining scene topic vectors in one embodiment;

[0034] Figure 8 This is a flowchart illustrating a method for determining a motion rhythm vector in one embodiment;

[0035] Figure 9 This is a flowchart illustrating a method for generating audio files in another embodiment;

[0036] Figure 10 This is a flowchart illustrating the audio file generation method in another embodiment;

[0037] Figure 11 This is a structural block diagram of an audio file generation device in one embodiment;

[0038] Figure 12 This is an internal structural diagram of a computer device in one embodiment;

[0039] Figure 13 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0041] It should be noted that the terms "comprising" and "having," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusion. The term "multiple" as used in this application refers to two or more. The term "and / or" as used in this application refers to one of the solutions, or any combination of multiple solutions.

[0042] In scenarios involving the sharing of multimedia content such as videos and images, background music, as a key auditory element, provides an optimized path for the expression of multimedia content. Users can enrich the expression of multimedia content by adding background music.

[0043] In existing technologies, users mostly manually select music from a preset music library to add background music to multimedia content. However, since the number of songs in the preset music library is limited, there may not be any music suitable for the multimedia content, resulting in a low match between the background music and the multimedia content, which affects the user experience.

[0044] In view of this, this application provides an audio file generation method. In response to a multimedia content input event, it acquires target multimedia content, then performs multimodal feature analysis on the target multimedia content based on a feature analysis model to obtain its multimodal features. Next, it determines the audio control parameters of the target multimedia content based on these multimodal features. Finally, it generates an audio file of the target multimedia content based on these audio control parameters. Because the audio file generation method provided by this application generates an audio file of multimedia content based on the multimedia content itself, this audio file has a higher degree of matching with the multimedia content compared to audio files obtained from a preset music library in existing technologies. This avoids the problem of low matching accuracy and improves the user experience.

[0045] The audio file generation method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network.

[0046] The execution subject of the audio file generation method provided in this application embodiment can be a computer device, which can be a terminal 102 or a server 104. Optionally, when the execution subject is a terminal 102, the terminal responds to a multimedia content input event, acquires the target multimedia content, performs multimodal feature analysis on the target multimedia content based on a feature analysis model to acquire the multimodal features of the target multimedia content, then determines the audio control parameters of the target multimedia content based on the multimodal features, and finally generates an audio file of the target multimedia content based on the audio control parameters, so as to use the audio file as background music for the target multimedia content. When the executing entity is server 104, server 104 responds to the multimedia content input event, communicates with terminal 102 via the network to obtain the target multimedia content, performs multimodal feature analysis on the target multimedia content using a feature analysis model to obtain the multimodal features of the target multimedia content, then determines the audio control parameters of the target multimedia content based on the multimodal features, and finally generates the audio file of the target multimedia content based on the audio control parameters, and communicates with terminal 102 via the network to send the audio file of the target multimedia content, or the target multimedia content with added audio files as background music, to terminal 102.

[0047] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearables. Portable wearable devices can be smartwatches, head-mounted displays, etc. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0048] For example, a client of the target application can be installed on terminal 102. This target application can be an application for publishing and browsing multimedia content. Specifically, the target application can be a social media application. Furthermore, this application is not limited to any particular form; it can be a parent application running on an operating system, a sub-application running within a parent application (such as a mini-program), or a webpage.

[0049] In one exemplary embodiment, such as Figure 2 As shown, an audio file generation method is provided, which includes the following steps:

[0050] Step 201: In response to the multimedia content input event, obtain the target multimedia content.

[0051] Optionally, multimedia content can be still images, moving images, videos, etc. For example, multimedia content can be pre-recorded by the user, currently being recorded by the user, or pre-stored by the user. The multimedia content input event can be used to input multimedia content. The target multimedia content is multimedia content for which background music needs to be added.

[0052] In some exemplary embodiments, a multimedia content input event can be executed when a user has a need to add background music to multimedia content.

[0053] Furthermore, in response to a multimedia content input event, the computer device can acquire the multimedia content indicated by the user's selected operation and identify that multimedia content as the target multimedia content.

[0054] Specifically, in an optional embodiment of this application, the computer device may further display a multimedia content input page, which includes a multimedia content input control. In response to a trigger operation on the multimedia content input control, at least one multimedia content is displayed, and in response to a trigger operation on any one of the at least one multimedia content, the multimedia content is identified as the target multimedia content.

[0055] In another optional embodiment of this application, the computer device may also display a multimedia content input page, which includes a multimedia content input area, and in response to a multimedia content paste operation on the multimedia content input area, the multimedia content indicated by the paste operation is determined as the target multimedia content.

[0056] In another optional embodiment of this application, the computer device may further display a multimedia content shooting page, which includes a multimedia content shooting control. In response to a trigger operation on the multimedia content shooting control, multimedia content is shot, and in response to the end of multimedia content shooting, the shot multimedia content is identified as the target multimedia content.

[0057] Step 202: Perform multimodal feature analysis on the target multimedia content based on the feature analysis model to obtain the multimodal features of the target multimedia content.

[0058] Optionally, the feature analysis model can be a convolutional neural network model, a recurrent neural network model, a long short-term memory network model, a neural network model based on a self-attention mechanism, a graph neural network model, or a hybrid model composed of the above models.

[0059] Multimodal features are used to characterize the features of target multimedia content in different modalities. Multimodal features can include visual features, scene features, motion features, etc.

[0060] In some exemplary embodiments, after acquiring target multimedia content, the computer device can perform multimodal feature analysis on the target multimedia content based on a feature analysis model to obtain the multimodal features of the target multimedia content.

[0061] Specifically, in one optional embodiment of this application, the computer device can input the target multimedia content into the feature analysis model, so that the feature analysis model performs multimodal feature analysis on the target multimedia content to obtain the multimodal features of the target multimedia content output by the feature analysis model.

[0062] In another optional embodiment of this application, the feature analysis model may include multiple sub-models, with different sub-models used to represent features of different modalities. The computer device may input the target multimedia content into the multiple sub-models to obtain the features of different modalities of the target multimedia content output by the multiple sub-models, and summarize the features of different modalities of the target multimedia content output by the multiple sub-models to obtain the multimodal features of the target multimedia content.

[0063] Step 203: Determine the audio control parameters of the target multimedia content based on multimodal features.

[0064] Optionally, audio control parameters refer to parameters used to control the generated audio file. For example, audio control parameters may include rhythm parameters, emotional tone parameters, structural hierarchy parameters, style adaptation parameters, instrument parameters, etc.

[0065] Rhythm parameters can include BPM (Beats Per Minute), beat type, rhythm density, etc.

[0066] BPM controls the speed of the generated audio file. For example, a BPM of 60-80 BPM produces a relaxed audio file, while a BPM of 120-140 BPM produces a brisk audio file. Beat type controls the strength of the generated audio file's rhythm. For example, a 4 / 4 beat generates a strong-weak-secondary strong-weak rhythm, while a 3 / 4 beat generates a strong-weak-weak rhythm. Rhythm density controls the number of notes / percussion instruments per unit of time in the generated audio file.

[0067] Emotional color parameters can include mode, harmony type, etc.

[0068] The key can be used to control the emotional tone of the generated audio file. For example, a major key produces a bright and cheerful audio file; a minor key produces a melancholic and serene audio file; a C major key produces a refreshing audio file; and a G major key produces an exhilarating audio file. The harmonic type can be used to control the emotional level of the generated audio file. For example, a triad harmonic produces a stable and warm emotional level; and a seventh chord harmonic produces a tense and lively emotional level.

[0069] Parameters for structural hierarchy can include volume, dynamic range, and instrument ratio.

[0070] The volume setting controls the volume of the generated audio file. The dynamic range setting controls the difference between the loudest and softest notes in the generated audio file. The instrument ratio setting controls the proportion of instruments in the melody, harmony, rhythm, and bass layers of the generated audio file.

[0071] Style adaptation parameters can include genre tags, etc.

[0072] The genre tag is used to control the arrangement logic and instrument combination of the generated audio file, such as pop, classical, electronic, folk, etc.

[0073] In some exemplary embodiments, after acquiring the multimodal features of the target multimedia content, the computer device can determine the audio control parameters of the target multimedia content based on the multimodal features.

[0074] Specifically, in an optional embodiment of this application, the computer device can directly input multimodal features into the audio control parameter determination model to obtain the audio control parameters of the target multimedia content output by the audio control parameter determination model.

[0075] In another optional embodiment of this application, the computer device may first perform feature fusion processing on the multimodal features to obtain fused features, and then input the fused features into the audio control parameter determination model to obtain the audio control parameters of the target multimedia content output by the audio control parameter determination model.

[0076] In the case where the multimodal features include three features, the fused feature can be represented as Pm=W1×V1⊕W2×V2⊕W3×V3, where ⊕ represents feature concatenation, W1, W2, and W3 are trainable weight matrices, and each weight matrix can be adjusted through interactive slider operation.

[0077] Step 204: Generate the audio file of the target multimedia content based on the audio control parameters.

[0078] Optionally, the audio file of the target multimedia content is also the background music for the target multimedia content.

[0079] In some exemplary embodiments, after determining the audio control parameters of the target multimedia content, the computer device can generate an audio file of the target multimedia content based on the audio control parameters.

[0080] Specifically, in one optional embodiment of this application, the computer device can input audio control parameters into the audio file generation model to obtain the audio file of the target multimedia content output by the audio file generation model.

[0081] The aforementioned audio file generation method responds to a multimedia content input event, acquires the target multimedia content, performs multimodal feature analysis on the target multimedia content based on a feature analysis model to obtain its multimodal features, determines audio control parameters based on these features, and finally generates an audio file of the target multimedia content based on these parameters. The audio file generation method provided in this application, because it generates an audio file of multimedia content based on the multimedia content itself, achieves a higher degree of matching with the multimedia content compared to audio files obtained from a preset music library in existing technologies, thus avoiding the problem of low matching accuracy.

[0082] In one exemplary embodiment, such as Figure 3 As shown, generating an audio file for the target multimedia content based on audio control parameters includes the following steps:

[0083] Step 301: Based on the target multimedia content, obtain the audio text information of the target multimedia content.

[0084] Optionally, the audio text information of the target multimedia content refers to lyrics.

[0085] In some exemplary embodiments, a computer device may acquire audio text information of target multimedia content based on the target multimedia content.

[0086] Specifically, in one optional embodiment of this application, the computer device can input the target multimedia content into the audio text information of the target multimedia content output in the audio text information determination model.

[0087] Step 302: Generate an audio file based on the audio control parameters and audio text information.

[0088] In some exemplary embodiments, after obtaining the audio text information of the target multimedia content based on the target multimedia content, the computer device can generate an audio file according to the audio control parameters and the audio text information.

[0089] Specifically, in one optional embodiment of this application, the computer device can input audio control parameters and audio text information into an audio file generation model to obtain an audio file output by the audio file generation model. The audio file generation model can be a generative adversarial network (GAN) model.

[0090] In this embodiment, the method of obtaining audio text information of the target multimedia content based on the target multimedia content and generating an audio file based on the audio control parameters and the audio text information has a high degree of matching with the target multimedia content and a high degree of richness, effectively improving the user experience.

[0091] In one exemplary embodiment, such as Figure 4 As shown, obtaining the audio text information of the target multimedia content, based on the target multimedia content, includes the following steps:

[0092] Step 401: Input the target multimedia content into the screen description model to obtain the timeline-based screen description information of the target multimedia content output by the screen description model.

[0093] Optionally, the image description model can be a temporal attention model, a convolutional long short-term memory network model, a residual network, a neural network model based on an attention mechanism, or a hybrid model composed of the above models.

[0094] Image description information refers to the standardized semantic description of the core visual elements of the target multimedia content at each time period in chronological order by the image description model, outputting temporally structured visual semantic data. The core visual elements can include scene type, main elements, action states, visual atmosphere, etc.

[0095] In some exemplary embodiments, a computer device may input target multimedia content into a picture description model to obtain timeline-based picture description information of the target multimedia content output by the picture description model.

[0096] Step 402: Determine the audio text information based on the screen description information.

[0097] In some exemplary embodiments, the computer device inputs the target multimedia content into the picture description model to obtain the timeline-based picture description information of the target multimedia content output by the picture description model, and then determines the audio text information based on the picture description information.

[0098] Specifically, in one optional embodiment of this application, the computer device can input screen description information into the audio text generation model to obtain audio text information output by the audio text generation model.

[0099] In another optional embodiment of this application, the computer device can perform semantic parsing on the screen description information to obtain target semantic features; determine the similarity between the target semantic features and each semantic feature in a preset semantic feature-audio text template mapping relationship list, wherein the semantic feature-audio text template mapping relationship list is used to indicate the mapping relationship between different semantic features and different audio text templates; determine the audio text template corresponding to the semantic feature with the highest similarity as the target audio model; and generate audio text information based on the target semantic features and the target audio model.

[0100] In this embodiment, the target multimedia content is input into the screen description model to obtain timeline-based screen description information of the target multimedia content output by the screen description model. The method of determining audio text information based on the screen description information makes the determined audio text information more compatible with the target multimedia content, thereby improving the matching degree of the generated audio file.

[0101] In one exemplary embodiment, such as Figure 5 As shown, the multimodal features include visual emotion vectors, scene theme vectors, and motion rhythm vectors; the feature analysis model includes a first analysis unit, a second analysis unit, and a third analysis unit; multimodal feature analysis is performed on the target multimedia content to obtain its multimodal features, including the following steps:

[0102] Step 501: Using the first analysis unit, determine the visual emotion vector based on the convolutional neural network and the target multimedia content.

[0103] Optionally, visual emotion vectors can be used to characterize the tendency and intensity of the emotional dimension of the target multimedia content.

[0104] In some exemplary embodiments, the computer device may utilize a first analysis unit to determine a visual emotion vector based on a convolutional neural network and target multimedia content.

[0105] Specifically, in an optional embodiment of this application, the computer device may use the first analysis unit to input the target multimedia content into a convolutional neural network to obtain the visual emotion vector output by the convolutional neural network.

[0106] In another optional embodiment of this application, the computer device may use the first analysis unit to input the target multimedia content into a convolutional neural network to obtain at least one of the color distribution information, facial expression information, and body movement intensity information of the target multimedia content output by the convolutional neural network; generate multidimensional emotion coordinates based on at least one of the color distribution information, facial expression information, and body movement intensity information; and determine a visual emotion vector based on the multidimensional emotion coordinates.

[0107] Step 502: Using the second analysis unit, determine the scene theme vector based on the hybrid model and target multimedia content.

[0108] Optionally, the hybrid model can be composed of a residual network model and a Transformer model. Scene topic vectors can be used to indicate scene topic information of the target multimedia content.

[0109] In some exemplary embodiments, the computer device may utilize a second analysis unit to determine scene topic vectors based on a hybrid model and target multimedia content.

[0110] Specifically, in one optional embodiment of this application, the computer device can input the target multimedia content into the hybrid model to obtain the scene topic vector output by the hybrid model.

[0111] In another optional embodiment of this application, the computer device may use the second analysis unit to input the target multimedia content into the hybrid model to obtain the scene semantic tags of the target multimedia content output by the hybrid model; and determine the scene topic vector based on the scene semantic tags.

[0112] Step 503: Using the third analysis unit, determine the motion rhythm vector based on optical flow and target multimedia content.

[0113] Optical flow refers to a method that quantifies the motion state of objects and dynamic changes in the image of a target multimedia content by calculating the displacement vectors of pixels within the image. Motion rhythm vectors can be used to indicate the motion rhythm information of the target multimedia content.

[0114] In some exemplary embodiments, the computer device may utilize a third analysis unit to determine the motion rhythm vector based on optical flow and target multimedia content.

[0115] Specifically, in an optional embodiment of this application, the computer device may utilize a third analysis unit to determine the inter-frame motion energy of the target multimedia content based on optical flow and the target multimedia content; generate a rhythm intensity curve and BPM recommendation information based on the inter-frame motion energy; and determine the motion rhythm vector based on the rhythm intensity curve and BPM recommendation information.

[0116] In this embodiment, a first analysis unit is used to determine visual emotion vectors based on convolutional neural networks and target multimedia content; a second analysis unit is used to determine scene theme vectors based on a hybrid model and target multimedia content; and a third analysis unit is used to determine motion rhythm vectors based on optical flow and target multimedia content. This method improves the accuracy of the determined multimodal features and thus improves the matching degree of the generated audio files.

[0117] In one exemplary embodiment, such as Figure 6 As shown, the visual sentiment vector is determined using the first analysis unit based on a convolutional neural network and the target multimedia content, including the following steps:

[0118] Step 601: Using the first analysis unit, input the target multimedia content into the convolutional neural network to obtain at least one of the color distribution information, facial expression information, and body movement intensity information of the target multimedia content output by the convolutional neural network.

[0119] Optionally, color distribution information can be used to characterize the visual color attributes and atmospheric features of the target multimedia content. For example, color distribution information may include RGB / HSV channel brightness distribution data, warm / cool pixel ratio, dominant color type and ratio, color contrast ratio, color uniformity quantification index, color saturation distribution range, etc.

[0120] Facial expression information can be used to characterize the emotional tendencies and intensity of individuals in target multimedia content. For example, facial expression information may include expression category labels, expression intensity quantification values, morphological features of key facial regions, expression duration, and information on the distribution of expressions among multiple individuals.

[0121] Body movement intensity information can be used to characterize the intensity of a person's body movements in target multimedia content. For example, body movement intensity information may include the displacement amplitude of limb key points, joint movement speed, limb movement frequency, proportion of the moving area in the frame, peak limb movement acceleration, changes in limb posture between adjacent frames, and intensity rating labels for the movement.

[0122] In some exemplary embodiments, the computer device may use a first analysis unit to input target multimedia content into a convolutional neural network to obtain at least one of color distribution information, facial expression information, and limb movement intensity information of the target multimedia content output by the convolutional neural network.

[0123] Step 602: Generate multidimensional emotion coordinates based on at least one of color distribution information, facial expression information, and body movement intensity information.

[0124] Optionally, the multidimensional emotional coordinate system can be a nine-dimensional emotional coordinate system. For example, the nine-dimensional emotional coordinate system includes pleasure, arousal, tension, warmth, epic feeling, seriousness, humor, elegance, and fear.

[0125] In some exemplary embodiments, after the computer device uses the first analysis unit to input the target multimedia content into the convolutional neural network to obtain at least one of the color distribution information, facial expression information, and body movement intensity information of the target multimedia content output by the convolutional neural network, it can generate multidimensional emotion coordinates based on at least one of the color distribution information, facial expression information, and body movement intensity information.

[0126] Specifically, in one optional embodiment of this application, the computer device can input at least one of color distribution information, facial expression information, and body movement intensity information into the multidimensional emotion coordinate determination model to obtain the multidimensional emotion coordinates output by the multidimensional emotion coordinate determination model.

[0127] Step 603: Determine the visual emotion vector based on the multidimensional emotion coordinates.

[0128] In some exemplary embodiments, after generating multidimensional emotion coordinates based on at least one of color distribution information, facial expression information, and body movement intensity information, the computer device can determine a visual emotion vector based on the multidimensional emotion coordinates.

[0129] Specifically, in one optional embodiment of this application, the computer device can directly determine the multidimensional emotion coordinates as a visual emotion vector.

[0130] In another optional embodiment of this application, the computer device may further filter emotional coordinates from a portion of the multi-dimensional emotional coordinates to serve as a visual emotional vector. The filtering rules may include scene adaptation priority filtering rules, music emotional relevance filtering rules, feature importance filtering rules, user-defined filtering rules, etc.

[0131] In this embodiment, the first analysis unit is used to input the target multimedia content into a convolutional neural network to obtain at least one of the color distribution information, facial expression information, and body movement intensity information of the target multimedia content output by the convolutional neural network; multidimensional emotion coordinates are generated based on at least one of the color distribution information, facial expression information, and body movement intensity information; the method of determining the visual emotion vector based on the multidimensional emotion coordinates improves the accuracy of the determined visual emotion vector, thereby improving the matching degree of the generated audio file.

[0132] In one exemplary embodiment, such as Figure 7As shown, using the second analysis unit, scene topic vectors are determined based on the hybrid model and target multimedia content, including the following steps:

[0133] Step 701: Using the second analysis unit, input the target multimedia content into the hybrid model to obtain the scene semantic tags of the target multimedia content output by the hybrid model.

[0134] Optionally, scene semantic tags can be used to indicate the scene type of the target multimedia content.

[0135] For example, scene semantic tags can include natural scene categories, such as grassland, seaside, forest, and mountain range; urban scene categories, such as street, office building, shopping mall, and park; indoor scene categories, such as coffee shop, bedroom, conference room, and cinema; functional scene categories, such as wedding, sports event, documentary, and advertising; and atmosphere scene categories, such as tranquil, lively, romantic, and suspenseful.

[0136] In some exemplary embodiments, the computer device may use a second analysis unit to input the target multimedia content into a hybrid model to obtain the scene semantic tags of the target multimedia content output by the hybrid model.

[0137] Step 702: Determine the scene topic vector based on the scene semantic tags.

[0138] In some exemplary embodiments, after the computer device uses the second analysis unit to input the target multimedia content into the hybrid model to obtain the scene semantic tags of the target multimedia content output by the hybrid model, it can determine the scene topic vector based on the scene semantic tags.

[0139] Specifically, in an optional embodiment of this application, the computer device can directly determine the scene semantic tags as scene topic vectors.

[0140] In another optional embodiment of this application, the computer device may perform a weighted summation of scene semantic tags to determine the scene topic vector.

[0141] In this embodiment, the second analysis unit is used to input the target multimedia content into the hybrid model to obtain the scene semantic tags of the target multimedia content output by the hybrid model; the method of determining the scene topic vector based on the scene semantic tags improves the accuracy of the determined scene topic vector, thereby improving the matching degree of the generated audio file.

[0142] In one exemplary embodiment, such as Figure 8 As shown, the motion rhythm vector is determined using the third analysis unit based on optical flow and target multimedia content, including the following steps:

[0143] Step 801: Using the third analysis unit, based on optical flow and the target multimedia content, determine the inter-frame motion energy of the target multimedia content.

[0144] Optionally, inter-frame motion energy refers to the quantized energy value calculated based on pixel motion amplitude and frequency between adjacent frames in the target multimedia content, which can be used to characterize the intensity of motion and dynamic change of adjacent frames.

[0145] In some exemplary embodiments, the computer device may utilize a third analysis unit to determine the inter-frame motion energy of the target multimedia content based on optical flow and the target multimedia content.

[0146] Specifically, in one optional embodiment of this application, the computer device can extract consecutive adjacent frame image pairs of the target multimedia content in a time sequence; calculate the displacement vector of the pixels in each image pair using the optical flow method; count the sum or average amplitude of the displacement vectors of all pixels in each image pair, and perform weighted calculations based on the proportion of moving pixels and displacement frequency to obtain the inter-frame motion energy of each image pair; determine the inter-frame motion energy of each image pair as the inter-frame motion energy of the target multimedia content, or average the inter-frame motion energy of each image pair to obtain the inter-frame motion energy of the target multimedia content.

[0147] Step 802: Generate rhythm intensity curves and BPM recommendation information based on inter-frame motion energy.

[0148] Optionally, the tempo intensity curve refers to a dynamic curve plotted along the time axis based on inter-frame motion energy, which can be used to characterize the temporal variation trend of the motion rhythm of the target multimedia content. BPM recommendation information can be used to indicate the suggested value for the number of beats per minute.

[0149] In some exemplary embodiments, after the computer device uses a third analysis unit to determine the inter-frame motion energy of the target multimedia content based on optical flow and the target multimedia content, it can generate a rhythm intensity curve and BPM recommendation information based on the inter-frame motion energy.

[0150] Specifically, in one optional embodiment of this application, the computer device can perform smoothing and denoising processing on the inter-frame motion energy to remove outliers and ensure the continuity and stability of the data; using the time axis as the horizontal axis and the preprocessed inter-frame motion energy as the vertical axis, a rhythm intensity curve is generated according to the time series, where the peak value of the curve corresponds to the time period of strong motion rhythm and the trough value corresponds to the time period of weak motion rhythm. The computer device can also statistically analyze the mean, peak frequency, and energy change period of the inter-frame motion energy to determine the rhythm period of the image motion; and determine BPM recommendation information based on the rhythm period.

[0151] In another optional embodiment of this application, the computer device may further input the inter-frame motion energy into the rhythm intensity curve generation model and the BPM recommendation information determination model, respectively, to obtain the rhythm intensity curve and BPM recommendation information output by the rhythm intensity curve generation model and the BPM recommendation information determination model, respectively.

[0152] Step 803: Determine the movement rhythm vector based on the rhythm intensity curve and BPM recommendation information.

[0153] In some exemplary embodiments, after generating a rhythm intensity curve and BPM recommendation information based on inter-frame motion energy, the computer device can determine a motion rhythm vector based on the rhythm intensity curve and BPM recommendation information.

[0154] Specifically, in an optional embodiment of this application, the computer device can directly determine the rhythm intensity curve and BPM recommendation information as the motion rhythm vector.

[0155] In another optional embodiment of this application, the computer device may extract features from the rhythm intensity curve to obtain rhythm intensity features; and perform weighted summation processing on the rhythm intensity features and BPM recommendation information to obtain a motion rhythm vector.

[0156] In this embodiment, the third analysis unit is used to determine the inter-frame motion energy of the target multimedia content based on optical flow and the target multimedia content; the motion rhythm vector is determined according to the rhythm intensity curve and BPM recommendation information; the method of determining the motion rhythm vector according to the rhythm intensity curve and BPM recommendation information improves the accuracy of the determined motion rhythm vector, thereby improving the matching degree of the generated audio file.

[0157] In one exemplary embodiment, such as Figure 9 As shown, generating an audio file for the target multimedia content based on audio control parameters includes the following steps:

[0158] Step 901: Generate at least one of melody information, harmony information, rhythm information, and bass information based on audio control parameters.

[0159] Optionally, melody information refers to a single-voice note sequence, which can be used to indicate the tune and melody of the generated audio file. Harmony information refers to multi-voice chord combinations, which can be used to indicate the harmonic color and harmonic support relationships of the generated audio file. Rhythm information refers to note values ​​and pause patterns, which can be used to indicate the beat strength and rhythmic movement of the generated audio file. Bass information refers to a low-frequency note sequence, which can be used to indicate the harmonic foundation and low-frequency support of the generated audio file.

[0160] In some exemplary embodiments, the computer device may generate at least one of melody information, harmony information, rhythm information, and bass information based on audio control parameters.

[0161] Specifically, in one optional embodiment of this application, the computer device can input audio control parameters into a melody generator based on StyleGAN (Style Generative Adversarial Network) to obtain a main melody MIDI (Musical Instrument Digital Interface) sequence and determine the main melody MIDI sequence as melody information; the computer device can also generate harmony information, rhythm information and bass information based on the instrument combination in the audio control parameters.

[0162] Step 902: Generate an audio file based on at least one of the following: melody information, harmony information, rhythm information, and bass information.

[0163] In some exemplary embodiments, after generating at least one of melody information, harmony information, rhythm information, and bass information based on audio control parameters, the computer device can generate an audio file based on at least one of the melody information, harmony information, rhythm information, and bass information.

[0164] Specifically, in one optional embodiment of this application, a computer device may input at least one of melody information, harmony information, rhythm information, and bass information into a generative adversarial network to obtain an audio file output by the generative adversarial network.

[0165] Furthermore, in an optional embodiment of this application, after obtaining the audio file, the computer device can add the audio file to the multimedia content to complete the background music for the multimedia content.

[0166] In this embodiment, the computer device generates at least one of melody information, harmony information, rhythm information, and bass information based on audio control parameters; the method of generating an audio file based on at least one of melody information, harmony information, rhythm information, and bass information makes the generated audio file not only highly matched with the target multimedia content, but also highly rich, effectively improving the user experience.

[0167] In one exemplary embodiment, such as Figure 10 As shown, another method for generating audio files is provided, which includes the following steps:

[0168] Step 1001: In response to the multimedia content input event, acquire the target multimedia content;

[0169] Step 1002: Using the first analysis unit in the feature analysis model, input the target multimedia content into the convolutional neural network to obtain at least one of the color distribution information, facial expression information, and body movement intensity information of the target multimedia content output by the convolutional neural network; generate multidimensional emotion coordinates based on at least one of the color distribution information, facial expression information, and body movement intensity information; determine the visual emotion vector based on the multidimensional emotion coordinates.

[0170] Step 1003: Using the second analysis unit in the feature analysis model, input the target multimedia content into the hybrid model to obtain the scene semantic tags of the target multimedia content output by the hybrid model; determine the scene topic vector based on the scene semantic tags;

[0171] Step 1004: Using the third analysis unit in the feature analysis model, based on optical flow and the target multimedia content, determine the inter-frame motion energy of the target multimedia content; generate a rhythm intensity curve and BPM recommendation information based on the inter-frame motion energy; determine the motion rhythm vector based on the rhythm intensity curve and BPM recommendation information.

[0172] Step 1005: Determine the audio control parameters of the target multimedia content based on multimodal features; input the target multimedia content into the picture description model to obtain the timeline-based picture description information of the target multimedia content output by the picture description model; determine the audio text information based on the picture description information.

[0173] Step 1006: Generate at least one of melody information, harmony information, rhythm information, and bass information based on audio control parameters; generate an audio file based on at least one of melody information, harmony information, rhythm information, and bass information, as well as audio text information.

[0174] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0175] Based on the same inventive concept, this application also provides an audio file generation apparatus for implementing the audio file generation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more audio file generation apparatus embodiments provided below can be found in the limitations of the audio file generation method described above, and will not be repeated here.

[0176] In one exemplary embodiment, such as Figure 11 As shown, an audio file generation device 1100 is provided, including: an acquisition module 1101, an execution module 1102, a determination module 1103, and a generation module 1104, wherein:

[0177] The acquisition module 1101 is used to acquire target multimedia content in response to a multimedia content input event;

[0178] The execution module 1102 is used to perform multimodal feature analysis on the target multimedia content based on the feature analysis model, and to obtain the multimodal features of the target multimedia content;

[0179] The determination module 1103 is used to determine the audio control parameters of the target multimedia content based on multimodal features;

[0180] The generation module 1104 is used to generate audio files for the target multimedia content based on audio control parameters.

[0181] In one embodiment, the generation module 1104 is specifically used to obtain the audio text information of the target multimedia content based on the target multimedia content; and generate an audio file according to the audio control parameters and the audio text information.

[0182] In one embodiment, the generation module 1104 is specifically used to input the target multimedia content into the picture description model to obtain the timeline-based picture description information of the target multimedia content output by the picture description model; and to determine the audio text information based on the picture description information.

[0183] In one embodiment, the multimodal features include a visual emotion vector, a scene theme vector, and a motion rhythm vector; the feature analysis model includes a first analysis unit, a second analysis unit, and a third analysis unit; the execution module 1102 is specifically used to determine the visual emotion vector based on a convolutional neural network and target multimedia content using the first analysis unit; to determine the scene theme vector based on a hybrid model and target multimedia content using the second analysis unit; and to determine the motion rhythm vector based on optical flow and target multimedia content using the third analysis unit.

[0184] In one embodiment, the execution module 1102 is specifically used to use the first analysis unit to input the target multimedia content into a convolutional neural network to obtain at least one of the color distribution information, facial expression information, and body movement intensity information of the target multimedia content output by the convolutional neural network; generate multidimensional emotion coordinates based on at least one of the color distribution information, facial expression information, and body movement intensity information; and determine the visual emotion vector based on the multidimensional emotion coordinates.

[0185] In one embodiment, the execution module 1102 is specifically used to use the second analysis unit to input the target multimedia content into the hybrid model to obtain the scene semantic tags of the target multimedia content output by the hybrid model; and to determine the scene topic vector based on the scene semantic tags.

[0186] In one embodiment, the execution module 1102 is specifically used to utilize the third analysis unit to determine the inter-frame motion energy of the target multimedia content based on optical flow and the target multimedia content; generate a rhythm intensity curve and BPM recommendation information based on the inter-frame motion energy; and determine the motion rhythm vector based on the rhythm intensity curve and BPM recommendation information.

[0187] In one embodiment, the generation module 1104 is specifically used to generate at least one of melody information, harmony information, rhythm information, and bass information based on audio control parameters; and to generate an audio file based on at least one of the melody information, harmony information, rhythm information, and bass information.

[0188] Each module in the aforementioned audio file generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0189] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements an audio file generation method.

[0190] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 13 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements an audio file generation method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0191] Those skilled in the art will understand that Figure 12 and Figure 13The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0192] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the audio file generation method described in any of the above embodiments.

[0193] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the audio file generation method described in any of the above embodiments.

[0194] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the audio file generation method described in any of the above embodiments.

[0195] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0196] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0197] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0198] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for generating audio files, characterized in that, The method includes: Responding to a multimedia content input event, the target multimedia content is retrieved; Based on the feature analysis model, multimodal feature analysis is performed on the target multimedia content to obtain the multimodal features of the target multimedia content; The audio control parameters of the target multimedia content are determined based on the multimodal features; Based on the audio control parameters, an audio file for the target multimedia content is generated.

2. The method according to claim 1, characterized in that, The step of generating the audio file for the target multimedia content based on the audio control parameters includes: Based on the target multimedia content, obtain the audio text information of the target multimedia content; The audio file is generated based on the audio control parameters and the audio text information.

3. The method according to claim 2, characterized in that, Based on the target multimedia content, the audio text information of the target multimedia content is obtained, including: The target multimedia content is input into the screen description model to obtain the timeline-based screen description information of the target multimedia content output by the screen description model. The audio text information is determined based on the image description information.

4. The method according to any one of claims 1 to 3, characterized in that, The multimodal features include visual emotion vectors, scene theme vectors, and motion rhythm vectors; the feature analysis model includes a first analysis unit, a second analysis unit, and a third analysis unit; the multimodal feature analysis of the target multimedia content to obtain the multimodal features of the target multimedia content includes: Using the first analysis unit, the visual emotion vector is determined based on the convolutional neural network and the target multimedia content; Using the second analysis unit, the scene topic vector is determined based on the hybrid model and the target multimedia content; The motion rhythm vector is determined using the third analysis unit based on optical flow and the target multimedia content.

5. The method according to claim 4, characterized in that, The step of determining the visual emotion vector based on a convolutional neural network and the target multimedia content using the first analysis unit includes: Using the first analysis unit, the target multimedia content is input into the convolutional neural network to obtain at least one of the color distribution information, facial expression information, and body movement intensity information of the target multimedia content output by the convolutional neural network. Multidimensional emotion coordinates are generated based on at least one of the color distribution information, facial expression information, and body movement intensity information. The visual emotion vector is determined based on the multidimensional emotion coordinates.

6. The method according to claim 4, characterized in that, The step of using the second analysis unit to determine the scene topic vector based on the hybrid model and the target multimedia content includes: Using the second analysis unit, the target multimedia content is input into the hybrid model to obtain the scene semantic tags of the target multimedia content output by the hybrid model; The scene topic vector is determined based on the scene semantic tags.

7. The method according to claim 4, characterized in that, The step of determining the motion rhythm vector using the third analysis unit based on optical flow and the target multimedia content includes: Using the third analysis unit, based on the optical flow method and the target multimedia content, the inter-frame motion energy of the target multimedia content is determined; Based on the inter-frame motion energy, a rhythm intensity curve and BPM recommendation information are generated; The movement rhythm vector is determined based on the rhythm intensity curve and the BPM recommendation information.

8. The method according to claim 1, characterized in that, The step of generating the audio file for the target multimedia content based on the audio control parameters includes: Generate at least one of melody information, harmony information, rhythm information, and bass information based on the audio control parameters; The audio file is generated based on at least one of the melody information, harmony information, rhythm information, and bass information.

9. An audio file generation device, characterized in that, The device includes: The acquisition module is used to acquire target multimedia content in response to multimedia content input events; The execution module is used to perform multimodal feature analysis on the target multimedia content based on the feature analysis model, and obtain the multimodal features of the target multimedia content; The determining module is used to determine the audio control parameters of the target multimedia content based on the multimodal features; The generation module is used to generate an audio file of the target multimedia content based on the audio control parameters.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images