Audio generation method

Abstract

The embodiment of the invention provides a method for generating an audio, which is used for realizing audio and picture synchronization of a to-be-processed video. The method provided by the embodiment of the invention comprises the following steps: extracting an element image frame containing a target image element from a to-be-processed video; generating a motion track of the target image element in the to-be-processed video based on the position of the target image element in each element image frame and the playing time point of each element image frame in the to-be-processed video; obtaining a target sound source matched with the target image element; executing space rendering on the target sound source by adopting a rendering mode matched with the motion trail of the target image element to obtain a target sound source after space rendering; generating an audio file corresponding to the to-be-processed video based on the target sound source after space rendering; the audio file is used for synchronously playing the target sound source after space rendering matched with the target image element and the position of the target image element in the target element image frame when the to-be-processed video is played to the target element image frame in the element image frames.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular to a method for generating audio. Background Art

[0002] Nowadays, more and more users like to rest in the car. Audio and video with white noise elements can help users quickly reach a relaxed state.

[0003] However, currently when playing a video file, the same white noise audio is generally played randomly along with the same video file, and the white noise audio is often irrelevant to the video image, or the relevance is poor. Summary of the invention

[0004] The embodiment of the present invention provides a method for generating audio, which is used to achieve synchronous playback of an audio file in a video to be processed and a target image in the video to be processed, that is, to achieve synchronization of audio and image in the video to be processed.

[0005] A first aspect of an embodiment of the present application provides a method for generating audio, including:

[0006] Extracting an element image frame containing a target image element from the video to be processed; the target image element is an image element capable of generating a sound effect;

[0007] Based on the position of the target image element in each element image frame and the corresponding playback time point of each element image frame in the video to be processed, generating a motion trajectory of the target image element in the video to be processed;

[0008] Acquire a target sound source matching the target image element;

[0009] Performing spatial rendering on the target sound source in a rendering manner that matches the motion trajectory of the target image element to obtain a target sound source after spatial rendering;

[0010] An audio file corresponding to the video to be processed is generated based on the target sound source after spatial rendering; the audio file is used to synchronously play the target sound source after spatial rendering that matches the target image element and the position of the target image element in the target element image frame when the video to be processed is played to the target element image frame in the element image frame.

[0011] A second aspect of an embodiment of the present application provides a device for generating audio, including:

[0012] An extraction unit, used to extract an element image frame containing a target image element from the video to be processed; the target image element is an image element capable of generating a sound effect;

[0013] A generating unit, configured to generate a motion trajectory of the target image element in the video to be processed based on a position of the target image element in each element image frame and a corresponding playback time point of each element image frame in the video to be processed;

[0014] An acquisition unit, used to acquire a target sound source matching the target image element;

[0015] A rendering unit, configured to perform spatial rendering on the target sound source in a rendering manner that matches the motion trajectory of the target image element, to obtain a target sound source after spatial rendering;

[0016] The generation unit is also used to generate an audio file corresponding to the video to be processed based on the target sound source after spatial rendering; the audio file is used to synchronously play the target sound source after spatial rendering that matches the target image element and the position of the target image element in the target element image frame when the video to be processed is played to the target element image frame in the element image frame.

[0017] A third aspect of an embodiment of the present application provides a computer device, including: a processor, which, when executing a computer program stored in a memory, is used to implement the method for generating audio provided in the first aspect of the embodiment of the present application.

[0018] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is used to implement the method for generating audio provided in the first aspect of an embodiment of the present application.

[0019] A fifth aspect of an embodiment of the present application provides a computer program product having computer instructions stored thereon. When the computer instructions are executed by a processor, they are used to implement the method for generating audio provided in the first aspect of an embodiment of the present application.

[0020] It can be seen from the above technical solutions that the embodiments of the present invention have the following advantages:

[0021] In an embodiment of the present application, an element image frame containing a target image element is extracted from a video to be processed; the target image element is an image element that can produce sound effects; based on the position of the target image element in each element image frame and the corresponding playback time point of each element image frame in the video to be processed, a motion trajectory of the target image element in the video to be processed is generated; a target sound source matching the target image element is obtained; spatial rendering is performed on the target sound source using a rendering method matching the motion trajectory of the target image element to obtain a spatially rendered target sound source; an audio file corresponding to the video to be processed is generated based on the spatially rendered target sound source; the audio file is used to synchronously play the target image element and the spatially rendered target sound source matching the position of the target image element in the target element image frame when the video to be processed is played to the target element image frame in the element image frame.

[0022] Because the audio file generated by the embodiment of the present application, when the video to be processed is played to the target element image frame containing the target image, the spatially rendered target sound source that matches the target image element and the position of the target image element in the target element image frame is played synchronously, thereby achieving synchronous playback of the audio file in the video to be processed and the target image in the video to be processed, that is, achieving audio and picture synchronization in the video to be processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of the architecture of a system for generating audio in an embodiment of the present application;

[0024] Figure 2 A schematic diagram of an embodiment of a method for generating audio in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of a target image frame and a target image element in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of the motion trajectory of a target image element in multiple image frames in an embodiment of the present application;

[0027] Figure 5 This is a schematic diagram of an embodiment of spatially rendering a target sound source based on a channel-based rendering method in an embodiment of the present application;

[0028] Figure 6 A schematic diagram of the process of obtaining a 5.1 standard multi-channel target sound source in an embodiment of the present application;

[0029] Figure 7 This is a schematic diagram of an embodiment of performing spatial rendering on a target sound source in an object-based rendering manner in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of an embodiment of generating an extended audio file in an embodiment of the present application;

[0031] Fig. 9 A schematic diagram of another embodiment of generating an extended audio file in an embodiment of the present application;

[0032] Fig.10 A schematic diagram of another embodiment of generating an extended audio file in an embodiment of the present application;

[0033] Fig.11 A schematic diagram of an embodiment of a device for generating audio in an embodiment of the present application. DETAILED DESCRIPTION

[0034] An embodiment of the present invention provides a method for generating audio, which is used to automatically generate an audio file that is played synchronously with the target image element in the element image frame of the video to be processed based on the target image element in the video to be processed, thereby realizing the synchronous playback of the audio file in the video to be processed and the target image in the video to be processed, that is, realizing the synchronization of sound and picture in the video to be processed.

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] The terms "first", "second", "third", "fourth", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] The embodiment of the present application provides a method for generating audio, the general principle of the scheme is: when a certain video needs to be played, the video to be processed is downloaded locally or from the network, and the element image frame of the target image element is extracted from the video to be processed; the target image element is an image element that can produce sound effects; based on the position of the target image element in each element image frame and the corresponding playback time point of each element image frame in the video to be processed, the motion trajectory of the target image element in the video to be processed is generated; the target sound source matching the target image element is obtained; the target sound source is spatially rendered using a rendering method matching the motion trajectory of the target image element, and the target sound source after spatial rendering is obtained. A sound source generates an audio file corresponding to the video to be processed based on the target sound source after spatial rendering; the audio file is used to synchronously play the target sound source after spatial rendering that matches the target image element and the position of the target image element in the target element image frame when the video to be processed is played to the target element image frame in the element image frame. Because the audio file generated by the embodiment of the present application can synchronously play the target sound source after spatial rendering that matches the target image element and the position of the target image element in the target element image frame when the video to be processed is played to the target element image frame in the element image frame, thereby realizing the synchronous playback of audio and target image in the video.

[0038] In order to better implement the above-mentioned method for generating audio, the present application embodiment provides a system for generating audio. Figure 1 , Figure 1 A schematic diagram of the architecture of a system for generating audio provided in an embodiment of the present application. The system for generating audio may include at least one terminal device 101 and a server 102; different types of applications may be installed on the terminal device 101, for example, QQ music, a video player, an instant messaging application, a live broadcast application, a conference communication application, etc. may be installed on the terminal device 101; the terminal device 101 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart car, etc. The server 102 may be used to store application data, audio data, video data or image data generated by different types of applications of the terminal device 101. The server 102 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms, etc.

[0039] In which, the above-mentioned method for generating audio is executed by the terminal device 101 or the server 102. When the method for generating audio is executed by the terminal device 101, the videos that the terminal device 101 needs to play in different types of applications can be included in the server. Then, when the terminal device 101 needs to generate audio, the terminal device 101 can obtain the video to be played from the server 102, and then the terminal device 101 extracts multiple image frames from the video, and based on the target image frame in the multiple image frames, extracts the element image frame containing the target image element, and then extracts the motion estimation of the target image element in the video to be processed, obtains the target sound source matching the target image element, and performs spatial rendering on the target sound source using a rendering method matching the motion trajectory of the target image element to obtain the target sound source after spatial rendering, and finally generates an audio file corresponding to the video to be processed based on the target sound source after spatial rendering, wherein the audio file is used to synchronously play the target sound source after spatial rendering that matches the target image element and the position of the target image element in the target element image frame when the video to be processed is played to the target element image frame containing the image element, thereby realizing the synchronous playback of audio and target image in the video to be processed.

[0040] For ease of understanding, the method for generating audio in the embodiment of the present application is described below. Figure 2 , an embodiment of the method for generating audio in the embodiment of the present application includes:

[0041] 201. Extracting an element image frame containing a target image element from a video to be processed; the target image element is an image element capable of generating a sound effect;

[0042] The execution subject of the embodiment of the present application can be a client, a server, or an APP installed in the client or the server. There is no specific limitation on the type of the execution subject.

[0043] For the convenience of description, the embodiment of the present application is described with the client as the execution entity. If the client needs to play a video, it can obtain the video to be processed locally or on the network, and extract the element image frame containing the target image element from the video to be processed, wherein the target image element is an image element that can produce sound effects.

[0044] Specifically, when the video to be processed contains mountains, water, birds and trees, the target image elements here can be water, birds or trees, because these elements can have corresponding water sounds, bird sounds or wind sounds. Of course, the examples here are only explanations of the target image elements rather than limitations. For example, the video to be processed can also contain rain, because rain can also correspond to the sound effect of rain. For ease of understanding, Figure 3 A schematic diagram of a target image element in an element image frame is given.

[0045] Specifically, because a video is composed of many image frames, the embodiment of the present application can be performed by the client to divide the video into multiple image frames, thereby obtaining multiple image frames in the video, and then extracting element image frames containing target image elements from the multiple image frames.

[0046] Furthermore, when extracting elemental image frames containing target image elements, the client can receive elemental image frames uploaded manually, or the client can automatically extract elemental image frames containing target image elements from the video based on an algorithm. There is no specific restriction on the method of extracting elemental image frames.

[0047] 202. Generate a motion trajectory of the target image element in the video to be processed based on the position of the target image element in each element image frame and the corresponding playback time point of each element image frame in the video to be processed;

[0048] After extracting the element image frames containing the target image elements, the embodiment of the present application further generates the motion trajectory of the target image elements in the video to be processed based on the position of the target image elements in each element image frame and the playback time point of each element image frame in the video to be processed.

[0049] As an optional embodiment, when extracting the motion trajectory of the target image element in multiple image frames, the position coordinates of the target image element in the multiple element image frames may be determined respectively, and then the position coordinates of the target image element in each image frame are connected according to the order in which the multiple element image frames are played in the video, so as to obtain the motion trajectory of the target image element in the multiple image frames. For ease of understanding, Figure 4 A schematic diagram of the motion trajectory of the target image element in multiple element image frames is given.

[0050] 203. Acquire a target sound source that matches the target image element;

[0051] After obtaining the target image element and the motion trajectory of the target image element in multiple image frames, the target sound source matching the target image element is further obtained from the material library (wherein a large number of sound sources are stored in the material library), wherein the material library can be stored locally on the client or on the server. When the executing subject of the embodiment of the present application is an APP, the material library can also be a small plug-in embedded in the APP. There is no specific restriction on the storage location and form of expression of the material library.

[0052] Furthermore, the process of how to obtain a target sound source matching a target image element from a material library will be described in the following embodiments, and no specific limitation is made here.

[0053] 204. Perform spatial rendering on the target sound source using a rendering method that matches the motion trajectory of the target image element to obtain a target sound source after spatial rendering.

[0054] After obtaining the motion trajectory of the target image element in multiple element image frames and the target sound source matching the target image element, spatial rendering is performed on the target sound source based on the motion trajectory of the target image element using a rendering method matching the motion trajectory of the target image element, thereby obtaining the target sound source after spatial rendering.

[0055] Specifically, the rendering method in the embodiments of the present application includes a channel-based rendering method or an object-based rendering method. The process of spatially rendering the target sound source according to the motion trajectory of the target image element using a rendering method that matches the motion trajectory of the target image element to obtain the target sound source after spatial rendering will be described in detail in the following embodiments and will not be repeated here.

[0056] 205. Generate an audio file corresponding to the video to be processed based on the target sound source after spatial rendering; the audio file is used to synchronously play the target sound source after spatial rendering that matches the target image element and the position of the target image element in the target element image frame when the video to be processed is played to the target element image frame in the element image frame.

[0057] In step 204, the target sound source after spatial rendering is obtained, and then an audio file corresponding to the video to be processed is further generated based on the target sound source after spatial rendering, so as to be used for synchronously playing the target sound source after spatial rendering that matches the target image element and the position of the target image element in the target element image frame when the video to be processed is played to the target image element frame.

[0058] Specifically, when generating an audio file corresponding to the video to be processed based on the target sound source after spatial rendering, the target sound source after spatial rendering can be aligned, superimposed or adaptively volume balanced with the target image element frame where the target image element is located to generate an audio file corresponding to the video to be processed.

[0059] In an embodiment of the present application, an element image frame containing a target image element is extracted from a video to be processed; the target image element is an image element that can produce sound effects; based on the position of the target image element in each element image frame and the corresponding playback time point of each element image frame in the video to be processed, a motion trajectory of the target image element in the video to be processed is generated; a target sound source matching the target image element is obtained; spatial rendering is performed on the target sound source using a rendering method matching the motion trajectory of the target image element to obtain a spatially rendered target sound source; an audio file corresponding to the video to be processed is generated based on the spatially rendered target sound source; the audio file is used to synchronously play the target image element and the spatially rendered target sound source matching the position of the target image element in the target element image frame when the video to be processed is played to the target element image frame in the element image frame.

[0060] Because the audio file generated by the embodiment of the present application, when the video to be processed is played to the target element image frame containing the target image, the spatially rendered target sound source that matches the target image element and the position of the target image element in the target element image frame is played synchronously, thereby achieving synchronous playback of the audio file in the video to be processed and the target image in the video to be processed, that is, achieving audio and picture synchronization in the video to be processed.

[0061] based on Figure 2 In step 204 of the embodiment, the process of performing spatial rendering on the target sound source based on the motion trajectory of the target image is described in detail below:

[0062] Specifically, the motion trajectory of the target image element in the video to be processed includes a static motion trajectory and a dynamic motion trajectory, wherein the static motion trajectory means that the position of the target image element in the video to be processed does not change with time, while the dynamic motion trajectory means that the position of the target image element in the video to be processed changes with time.

[0063] Furthermore, the methods of performing spatial rendering on the target sound source in the embodiments of the present application include a channel-based rendering method and an object-based rendering method, wherein the channel-based rendering method renders the target sound source on the channel based on a preset mixing framework, while the object-based rendering method renders the target sound source based on the position of the target image element in the video and a preset multi-channel sound source format.

[0064] As an optional embodiment, if the motion trajectory of the target image element includes a static motion trajectory, the channel-based rendering method is used to perform spatial rendering on the target sound source. If the motion trajectory of the target image element includes the dynamic motion trajectory, the object-based rendering method is used to perform spatial rendering on the target sound source. The following describes the process of performing spatial rendering on the target sound source from two aspects, please refer to Figure 5 and Figure 7 :

[0065] 1. The motion trajectory is static, and the channel-based rendering method is used to render the target sound source in space. Please refer to Figure 5 :

[0066] 501. Separate relevant signals and irrelevant signals from left and right sound channels of the target sound source using an upmixing algorithm;

[0067] Generally, the acquired videos are stereo, and stereo includes left and right channels. When the motion trajectory of the target image element is static, an upmixing algorithm is used to separate the relevant signals from the left and right channels of the target sound source.

[0068] Specifically, the upmixing algorithm is an algorithm based on digital signal processing, which mainly separates related signals and unrelated signals from stereo, and then places the related signals and unrelated signals in different channels according to a specific mixing framework, thereby generating audio files of different formats, such as generating 3.1, 5.1, 5.1.4 or 7.1.4 multi-channel audio files.

[0069] As an optional embodiment, when the upmixing algorithm is used in the embodiment of the present application, the Mid / Side decomposition algorithm or the NLMS adaptive filter is combined to find the correlated signals and uncorrelated signals in the left and right channels. When the Mid / Side decomposition algorithm is used, the collected correlated signal refers to the center signal of the stereo sound image (that is, the Mid signal), and the uncorrelated signal refers to the side signals on both sides of the stereo sound image (that is, the Side signal).

[0070] Specifically, in the Mid / Side stereo system, the Mid channel is located at the center of the stereo sound image, and its position will move closer to the center as the Mid channel signal increases. The Side channels are set on both sides of the stereo sound image, and their positions expand to both sides as the Side channel signal increases, making the sound wider.

[0071] When the NLMS adaptive filter is used, the input stereo signal x(n) is filtered to find the correlation signal and the error signal from the stereo signal.

[0072] For ease of understanding, the following describes the Mid / Side decomposition algorithm and the NLMS adaptive filter by taking the generation of a 5.1 audio file as an example:

[0073] The 5.1 audio file includes the left (Left), center (Center), right (Right), subwoofer (LFE), left surround (Left Surround) and right surround (Right Surround) channels, among which the Mid / Side decomposition formula is as follows:

[0074]

[0075] Among them, x L (n) is the left channel signal in the input stereo signal, x R (n) is the right channel signal of the input stereo signal, the Mid signal is used to generate the relevant signal in the embodiment of the present application, and the Side signal is used to generate the unrelated signal in the embodiment of the present application.

[0076] Furthermore, the specific formula of the NLMS adaptive filter is as follows:

[0077] y(n)=w T (n)x(n); e(n)=d(n)-y(n);

[0078]

[0079] Among them, x(n) refers to the input stereo signal, w T (n) is the weight coefficient, y(n) is the correlation signal, d(n) is a fixed value, e(n) is the error signal, and w(n+1) is the weight coefficient after iteration.

[0080] Among them, NLMS is a variant of LMS filter, which is used to filter the input stereo signal x(n) to find the correlation signal and error signal from the stereo signal. Compared with the LMS filter, the NLMS filter improves the problem of LMS being sensitive to input dynamics. NLMS finds the correlation signal and error signal by continuously optimizing the minimum mean square error between the expected signal and the input signal.

[0081] 502. Place the relevant signal and the irrelevant signal in different channels according to a preset mixing framework to obtain a target sound source after spatial rendering.

[0082] After the relevant signal is obtained in step 501, the relevant signal and the unrelated signal are processed accordingly according to the preset mixing framework to obtain the target sound source after spatial rendering.

[0083] The upmixing algorithm also includes a mixing framework, and the mixing framework includes signals, mapping rules of each channel, and post-processing of the signal. For ease of understanding, the following takes a 5.1 audio file as an example to illustrate the processing process of using the mixing framework in the upmixing algorithm:

[0084] Specifically, the 5.1 audio file includes a center signal, a left front signal, a right front signal, a left surround signal, a right surround signal and a subwoofer. When the input is a stereo left and right channel signal, the output signal of the left front channel in the 5.1 audio file is the original stereo left channel signal minus 3dB, the output signal of the right front channel is the original stereo right channel signal minus 3dB, and the output of the center signal in the 5.1 audio file is the relevant signal in step 501 minus 2dB, the LFE is the signal passed through the low-pass filter with a cutoff frequency of 120Hz minus 4dB, and the output signal of the left surround channel in the 5.1 audio file is the irrelevant signal minus 2dB, and the output signal of the right surround channel is also the irrelevant signal minus 2dB.

[0085] Furthermore, in order to enhance the surround effect of the left and right surround channel signals, after obtaining the output signal of the left surround channel and the output signal of the right surround channel in the 5.1 audio file, the output signal of the left surround channel and the output signal of the right surround channel are further subjected to decorrelation processing, wherein the decorrelation processing here includes but is not limited to passing the output signal of the left surround channel and the output signal of the right surround channel through an all-pass filter, or delaying the sampling points in the time domain, etc.

[0086] For ease of understanding, Figure 6 A schematic diagram of the process of obtaining the 5.1-format multi-channel target sound source is given.

[0087] 2. The motion trajectory is dynamic, and the object-based rendering method is used to perform spatial rendering on the target sound source. Figure 7 :

[0088] 701. Input the target sound source, the motion trajectory of the target image element and the preset multi-channel music format into the spatial renderer to obtain the spatially rendered target sound source output by the spatial renderer.

[0089] If the motion trajectory of the target image element in the multi-frame image is dynamic, the target sound source, the motion trajectory of the target image element and the preset multi-channel music format are input into the spatial renderer to obtain the spatially rendered target sound source output by the spatial renderer.

[0090] Specifically, the target sound source is first analyzed to obtain the sound channel information of the target sound source, and multiple position information of the target sound source in the entire video is determined according to the motion trajectory of the target image element.

[0091] Then, according to the multiple position information of the target sound source, the weight of the target sound source in each channel in the multi-channel music format is calculated, and the calculation of the weight can be achieved through various acoustic models and algorithms, such as the VBAP (Vector Base Amplitude Panning) algorithm or the Ambisonics algorithm.

[0092] The multi-channel music format is further determined. Taking the 7.1.4-channel music format as an example, in this multi-channel music format, it includes 7 horizontal channels, namely front left channel, front center channel, front right channel, left channel, right channel, rear left channel and rear right channel, 1 low-frequency enhancement (LEF) channel and 4 height channels, namely front left high channel, front right high channel, rear left high channel and rear right high channel.

[0093] Finally, the target sound source of each position information is multiplied by its weight in each channel, and then the multiplied result is superimposed on the corresponding channel, so that each channel includes a mixed signal of the target audio from different position information, and finally the mixed signal of each channel is mixed to obtain the multi-channel target sound source output by the spatial renderer.

[0094] For easier understanding, the following example is given:

[0095] Assuming that the motion trajectory of the target image element in the video contains three position information, the corresponding motion trajectory of the target sound source in the video also contains three position information. Assuming that the three position information are (x1, y1), (x2, y2) and (x3, y3), the weights of the target sound source at (x1, y1) in the 12 channels of the 7.1.4 music format are calculated respectively, and then the target sound source at (x1, y1) is multiplied by the respective weights, and the multiplied results are superimposed on the above 12 channels respectively, and then the target sound source at (x2, y2) in the 12 channels of the 7.1.4 music format is calculated. The weights in the 7.1.4 music format are calculated, and then the target sound source at (x2, y2) is multiplied by the respective weights, and the multiplied results are superimposed on the 12 channels respectively. Finally, the weights of the target sound source at (x3, y3) in the 12 channels of the 7.1.4 music format are calculated, and then the target sound source at (x3, y3) is multiplied by the respective weights, and the multiplied results are superimposed on the 12 channels respectively. In this way, each channel contains mixed signals of target sound sources from different position information. Finally, the mixed signals of each channel are mixed to obtain the target sound source after spatial rendering that is finally output in the embodiment of the present application.

[0096] As an optional embodiment, before executing step 701, the target sound source may be preprocessed according to the motion trajectory of the target image element, wherein the preprocessing includes at least one of reverberation processing, delay processing, compression processing and volume automation processing.

[0097] Furthermore, if the motion trajectory of the target image element in multiple image frames is a motion along a first direction, volume automation processing and compression processing are performed on the target sound source, wherein the first direction is along the horizontal direction of the image frame, that is, a direction parallel to the playback scroll bar of the video, and if the motion trajectory of the target image element in multiple image frames is a motion along a second direction, reverberation processing and delay processing are performed on the target sound source, wherein the second direction is perpendicular to the first direction, that is, the second direction is along the longitudinal direction of the image frame.

[0098] Assuming that the target sound source in the first direction is the sound of birds, it is easy to understand that as the bird moves in the first direction, the bird's cry will become louder and louder. Therefore, performing volume automation and compression processing on the target sound source in the first direction can enhance the scene effect of the target sound source. Assuming that the target sound source in the second direction is the sound of flowing water, reverberation and delay processing are performed on the sound of flowing water. Because the sound of water heard at different positions along the longitudinal direction is different, performing reverberation and delay processing on the target sound source in the second direction can enhance the element influence effect of the target image element (i.e., flowing water).

[0099] based on Figure 5 or Figure 7 In the described embodiment, if the target image frame includes at least two target image elements, after obtaining at least two spatially rendered target sound sources, each spatially rendered target sound source corresponding to the video to be processed is aligned at the timestamp of the respective target image elements according to the timestamps in the motion trajectories of the target image elements, and the aligned multiple spatially rendered target sound sources are superimposed to obtain a superimposed sound source.

[0100] Furthermore, in order to match the volume of each target sound source in the at least two target sound sources, the superimposed sound source may be subjected to adaptive volume balancing processing to obtain an audio file corresponding to the video to be processed.

[0101] For easier understanding, the following example is given:

[0102] Assuming that the target image elements included in the target image frame are flowing water and birds, after obtaining the spatially rendered sound of flowing water and the spatially rendered sound of birds, the spatially rendered sound of flowing water and the flowing water image element in the video are aligned at the timestamp according to the timestamp when the flowing water appears in the video to be processed, and then the spatially rendered bird sound and the bird image element in the video are aligned at the timestamp according to the timestamp when the bird appears in the video, and finally the aligned bird sound and the aligned flowing water sound are superimposed to obtain the superimposed sound source.

[0103] Furthermore, in order to match the volume of bird calls and flowing water sounds and prevent the volume disharmony caused by one sound being too loud and the other too soft, the superimposed sound sources can also be adaptively volume balanced to obtain an audio file corresponding to the video to be processed.

[0104] based on Figure 5 or Figure 7 In the described embodiment, after obtaining the audio file corresponding to the video to be processed, if the client detects that the video is in a loop playback mode, in order to reduce the fatigue caused by the image elements in the video of the audio file corresponding to the video to be processed during the loop playback process, an extended audio file can also be generated based on the audio file of the video to be processed, wherein the audio file is used to play when the video to be processed is played for the first time, and the extended audio file is used to play when the video to be processed is played in a loop.

[0105] As an optional embodiment, when generating an extended audio file based on the audio file of the video to be processed, it can be to obtain an audio adjustment element for the video to be processed, the audio adjustment element includes at least one of the target light music, the associative element sound source of the target image element and the sampling segment of the target sound source; the audio adjustment element is superimposed on the audio file to obtain an extended audio file, wherein the audio file is used to play when the video to be processed is played for the first time, and the extended audio file is used to play when the video to be processed is played in a loop.

[0106] Specifically, when the audio adjustment element includes different elements (such as target light music, an associative element sound source of a target image element, and a sampled segment of a target sound source), how to superimpose the specific audio adjustment element with the audio file to obtain an extended audio file will be described in detail from the following three aspects:

[0107] 1. If the audio adjustment element includes the target light music

[0108] See also Figure 8 , superimposing the audio adjustment element with the audio file to obtain an embodiment of the extended audio file, including:

[0109] 801. Selecting target light music from a preset light music library based on a similarity strategy;

[0110] In order to avoid the fatigue caused by repeated playback of the adaptive target sound source during the playback of multiple image frames, the embodiment of the present application can also select target light music from a preset light music library based on a similarity strategy, and superimpose the target light music after the adaptive target sound source.

[0111] As an optional embodiment, the process of selecting target light music from a preset light music library based on a similarity strategy is as follows:

[0112] S1, extracting the scene semantic vector of the video to be processed;

[0113] Specifically, when extracting the scene semantic vector of the video to be processed, the word vector model is used as a model for converting natural language into word vectors so that the word vectors can be recognized by computers. Specifically, the word vector model in the embodiments of the present application can be a common model such as Word2Vec, GloVe or Bert.

[0114] As an optional embodiment, when extracting the scene semantic vector of the video to be processed, all target image elements in multiple image frames can be extracted respectively, and then the identifier of each target image element is obtained. The identifier here can be a natural language identifier such as the name or identification code of the target image element, and then the identifier of each target image element is converted into a semantic vector using a word vector model, thereby obtaining multiple semantic vectors of each target image element, such as in Figure 3 The target image elements in the video include birds, trees and flowing water. The natural language identifiers of these target image elements are converted into their respective semantic vectors, and then the average value is calculated based on the semantic vectors of all target image elements. The average value is the scene semantic vector of the video.

[0115] S2, extracting the semantic vector of each light music scene label in the light music library to obtain multiple light music scene semantic vectors;

[0116] Similar to step S1, because each light music in the light music library has a scene label, and these scene labels are natural language labels, the semantic vector of the scene of each light music can be extracted to obtain multiple scene semantic vectors of light music. As an optional embodiment, the scene label of each light music can be input into a pre-trained word vector model, and then multiple scene semantic vectors corresponding to multiple light music can be obtained.

[0117] Among them, the process of converting the scene label of light music into a scene semantic vector is similar to the process of converting the identifier of each target image element into a semantic vector in step S1. The process of converting natural language into semantic vectors using a word vector model has been described in the prior art and will not be repeated here.

[0118] S3, respectively calculating the similarity between the scene semantic vector of the video and the scene semantic vector of each light music to obtain multiple similarity values;

[0119] In order to obtain the target light music that best matches each target image element in the video, the embodiment of the present application can adopt a similarity algorithm to calculate the similarity between the scene semantic vector of the video and the scene semantic vectors of multiple light music to obtain multiple similarity values, wherein the method for calculating the similarity includes but is not limited to the cosine similarity method and the distance similarity method (Euclidean distance or Manhattan distance). No specific limitation is made to the algorithm for calculating the similarity here.

[0120] As an optional embodiment, when calculating the similarity value, a cosine similarity calculation method may be used to calculate the similarity between the scene semantic vector of the video and the scene semantic vectors of multiple light music, such as using the following formula for calculation:

[0121]

[0122] Among them, A i , B i Represent the components of vectors A and B respectively.

[0123] S4. Determine the light music corresponding to the similarity value that meets the preset standard among the multiple similarity values ​​as the target light music.

[0124] After obtaining a plurality of similarity values ​​between the scene semantic vector of the video and the scene semantic vectors of a plurality of light music in step S3, the light music corresponding to the similarity value meeting the preset standard can be determined as the target light music.

[0125] As an optional embodiment, the light music corresponding to the highest similarity value may be determined as the target light music, or one light music may be randomly selected from a plurality of light music satisfying a preset similarity threshold as the target light music.

[0126] 802. Process the target light music by using a channel-based rendering method to obtain processed multi-channel light music audio;

[0127] After determining the target light music from the light music library in step 801, the target light music is further processed based on the channel rendering method to obtain the processed multi-channel light music audio. Figure 5 The channel-based rendering method for the target sound source described in the embodiment is similar and will not be described in detail here.

[0128] 803. Superimpose the processed multi-channel light music sound source with the sound source of the audio file to obtain an extended audio file.

[0129] After obtaining the processed multi-channel light music audio, in order to avoid the problems of monotony and fatigue caused by repeated playback, the embodiment of the present application can superimpose the processed multi-channel light music sound source with the sound source of the audio file to obtain an extended audio file.

[0130] Specifically, after the processed multi-channel light music sound source is superimposed with the sound source of the audio file, the processed multi-channel light music sound source is played while the processed multi-channel light music sound source is played.

[0131] The embodiment of the present application can superimpose the processed multi-channel light music sound source with the sound source of the audio file to obtain an extended audio file. The extended audio file can enhance the immersion and layering brought by the sound source and reduce the fatigue caused by loop playback.

[0132] As an optional embodiment, in order to enhance the connectivity and fusion between the multi-channel light music and the multi-channel target sound source, the embodiment of the present application may also be to superimpose the processed multi-channel light music sound source with the sound source of the audio file in a fade-in and fade-out manner to obtain an extended audio file, so as to avoid the abruptness caused by the pauses between multiple music.

[0133] Because the embodiment of the present application generates an extended audio file based on the introduction of light music, and this processed multi-channel light music source can improve the quality of the audio on the one hand, and on the other hand, light music can fully relax the body and mind and enhance the user's physical and mental pleasure.

[0134] 2. If the audio adjustment element includes the target light music

[0135] See also Fig. 9 , superimposing the audio adjustment element with the audio file to obtain an embodiment of the extended audio file, including:

[0136] 901. Randomly extract a plurality of target sound source segments of equal length from the target sound source;

[0137] Different from Figure 8 The described method of generating an extended audio file, the embodiment of the present application can directly randomly extract multiple target sound source segments of equal length from the target sound source, and generate an extended audio file based on the target sound source segments, thereby improving the convenience of generating an extended audio file.

[0138] Specifically, when using the target sound source to generate an extended audio file, multiple target sound source segments of equal length can be randomly extracted from the target sound source. For example, when the target sound source is a 10s sound source, multiple target sound source segments of equal length can be randomly determined from the 10s sound source, where the target sound source segment can be 2s, 3s or 4s, etc. There is no specific restriction on the length of the target sound source segment here.

[0139] 902. If the motion trajectory of the target image element corresponding to the target sound source includes a static motion trajectory, rendering the multiple target sound source segments of equal length using a channel-based rendering method to obtain multiple spatially rendered target sound source segments;

[0140] After obtaining a plurality of target sound source segments of equal length in step 901, the target sound source segments may be further rendered according to the motion trajectory of the target image element corresponding to the target sound source. If the motion trajectory of the target image element corresponding to the target sound source includes a static motion trajectory, the plurality of target sound source segments of equal length may be rendered using a channel-based rendering method to obtain a plurality of spatially rendered target sound source segments. The process of rendering the plurality of target sound source segments of equal length using a channel-based rendering method is similar to the process of rendering the plurality of target sound source segments of equal length using a channel-based rendering method. Figure 5 The descriptions in the embodiments are similar and can be referenced to each other, and will not be repeated here.

[0141] 903. If the motion trajectory of the target image element corresponding to the target sound source includes a dynamic motion trajectory, rendering the multiple target sound source segments of equal length in an object-based rendering manner to obtain multiple spatially rendered target sound source segments;

[0142] After obtaining a plurality of target sound source segments of equal length in step 901, if the motion trajectory of the target image element corresponding to the target sound source includes a dynamic motion trajectory, the plurality of target sound source segments of equal length are rendered using an object-based rendering method to obtain a plurality of target sound source segments after spatial rendering, wherein the process of rendering the plurality of target sound source segments of equal length using an object-based rendering method is similar to Figure 7 The descriptions in the embodiments are similar and can be referenced to each other, and will not be repeated here.

[0143] 904. Obtain a first timestamp at which the target image element corresponding to the target sound source starts playing in the video and a second timestamp at which the playing ends;

[0144] After obtaining a plurality of spatially rendered target sound source segments, a first timestamp of when the target image element corresponding to the target sound source starts playing and a second timestamp of when the target image element ends playing in the video is further obtained.

[0145] Specifically, when the target image element is a bird, a first timestamp when the bird starts playing and a second timestamp when the bird ends playing in the video are obtained. When obtaining the first timestamp and the second timestamp, the client can determine the first image frame where the bird appears and the second image frame where the bird disappears based on multiple image frames in the video through image segmentation, and then obtain the first timestamp when the first image frame appears in the video and the second timestamp when the second image frame appears in the video.

[0146] 905. Randomly superimpose one or more of the multiple spatially rendered target sound source segments between the first timestamp and the second timestamp.

[0147] After the first timestamp and the second timestamp are determined, one or more of the multiple spatially rendered target sound source segments may be randomly superimposed between the first timestamp and the second timestamp.

[0148] Because the embodiment of the present application generates extended audio directly based on the sampled segments of the target sound source, compared with the method of selecting light music, it reduces the process of obtaining the target light music, thereby improving the convenience of generating extended audio files.

[0149] Further, based on Fig. 9 In the embodiment described above, when the duration of the target sound source far exceeds the duration of the target image element appearing in the video, for example, the bird call sound source is 3 minutes, but the bird flying in the video is only 10 seconds, and there are only 2 minutes of valid bird calls in the 3-minute bird call sound source, in order to improve the effectiveness of obtaining the bird call included in the target sound source segment, the embodiment of the present application may further perform the following steps before step 901:

[0150] If the duration of the target sound source is greater than the time difference between the first timestamp and the second timestamp, a valid sound source segment is extracted from the target sound source, and then when executing step 901, a plurality of target sound source segments of equal length are randomly extracted from the valid sound source segments.

[0151] Specifically, because the duration of the target sound source is greater than the time difference between the first timestamp and the second timestamp, that is, the duration of the target sound source is greater than the duration that the target image element appears in the video, in order to obtain a valid target sound source segment, a valid sound source segment is further extracted from the target sound source.

[0152] As an optional embodiment, when extracting a valid sound source segment from a target sound source, audio framing may be performed on the target sound source to obtain a multi-frame sound source signal, and then the energy value of each frame of the sound source signal is calculated to obtain multiple energy values ​​of the multi-frame sound source signal, and then a plurality of target energy values ​​greater than a preset energy threshold (the preset energy threshold here is used to characterize the presence of the target image element in the target sound source) are selected from the multiple energy values, and then the multi-frame sound source signals corresponding to the multiple target energy values ​​are determined as valid sound source signals, and then all timestamps of the valid sound source signal are determined from the video, and according to all timestamps, the valid sound source segment is extracted from the target sound source.

[0153] The goal of audio framing is to divide the continuous speech signal into short time frames so that each frame can be further analyzed and processed. The commonly used framing method is to use a fixed-length window to slide the window on the target sound source to divide the signal. Usually, the framing window length is selected to be 20-30ms, which can ensure that the statistical characteristics of the language signal remain relatively stable in a short time.

[0154] Specifically, when calculating the energy value of each frame, it can be calculated by the following formula:

[0155]

[0156] In the embodiment of the present application, before randomly extracting multiple target sound source segments of equal length from the target sound source, valid sound source segments are first extracted from the target sound source, thereby ensuring the probability of extracting valid sound source segments from the target sound source segments and improving the effectiveness of obtaining the target sound source segments.

[0157] 3. If the audio adjustment element includes the associative element sound source of the target image element

[0158] See also Fig.10 , superimposing the audio adjustment element with the audio file to obtain an embodiment of the extended audio file, including:

[0159] 1001. Acquire multiple associative element sound sources of a target image element;

[0160] Different from Figure 8 and Fig. 9The described embodiment, when generating an extended audio file, the embodiment of the present application generates the extended audio file based on the associative element sound source of the target image element, thereby avoiding the monotony and fatigue brought by the target sound source during continuous playback, wherein the associative element here is an image element associated with the target image element. For example, when the target image element is a forest or rain, the associative element can be thunder, wind or animal calls, etc., and the associative element sound source here is a sound source containing the associative element. For example, when the associative element is thunder, the associative element sound source is a sound source containing thunder, and when the associative element is wind, the associative element sound source is a sound source containing wind, etc.

[0161] As an optional embodiment, when obtaining the associative element sound source of the target image element, the semantic vector of the target image element identifier can be calculated using a preset word vector model (such as Word2Vec, GloVe, Bert and other models). The target image element identifier here refers to the natural language identifier of the target image element, and the process of calculating the semantic vector of the target image element identifier using the word vector model is similar to that described in the prior art and will not be repeated here.

[0162] After obtaining the semantic vector of the target image element identifier, the same word vector model is further used to calculate the semantic vectors of all sound source identifiers in the sound source library. All sound sources contained in the sound source library have their own sound source identifiers, so the word vector model can be used to calculate the semantic vectors of all sound source identifiers in the sound source library. Then, according to the similarity value calculation method, the similarity value between the semantic vector of the target image element identifier and the semantic vectors of all sound source identifiers is calculated (such as cosine similarity or distance similarity, etc.), thereby obtaining multiple similarity values ​​accordingly.

[0163] After obtaining a plurality of similarity values, a plurality of target similarities greater than a preset similarity threshold may be determined from the plurality of similarity values, and a plurality of sound sources corresponding to the plurality of target similarity values ​​may be further determined as a plurality of associative element sound sources.

[0164] For easier understanding, the following example is given:

[0165] Assuming that there are 100 sound sources in the sound source library, 100 similarity values ​​between the semantic vector of the target sound source identifier and the semantic vectors of the 100 sound source identifiers are calculated respectively. Assuming that there are 5 similarity values ​​greater than the similarity threshold, the 5 sound sources corresponding to these 5 similarity values ​​are determined as associative element sound sources.

[0166] It should be noted that when determining multiple associative element sound sources, the number of associated element sound sources can be changed by changing the size of the similarity threshold. For example, when it is determined that there are 5 associative element sound sources, but only 3 associative elements are needed, the number of associative elements can be reduced by increasing the similarity threshold. If more associative elements (such as 8) are needed, the number of associative elements can be increased by reducing the similarity threshold.

[0167] 1002. Randomly arrange and combine the multiple associative element sound sources to obtain associative element sound sources combined in a first order;

[0168] In order to further enhance the richness and diversity of the associative elements, a plurality of associative element sound sources may be randomly arranged and combined to obtain associative element sound sources combined in a first order.

[0169] Because the types and quantities of associative element sound sources after random combination are greater than those of single associative elements, the content of the sound source is richer. The random combination method also ensures that the combined associative element sound sources obtained are different each time in the loop, thereby further improving the diversity and richness of the target sound source during the loop process.

[0170] 1003. Obtain a first timestamp at which the target image element starts playing and a second timestamp at which the target image element ends playing in the video;

[0171] Similar to step 903, in order to achieve the effect of synchronization between sound and picture, that is, in order to achieve the synchronous playback of the target sound source when the target image element is played in the image of the video, the embodiment of the present application first obtains the first timestamp and the second timestamp when the target image element starts to be played in the video, and further provides a technical basis for superimposing the combined associative element sound source outside the first timestamp and the second timestamp.

[0172] 1004. Obtain any target timestamp before the first timestamp or after the second timestamp, and superimpose the combined associative element sound sources at any target timestamp of the audio file in a first order.

[0173] After obtaining the first timestamp and the second timestamp of the target image element played in the video, because the embodiment of the present application generates an extended audio file based on the associative element sound source of the target image element, during the loop playback process, the associative element sound source is superimposed and combined in the first order at any target timestamp before the first timestamp and after the second timestamp.

[0174] Specifically, because the associative element sound source is an element sound source associated with the target image element, the embodiment of the present application can superimpose the combined associative element sound source in a first order at any target timestamp except the time period between the first timestamp and the second timestamp, thereby enhancing the diversity and richness of the sound source content during the loop playback of the video, reducing the monotony of the sound source and the fatigue caused by the monotony.

[0175] In an embodiment of the present application, a randomly combined associative element sound source is superimposed at any target timestamp except the time period between the first timestamp and the second timestamp. Compared with the target sound source played in a loop, the randomly combined associative element sound source improves the diversity and richness of the sound source during the image playback process, and further reduces the fatigue caused by the single loop playback of the target sound source during the loop playback process.

[0176] based on Fig.10 In the described embodiment, when executing step 1002, in order to enhance the layering of the sound of the associative element sound source after random combination, the embodiment of the present application can also perform spatial rendering on the multiple associative element sound sources based on the object rendering method after obtaining the multiple associative element sound sources. Specifically, when performing the spatial rendering, each associative element sound source, the motion trajectory of each randomly initialized target image element and the preset music format (such as 5.1.4, 7.1.4, etc.) are input into the spatial renderer to obtain multiple associative element sound sources after spatial rendering, and then the multiple associative element sound sources after spatial rendering are randomly arranged and combined.

[0177] Specifically, for the process of performing spatial rendering of the associative element sound source based on object rendering, Figure 7 The process of performing spatial rendering of the target sound source based on object rendering in the embodiment is similar and will not be repeated here.

[0178] Further, based on Fig.10 In the described embodiment, when executing step 1002, if after a plurality of associative image element sound sources are randomly arranged and combined, the resulting associative element sound source after combination is greater than the playback duration of the target image element, a plurality of associative element sound source segments of equal length may be further randomly extracted from the plurality of associative element sound sources, and then the plurality of associative element sound source segments of equal length may be randomly combined, so that the playback duration of the randomly combined associative element sound source adapts to the playback duration of the target image element in the video.

[0179] It is understandable that, in various embodiments of the present invention, the sequence numbers of the above steps do not mean the order of execution. The order of execution of the steps should be determined by their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0180] The above describes in detail the method for generating audio in the embodiment of the present application. Next, the device for generating audio in the present application is described. Fig.11 , an embodiment of the device for generating audio in the embodiment of the present application includes:

[0181] The extraction unit 1101 is used to extract an element image frame containing a target image element from the video to be processed; the target image element is an image element that can generate a sound effect;

[0182] A generating unit 1102 is used to generate a motion trajectory of the target image element in the video to be processed based on the position of the target image element in each element image frame and the corresponding playback time point of each element image frame in the video to be processed;

[0183] An acquisition unit 1103 is used to acquire a target sound source matching the target image element;

[0184] A rendering unit 1104 is configured to perform spatial rendering on the target sound source in a rendering manner that matches the motion trajectory of the target image element, to obtain a target sound source after spatial rendering;

[0185] The generation unit 1102 is also used to generate an audio file corresponding to the video to be processed based on the target sound source after spatial rendering; the audio file is used to synchronously play the target sound source after spatial rendering that matches the target image element and the position of the target image element in the target element image frame when the video to be processed is played to the target element image frame in the element image frame.

[0186] Preferably, the motion trajectory includes a static motion trajectory and a dynamic motion trajectory, and the rendering method includes a channel-based rendering method and an object-based rendering method;

[0187] The rendering unit 1104 is specifically used for:

[0188] If the motion trajectory of the target image element includes the static motion trajectory, performing spatial rendering on the target sound source using the channel-based rendering method;

[0189] If the motion trajectory of the target image element includes the dynamic motion trajectory, the object-based rendering method is used to perform spatial rendering on the target sound source.

[0190] Preferably, the rendering unit 1104 is specifically used for:

[0191] An upmix algorithm is used to separate the relevant and irrelevant signals from the left and right sound channels of the target sound source;

[0192] The related signals and the unrelated signals are placed in different channels according to the preset mixing framework to obtain the target sound source after spatial rendering.

[0193] Preferably, the rendering unit 1104 is specifically used for:

[0194] The target sound source, the motion trajectory of the target image element and the preset multi-channel music format are input into the spatial renderer to obtain the spatially rendered target sound source output by the spatial renderer.

[0195] Preferably, the device further comprises:

[0196] The preprocessing unit 1105 is configured to perform volume automation processing and compression processing on the target sound source to obtain a processed target sound source if the motion trajectory of the target image element in the video to be processed is a motion along a first direction;

[0197] The preprocessing unit 1105 is further configured to perform reverberation processing and delay processing on the target sound source to obtain a processed target sound source if the motion trajectory of the target image element in the plurality of image frames is a motion along a second direction, wherein the second direction is perpendicular to the first direction;

[0198] The rendering unit 1104 is specifically used for:

[0199] The processed target sound source, the motion trajectory of the target image element and the preset multi-channel music format are input into the spatial renderer to obtain the spatially rendered target sound source output by the spatial renderer.

[0200] Preferably, the number of the target image elements is at least two, the number of the rendered target sound sources is at least two, and the generating unit 1102 is specifically configured to:

[0201] According to the timestamps in the motion trajectories of the target image elements, each rendered target sound source corresponding to the video to be processed is aligned at the timestamps of the target image elements respectively corresponding to the timestamps;

[0202] Superimposing the aligned and rendered target sound sources to obtain a superimposed sound source;

[0203] Adaptive volume balancing is performed on the superimposed sound source to obtain an audio file corresponding to the video to be processed.

[0204] Preferably, the device further comprises:

[0205] The acquisition unit 1103 is used to:

[0206] Acquire an audio adjustment element for the video to be processed, wherein the audio adjustment element includes at least one of target light music, an associative element sound source of the target image element, and a sample segment of the target sound source;

[0207] The superimposing unit 1106 is used for:

[0208] The audio adjustment element is superimposed on the audio file to obtain an extended audio file, wherein the audio file is used to be played when the video to be processed is played for the first time, and the extended audio file is used to be played when the video to be processed is played in a loop.

[0209] Preferably, if the audio adjustment element includes the target light music, the acquiring unit 1106 is specifically configured to:

[0210] Select target light music from a preset light music library based on a similarity strategy;

[0211] The superposition unit 1106 is specifically used for:

[0212] Processing the target light music using a channel-based rendering method to obtain a processed multi-channel light music sound source;

[0213] The processed multi-channel light music sound source is superimposed on the sound source of the audio file to obtain an extended audio file.

[0214] Preferably, the acquisition unit 1103 is specifically used for:

[0215] Extracting a scene semantic vector of the video to be processed;

[0216] Extracting the semantic vector of each light music scene label in the light music library to obtain a plurality of light music scene semantic vectors;

[0217] Respectively calculating the similarity between the scene semantic vector of the video to be processed and the scene semantic vector of each light music to obtain a plurality of similarity values;

[0218] The light music corresponding to the similarity value that meets the preset standard among the multiple similarity values ​​is determined as the target light music.

[0219] Preferably, the acquisition unit 1103 is specifically used for:

[0220] Extracting a plurality of element image frames from the video to be processed;

[0221] Extracting all target image elements from the plurality of element image frames;

[0222] Get the identifier of each target image element;

[0223] Calculating the semantic vector corresponding to the identifier of each target image element to obtain multiple semantic vectors of each target image element;

[0224] An average value of the multiple semantic vectors is calculated to obtain a scene semantic vector of the video to be processed.

[0225] Preferably, the superposition unit 1106 is further used for:

[0226] The processed multi-channel light music audio is superimposed on the audio file in a fade-in and fade-out manner to obtain an extended audio file.

[0227] Preferably, if the audio adjustment element includes a sample segment of the target sound source, the acquiring unit 1103 is further configured to:

[0228] Randomly extract multiple target sound source segments of equal length from the target sound source;

[0229] The superposition unit 1106 is further used for:

[0230] If the motion trajectory of the target image element corresponding to the target sound source includes a static motion trajectory, rendering the multiple target sound source segments of equal length using a channel-based rendering method to obtain multiple spatially rendered target sound source segments;

[0231] If the motion trajectory of the target image element corresponding to the target sound source includes a dynamic motion trajectory, rendering the multiple target sound source segments of equal length using an object-based rendering method to obtain multiple spatially rendered target sound source segments;

[0232] Obtain a first timestamp of when the target image element corresponding to the target sound source starts playing and a second timestamp of when the playing ends in the video to be processed;

[0233] One or more of a plurality of spatially rendered target sound source segments are randomly superimposed between the first timestamp and the second timestamp of the audio file.

[0234] Preferably, before randomly extracting a plurality of target sound source segments of equal length from the target sound source, the acquiring unit 1103 is further configured to:

[0235] If the duration of the target sound source is greater than the time difference between the first timestamp and the second timestamp, extracting a valid sound source segment from the target sound source;

[0236] A plurality of target sound source segments of equal length are randomly extracted from the valid sound source segments.

[0237] Preferably, the extraction unit 1101 is specifically used for:

[0238] Performing audio framing on the target sound source to obtain a multi-frame sound source signal;

[0239] Calculating the energy value of each frame of the sound source signal to obtain multiple energy values ​​of multiple frames of the sound source signal;

[0240] Selecting a plurality of target energy values ​​greater than a preset energy threshold from the plurality of energy values;

[0241] Determine the multiple frames of sound source signals corresponding to multiple target energy values ​​as valid sound source signals;

[0242] Get all timestamps of valid audio source signals;

[0243] According to all timestamps, extract valid audio source segments from the target audio source.

[0244] Preferably, if the audio adjustment element includes the associative element sound source of the target image element, the acquisition unit 1103 is specifically configured to:

[0245] Extracting the semantic vector of the target image element identifier;

[0246] Extract the semantic vectors of all sound source identifiers in the sound source library;

[0247] Calculating similarity values ​​between the semantic vector of the target image element identifier and the semantic vectors of all the music identifiers to obtain multiple similarity values;

[0248] Determining a plurality of target similarity values ​​greater than a preset similarity threshold from the plurality of similarity values;

[0249] The multiple sound sources corresponding to the multiple target similarity values ​​are determined as multiple associative element sound sources of the target image element.

[0250] Preferably, if the audio adjustment element includes the associative element sound source of the target image element, the acquisition unit 1103 is specifically configured to:

[0251] Acquire multiple associative element sound sources of the target image element;

[0252] Randomly arranging and combining the multiple associative element sound sources to obtain associative element sound sources combined in a first order;

[0253] Obtain a first timestamp at which the target image element starts playing and a second timestamp at which the target image element ends playing in the video to be processed;

[0254] Obtain any target timestamp before the first timestamp and after the second timestamp;

[0255] The combined associative element sound source is superimposed at any target time stamp of the audio file in the first order.

[0256] Preferably, before randomly arranging and combining the multiple associative element sound sources, the rendering unit 1104 is further used to:

[0257] Inputting each associative element sound source among the multiple associative element sound sources, the motion trajectory of each randomly initialized associative element sound source, and a preset music format into a spatial renderer to obtain multiple spatially rendered associative element sound sources;

[0258] The associative element sound sources rendered in the multiple spaces are randomly arranged and combined.

[0259] Preferably, if the playing time of the associative element sound source is greater than the time difference between the first timestamp and the second timestamp, before randomly sorting and combining the multiple associative element sound sources, the extraction unit 1101 is further used to:

[0260] Randomly extract multiple equal-length associative element sound source segments from the associative element sound source;

[0261] The rendering unit 1104 is further configured to:

[0262] The multiple associative element sound source segments of equal length are randomly arranged and combined.

[0263] For the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0264] In an embodiment of the present application, in an embodiment of the present application, an element image frame containing a target image element is extracted from the video to be processed by an extraction unit 1101; the target image element is an image element that can produce sound effects; a motion trajectory of the target image element in the video to be processed is generated by a generation unit 1102 based on the position of the target image element in each element image frame and the corresponding playback time point of each element image frame in the video to be processed, a target sound source matching the target image element is obtained by an acquisition unit 1103, and a rendering unit 1104 performs spatial rendering on the target sound source using a rendering method matching the motion trajectory of the target image element to obtain a spatially rendered target sound source; an audio file corresponding to the video to be processed is generated based on the spatially rendered target sound source; the audio file is used to synchronously play the target image element and the spatially rendered target sound source matching the position of the target image element in the target element image frame when the video to be processed is played to the target element image frame in the element image frame.

[0265] Because the audio file generated by the embodiment of the present application, when the video to be processed is played to the target element image frame containing the target image, the spatially rendered target sound source that matches the target image element and the position of the target image element in the target element image frame is played synchronously, thereby achieving synchronous playback of the audio file in the video to be processed and the target image in the video to be processed, that is, achieving audio and picture synchronization in the video to be processed.

[0266] The above describes the apparatus for generating audio in the embodiment of the present invention from the perspective of modular functional entities. The following describes the computer apparatus in the embodiment of the present invention from the perspective of hardware processing:

[0267] An embodiment of the computer device in the embodiment of the present invention includes:

[0268] Processor and memory;

[0269] When the memory is used to store a computer program and the processor is used to execute the computer program stored in the memory, the method steps described in the above method embodiment can be implemented.

[0270] It is understandable that when the processor in the computer device described above executes the computer program, it can also implement the functions of the various units in the corresponding device embodiments described above, which will not be repeated here. Exemplarily, the computer program can be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program in the device for generating audio. For example, the computer program can be divided into the various units in the above-mentioned device for generating audio, and each unit can implement the specific functions described in the above-mentioned corresponding device for generating audio.

[0271] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that a processor and a memory are merely examples of computer devices and do not constitute a limitation of the computer device. The computer device may include more or fewer components, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, and the like.

[0272] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0273] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the computer device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0274] The present invention also provides a computer-readable storage medium, which is used to implement the function of the device for generating audio, and a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the processor can be used to execute the method steps described in the above method embodiment.

[0275] It is understandable that if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a corresponding computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned corresponding embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0276] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0277] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0278] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0279] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating audio, characterized in that include: Extracting an element image frame containing a target image element from a video to be processed; The target image element is an image element that can produce sound effects; Based on the position of the target image element in each element image frame and the corresponding playback time point of each element image frame in the video to be processed, generating a motion trajectory of the target image element in the video to be processed; Acquire a target sound source matching the target image element; Performing spatial rendering on the target sound source in a rendering manner that matches the motion trajectory of the target image element to obtain a target sound source after spatial rendering; Generate an audio file corresponding to the video to be processed based on the target sound source after the spatial rendering; The audio file is used to synchronously play the target sound source after spatial rendering that matches the target image element and the position of the target image element in the target element image frame when the video to be processed is played to the target element image frame in the element image frame.

2. The method according to claim 1, characterized in that The motion trajectory includes a static motion trajectory and a dynamic motion trajectory, and the rendering method includes a channel-based rendering method and an object-based rendering method; The step of performing spatial rendering on the target sound source in a rendering manner that matches the motion trajectory of the target image element to obtain the target sound source after spatial rendering includes: If the motion trajectory of the target image element includes the static motion trajectory, performing spatial rendering on the target sound source using the channel-based rendering method to obtain a target sound source after spatial rendering; If the motion trajectory of the target image element includes the dynamic motion trajectory, the object-based rendering method is used to perform spatial rendering on the target sound source to obtain the target sound source after spatial rendering.

3. The method according to claim 2, characterized in that The step of performing spatial rendering on the target sound source by using the channel-based rendering method to obtain the target sound source after spatial rendering includes: An upmixing algorithm is used to separate relevant signals from irrelevant signals from left and right sound channels of the target sound source; The related signal and the unrelated signal are placed in different channels according to a preset mixing framework to obtain a target sound source after spatial rendering.

4. The method according to claim 2, characterized in that: The step of performing spatial rendering on the target sound source by using the object-based rendering method to obtain the target sound source after spatial rendering includes: The target sound source, the motion trajectory of the target image element and a preset multi-channel music format are input into a spatial renderer to obtain the spatially rendered target sound source output by the spatial renderer.

5. The method according to claim 4, characterized in that The method further comprises: If the motion trajectory of the target image element in the video to be processed is a motion along a first direction, performing volume automation processing and compression processing on the target sound source to obtain a processed target sound source; If the motion trajectory of the target image element in the plurality of image frames is a motion along a second direction, performing reverberation processing and delay processing on the target sound source to obtain a processed target sound source; the second direction is perpendicular to the first direction; The step of inputting the target sound source, the motion trajectory of the target image element and the preset multi-channel music format into the spatial renderer comprises: The processed target sound source, the motion trajectory of the target image element and the preset multi-channel music format are input into the space renderer.

6. The method according to claim 1, characterized in that The number of the target image elements is at least two, the number of the rendered target sound sources is at least two, and the generating of the audio file corresponding to the video to be processed based on the rendered target sound sources includes: According to the timestamps in the motion trajectories of the target image elements, each rendered target sound source corresponding to the video to be processed is aligned at the timestamps of the target image elements respectively corresponding to the timestamps; Superimposing the aligned and rendered target sound sources to obtain a superimposed sound source; Adaptive volume balancing is performed on the superimposed sound source to obtain an audio file corresponding to the video to be processed.

7. The method according to claim 6, characterized in that If the video to be processed is played in a loop, after obtaining the audio file corresponding to the video to be processed, the method further includes: Acquire an audio adjustment element for the video to be processed, wherein the audio adjustment element includes at least one of target light music, an associative element sound source of the target image element, and a sample segment of the target sound source; The audio adjustment element is superimposed on the audio file to obtain an extended audio file, wherein the audio file is used to be played when the video to be processed is played for the first time, and the extended audio file is used to be played when the video to be processed is played in a loop.

8. The method according to claim 7, characterized in that If the audio adjustment element includes the target light music, the step of obtaining the audio adjustment element for the video to be processed includes: Select target light music from a preset light music library based on a similarity strategy; The step of superimposing the audio adjustment element with the audio file to obtain an extended audio file includes: Processing the target light music using a channel-based rendering method to obtain a processed multi-channel light music sound source; The processed multi-channel light music sound source is superimposed on the sound source of the audio file to obtain an extended audio file.

9. The method according to claim 8, characterized in that Based on the similarity strategy, target light music is selected from the preset light music library, including: Extracting a scene semantic vector of the video to be processed; Extracting the semantic vector of each light music scene label in the light music library to obtain a plurality of light music scene semantic vectors; Respectively calculating the similarity between the scene semantic vector of the video to be processed and the scene semantic vector of each light music to obtain a plurality of similarity values; The light music corresponding to the similarity value that meets the preset standard among the multiple similarity values ​​is determined as the target light music.

10. The method according to claim 9, characterized in that The step of extracting the scene semantic vector of the video to be processed includes: Extracting a plurality of element image frames from the video to be processed; Extracting all target image elements from the plurality of element image frames; Get the identifier of each target image element; Calculating the semantic vector corresponding to the identifier of each target image element to obtain multiple semantic vectors of each target image element; An average value of the multiple semantic vectors is calculated to obtain a scene semantic vector of the video to be processed.

11. The method according to claim 8, characterized in that The processed multi-channel light music sound source is superimposed on the sound source of the audio file to obtain an extended audio file, including: The processed multi-channel light music audio is superimposed on the audio file in a fade-in and fade-out manner to obtain an extended audio file.

12. The method according to claim 7, characterized in that If the audio adjustment element includes a sample segment of the target sound source, obtaining the audio adjustment element for the video to be processed includes: Randomly extracting a plurality of target sound source segments of equal length from the target sound source; The step of superimposing the audio adjustment element with the audio file to obtain an extended audio file includes: If the motion trajectory of the target image element corresponding to the target sound source includes a static motion trajectory, rendering the multiple target sound source segments of equal length using a channel-based rendering method to obtain multiple spatially rendered target sound source segments; If the motion trajectory of the target image element corresponding to the target sound source includes a dynamic motion trajectory, rendering the multiple target sound source segments of equal length using an object-based rendering method to obtain multiple spatially rendered target sound source segments; Obtain a first timestamp of when the target image element corresponding to the target sound source starts playing and a second timestamp of when the playing ends in the video to be processed; One or more of a plurality of spatially rendered target sound source segments are randomly superimposed between the first timestamp and the second timestamp of the audio file.

13. The method according to claim 12, characterized in that Before randomly extracting a plurality of target sound source segments of equal length from the target sound source, the method further includes: If the duration of the target sound source is greater than the time difference between the first timestamp and the second timestamp, extracting a valid sound source segment from the target sound source; The step of randomly extracting a plurality of target sound source segments of equal length from the target sound source comprises: A plurality of target sound source segments of equal length are randomly extracted from the valid sound source segments.

14. The method according to claim 13, characterized in that The extracting a valid sound source segment from the target sound source includes: Performing audio framing on the target sound source to obtain a multi-frame sound source signal; Calculating the energy value of each frame of the sound source signal to obtain multiple energy values ​​of multiple frames of the sound source signal; Selecting a plurality of target energy values ​​greater than a preset energy threshold from the plurality of energy values; Determine the multiple frames of sound source signals corresponding to the multiple target energy values ​​as valid sound source signals; Obtain all timestamps of the valid audio source signal; According to all the timestamps, valid sound source segments are cut out from the target sound source.

15. The method according to claim 7, characterized in that If the audio adjustment element includes an associative element sound source of the target image element, the step of obtaining the audio adjustment element for the video to be processed includes: Extracting the semantic vector of the target image element identifier; Extract the semantic vectors of all sound source identifiers in the sound source library; Calculating similarity values ​​between the semantic vector of the target image element identifier and the semantic vectors of all the music identifiers to obtain multiple similarity values; Determining a plurality of target similarity values ​​greater than a preset similarity threshold from the plurality of similarity values; The multiple sound sources corresponding to the multiple target similarity values ​​are determined as multiple associative element sound sources of the target image element.

16. The method according to claim 7, characterized in that If the audio adjustment element includes the associative element sound source of the target image element, superimposing the audio adjustment element with the audio file to obtain an extended audio file includes: Acquire multiple associative element sound sources of the target image element; Randomly arranging and combining the multiple associative element sound sources to obtain associative element sound sources combined in a first order; Obtain a first timestamp at which the target image element starts playing and a second timestamp at which the target image element ends playing in the video to be processed; Obtain any target timestamp before the first timestamp or after the second timestamp; The combined associative element sound source is superimposed at any target time stamp of the audio file in the first order.

17. The method according to claim 16, characterized in that Before randomly arranging and combining the multiple associative element sound sources, the method further includes: Inputting each associative element sound source among the multiple associative element sound sources, the motion trajectory of each randomly initialized associative element sound source, and a preset music format into a spatial renderer to obtain multiple spatially rendered associative element sound sources; Randomly arranging and combining the multiple associative element sound sources includes: The associative element sound sources rendered in the multiple spaces are randomly arranged and combined.

18. The method according to claim 17, characterized in that If the playing time of the associative element sound source is greater than the time difference between the first timestamp and the second timestamp, before randomly sorting and combining the multiple associative element sound sources, the method further includes: Randomly extracting a plurality of associative element sound source segments of equal length from the associative element sound source; The randomly arranging and combining the multiple associative element sound sources comprises: The multiple associative element sound source segments of equal length are randomly arranged and combined.

19. A computer device comprising a processor, characterized in that: When the processor executes the computer program stored in the memory, the processor is configured to implement the method for generating audio according to any one of claims 1 to 18.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it is used to implement the method for generating audio according to any one of claims 1 to 18.

21. A computer program product having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, they are used to implement the method for generating audio according to any one of claims 1 to 18.

Images