Method and electronic device for generating visual animation based on audio

Abstract

The present invention provides a method for generating visual animation based on audio, the method comprising: reading audio data and converting the audio data into a numpy array format; using Fourier transform to convert the audio signal from the time domain to the frequency domain and remove high-frequency noise and clutter; identifying the beat position in the audio signal and converting it into Byte data; converting each Byte data into a corresponding action instruction; querying action information corresponding to the action instruction from a database; converting the action instruction into an animation instruction according to the queried action information; generating an animation according to the animation instruction and saving it as a video file. The present invention can adaptively obtain actions similar to the sound style, and can meet personalized needs.

Description

Technical Field

[0001] The present invention relates to the technical field of animation generation, and in particular to a method and an electronic device for generating visual animation based on audio. Background Art

[0002] It should be noted that the contents recorded in this section do not represent all prior art.

[0003] Currently, animations are produced and generated by technicians based on their experience, and then matched with relevant music. This method cannot automatically adapt to different styles and personalized needs. Summary of the invention

[0004] In view of this, in order to solve one of the technical problems in the related art to a certain extent, it is necessary to provide a method, device, electronic device and storage medium for generating visual animation based on audio, which can adaptively obtain actions similar to the sound style and meet personalized needs.

[0005] The present invention provides a method for generating visual animation based on audio, the method comprising:

[0006] Read audio data and convert it into numpy array format;

[0007] Use Fourier transform to convert audio signals from time domain to frequency domain and remove high frequency noise and clutter;

[0008] Identify the beat position in the audio signal and convert it into Byte data;

[0009] Convert each Byte data into corresponding action instructions;

[0010] Querying action information corresponding to the action instruction from a database;

[0011] According to the queried action information, the action instructions are converted into animation instructions;

[0012] Generate animations according to animation instructions and save them as video files.

[0013] Furthermore, the method of converting the audio signal from the time domain to the frequency domain using Fourier transform and removing high-frequency noise and clutter includes:

[0014] Use numpy.fft.fft() inverse FFT in the Python math library numpy to convert the audio data from the time domain to the frequency domain, and use numpy.fft.ifft() to remove high-frequency noise and clutter.

[0015] Furthermore, the step of identifying the beat position in the audio signal and converting it into Byte data includes:

[0016] Load the audio file after conversion and removal of high-frequency noise and clutter;

[0017] Apply Onset Detection algorithm to obtain the beat position;

[0018] Convert the beat's time into a sample index;

[0019] Convert the beat sample index to byte position and get the Byte data.

[0020] Furthermore, converting the action instructions into animation instructions according to the queried action information includes:

[0021] Use Python library cv2 to convert base64 animation frames back into OpenCV animation instructions; the action instruction creation process is: use Python library cv2 to convert animation frames into JPG format, and finally convert JPG format into base64 format string, and encode it 0-127 and store it in the database. When the encoding of the animation file is 0, it means that the animation file will be used when the Byte instruction is 0.

[0022] Furthermore, the use of the Python library cv2 to reconvert the base64 animation frames into OpenCV animation instructions includes:

[0023] Decode the base64 string into JPG data;

[0024] Decode JPG data into OpenCV animation instructions.

[0025] The present invention provides a device for generating visual animation based on audio, the device comprising:

[0026] The reading module is used to read audio data and convert it into numpy array format;

[0027] A first conversion module, for converting the audio signal from the time domain to the frequency domain using Fourier transform and removing high frequency noise and clutter;

[0028] The recognition module is used to identify the beat position in the audio signal and convert it into Byte data;

[0029] The second conversion module is used to convert each Byte data into a corresponding action instruction;

[0030] A query module, used for querying action information corresponding to the action instruction from a database;

[0031] A third conversion module is used to convert the action instructions into animation instructions according to the queried action information;

[0032] The generation module is used to generate animation according to the animation instructions and save it as a video file.

[0033] Furthermore, the method of converting the audio signal from the time domain to the frequency domain using Fourier transform and removing high-frequency noise and clutter includes:

[0034] Use numpy.fft.fft() inverse FFT in the Python math library numpy to convert the audio data from the time domain to the frequency domain, and use numpy.fft.ifft() to remove high-frequency noise and clutter.

[0035] Furthermore, the step of identifying the beat position in the audio signal and converting it into Byte data includes:

[0036] Load the audio file after conversion and removal of high-frequency noise and clutter;

[0037] Apply Onset Detection algorithm to obtain the beat position;

[0038] Convert the beat's time into a sample index;

[0039] Convert the beat sample index to byte position and get the Byte data.

[0040] The present invention provides an electronic device, comprising:

[0041] Processor; and

[0042] A storage device includes processor executable instructions, and when the executable instructions are executed by the processor, the steps of the method for generating visual animation based on audio are executed.

[0043] The present invention provides a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the method for generating visual animation based on audio are implemented.

[0044] It can be seen from the above scheme that the present invention reads audio data and converts the audio data into numpy array format; uses Fourier transform to convert the audio signal from time domain to frequency domain and remove high-frequency noise and clutter; identifies the beat position in the audio signal and converts it into Byte data; converts each Byte data into a corresponding action instruction; queries the action information corresponding to the action instruction from the database; converts the action instruction into an animation instruction according to the queried action information; generates an animation according to the animation instruction and saves it as a video file. The present invention maps each byte to an action, realizes the conversion between sound and action, and can master and create a variety of rich actions through simple voice commands or songs. By identifying different actions and beats, more efficient and accurate action recognition and animation generation can be achieved, and it can also automatically adapt to different animation styles and personalized needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The present invention is a method flow chart of one embodiment of the present invention.

[0046] Figure 2 It is a schematic structural diagram of an electronic device according to one embodiment of the present invention.

[0047] Figure 3 It is a schematic structural diagram of an apparatus for generating visual animation based on audio according to an embodiment of the present invention.

[0048] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention. It is understood that the drawings are only provided for reference and explanation, and are not used to limit the present invention.

[0050] like Figure 1 As shown, the method for generating visual animation based on audio provided by the present invention may include the following steps.

[0051] S110: Read audio data and convert the audio data into a numpy array format.

[0052] S120: Use Fourier transform to convert the audio signal from the time domain to the frequency domain and remove high-frequency noise and clutter.

[0053] In this step, we use the numpy.fft.fft() inverse FFT in the Python math library numpy to convert the audio data from the time domain to the frequency domain, and use numpy.fft.ifft() to remove high-frequency noise and clutter.

[0054] S130: Identify the beat position in the audio signal and convert it into Byte data.

[0055] Specifically, this step includes the following steps S131-S134:

[0056] S131: Loading the audio file after conversion and removal of high-frequency noise and clutter.

[0057] S132: Apply Onset Detection algorithm to obtain the beat position.

[0058] S133: Convert the beat time into a sample index.

[0059] S134: Convert the beat sample index into a byte position and obtain Byte data.

[0060] S140: Convert each Byte data into a corresponding action instruction.

[0061] S150: Query the database for action information corresponding to the action instruction.

[0062] S160: Converting the action instruction into an animation instruction according to the retrieved action information.

[0063] Specifically, this step includes: using the Python library cv2 to reconvert the base64 animation frame into OpenCV animation instructions; wherein, the action instruction creation process is: using the Python library cv2 to convert the animation frame into JPG format, and finally converting the JPG format into a base64 format string, encoding it 0-127 and storing it in the database. When the encoding of the animation file is 0, it means that the animation file will be used when the Byte instruction is 0.

[0064] Furthermore, using the Python library cv2 to convert the base64 animation frames back into OpenCV animation instructions may specifically include: decoding the base64 string into JPG data, and then decoding the JPG data into OpenCV animation instructions.

[0065] S170: Generate an animation according to the animation instruction and save it as a video file.

[0066] The present invention maps each byte to an action, realizing the conversion between sound and action. Various rich actions can be mastered and created through simple voice commands or songs. By identifying different actions and beats, more efficient and accurate action recognition and animation generation can be achieved, and it can also automatically adapt to different animation styles and personalized needs.

[0067] Figure 2 A structural diagram of an embodiment of an electronic device 100 is provided for the present invention. The electronic device 100 may include a processor 10 and a storage device 20 , and also includes a computer program stored in the storage device 20 and executable on the processor 10 .

[0068] The processor 10 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the electronic device 100, and various interfaces and lines are used to connect various parts of the entire electronic device 100.

[0069] The storage device 20 can be used to store the computer program and / or module. The processor 10 realizes various functions of the electronic device by running or executing the computer program and / or module stored in the storage device 20 and calling the data stored in the storage device 20. The storage device 20 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, a program required for at least one function, etc.; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the storage device 20 can include a high-speed random access storage device, and can also include a non-volatile storage device, such as a hard disk, a memory, a plug-in hard disk, a smart memory card, a secure digital card, a flash memory card, at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0070] Figure 3 The invention provides a structural diagram of a device 200 for generating visual animation based on audio, wherein the device 200 for generating visual animation based on audio may include a reading module 21, a first conversion module 22, a recognition module 23, a second conversion module 24, a query module 25, a third conversion module 26, and a generation module 27.

[0071] A reading module 21, used for reading audio data and converting the audio data into a numpy array format;

[0072] A first conversion module 22, for converting the audio signal from the time domain to the frequency domain using Fourier transform and removing high frequency noise and clutter;

[0073] The recognition module 23 is used to recognize the beat position in the audio signal and convert it into Byte data;

[0074] The second conversion module 24 is used to convert each Byte data into a corresponding action instruction;

[0075] A query module 25, used to query the action information corresponding to the action instruction from the database;

[0076] The third conversion module 26 is used to convert the action instruction into an animation instruction according to the queried action information;

[0077] The generating module 27 is used to generate the animation according to the animation instruction and save it as a video file.

[0078] Furthermore, the method of converting the audio signal from the time domain to the frequency domain using Fourier transform and removing high-frequency noise and clutter includes:

[0079] Use numpy.fft.fft() inverse FFT in the Python math library numpy to convert the audio data from the time domain to the frequency domain, and use numpy.fft.ifft() to remove high-frequency noise and clutter.

[0080] Among them, numpy.fft.fft() is used to perform the inverse Cooley-Tukey FFT operation to transform the signal.

[0081] Furthermore, the step of identifying the beat position in the audio signal and converting it into Byte data includes:

[0082] Load the audio file after conversion and removal of high-frequency noise and clutter;

[0083] Apply Onset Detection algorithm to obtain the beat position;

[0084] Convert the beat's time into a sample index;

[0085] Convert the beat sample index to byte position and get the Byte data.

[0086] Furthermore, converting the action instructions into animation instructions according to the queried action information includes:

[0087] Use Python library cv2 to convert base64 animation frames back into OpenCV animation instructions; the action instruction creation process is: use Python library cv2 to convert animation frames into JPG format, and finally convert JPG format into base64 format string, and encode it 0-127 and store it in the database. When the encoding of the animation file is 0, it means that the animation file will be used when the Byte instruction is 0.

[0088] Furthermore, the use of the Python library cv2 to reconvert the base64 animation frames into OpenCV animation instructions includes:

[0089] Decode the base64 string into JPG data;

[0090] Decode JPG data into OpenCV animation instructions.

[0091] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for generating visual animation based on audio are implemented.

[0092] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating visual animation based on audio, characterized in that: The method comprises: Read audio data and convert it into numpy array format; Use Fourier transform to convert audio signals from time domain to frequency domain and remove high frequency noise and clutter; Identify the beat position in the audio signal and convert it into Byte data; Convert each Byte data into corresponding action instructions; Querying action information corresponding to the action instruction from a database; According to the queried action information, the action instructions are converted into animation instructions; Generate animation according to animation instructions and save it as a video file; The step of identifying the beat position in the audio signal and converting it into Byte data includes: Load the audio file after conversion and removal of high-frequency noise and clutter; Apply Onset Detection algorithm to obtain the beat position; Convert the beat's time into a sample index; Convert the beat sample index to byte position and get the Byte data.

2. The method for generating visual animation based on audio according to claim 1, characterized in that: The method of using Fourier transform to convert the audio signal from the time domain to the frequency domain and remove high-frequency noise and clutter includes: Use numpy.fft.fft() inverse FFT in the Python math library numpy to convert the audio data from the time domain to the frequency domain, and use numpy.fft.ifft() to remove high-frequency noise and clutter.

3. The method for generating visual animation based on audio according to claim 1, characterized in that: The step of converting the action instructions into animation instructions according to the queried action information includes: Use Python library cv2 to convert base64 animation frames back into OpenCV animation instructions; the action instruction creation process is: use Python library cv2 to convert animation frames into JPG format, and finally convert JPG format into base64 format string, and encode it 0-127 and store it in the database. When the encoding of the animation file is 0, it means that the animation file will be used when the Byte instruction is 0.

4. The method for generating visual animation based on audio according to claim 3, characterized in that: The Python library cv2 is used to convert the base64 animation frames back into OpenCV animation instructions, including: Decode the base64 string into JPG data; Decode JPG data into OpenCV animation instructions.

5. A device for generating visual animation based on audio, characterized in that: The device comprises: The reading module is used to read audio data and convert it into numpy array format; A first conversion module, for converting the audio signal from the time domain to the frequency domain using Fourier transform and removing high frequency noise and clutter; The recognition module is used to identify the beat position in the audio signal and convert it into Byte data; The second conversion module is used to convert each Byte data into a corresponding action instruction; A query module, used for querying action information corresponding to the action instruction from a database; A third conversion module, used to convert the action instructions into animation instructions according to the queried action information; A generation module, used for generating animation according to animation instructions and saving it as a video file; The step of identifying the beat position in the audio signal and converting it into Byte data includes: Load the converted audio file and remove the high-frequency noise and clutter; Apply Onset Detection algorithm to obtain the beat position; Convert the beat's time into a sample index; Convert the beat sample index to byte position and get the Byte data.

6. The device for generating visual animation based on audio according to claim 5, characterized in that: The method of converting the audio signal from the time domain to the frequency domain and removing high-frequency noise and clutter using Fourier transform includes: Use numpy.fft.fft() inverse FFT in the Python math library numpy to convert the audio data from the time domain to the frequency domain, and use numpy.fft.ifft() to remove high-frequency noise and clutter.

7. An electronic device, characterized in that: The electronic device comprises: Processor; and A storage device, the storage device comprising processor executable instructions, wherein the executable instructions are used to execute the steps of the method for generating visual animation based on audio as described in any one of claims 1-4 when executed by the processor.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generating visual animation based on audio as described in any one of claims 1 to 4 are implemented.

Images