Audio data extraction method and device and storage medium

Abstract

The invention relates to the technical field of audio processing, in particular to an audio data extraction method and device and a storage medium, and the method comprises the steps: receiving a video stream, and reading multimedia data based on a video address; processing the multimedia data to obtain audio data; decoding the audio data to obtain audio coded data; the audio encoded data is assembled into an audio stream. According to the method provided by the invention, the audio data in the stream address are analyzed and extracted by reading the stream address, and the audio data are packaged into the media stream form data which can be used for converging with the corresponding video track.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular to an audio data extraction method, device and storage medium. Background Art

[0002] The digital audio matrix system is generally a device consisting of multiple input ports and multiple output ports. It encodes the audio analog signal into a digital signal, then performs various DSP audio processing, and finally outputs the audio analog signal to the power amplifier for playback through analog-to-digital conversion. The digital audio matrix system manages and controls the transmission and distribution of audio signals, and provides signal routing, switching, distribution and control functions to achieve flexible operation and management of multi-source and multi-target audio. Therefore, the digital audio matrix system is widely used in control centers, command and dispatch centers, multimedia conference rooms, remote video conferencing centers, exhibition halls, banquet halls, stadiums and other scenes to meet the various application requirements of these scenes for public sound reinforcement systems.

[0003] At present, the DSP audio processing process in the digital audio matrix system includes multiple processing links, such as decoding, post-processing, channel sorting, etc., and the execution of these processes is in a serial relationship. In some application scenarios, it is necessary to process the audio data in the multimedia data separately, and an audio data extraction method, device and storage medium are urgently needed. Summary of the invention

[0004] The purpose of the present invention is to provide an audio data extraction method to extract video stream data and process the audio data therein separately.

[0005] In order to solve the above technical problems, the present invention provides an audio data extraction method, comprising:

[0006] Receive video stream and read multimedia data based on video address;

[0007] Process multimedia data to obtain audio data;

[0008] Decode the audio data to obtain audio encoding data;

[0009] Assembles audio encoded data into an audio stream.

[0010] Optionally, the process of reading multimedia data is as follows:

[0011] Request the target video address, continuously read the video stream, and store each byte of the video stream into the array buffer to form multimedia data.

[0012] Optionally, the multimedia data is formed by merging all array buffers.

[0013] Optionally, after merging an array buffer, get the new buffer and set the offset and buffer length of the new buffer.

[0014] Optionally, the process of processing multimedia data to obtain audio data is as follows:

[0015] The multimedia data is received, and the audio data and the video data are distinguished according to the hexadecimal data header, and the multimedia data is divided into the audio data and the video data.

[0016] Optionally, the process of decoding audio data is as follows:

[0017] The audio processing module is called to asynchronously decode the audio data to obtain audio encoding data, wherein the audio encoding data includes audio information.

[0018] Optionally, the audio information includes audio length, number of channels, sampling rate and duration.

[0019] Optionally, the process of assembling the audio coded data into an audio stream is as follows:

[0020] Set the audio information of the audio encoding data;

[0021] Performing circular cutting on the audio coding data based on the sampling rate to obtain the audio coding data of each channel;

[0022] Establishing an audio buffer, wherein parameters of the audio buffer include channels, buffer length and sampling rate;

[0023] Filling the audio encoding data into the audio buffer;

[0024] Assign the audio information of the audio buffer to obtain the audio stream.

[0025] In addition, the present invention also provides an electronic device, comprising:

[0026] Memory, used to store computer programs;

[0027] The processor is used to execute the computer program to implement the steps of the audio data extraction method as described above.

[0028] In addition, the present invention also provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the method for extracting audio data as described above are implemented.

[0029] The audio data extraction method provided by the present invention adopts the method of obtaining the read stream address to parse and extract the audio PCM data therein, and the audio data is encapsulated into media stream data by the method of the present invention, which can be used to merge with the corresponding video track. In addition, the present invention can also perform noise reduction, noise elimination and other processing on the extracted PCM data to realize the separate processing of audio data in multimedia data.

[0030] The present invention also provides an electronic device and a computer-readable storage medium, which also have the above-mentioned beneficial effects and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 A flowchart of an audio data extraction method provided by an embodiment of the present invention;

[0033] Figure 2 The present invention provides a structural block diagram of an electronic device. DETAILED DESCRIPTION

[0034] The core of the present invention is to provide an audio data extraction method. In the prior art, the traditional audio data processing method is mainly based on aligning audio data, but there is no separate processing method for audio data. In some audio data processing scenarios, before merging audio data, it is necessary to perform operations such as denoising and eliminating noise on the audio data.

[0035] The present invention provides an audio data extraction method, which includes receiving a video stream, reading multimedia data based on a video address; processing the multimedia data to obtain audio data; decoding the audio data to obtain audio encoding data; and assembling the audio encoding data into an audio stream.

[0036] The present invention extracts audio data from multimedia data, obtains audio coding data by decoding the audio data, and the audio coding data can be processed accordingly, thereby realizing separate processing of audio data in multimedia data.

[0037] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0038] Please refer to Figure 1 , Figure 1 The present invention provides a flowchart of an audio data extraction method.

[0039] See also Figure 1 In an embodiment of the present invention, the audio data extraction method includes:

[0040] S1: Receive video stream and read multimedia data based on the video address.

[0041] Furthermore, the process of reading multimedia data in this embodiment is as follows:

[0042] Request the target video address, continuously read the video stream, and store each byte of the video stream into the array buffer to form multimedia data.

[0043] In a more specific embodiment, fetch (an API for network request, built on Promise) is used to request the target mp4 video address, and the getReader of the returned result is used as the current reader to lock the multimedia data.

[0044] It should be noted that, in the process of extracting video stream data, the method of this embodiment implements continuous reading by setting the reading state to while, and fills each byte into the array buffer.

[0045] Furthermore, the multimedia data is formed by merging all array buffers, and the starting position of the multimedia data is set to 0. In this embodiment, the audio data and the video data can be distinguished according to the hexadecimal data header.

[0046] In some more specific implementation cases, the process of array buffer merging is as follows:

[0047] After merging an array buffer, a new buffer is obtained, the offset and buffer length of the new buffer are set, and then another data buffer is merged. The above process is repeated until all array buffers are merged to obtain multimedia data.

[0048] S2: Process the multimedia data to obtain audio data.

[0049] Specifically, the process of processing multimedia data to obtain audio data is as follows:

[0050] The multimedia data is received, and the audio data and the video data are distinguished according to the hexadecimal data header, and the multimedia data is divided into the audio data and the video data.

[0051] It should be noted that in this embodiment, AudioContext (one of the core objects of the Web Audio API, used to process and operate audio in a web page) is used to process multimedia data, and AudioContext is used to generate, process and control audio signals.

[0052] In this embodiment, the audio components include waveform, frequency, amplitude, sample rate, bit depth, channels, timbre, duration, audio format and effects processing.

[0053] S3: Decode the audio data to obtain audio encoding data.

[0054] Specifically, the process of decoding audio data is as follows:

[0055] The audio processing module is called to asynchronously decode the audio data to obtain audio encoding data, wherein the audio encoding data includes audio information.

[0056] The audio processing module in this embodiment is AudioContext, and the audio coding data is specifically PCM data (Pulse Code Modulation). PCM data is a technology that uses digital signals to represent analog signals and is applied to the digitization process of audio and video data.

[0057] In some specific embodiments, the audio information includes audio length, number of channels, sampling rate and duration.

[0058] S4: Assemble the audio encoding data into an audio stream.

[0059] Optionally, the process of assembling the audio coded data into an audio stream is as follows:

[0060] Setting audio information of the audio coding data, the audio signal in this embodiment includes but is not limited to audio length, number of channels, sampling rate and duration;

[0061] The audio coded data is circularly cut based on the sampling rate to obtain the audio coded data of each channel. It should be noted that the process of circularly cutting the audio coded data in this embodiment adopts Float32Array (a typed array in JavaScript, used to represent an array containing 32-bit floating-point numbers. Float32Array is specially used to process binary data and is usually used for high-performance computing and processing of audio data).

[0062] An audio buffer is established, wherein the parameters of the audio buffer include channels, buffer length and sampling rate. The audio buffer in this embodiment is created using AudioContext;

[0063] Filling the audio encoding data into the audio buffer, it should be noted that the audio information of the audio buffer at this time includes at least the channel, buffer length and sampling rate;

[0064] Assign the audio information of the audio buffer to obtain the audio stream.

[0065] An audio data extraction method provided by an embodiment of the present invention, the audio coded data in the audio buffer in this embodiment can be played by a speaker or a media element to realize the separation of audio data in a video stream, and the audio coded data can also be converted into an audio stream, and the audio stream can perform operations such as audio and video data alignment and confluence. The method of this embodiment can quickly separate audio and video, significantly improving the processing speed, and the user can select a specific video stream for extraction to meet different application requirements. In addition, the method of this embodiment can also integrate a variety of audio processing technologies, and realize the separate processing of audio data in a video stream by performing noise reduction, equalization or other processing methods on the audio coded data.

[0066] An electronic device provided by an embodiment of the present invention is introduced below. The electronic device described below and the audio data extraction method described above can refer to each other.

[0067] Please refer to Figure 2 , Figure 2 This is a structural block diagram of an electronic device provided by an embodiment of the present invention. Figure 2 , the electronic device 100 may include a memory 101 , a processor 102 , and a communication bus 103 ; the memory 101 and the processor 102 are connected via the communication bus 103 .

[0068] The memory 101 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 101 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the functions of the audio data processing method provided in the above embodiment, etc.; the data storage area may store data involved in the audio data extraction method provided in the above embodiment, etc.;

[0069] The processor 102 may include one or more processing cores. The processor 102 calls the data stored in the memory 101 by running or executing the instructions, programs, code sets or instruction sets stored in the memory 101, and performs various functions and processes data of the present application. The processor 102 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller and a microprocessor. It is understandable that for different devices, the electronic device used to implement the above-mentioned processor 102 function can also be other, and the embodiments of the present application are not specifically limited.

[0070] The communication bus 103 may include a path to transmit information between the above components. The communication bus 103 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 103 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one double arrow is used in the diagram, but it does not mean that there is only one bus or one type of bus. Figure 2 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0071] An embodiment of the present application provides a computer-readable storage medium storing a computer program that can be loaded and executed by a processor to implement the audio data processing method provided in the above embodiment.

[0072] In this embodiment, the computer-readable storage medium may be a tangible device that holds and stores instructions used by the instruction execution device. The computer-readable storage medium may be, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof. Specifically, the computer-readable storage medium may be a portable computer disk, a hard disk, a USB flash drive, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a podium random access memory (SRAM), a portable compact disk read-only memory (CDROM), a digital versatile disk (DVD), a memory stick, a floppy disk, an optical disk, a magnetic disk, a mechanical encoding device, and any combination thereof.

[0073] In the embodiments provided in the present application, it should be understood that the disclosed systems and devices can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0074] In addition, it is to be understood that relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprises," "comprising," or any other variation thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or elements that are inherent to such process, method, article, or device.

[0075] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for extracting audio data, characterized in that: include: Receive video stream and read multimedia data based on video address; Process multimedia data to obtain audio data; Decode the audio data to obtain audio encoding data; Assembles audio encoded data into an audio stream.

2. The audio data extraction method according to claim 1, characterized in that: The process of reading multimedia data is as follows: Request the target video address, continuously read the video stream, and store each byte of the video stream into the array buffer to form multimedia data.

3. The audio data extraction method according to claim 2, characterized in that: The multimedia data is formed by merging all array buffers.

4. The audio data extraction method according to claim 3, characterized in that: After merging an array buffer, get the new buffer and set the offset and buffer length of the new buffer.

5. The audio data extraction method according to claim 4, characterized in that: The process of processing multimedia data to obtain audio data is as follows: The multimedia data is received, and the audio data and the video data are distinguished according to the hexadecimal data header, and the multimedia data is divided into the audio data and the video data.

6. The audio data extraction method according to claim 5, characterized in that: The process of decoding audio data is as follows: The audio processing module is called to asynchronously decode the audio data to obtain audio encoding data, wherein the audio encoding data includes audio information.

7. The audio data extraction method according to claim 6, characterized in that: The audio information includes audio length, number of channels, sampling rate and duration.

8. The audio data extraction method according to claim 7, characterized in that: The process of assembling audio encoded data into an audio stream is as follows: Set the audio information of the audio encoding data; Performing circular cutting on the audio coding data based on the sampling rate to obtain the audio coding data of each channel; Establishing an audio buffer, wherein parameters of the audio buffer include channels, buffer length and sampling rate; Filling the audio encoding data into the audio buffer; Assign the audio information of the audio buffer to obtain the audio stream.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the method for extracting audio data as claimed in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein, when the computer program is executed by a processor, the steps of the method for extracting audio data as described in any one of claims 1 to 8 are implemented.

Images