Audio processing method, device and storage medium

Abstract

The embodiment of the present disclosure provides an audio processing method, device and storage medium. In the preview process of a video editing draft, audio decoding is performed on each track by using an audio decoding thread corresponding to each track, respectively, the decoded data is cached in a first cache unit corresponding to each track, respectively, sound effect processing is performed on the decoded data by using a sound effect processing thread corresponding to each track, respectively, the sound effect processing result is cached in a second cache unit corresponding to each track, respectively, the sound effect processing result is read from the second cache unit corresponding to each track by using a preview thread, mixed, and played. Through concurrent processing of multiple tracks, the audio processing efficiency in the preview process can be improved, the fluency of the preview process can be improved, the performance bottleneck of the traditional serial decoding and sound effect processing mode can be broken through, and the maximum utilization of the computing resources and storage resources of the device can be realized.

Description

Technical Field

[0001] This disclosure relates to the field of computer and network communication technology, and in particular to an audio processing method, device and storage medium. Background Technology

[0002] Non-linear editing is a digital production of video using computers. It can quickly and accurately access and edit materials directly from the computer's hard drive in the form of frames or files, overcoming the shortcomings of linear editing. It also requires less equipment, has a user-friendly interface, and improves video production efficiency.

[0003] In non-linear editing, video editing drafts can usually be previewed. During the preview process, audio decoding and sound effects processing are required. However, if the video editing draft has multiple audio tracks, the processing of some sound effects may take a long time, which will prevent the preview process from proceeding smoothly and affect the preview effect. Summary of the Invention

[0004] This disclosure provides an audio processing method, device, and storage medium to improve the audio processing efficiency and smoothness of the preview process.

[0005] In a first aspect, embodiments of this disclosure provide an audio processing method, including:

[0006] During the preview of the video editing draft, the audio decoding thread corresponding to each track of the video editing draft is used to decode the audio of each track, and the decoded data is cached into the first cache unit corresponding to each track.

[0007] Each track's corresponding audio processing thread reads the decoded data from its corresponding first buffer unit for audio processing, and caches the audio processing results into the corresponding second buffer unit for each track.

[0008] The preview thread reads the audio processing results from the second buffer unit corresponding to each track, mixes them, and plays the mixed results.

[0009] In a second aspect, embodiments of this disclosure provide an audio processing device, including:

[0010] The decoding unit is used to perform audio decoding on each track of the video editing draft using the audio decoding thread corresponding to each track during the preview process, and to cache the decoded data into the first cache unit corresponding to each track.

[0011] The audio processing unit is used to read the decoded data from the corresponding first buffer unit for audio processing using the audio processing thread corresponding to each track, and to cache the audio processing results into the corresponding second buffer unit for each track.

[0012] The preview unit is used to read the audio processing results from the second buffer unit corresponding to each track using the preview thread, mix them, and play the mixed results.

[0013] Thirdly, embodiments of this disclosure provide an electronic device, including: at least one processor and a memory;

[0014] The memory stores computer-executed instructions;

[0015] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the audio processing method as described in the first aspect and various possible designs of the first aspect.

[0016] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the audio processing method described in the first aspect and various possible designs of the first aspect.

[0017] Fifthly, embodiments of this disclosure provide a computer program product, including computer execution instructions, which, when executed by a processor, implement the audio processing method described in the first aspect and various possible designs of the first aspect.

[0018] The audio processing method, device, and storage medium provided in this disclosure improve the audio processing efficiency and smoothness of the preview process by employing an audio decoding thread corresponding to each track of the video editing draft during the preview process. The decoded data is then cached in a first cache unit corresponding to each track. A sound effect processing thread corresponding to each track reads the decoded data from the corresponding first cache unit, performs sound effect processing, and caches the results in a second cache unit corresponding to each track. A preview thread reads the sound effect processing results from the second cache units corresponding to each track, mixes them, and plays the mixed results. By processing multiple tracks concurrently, the audio processing efficiency and smoothness of the preview process are improved, overcoming the performance bottlenecks of traditional serial decoding and sound effect processing modes, and maximizing the utilization of device computing and storage resources. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic flowchart of an audio processing method provided in an embodiment of the present disclosure;

[0021] Figure 2 This is a schematic flowchart of an audio processing method provided in another embodiment of the present disclosure;

[0022] Figure 3 This is a schematic flowchart of an audio processing method provided in another embodiment of the present disclosure;

[0023] Figure 4 This is a schematic flowchart of an audio processing method provided in another embodiment of the present disclosure;

[0024] Figure 5 This is a structural block diagram of an audio processing device provided in an embodiment of the present disclosure;

[0025] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0027] In non-linear editing, it is usually possible to preview the video editing draft. During the preview process, audio decoding and sound effects processing are required.

[0028] In one technical solution, a serial processing approach is typically used. For the current frame, the first track's current frame is obtained, audio is decoded and sound effects are processed, and then buffered. The second track's current frame is then obtained, audio is decoded and sound effects are processed, and then buffered. This process continues until all tracks have completed audio decoding and sound effects processing for the current frame. Then, the data corresponding to each track is read from the buffer, mixed, and then the audio is played.

[0029] However, if the video editing draft contains multiple audio tracks, some sound effects may take a long time to process, resulting in a longer audio processing time for each frame, which can cause the preview process to be unsmooth and affect the preview effect.

[0030] To address the aforementioned technical problems, this disclosure provides an audio processing method. During the preview process of a video editing draft, an audio decoding thread corresponding to each track of the draft decodes the audio for each track, caching the decoded data into a first buffer unit corresponding to each track. An audio effect processing thread corresponding to each track reads the decoded data from the corresponding first buffer unit, performs audio effect processing, and caches the results into a second buffer unit corresponding to each track. A preview thread reads the audio effect processing results from the second buffer units corresponding to each track, mixes them, and plays the mixed result. By processing multiple tracks concurrently, the audio processing efficiency and smoothness of the preview process can be improved, overcoming the performance bottlenecks of traditional serial decoding and audio effect processing modes, and maximizing the utilization of device computing and storage resources.

[0031] The audio processing method disclosed herein can be applied to any electronic device capable of non-linear editing, and its flowchart is shown below. Figure 1 As shown, in the audio processing, an audio decoding thread, a first buffer unit, a sound effects processing thread, and a second buffer unit are created for each track of the video editing draft. In addition, a preview thread is also created. The audio decoding thread for each track decodes the audio for each track and buffers the decoded data into the first buffer unit for each track. The sound effects processing thread for each track reads the decoded data from the first buffer unit for sound effects processing and buffers the sound effects processing results into the second buffer unit for each track. The preview thread reads the sound effects processing results from the second buffer units for each track, mixes them, and plays the mixed results.

[0032] It should be noted that the user information and data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0033] The audio processing method of this disclosure will be described in detail below with reference to specific embodiments.

[0034] refer to Figure 2 and Figure 3 , Figure 2 This is a schematic flowchart of an audio processing method provided in an embodiment of the present disclosure. Figure 3This is a schematic diagram of threads and buffer units in an audio processing method (showing only the audio decoding thread, sound effect processing thread, first buffer unit, and second buffer unit for one track). The method of this embodiment can be applied to electronic devices such as terminal devices or servers. This audio processing method includes:

[0035] S201. During the preview of the video editing draft, the audio decoding thread corresponding to each track of the video editing draft is used to perform audio decoding on each track, and the decoded data is cached into the first cache unit corresponding to each track.

[0036] In this embodiment, when previewing a video editing draft, an audio decoding thread can be created for each track of the video editing draft, and the audio decoding of each track can be performed by the audio decoding thread corresponding to each track.

[0037] Furthermore, a buffer unit, designated as the first buffer unit, is created for each track in the video editing draft. This buffer unit stores the decoding data corresponding to each track. Specifically, after each track's audio decoding thread performs audio decoding, the decoded data is buffered into the first buffer unit corresponding to that track. The first buffer units for different tracks can store the same number of frames of decoded data; for example, each first buffer unit for different tracks can store 30 frames of decoded data. This ensures that the audio decoding process between different tracks can be performed in parallel, allowing subsequent steps to retrieve the same frame of decoded data from the first buffer units corresponding to different tracks.

[0038] Optionally, the first buffer unit can be a First-In-First-Out (FIFO) buffer unit. In subsequent steps, decoded data can be read from the first buffer unit in sequence, and each frame of decoded data is deleted from the first buffer unit after being read, thereby ensuring the continuous progress of the audio decoding process. Optionally, the first buffer unit can be a FIFO ring buffer. The advantage of using a ring buffer is that when a frame of decoded data is taken out (read and deleted) from the ring buffer, the remaining decoded data does not need to be moved to the storage position. Of course, in this embodiment, the first buffer unit can also be a non-ring buffer of FIFO, in which case the remaining decoded data needs to be moved forward sequentially to the storage position after a frame of decoded data is taken out (read and deleted).

[0039] S202. The audio processing thread corresponding to each track reads the decoded data from the corresponding first buffer unit for audio processing, and caches the audio processing results into the corresponding second buffer unit for each track.

[0040] In this embodiment, a sound effects processing thread (ProcessThread) is created for each track of the video editing draft. This thread can read decoded data from the corresponding first buffer unit and perform sound effects processing. Of course, if a certain segment of a certain track does not have sound effects set, the sound effects processing thread for that track does not need to perform sound effects processing after reading the decoded data from the corresponding first buffer unit. The specific sound effects processing procedure is not limited here.

[0041] In addition, a buffer unit, designated as the second buffer unit, is created for each track in the video editing draft. This buffer unit stores the audio processing results corresponding to each track. Specifically, the audio processing thread for each track reads the decoded data from the corresponding first buffer unit, performs audio processing, and then caches the results in the second buffer unit for each track. The second buffer units for different tracks can store the same number of frames of decoded data; for example, each second buffer unit for different tracks can store 30 frames of audio processing results. This ensures that the audio processing between different tracks can be performed in parallel, allowing subsequent steps to retrieve the audio processing results for the same frame from the second buffer units of different tracks.

[0042] Optionally, the second buffer unit can also be a first-in-first-out (FIFO) buffer unit. In subsequent steps, the audio processing results can be read from the second buffer unit in sequence, and each frame of audio processing result is deleted from the second buffer unit after being read, thereby ensuring the continuous progress of the audio processing process. Optionally, the second buffer unit can be a FIFO ring buffer; of course, in this embodiment, the second buffer unit can also be a non-ring buffer with FIFO, in which case, after one frame of audio processing result is retrieved (read and deleted), the remaining audio processing results need to be moved forward in sequence to the next storage location.

[0043] S203. The preview thread reads the sound effect processing results from the second buffer unit corresponding to each track, mixes them, and plays the mixed results.

[0044] In this embodiment, a preview thread, or driving thread, is created as the main driving thread for the preview process. It reads the audio processing results from the second buffer unit corresponding to each track and performs mixing to achieve audio mixing between different tracks. Finally, the mixed result is played to achieve audio playback during the preview process.

[0045] In one alternative embodiment, the preview thread can read the audio effect processing result corresponding to the current frame from the second buffer unit corresponding to each track, mix them, and play the mixed result. That is, the preview thread mixes one frame and plays one frame at a time.

[0046] In another alternative embodiment, such as Figure 4 As shown, a buffer unit, referred to as the third buffer unit, can be created to buffer the mixing results. That is, the preview thread reads the audio effect processing results corresponding to the same frame from the second buffer unit corresponding to each track, mixes them, and buffers the mixed results in the third buffer unit. Then, it reads the mixed results corresponding to the current frame from the third buffer unit for playback. By mixing the multi-track audio effect processing results in advance through the third buffer unit, the smoothness of the caching process is ensured.

[0047] Optionally, the third buffer unit can also be a first-in-first-out (FIFO) buffer unit. During playback, the mixed results can be read from the third buffer unit in sequential order, and each frame of mixed result is deleted from the third buffer unit after being read, thus ensuring the continuous progress of the mixing process. Optionally, the third buffer unit can be a FIFO ring buffer; of course, in this embodiment, the third buffer unit can also be a non-ring buffer with FIFO, in which case after one frame of mixed result is retrieved (read and deleted), the remaining mixed results need to be moved forward in storage position sequentially.

[0048] The audio processing method provided in this embodiment, during the preview process of a video editing draft, employs an audio decoding thread corresponding to each track of the video editing draft to decode the audio for each track separately, caching the decoded data into a first buffer unit corresponding to each track; employs an audio effect processing thread corresponding to each track to read the decoded data from the corresponding first buffer unit for audio effect processing, caching the audio effect processing results into a second buffer unit corresponding to each track; and employs a preview thread to read the audio effect processing results from the second buffer units corresponding to each track, mixes them, and plays the mixed results. By processing multiple tracks concurrently, the audio processing efficiency and smoothness of the preview process can be improved, overcoming the performance bottlenecks of traditional serial decoding and audio effect processing modes, and maximizing the utilization of device computing and storage resources.

[0049] Based on any of the above embodiments, when the audio decoding thread corresponding to each track of the video editing draft performs audio decoding on each track in step S201, it may specifically include:

[0050] If the first buffer unit corresponding to any track is not full, then the audio decoding thread corresponding to that track will continue to decode the audio for that track; or

[0051] If the first buffer unit corresponding to any track is full, then the audio decoding thread corresponding to that track will be paused from decoding the audio for that track.

[0052] In this embodiment, since the storage space of the first buffer unit corresponding to each track is limited, for any track, if the first buffer unit corresponding to that track is not full, the audio decoding thread corresponding to that track continues to perform audio decoding on that track, and the decoded data can continue to be written into the first buffer unit corresponding to that track; if the first buffer unit corresponding to that track is full, the decoded data cannot continue to be written into the first buffer unit corresponding to that track, and at this time, a blockage occurs, that is, the audio decoding thread corresponding to that track is blocked, and the audio decoding thread corresponding to that track is suspended until the first buffer unit corresponding to that track has space, and the audio decoding thread corresponding to that track can continue to perform audio decoding on that track.

[0053] Optionally, a semaphore can be used, where a semaphore is essentially a counter. In this embodiment, the semaphore can be used to represent the number of storage locations in the first buffer unit. When the semaphore is 0, that is, the first buffer unit is full, the audio decoding thread is blocked.

[0054] Based on any of the above embodiments, the step S202, which involves using a sound effect processing thread corresponding to each track to read decoded data from the corresponding first buffer unit for sound effect processing, may specifically include:

[0055] If the first buffer unit corresponding to any track is not empty, and the second buffer unit corresponding to that track is not full, then the audio processing thread corresponding to that track continues to read decoded data from the first buffer unit corresponding to that track for audio processing; or

[0056] If the first buffer unit corresponding to any track is empty, and / or the second buffer unit corresponding to that track is full, then the audio processing thread corresponding to that track will pause reading decoded data from the first buffer unit corresponding to that track for audio processing.

[0057] In this embodiment, since the audio effect processing thread of any track reads decoded data from the first buffer unit corresponding to that track for audio effect processing and caches the audio effect processing result in the second buffer unit corresponding to that track, the audio effect processing thread can proceed only if there is decoded data available to be read in the first buffer unit (i.e., the first buffer unit is not empty) and the second buffer unit has storage space to cache the audio effect processing result (i.e., the second buffer unit is not full). Therefore, if the first buffer unit corresponding to that track is not empty and the second buffer unit corresponding to that track is not full, the audio effect processing thread corresponding to that track continues to read decoded data from the first buffer unit corresponding to that track. The audio processing thread reads decoded data from the first buffer unit for audio effect processing. Otherwise, if the first buffer unit corresponding to the track is empty and / or the second buffer unit corresponding to the track is full, the audio effect processing thread corresponding to the track is blocked, and the use of the audio effect processing thread corresponding to the track to read decoded data from the first buffer unit corresponding to the track for audio effect processing is suspended. In other words, the audio effect processing result is no longer written to the second buffer unit until the first buffer unit corresponding to the track is not empty and the second buffer unit corresponding to the track is not full. Only then will the audio effect processing thread corresponding to the track continue to read decoded data from the first buffer unit corresponding to the track for audio effect processing.

[0058] Optionally, the above process can be implemented using semaphores. One semaphore is used to represent the readable decoded data in the first buffer unit corresponding to the track. When the semaphore is 0, the first buffer unit corresponding to the track is empty, and the audio processing thread corresponding to the track is blocked. Another semaphore is used to represent the number of storage locations in the second buffer unit. When the semaphore is 0, that is, the second buffer unit is full, the audio processing thread corresponding to the track is blocked.

[0059] Based on any of the above embodiments, S203, which describes using a preview thread to read the sound effect processing results from the second buffer unit corresponding to each track for mixing, may specifically include:

[0060] If the second buffer unit corresponding to each track is not empty, the preview thread continues to read the audio processing results from the second buffer unit corresponding to each track for mixing; or

[0061] If the second buffer unit corresponding to any track is empty, the preview thread will pause and read the audio processing results from the second buffer unit corresponding to each track for mixing.

[0062] In this embodiment, since the preview thread reads the same frame of audio processing results from the second buffer unit corresponding to each track for mixing, and the processing speed of the audio decoding thread and / or audio processing thread corresponding to each track may differ, if the second buffer unit corresponding to each track is not empty, it means that there is audio processing result corresponding to the same frame to be read in the second buffer unit corresponding to each track. Therefore, the preview thread can continue to read the same frame of audio processing results from the second buffer unit corresponding to each track for mixing. If the second buffer unit corresponding to any track is empty, it means that there is no audio processing result corresponding to the frame to be read in the second buffer unit corresponding to that track, while there is audio processing result corresponding to the frame to be read in the second buffer unit corresponding to other tracks. At this time, the preview thread is blocked, that is, the use of the preview thread to read audio processing results from the second buffer unit corresponding to each track for mixing is paused until the second buffer unit corresponding to each track is not empty, and then the use of the preview thread to read audio processing results from the second buffer unit corresponding to each track for mixing continues.

[0063] Optionally, the above process can be implemented using semaphores. In this process, a semaphore is used to represent the readable audio processing result in the second buffer unit corresponding to each track. If the semaphore of any track is 0, the second buffer unit corresponding to that track is empty, and the preview thread is blocked.

[0064] Furthermore, in a scenario where a third buffer unit is created, and the preview thread reads the audio effect processing results corresponding to the same frame from the second buffer unit corresponding to each track for mixing, and caches the mixed results in the third buffer unit, then reads the mixed results corresponding to the current frame from the third buffer unit for playback, if the second buffer units corresponding to each track are not empty and the third buffer unit is not full, then the preview thread continues to read the audio effect processing results from the second buffer units corresponding to each track for mixing; or

[0065] If the second buffer unit corresponding to any track is empty, and / or the third buffer unit is full, then the mixing process will be paused, and the preview thread will read the audio processing results from the second buffer unit corresponding to each track.

[0066] In this embodiment, similar to the above embodiment, if the third cache unit is full, the preview thread will also be blocked, and the use of the preview thread to read the sound effect processing results from the second cache unit corresponding to each track for mixing will be suspended, and the mixing results will no longer be written to the third storage unit.

[0067] Optionally, another semaphore can be used to represent the number of storage locations in the third cache unit. When the semaphore is 0, that is, the third cache unit is full, the preview thread is blocked.

[0068] In the above embodiments, the audio decoding thread and the audio effect processing thread are driven by the preview thread continuously consuming the audio effect processing results read from the second buffer unit corresponding to each track. This achieves self-driving of audio processing during the preview process and has a very high overall flexibility. Furthermore, by adjusting the size of the buffer unit, the performance limits of audio decoding and audio effect processing can be significantly improved.

[0069] It should be noted that the blocking of the audio decoding thread of any track does not affect the audio decoding thread of other tracks, and the blocking of the sound effect processing thread of any track does not affect the sound effect processing thread of other tracks. However, by running and blocking the preview thread, it can be ensured that the sound effect processing results corresponding to the same frame can be read from the second buffer unit of each track for mixing, thereby driving the audio decoding thread and sound effect processing thread of multiple tracks.

[0070] Corresponding to the audio processing method in the above embodiments, Figure 5 This is a structural block diagram of an audio processing device provided according to an embodiment of the present disclosure. For ease of explanation, only the parts relevant to the embodiments of the present disclosure are shown. (Refer to...) Figure 5 The audio processing device 500 includes a decoding unit 501, a sound effect processing unit 502, and a preview unit 503.

[0071] The decoding unit 501 is used to perform audio decoding on each track of the video editing draft by using the audio decoding thread corresponding to each track of the video editing draft during the preview process, and to cache the decoded data into the first cache unit corresponding to each track.

[0072] The sound effect processing unit 502 is used to read the decoded data from the corresponding first buffer unit for sound effect processing using the sound effect processing thread corresponding to each track, and to cache the sound effect processing results into the second buffer unit corresponding to each track.

[0073] The preview unit 503 is used to read the sound effect processing results from the second buffer unit corresponding to each track using the preview thread, mix them, and play the mixed results.

[0074] In one or more embodiments of this disclosure, when the decoding unit 501 performs audio decoding on each track using the audio decoding thread corresponding to each track of the video editing draft, it is further configured to:

[0075] If the first buffer unit corresponding to any track is not full, then the audio decoding thread corresponding to that track will continue to decode the audio for that track; or

[0076] If the first buffer unit corresponding to any track is full, then the audio decoding thread corresponding to that track will be paused from decoding the audio for that track.

[0077] In one or more embodiments of this disclosure, when the audio effect processing unit 502 reads decoded data from the corresponding first buffer unit for audio effect processing using the audio effect processing thread corresponding to each track, it is further configured to:

[0078] If the first buffer unit corresponding to any track is not empty, and the second buffer unit corresponding to that track is not full, then the audio processing thread corresponding to that track continues to read decoded data from the first buffer unit corresponding to that track for audio processing; or

[0079] If the first buffer unit corresponding to any track is empty, and / or the second buffer unit corresponding to that track is full, then the audio processing thread corresponding to that track will pause reading decoded data from the first buffer unit corresponding to that track for audio processing.

[0080] In one or more embodiments of this disclosure, when the preview thread reads the audio processing results from the second buffer unit corresponding to each track for mixing, it is further configured to:

[0081] If the second buffer unit corresponding to each track is not empty, the preview thread continues to read the audio processing results from the second buffer unit corresponding to each track for mixing; or

[0082] If the second buffer unit corresponding to any track is empty, the preview thread will pause and read the audio processing results from the second buffer unit corresponding to each track for mixing.

[0083] In one or more embodiments of this disclosure, when the preview unit 503 reads the sound effect processing results from the second buffer unit corresponding to each track using the preview thread, mixes them, and plays the mixed results, it is further configured to:

[0084] The preview thread reads the audio effect processing results corresponding to the current frame from the second buffer unit corresponding to each track, mixes them, and plays the mixed results.

[0085] In one or more embodiments of this disclosure, when the preview unit 503 reads the sound effect processing results from the second buffer unit corresponding to each track using the preview thread, mixes them, and plays the mixed results, it is further configured to:

[0086] The preview thread reads the audio effect processing results corresponding to the same frame from the second buffer unit corresponding to each track, mixes them, and caches the mixed results in the third buffer unit. The mixed results corresponding to the current frame are then read from the third buffer unit for playback.

[0087] In one or more embodiments of this disclosure, each cache unit is a first-in-first-out cache unit.

[0088] In one or more embodiments of this disclosure, each buffer unit is a first-in-first-out (FIFO) ring buffer.

[0089] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0090] refer to Figure 6 The diagram illustrates a structural schematic of an electronic device 600 suitable for implementing embodiments of the present disclosure. The electronic device 600 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0091] like Figure 6 As shown, electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0092] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0093] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0094] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0095] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0096] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0097] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0098] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0099] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0100] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0101] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0102] In a first aspect, according to one or more embodiments of this disclosure, an audio processing method is provided, comprising:

[0103] During the preview of the video editing draft, the audio decoding thread corresponding to each track of the video editing draft is used to decode the audio of each track, and the decoded data is cached into the first cache unit corresponding to each track.

[0104] Each track's corresponding audio processing thread reads the decoded data from its corresponding first buffer unit for audio processing, and caches the audio processing results into the corresponding second buffer unit for each track.

[0105] The preview thread reads the audio processing results from the second buffer unit corresponding to each track, mixes them, and plays the mixed results.

[0106] According to one or more embodiments of this disclosure, the step of using an audio decoding thread corresponding to each track of the video editing draft to perform audio decoding on each track respectively further includes:

[0107] If the first buffer unit corresponding to any track is not full, then the audio decoding thread corresponding to that track will continue to decode the audio for that track; or

[0108] If the first buffer unit corresponding to any track is full, then the audio decoding thread corresponding to that track will be paused from decoding the audio for that track.

[0109] According to one or more embodiments of this disclosure, the step of using a sound effect processing thread corresponding to each track to read decoded data from the corresponding first buffer unit for sound effect processing further includes:

[0110] If the first buffer unit corresponding to any track is not empty, and the second buffer unit corresponding to that track is not full, then the audio processing thread corresponding to that track continues to read decoded data from the first buffer unit corresponding to that track for audio processing; or

[0111] If the first buffer unit corresponding to any track is empty, and / or the second buffer unit corresponding to that track is full, then the audio processing thread corresponding to that track will pause reading decoded data from the first buffer unit corresponding to that track for audio processing.

[0112] According to one or more embodiments of this disclosure, the step of using a preview thread to read the audio processing results from the second buffer unit corresponding to each track for mixing further includes:

[0113] If the second buffer unit corresponding to each track is not empty, the preview thread continues to read the audio processing results from the second buffer unit corresponding to each track for mixing; or

[0114] If the second buffer unit corresponding to any track is empty, the preview thread will pause and read the audio processing results from the second buffer unit corresponding to each track for mixing.

[0115] According to one or more embodiments of this disclosure, the step of using a preview thread to read the audio processing results from the second buffer unit corresponding to each track, mixing them, and playing the mixed results includes:

[0116] The preview thread reads the audio effect processing results corresponding to the current frame from the second buffer unit corresponding to each track, mixes them, and plays the mixed results.

[0117] According to one or more embodiments of this disclosure, the step of using a preview thread to read the audio processing results from the second buffer unit corresponding to each track, mixing them, and playing the mixed results includes:

[0118] The preview thread reads the audio effect processing results corresponding to the same frame from the second buffer unit corresponding to each track, mixes them, and caches the mixed results in the third buffer unit. The mixed results corresponding to the current frame are then read from the third buffer unit for playback.

[0119] According to one or more embodiments of this disclosure, each cache unit is a first-in-first-out cache unit.

[0120] According to one or more embodiments of this disclosure, each buffer unit is a first-in-first-out (FIFO) ring buffer.

[0121] Secondly, according to one or more embodiments of this disclosure, an audio processing device is provided, comprising:

[0122] The decoding unit is used to perform audio decoding on each track of the video editing draft using the audio decoding thread corresponding to each track during the preview process, and to cache the decoded data into the first cache unit corresponding to each track.

[0123] The audio processing unit is used to read the decoded data from the corresponding first buffer unit for audio processing using the audio processing thread corresponding to each track, and to cache the audio processing results into the corresponding second buffer unit for each track.

[0124] The preview unit is used to read the audio processing results from the second buffer unit corresponding to each track using the preview thread, mix them, and play the mixed results.

[0125] According to one or more embodiments of this disclosure, when the decoding unit performs audio decoding on each track using the audio decoding thread corresponding to each track of the video editing draft, it is further configured to:

[0126] If the first buffer unit corresponding to any track is not full, then the audio decoding thread corresponding to that track will continue to decode the audio for that track; or

[0127] If the first buffer unit corresponding to any track is full, then the audio decoding thread corresponding to that track will be paused from decoding the audio for that track.

[0128] According to one or more embodiments of this disclosure, when the audio effect processing unit reads decoded data from the corresponding first buffer unit for audio effect processing using the audio effect processing thread corresponding to each track, it is further configured to:

[0129] If the first buffer unit corresponding to any track is not empty, and the second buffer unit corresponding to that track is not full, then the audio processing thread corresponding to that track continues to read decoded data from the first buffer unit corresponding to that track for audio processing; or

[0130] If the first buffer unit corresponding to any track is empty, and / or the second buffer unit corresponding to that track is full, then the audio processing thread corresponding to that track will pause reading decoded data from the first buffer unit corresponding to that track for audio processing.

[0131] According to one or more embodiments of this disclosure, when the audio processing results are read from the second buffer unit corresponding to each track by the preview thread for mixing, it is further configured to:

[0132] If the second buffer unit corresponding to each track is not empty, the preview thread continues to read the audio processing results from the second buffer unit corresponding to each track for mixing; or

[0133] If the second buffer unit corresponding to any track is empty, the preview thread will pause and read the audio processing results from the second buffer unit corresponding to each track for mixing.

[0134] According to one or more embodiments of this disclosure, when the preview unit reads the audio processing results from the second buffer unit corresponding to each track using a preview thread, mixes them, and plays the mixed results, it is further configured to:

[0135] The preview thread reads the audio effect processing results corresponding to the current frame from the second buffer unit corresponding to each track, mixes them, and plays the mixed results.

[0136] According to one or more embodiments of this disclosure, when the preview unit reads the audio processing results from the second buffer unit corresponding to each track using a preview thread, mixes them, and plays the mixed results, it is further configured to:

[0137] The preview thread reads the audio effect processing results corresponding to the same frame from the second buffer unit corresponding to each track, mixes them, and caches the mixed results in the third buffer unit. The mixed results corresponding to the current frame are then read from the third buffer unit for playback.

[0138] According to one or more embodiments of this disclosure, each cache unit is a first-in-first-out cache unit.

[0139] According to one or more embodiments of this disclosure, each buffer unit is a first-in-first-out (FIFO) ring buffer.

[0140] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0141] The memory stores computer-executed instructions;

[0142] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the audio processing method as described in the first aspect and various possible designs of the first aspect.

[0143] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, which, when executed by a processor, implement the audio processing method described in the first aspect and various possible designs of the first aspect.

[0144] Fifthly, according to one or more embodiments of the present disclosure, a computer program product is provided, including computer execution instructions that, when executed by a processor, implement the audio processing method described in the first aspect and various possible designs of the first aspect.

[0145] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0146] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0147] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An audio processing method, characterized by, The method comprises the following steps: During previewing the video editing draft, audio decoding threads corresponding to each track of the video editing draft are used to respectively perform audio decoding on each track, and the decoded data is respectively cached into a first cache unit corresponding to each track, wherein if the first cache unit corresponding to any track is full, audio decoding of the track using the audio decoding thread corresponding to the track is paused; Audio effect processing threads corresponding to each track are used to respectively read the decoded data from the first cache unit corresponding to each track to perform audio effect processing, and the audio effect processing result is respectively cached into a second cache unit corresponding to each track, wherein if the first cache unit corresponding to any track is empty and / or the second cache unit corresponding to the track is full, reading the decoded data from the first cache unit corresponding to the track using the audio effect processing thread corresponding to the track to perform audio effect processing is paused; A preview thread is used to read the audio effect processing result from the second cache unit corresponding to each track to perform mixing, and the mixing result is played. If the second cache unit corresponding to any track is empty, reading the audio effect processing result from the second cache unit corresponding to each track using the preview thread to perform mixing is paused.

2. The method of claim 1, wherein, The method further comprises the following steps: If the first cache unit corresponding to any track is not full, audio decoding of the track using the audio decoding thread corresponding to the track is continued.

3. The method of claim 1, wherein, The method further comprises the following steps: If the first cache unit corresponding to any track is not empty and the second cache unit corresponding to the track is not full, reading the decoded data from the first cache unit corresponding to the track using the audio effect processing thread corresponding to the track to perform audio effect processing is continued.

4. The method of claim 1, wherein, The method further comprises the following steps: If the second cache unit corresponding to each track is not empty, reading the audio effect processing result from the second cache unit corresponding to each track using the preview thread to perform mixing is continued.

5. The method of claim 1, wherein, The method further comprises the following steps: The preview thread reads the audio effect processing result corresponding to a current frame from the second cache unit corresponding to each track to perform mixing, and plays the mixing result.

6. The method of claim 1, wherein, The preview thread reads the audio effect processing result corresponding to the same frame from the second cache unit corresponding to each track to perform mixing, and caches the mixing result into a third cache unit, reads the mixing result corresponding to a current frame from the third cache unit to perform playing. Each cache unit is a first-in-first-out cache unit.

7. The method according to any one of claims 1 to 6, characterized in that, Each cache unit is a first-in-first-out ring buffer.

8. The method of claim 7, wherein, The method comprises the following steps:

9. An audio processing device, characterized by ​ The decoding unit is configured to, during previewing of the video editing draft, perform audio decoding on each track of the video editing draft by using an audio decoding thread corresponding to the track, and store the decoded data in a first buffer corresponding to the track, wherein if the first buffer corresponding to any track is full, the audio decoding thread corresponding to the track is suspended. The audio effect processing unit is configured to perform audio effect processing on the decoded data in the first buffer corresponding to each track by using an audio effect processing thread corresponding to the track, and store the audio effect processing result in a second buffer corresponding to the track, wherein if the first buffer corresponding to any track is empty and / or the second buffer corresponding to the track is full, the audio effect processing thread corresponding to the track is suspended. The preview unit is configured to mix the audio effect processing results in the second buffer corresponding to each track by using a preview thread, and play the mixed result. If the second buffer corresponding to any track is empty, the preview thread is suspended.

10. An electronic device, comprising: The method comprises the following steps: obtaining a video editing draft; performing audio decoding on each track of the video editing draft by using an audio decoding thread corresponding to the track, and storing the decoded data in a first buffer corresponding to the track; performing audio effect processing on the decoded data in the first buffer corresponding to each track by using an audio effect processing thread corresponding to the track, and storing the audio effect processing result in a second buffer corresponding to the track; 11. A computer readable storage medium, characterized in that, mixing the audio effect processing results in the second buffer corresponding to each track by using a preview thread, and playing the mixed result.

12. A computer program product, characterised in that, The method comprises the following steps: obtaining a video editing draft; performing audio decoding on each track of the video editing draft by using an audio decoding thread corresponding to the track, and storing the decoded data in a first buffer corresponding to the track; performing audio effect processing on the decoded data in the first buffer corresponding to each track by using an audio effect processing thread corresponding to the track, and storing the audio effect processing result in a second buffer corresponding to the track; mixing the audio effect processing results in the second buffer corresponding to each track by using a preview thread, and playing the mixed result. The computer readable storage medium stores computer executable instructions, and when the processor executes the computer executable instructions, the method in any one of claims 1-8 is implemented. The computer readable storage medium stores computer executable instructions, and when the processor executes the computer executable instructions, the method in any one of claims 1-8 is implemented.

Images