Audio processing method, and device and storage medium

Abstract

Provided are an audio processing method, and a device and a storage medium. The audio processing method comprises: during the process of previewing a video editing draft, performing audio decoding on each track by using an audio decoding thread corresponding to each track of the video editing draft, and caching decoded data in a first cache unit corresponding to each track (S201); by using a sound-effect processing thread corresponding to each track, reading, from the corresponding first caching unit, the decoded data for sound-effect processing, and caching a sound-effect processing result in a second cache unit corresponding to each track (S202); and by using a preview thread, reading, from the second cache unit corresponding to each track, the sound-effect processing result for mixing, and playing a mixing result (S203). By means of performing concurrent processing on a plurality of tracks, the audio processing efficiency during a preview process can be improved, the fluency of the preview process is improved, the performance bottleneck of traditional serial decoding and sound-effect processing modes is broken through, and the maximum utilization of device computing resources and storage resources of a device is realized.

Description

Audio processing method, device and storage medium

[0001] This application claims priority to Chinese Patent Application No. 202311121272.2 filed on August 31, 2023, and the contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0002] Embodiments of the present disclosure relate to an audio processing method, device, and storage medium. Background Art

[0003] Non-linear editing is the digital production of videos with the help of computers. It can quickly and accurately access materials in the form of frames or files directly from the computer's hard disk for editing, overcoming the shortcomings of linear editing. It also takes up less equipment and has a user-friendly human-computer interface, thereby improving video production efficiency.

[0004] During the non-linear editing process, the video editing draft can usually be previewed. During the preview process, audio decoding and sound effect processing are required. However, if the video editing draft contains multiple audio tracks, the processing of certain sound effects may also take a long time, and the preview process cannot proceed smoothly, affecting the preview effect.

[0005] Summary of the Invention

[0006] The embodiments of the present disclosure provide an audio processing method, device, and storage medium to improve the audio processing efficiency and the smoothness of the preview process.

[0007] In a first aspect, an embodiment of the present disclosure provides an audio processing method, comprising:

[0008] During the preview of the video editing draft, the audio decoding thread corresponding to each track of the video editing draft is used to respectively decode the audio of each track, and the decoded data is respectively cached in the first cache unit corresponding to each track;

[0009] Using the sound effect processing thread corresponding to each track to read the decoded data from the corresponding first cache unit for sound effect processing, and cache the sound effect processing results in the second cache unit corresponding to each track;

[0010] The preview thread is used to read the sound effect processing results from the second cache unit corresponding to each track, mix them, and play the mixed results.

[0011] In a second aspect, an embodiment of the present disclosure provides an audio processing device, including:

[0012] a decoding unit configured to, during a preview of the video editing draft, use an audio decoding thread corresponding to each track of the video editing draft to respectively decode the audio of each track, and cache the decoded data in a first cache unit corresponding to each track;

[0013] The sound effect processing unit is configured to use the sound effect processing thread corresponding to each track to read the decoded data from the corresponding first cache unit for sound effect processing, and cache the sound effect processing results in the second cache unit corresponding to each track;

[0014] The preview unit is configured to use a preview thread to read the sound effect processing results from the second cache unit corresponding to each track, mix them, and play the mixed results.

[0015] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

[0016] The memory stores computer-executable instructions;

[0017] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the audio processing method described in the first aspect and various possible designs of the first aspect.

[0018] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the audio processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0019] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, comprising computer-executable instructions. When a processor executes the computer-executable instructions, the audio processing method as described in the first aspect and various possible designs of the first aspect is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, a brief introduction to the drawings required for use in the embodiments will be given below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] FIG1 is a schematic flow chart of an audio processing method according to an embodiment of the present disclosure;

[0022] FIG2 is a flow chart of an audio processing method according to another embodiment of the present disclosure;

[0023] FIG3 is a flow chart of an audio processing method provided by another embodiment of the present disclosure;

[0024] FIG4 is a schematic flow chart of an audio processing method provided by another embodiment of the present disclosure;

[0025] FIG5 is a structural block diagram of an audio processing device provided by an embodiment of the present disclosure; and

[0026] FIG6 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0028] During the non-linear editing process, the video editing draft can usually be previewed, and audio decoding and sound effect processing are required during the preview process.

[0029] In one technical solution, a serial solution is usually adopted for processing. For the current frame, the current frame of the first track is obtained for audio decoding and sound effect processing, and then cached. Then, the current frame of the second track is obtained for audio decoding and sound effect processing, and then cached. After all tracks complete audio decoding and sound effect processing for the current frame, the data corresponding to each track is read from the cache, mixed, and then audio playback is performed.

[0030] However, if the video editing draft contains multiple audio tracks and some sound effects may take a long time to process, the processing of one frame of audio will take a long time, resulting in the preview process not being able to proceed smoothly, affecting the preview effect.

[0031] In order to solve the above technical problems, the present disclosure provides an audio processing method. During the preview of a video editing draft, the audio decoding thread corresponding to each track of the video editing draft is used to perform audio decoding on each track respectively, and the decoded data is cached in the first cache unit corresponding to each track respectively; the sound effect processing thread corresponding to each track is used to read the decoded data from the corresponding first cache unit respectively for sound effect processing, and the sound effect processing results are cached in the second cache unit corresponding to each track respectively; the preview thread is used to read the sound effect processing results from the second cache unit corresponding to each track for mixing, and the mixed results are played. By concurrently processing multiple tracks, the audio processing efficiency of the preview process can be improved, the smoothness of the preview process can be improved, the performance bottleneck of the traditional serial decoding and sound effect processing mode can be broken through, and the maximum utilization of the device computing resources and storage resources can be achieved.

[0032] The audio processing method provided by the present invention is applied to any electronic device capable of non-linear editing, and its flow chart is shown in Figure 1, wherein during the audio processing process, an audio decoding thread, a first cache unit, a sound effect processing thread, and a second cache unit are created for each track of the video editing draft. In addition, a preview thread is created, and the audio decoding thread corresponding to each track is used to perform audio decoding on each track, and the decoded data is cached in the first cache unit corresponding to each track; the sound effect processing thread corresponding to each track is used to read the decoded data from the corresponding first cache unit for sound effect processing, and the sound effect processing results are cached in the second cache unit corresponding to each track; the preview thread is used to read the sound effect processing results from the second cache unit corresponding to each track for mixing, and the mixed results are played.

[0033] It should be noted that the user information and data involved in the embodiments of the present disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0034] The audio processing method of the present disclosure will be described in detail below with reference to specific embodiments.

[0035] Referring to Figures 2 and 3 , Figure 2 is a schematic flow chart of an audio processing method provided by an embodiment of the present disclosure, and Figure 3 is a schematic diagram of threads and cache units in the audio processing method (only the audio decoding thread, sound effect processing thread, first cache unit, and second cache unit for one track are shown). The method of this embodiment can be applied to electronic devices such as terminal devices or servers, and the audio processing method includes:

[0036] S201. During the preview of the video editing draft, the audio decoding thread corresponding to each track of the video editing draft is used to perform audio decoding on each track respectively, and the decoded data is cached in the first cache unit corresponding to each track respectively.

[0037] In this embodiment, when previewing the video editing draft, an audio decoding thread (Decode Thread) may be created for each track of the video editing draft, and audio decoding of each track may be performed separately through the audio decoding thread corresponding to each track.

[0038] In addition, a cache unit is created for each track in the video editing draft, denoted as the first cache unit, and is configured to store the decoded data corresponding to each track. That is, after the audio decoding thread corresponding to each track performs audio decoding on each track, the decoded data can be cached in the first cache unit corresponding to each track. The first cache units corresponding to different tracks can store the same number of frames of decoded data. For example, the first cache units corresponding to different tracks can each store 30 frames of decoded data. This ensures that the audio decoding process for different tracks can be performed in parallel, allowing subsequent steps to obtain decoded data corresponding to the same frame from the first cache units corresponding to different tracks.

[0039] Optionally, the first cache unit may be a first-in-first-out (FIFO) cache unit. In subsequent steps, decoded data may be read from the first cache unit in sequence, and after a frame of decoded data is read, the decoded data of that frame is deleted from the first cache unit, thereby ensuring the continuity of the audio decoding process. Optionally, the first cache unit may be a first-in-first-out (FIFO) ring buffer. The advantage of using a ring buffer is that after a frame of decoded data in the ring buffer is taken out (read and deleted), the remaining decoded data does not need to be moved to a new storage location. Of course, in this embodiment, the first cache unit may also be a non-circular first-in-first-out (FIFO) buffer. In this case, after a frame of decoded data is taken out (read and deleted), the remaining decoded data needs to be moved forward in sequence.

[0040] S202: Use the sound effect processing thread corresponding to each track to read the decoded data from the corresponding first cache unit to perform sound effect processing, and cache the sound effect processing results in the second cache unit corresponding to each track.

[0041] In this embodiment, a sound effect processing thread is created for each track in the video editing draft, which can read decoded data from the corresponding first buffer unit for sound effect processing. Of course, if a certain segment of a track does not have sound effects configured, the sound effect processing thread corresponding to that track reads the decoded data from the corresponding first buffer unit without sound effect processing. The specific sound effect processing process is not limited here.

[0042] In addition, a cache unit is created for each track of the video editing draft, recorded as a second cache unit, and is configured to store the sound effect processing results corresponding to each track. That is, the sound effect processing thread corresponding to each track reads the decoded data from the corresponding first cache unit for sound effect processing, and caches the sound effect processing results in the second cache unit corresponding to each track. The second cache units corresponding to different tracks can store the same number of frames of decoded data. For example, the second cache units corresponding to different tracks can store 30 frames of sound effect processing results, thereby ensuring that the sound effect processing processes between different tracks can be carried out in parallel, so that subsequent steps can obtain the sound effect processing results corresponding to the same frame from the second cache units corresponding to different tracks.

[0043] Optionally, the second cache unit may be a first-in, first-out (FIFO) cache unit. In subsequent steps, the sound effect processing results may be read from the second cache unit in order, and after a frame of sound effect processing results is read, the frame of sound effect processing results is deleted from the second cache unit, thereby ensuring the continuity of the sound effect processing process. Optionally, the second cache unit may be a first-in, first-out (FIFO) ring buffer. Of course, in this embodiment, the second cache unit may also be a non-first-in, first-out (FIFO) ring buffer. In this case, after a frame of sound effect processing results is retrieved (read and deleted), the remaining sound effect processing results need to be moved to the previous storage location in sequence.

[0044] S203: Using a preview thread, read the sound effect processing results from the second cache unit corresponding to each track, mix them, and play the mixed results.

[0045] In this embodiment, a preview thread, or a driving thread, is created as the main driving thread of the preview process. The sound effect processing results are read from the second cache unit corresponding to each track for mixing (Compose), thereby achieving mixing between different tracks. Finally, the mixed results are played to achieve audio playback during the preview process.

[0046] In an optional embodiment, a preview thread may be used to read the sound effect processing result corresponding to the current frame from the second cache unit corresponding to each track, mix the result, and play the mixed result, that is, the preview thread plays one frame after mixing one frame.

[0047] In another optional embodiment, as shown in FIG4 , a cache unit may be created, recorded as a third cache unit, which is configured to cache the mixing results. That is, a preview thread is used to read the sound effect processing results corresponding to the same frame from the second cache units corresponding to each track for mixing, and the mixed results are cached in the third cache unit. The mixed results corresponding to the current frame are read from the third cache unit for playback. The multi-track sound effect processing results are mixed in advance through the third cache unit to ensure the smoothness of the caching process.

[0048] Optionally, the third cache unit may be a first-in, first-out (FIFO) cache unit. During playback, the mixed results may be read from the third cache unit in order, and after a frame of mixed results is read, the mixed results of that frame are deleted from the third cache unit, thereby ensuring the continuity of the mixing process. Optionally, the third cache unit may be a first-in, first-out (FIFO) ring buffer. Of course, in this embodiment, the third cache unit may also be a first-in, first-out (FIFO) non-ring buffer. In this case, after a frame of mixed results is retrieved (read and deleted), the remaining mixed results need to be moved forward in order.

[0049] The audio processing method provided in this embodiment uses the following methods: during the preview of a video editing draft, the audio decoding thread corresponding to each track of the video editing draft is used to perform audio decoding on each track, and the decoded data is cached in the first cache unit corresponding to each track; the sound effect processing thread corresponding to each track is used to read the decoded data from the corresponding first cache unit for sound effect processing, and the sound effect processing results are cached in the second cache unit corresponding to each track; the preview thread is used to read the sound effect processing results from the second cache unit corresponding to each track, mix them, and play the mixed results. By concurrently processing multiple tracks, the audio processing efficiency of the preview process can be improved, the smoothness of the preview process can be improved, the performance bottleneck of the traditional serial decoding and sound effect processing mode can be broken through, and the maximum utilization of the device computing resources and storage resources can be achieved.

[0050] Based on any of the above embodiments, when S201 uses the audio decoding thread corresponding to each track of the video editing draft to respectively decode the audio of each track, the method may specifically include:

[0051] If the first cache unit corresponding to any track is not full, continue to use the audio decoding thread corresponding to the track to perform audio decoding on the track; or

[0052] If the first cache unit corresponding to any track is full, the audio decoding thread corresponding to the track is suspended from performing audio decoding on the track.

[0053] In this embodiment, since the storage space of the first cache unit corresponding to each track is limited, for any track, if the first cache unit corresponding to the track is not full, the audio decoding thread corresponding to the track will continue to be used to perform audio decoding on the track, and the decoded data can continue to be written into the first cache unit corresponding to the track; if the first cache unit corresponding to the track is full, the decoded data cannot continue to be written into the first cache unit corresponding to the track, and blocking occurs at this time, that is, the audio decoding thread corresponding to the track is blocked, and the audio decoding thread corresponding to the track is suspended from performing audio decoding on the track until there is space in the first cache unit corresponding to the track, and the audio decoding thread corresponding to the track can continue to be used to perform audio decoding on the track.

[0054] Optionally, a semaphore method can be used, where the semaphore is essentially a counter. In this embodiment, the semaphore can be used to represent the number of storage locations in the first cache unit. When the semaphore is 0, that is, the first cache unit is full, the audio decoding thread is blocked.

[0055] Based on any of the above embodiments, S202 uses the sound effect processing thread corresponding to each track to read the decoded data from the corresponding first cache unit to perform sound effect processing, which may specifically include:

[0056] If the first cache unit corresponding to any track is not empty and the second cache unit corresponding to the track is not full, then continue to use the sound effect processing thread corresponding to the track to read the decoded data from the first cache unit corresponding to the track for sound effect processing; or

[0057] If the first cache unit corresponding to any track is empty and / or the second cache unit corresponding to the track is full, the audio processing thread corresponding to the track is suspended to read decoded data from the first cache unit corresponding to the track for audio processing.

[0058] In this embodiment, since the sound effect processing thread of any track reads the decoded data from the first cache unit corresponding to the track for sound effect processing, and caches the sound effect processing result in the second cache unit corresponding to the track, the condition for the sound effect processing thread to proceed is that there is decoded data that can be read in the first cache unit (that is, the first cache unit is not empty), and there is storage space in the second cache unit to cache the sound effect processing result (that is, the second cache unit is not full). Therefore, if the first cache unit corresponding to the track is not empty and the second cache unit corresponding to the track is not full, the sound effect processing thread corresponding to the track will continue to read the decoded data from the second cache unit corresponding to the track. The decoded data is read from a cache unit for sound effect processing; otherwise, if the first cache unit corresponding to the track is empty, and / or the second cache unit corresponding to the track is full, the sound effect processing thread corresponding to the track is blocked, and the sound effect processing thread corresponding to the track is suspended from reading the decoded data from the first cache unit corresponding to the track for sound effect processing, and the sound effect processing result is no longer written to the second cache unit, until the first cache unit corresponding to the track is not empty and the second cache unit corresponding to the track is not full, then the sound effect processing thread corresponding to the track is continued to read the decoded data from the first cache unit corresponding to the track for sound effect processing.

[0059] Optionally, the above process can be implemented by means of semaphores, wherein a semaphore is used to represent the decoded data that can be read in the first cache unit corresponding to the track. When the semaphore is 0, the first cache unit corresponding to the track is empty, and the sound effect processing thread corresponding to the track is blocked; another semaphore is used to represent the number of storage locations in the second cache unit. When the semaphore is 0, that is, the second cache unit is full, the sound effect processing thread corresponding to the track is blocked.

[0060] Based on any of the above embodiments, S203 uses a preview thread to read the sound effect processing results from the second cache unit corresponding to each track for mixing, which may specifically include:

[0061] If the second cache units corresponding to the various tracks are not empty, then continue to use the preview thread to read the sound effect processing results from the second cache units corresponding to the various tracks for mixing; or

[0062] If the second cache unit corresponding to any track is empty, the preview thread is paused to read the sound effect processing results from the second cache units corresponding to each track for mixing.

[0063] In this embodiment, since the preview thread reads the same frame of sound effect processing results from the second cache units corresponding to each track for mixing, and the processing speeds of the audio decoding threads and / or sound effect processing threads corresponding to each track may vary, if the second cache units corresponding to each track are not empty, it means that the second cache units corresponding to each track contain sound effect processing results corresponding to the same frame to be read. Therefore, the preview thread can continue to read the same frame of sound effect processing results from the second cache units corresponding to each track for mixing. If the second cache unit corresponding to any track is empty, it means that the second cache unit corresponding to this track does not contain sound effect processing results corresponding to the frame to be read, while the second cache units corresponding to other tracks contain sound effect processing results corresponding to the frame to be read. In this case, the preview thread is blocked, that is, the preview thread is suspended from reading the sound effect processing results from the second cache units corresponding to each track for mixing until the second cache units corresponding to each track are not empty. At this time, the preview thread continues to read the sound effect processing results from the second cache units corresponding to each track for mixing.

[0064] Optionally, the above process can be implemented by using a semaphore, wherein a semaphore is used to represent the sound effect processing results that can be read in the second cache unit corresponding to each track. If the semaphore of any track is 0, the second cache unit corresponding to the track is empty, and the preview thread is blocked.

[0065] Furthermore, in a scenario where a third cache unit is created, the preview thread is used to read the sound effect processing results corresponding to the same frame from the second cache units corresponding to each track for mixing, and the mixed results are cached in the third cache unit, and the mixed results corresponding to the current frame are read from the third cache unit for playback, if the second cache units corresponding to each track are not empty and the third cache unit is not full, the preview thread is continued to read the sound effect processing results from the second cache units corresponding to each track for mixing; or

[0066] If the second cache unit corresponding to any track is empty, and / or the third cache unit is full, the preview thread is paused to read the sound effect processing results from the second cache units corresponding to each track for mixing.

[0067] In this embodiment, similar to the above embodiment, if the third cache unit is full, the preview thread will be blocked, and the preview thread will be suspended from reading the sound effect processing results from the second cache unit corresponding to each track for mixing, and the mixing results will no longer be written to the third storage unit.

[0068] Optionally, another semaphore may be used to represent the number of storage locations in the third cache unit. When the semaphore is 0, that is, the third cache unit is full, the preview thread is blocked.

[0069] In the above embodiment, the preview thread continuously consumes the second cache unit corresponding to each track to read the sound effect processing results, thereby driving the audio decoding thread and the sound effect processing thread, realizing self-driving of audio processing during the preview process, and the overall flexibility is very high; and by adjusting the size of the cache unit, the audio decoding and sound effect processing performance limits can be significantly improved.

[0070] It should be noted that the blocking of the audio decoding thread of any track does not affect the progress of the audio decoding threads of other tracks, and the blocking of the sound effect processing thread of any track does not affect the sound effect processing threads of other tracks. By running and blocking the preview thread, it can be ensured that the sound effect processing results corresponding to the same frame can be read from the second cache unit corresponding to each track and mixed, thereby realizing the driving of the audio decoding threads and sound effect processing threads of multiple tracks.

[0071] Corresponding to the audio processing method described in the preceding embodiments, FIG5 is a block diagram of the structure of an audio processing device provided by an embodiment of the present disclosure. For ease of illustration, only the portions relevant to the embodiment of the present disclosure are shown. Referring to FIG5 , the audio processing device 500 includes a decoding unit 501, an audio effect processing unit 502, and a preview unit 503.

[0072] The decoding unit 501 is configured to, during the preview of the video editing draft, use the audio decoding thread corresponding to each track of the video editing draft to respectively decode the audio of each track, and cache the decoded data in the first cache unit corresponding to each track;

[0073] The sound effect processing unit 502 is configured to use the sound effect processing thread corresponding to each track to read the decoded data from the corresponding first buffer unit for sound effect processing, and cache the sound effect processing results in the second buffer unit corresponding to each track;

[0074] The preview unit 503 is configured to use a preview thread to read the sound effect processing results from the second cache unit corresponding to each track, mix them, and play the mixed results.

[0075] In one or more embodiments of the present disclosure, when the decoding unit 501 uses the audio decoding thread corresponding to each track of the video editing draft to perform audio decoding on each track respectively, it is further configured to:

[0076] If the first cache unit corresponding to any track is not full, continue to use the audio decoding thread corresponding to the track to perform audio decoding on the track; or

[0077] If the first cache unit corresponding to any track is full, the audio decoding thread corresponding to the track is suspended from performing audio decoding on the track.

[0078] In one or more embodiments of the present disclosure, when the sound effect processing unit 502 uses the sound effect processing thread corresponding to each track to read the decoded data from the corresponding first cache unit for sound effect processing, it is further configured to:

[0079] If the first cache unit corresponding to any track is not empty and the second cache unit corresponding to the track is not full, then continue to use the sound effect processing thread corresponding to the track to read the decoded data from the first cache unit corresponding to the track for sound effect processing; or

[0080] If the first cache unit corresponding to any track is empty and / or the second cache unit corresponding to the track is full, the audio processing thread corresponding to the track is suspended to read decoded data from the first cache unit corresponding to the track for audio processing.

[0081] In one or more embodiments of the present disclosure, when a preview thread is used to read the sound effect processing result from the second cache unit corresponding to each track for mixing, the system is further configured to:

[0082] If the second cache units corresponding to the various tracks are not empty, then continue to use the preview thread to read the sound effect processing results from the second cache units corresponding to the various tracks for mixing; or

[0083] If the second cache unit corresponding to any track is empty, the preview thread is paused to read the sound effect processing results from the second cache units corresponding to each track for mixing.

[0084] In one or more embodiments of the present disclosure, when the preview unit 503 uses the preview thread to read the sound effect processing results from the second cache unit corresponding to each track for mixing and playing the mixed results, it is further configured to:

[0085] The preview thread is used to read the sound effect processing result corresponding to the current frame from the second cache unit corresponding to each track, mix the sound effect processing result, and play the mixed result.

[0086] In one or more embodiments of the present disclosure, when the preview unit 503 uses the preview thread to read the sound effect processing results from the second cache unit corresponding to each track for mixing and playing the mixed results, it is further configured to:

[0087] The preview thread is used to read the sound effect processing results corresponding to the same frame from the second cache unit corresponding to each track for mixing, and the mixed results are cached in the third cache unit, and the mixed results corresponding to the current frame are read from the third cache unit for playback.

[0088] In one or more embodiments of the present disclosure, each cache unit is a first-in-first-out cache unit.

[0089] In one or more embodiments of the present disclosure, each cache unit is a first-in-first-out ring buffer.

[0090] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0091] Referring to FIG6 , a schematic diagram of the structure of an electronic device 600 suitable for implementing an embodiment of the present disclosure is shown. The electronic device 600 may be a terminal device or a server. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG6 is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

[0092] As shown in FIG6 , the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0093] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 6 shows an electronic device 600 having various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0094] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0095] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0096] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0097] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0098] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0099] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0100] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0101] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0102] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0103] In a first aspect, according to one or more embodiments of the present disclosure, there is provided an audio processing method, comprising:

[0104] During the preview of the video editing draft, the audio decoding thread corresponding to each track of the video editing draft is used to respectively decode the audio of each track, and the decoded data is respectively cached in the first cache unit corresponding to each track;

[0105] Using the sound effect processing thread corresponding to each track to read the decoded data from the corresponding first cache unit for sound effect processing, and cache the sound effect processing results in the second cache unit corresponding to each track;

[0106] The preview thread is used to read the sound effect processing results from the second cache unit corresponding to each track, mix them, and play the mixed results.

[0107] According to one or more embodiments of the present disclosure, the audio decoding thread corresponding to each track of the video editing draft is used to respectively decode the audio of each track, further comprising:

[0108] If the first cache unit corresponding to any track is not full, continue to use the audio decoding thread corresponding to the track to perform audio decoding on the track; or

[0109] If the first cache unit corresponding to any track is full, the audio decoding thread corresponding to the track is suspended from performing audio decoding on the track.

[0110] According to one or more embodiments of the present disclosure, the audio effect processing thread corresponding to each track is used to read decoded data from the corresponding first cache unit for audio effect processing, further comprising:

[0111] If the first cache unit corresponding to any track is not empty and the second cache unit corresponding to the track is not full, then continue to use the sound effect processing thread corresponding to the track to read the decoded data from the first cache unit corresponding to the track for sound effect processing; or

[0112] If the first cache unit corresponding to any track is empty and / or the second cache unit corresponding to the track is full, the audio processing thread corresponding to the track is suspended to read decoded data from the first cache unit corresponding to the track for audio processing.

[0113] According to one or more embodiments of the present disclosure, a preview thread is used to read the sound effect processing result from the second cache unit corresponding to each track for mixing, and further includes:

[0114] If the second cache units corresponding to the various tracks are not empty, then continue to use the preview thread to read the sound effect processing results from the second cache units corresponding to the various tracks for mixing; or

[0115] If the second cache unit corresponding to any track is empty, the preview thread is paused to read the sound effect processing results from the second cache units corresponding to each track for mixing.

[0116] According to one or more embodiments of the present disclosure, a preview thread is used to read the sound effect processing results from the second cache unit corresponding to each track for mixing, and the mixed results are played, including:

[0117] The preview thread is used to read the sound effect processing result corresponding to the current frame from the second cache unit corresponding to each track, mix the sound effect processing result, and play the mixed result.

[0118] According to one or more embodiments of the present disclosure, a preview thread is used to read the sound effect processing results from the second cache unit corresponding to each track for mixing, and the mixed results are played, including:

[0119] The preview thread is used to read the sound effect processing results corresponding to the same frame from the second cache unit corresponding to each track for mixing, and the mixed results are cached in the third cache unit, and the mixed results corresponding to the current frame are read from the third cache unit for playback.

[0120] According to one or more embodiments of the present disclosure, each cache unit is a first-in-first-out cache unit.

[0121] According to one or more embodiments of the present disclosure, each cache unit is a first-in-first-out ring buffer.

[0122] In a second aspect, according to one or more embodiments of the present disclosure, there is provided an audio processing device, including:

[0123] a decoding unit configured to, during a preview of the video editing draft, decode the audio of each track in the video editing draft using an audio decoding thread corresponding to each track in the video editing draft, and cache the decoded data in a first cache unit corresponding to each track;

[0124] The sound effect processing unit is configured to use the sound effect processing thread corresponding to each track to read the decoded data from the corresponding first buffer unit for sound effect processing, and cache the sound effect processing results in the second buffer unit corresponding to each track;

[0125] The preview unit is used to read the sound effect processing results from the second cache unit corresponding to each track using a preview thread, mix them, and play the mixed results.

[0126] According to one or more embodiments of the present disclosure, when the decoding unit uses the audio decoding thread corresponding to each track of the video editing draft to respectively decode the audio of each track, it is further configured to:

[0127] If the first cache unit corresponding to any track is not full, continue to use the audio decoding thread corresponding to the track to perform audio decoding on the track; or

[0128] If the first cache unit corresponding to any track is full, the audio decoding thread corresponding to the track is suspended from performing audio decoding on the track.

[0129] According to one or more embodiments of the present disclosure, when the sound effect processing unit uses the sound effect processing thread corresponding to each track to read the decoded data from the corresponding first cache unit for sound effect processing, it is further configured to:

[0130] If the first cache unit corresponding to any track is not empty and the second cache unit corresponding to the track is not full, then continue to use the sound effect processing thread corresponding to the track to read the decoded data from the first cache unit corresponding to the track for sound effect processing; or

[0131] If the first cache unit corresponding to any track is empty and / or the second cache unit corresponding to the track is full, the audio processing thread corresponding to the track is suspended to read decoded data from the first cache unit corresponding to the track for audio processing.

[0132] According to one or more embodiments of the present disclosure, when a preview thread is used to read the sound effect processing results from the second cache unit corresponding to each track for mixing, it is also used to:

[0133] If the second cache units corresponding to the various tracks are not empty, then continue to use the preview thread to read the sound effect processing results from the second cache units corresponding to the various tracks for mixing; or

[0134] If the second cache unit corresponding to any track is empty, the preview thread is paused to read the sound effect processing results from the second cache units corresponding to each track for mixing.

[0135] According to one or more embodiments of the present disclosure, when the preview unit uses the preview thread to read the sound effect processing results from the second cache unit corresponding to each track for mixing and playing the mixed results, it is further configured to:

[0136] The preview thread is used to read the sound effect processing result corresponding to the current frame from the second cache unit corresponding to each track, mix the sound effect processing result, and play the mixed result.

[0137] According to one or more embodiments of the present disclosure, when the preview unit uses the preview thread to read the sound effect processing results from the second cache unit corresponding to each track for mixing and playing the mixed results, it is further configured to:

[0138] The preview thread is used to read the sound effect processing results corresponding to the same frame from the second cache unit corresponding to each track for mixing, and the mixed results are cached in the third cache unit, and the mixed results corresponding to the current frame are read from the third cache unit for playback.

[0139] According to one or more embodiments of the present disclosure, each cache unit is a first-in-first-out cache unit.

[0140] According to one or more embodiments of the present disclosure, each cache unit is a first-in-first-out ring buffer.

[0141] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0142] Memory stores computer-executable instructions;

[0143] At least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the audio processing method of the first aspect and various possible designs of the first aspect.

[0144] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the audio processing method as described in the first aspect and various possible designs of the first aspect are implemented.

[0145] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, comprising computer execution instructions. When a processor executes the computer execution instructions, the audio processing method of the first aspect and various possible designs of the first aspect are implemented.

[0146] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0147] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0148] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. An audio processing method, comprising: During the preview of the video editing draft, the audio decoding thread corresponding to each track of the video editing draft is used to respectively decode the audio of each track, and the decoded data is cached in the first cache unit corresponding to each track; Using the sound effect processing thread corresponding to each track to read the decoded data from the corresponding first cache unit to perform sound effect processing, and cache the sound effect processing results in the second cache unit corresponding to each track; The preview thread is used to read the sound effect processing results from the second cache unit corresponding to each track for mixing, and the mixed results are played.

2. The method according to claim 1, wherein: The step of using the audio decoding thread corresponding to each track of the video editing draft to respectively decode the audio of each track also includes: If the first cache unit corresponding to any track is not full, continue to use the audio decoding thread corresponding to the track to perform audio decoding on the track; or If the first cache unit corresponding to any track is full, the audio decoding thread corresponding to the track is suspended from performing audio decoding on the track.

3. The method according to claim 1, wherein: The method of using the sound effect processing thread corresponding to each track to read the decoded data from the corresponding first cache unit to perform sound effect processing also includes: If the first cache unit corresponding to any track is not empty and the second cache unit corresponding to the track is not full, then continue to use the sound effect processing thread corresponding to the track to read the decoded data from the first cache unit corresponding to the track for sound effect processing; or If the first cache unit corresponding to any track is empty, and / or the second cache unit corresponding to the track is full, the audio processing thread corresponding to the track is suspended to read the decoded data from the first cache unit corresponding to the track for audio processing.

4. The method according to claim 1, wherein: The method of using the preview thread to read the sound effect processing result from the second cache unit corresponding to each track for mixing also includes: If the second cache units corresponding to the tracks are not empty, the preview thread continues to read the sound effect processing results from the second cache units corresponding to the tracks for mixing; or If the second cache unit corresponding to any track is empty, the preview thread is paused to read the sound effect processing results from the second cache units corresponding to each track for mixing.

5. The method according to claim 1, wherein: The preview thread is used to read the sound effect processing results from the second cache unit corresponding to each track for mixing, and the mixed results are played, including: The preview thread is used to read the sound effect processing result corresponding to the current frame from the second cache unit corresponding to each track for mixing, and the mixed result is played.

6. The method according to claim 1, wherein: The preview thread is used to read the sound effect processing results from the second cache unit corresponding to each track for mixing, and the mixed results are played, including: The preview thread is used to read the sound effect processing results corresponding to the same frame from the second cache unit corresponding to each track for mixing, and the mixed results are cached in the third cache unit, and the mixed results corresponding to the current frame are read from the third cache unit for playback.

7. The method according to any one of claims 1 to 6, wherein: Each cache unit is a first-in-first-out cache unit.

8. The method according to claim 7, wherein: Each cache unit is a first-in-first-out circular buffer.

9. An audio processing device, comprising: A decoding unit is configured to respectively perform audio decoding on each track of the video editing draft using an audio decoding thread corresponding to each track of the video editing draft during previewing the video editing draft, and cache the decoded data in a first cache unit corresponding to each track; The sound effect processing unit is configured to use the sound effect processing thread corresponding to each track to read the decoded data from the corresponding first cache unit for sound effect processing, and cache the sound effect processing results respectively. to the second cache unit corresponding to each track; as well as The preview unit is configured to use a preview thread to read the sound effect processing results from the second cache unit corresponding to each track for mixing, and play the mixed results.

10. An electronic device, comprising: at least one processor and memory; Wherein, the memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising computer executable instructions, wherein: When the processor executes the computer-executable instructions, the method according to any one of claims 1 to 8 is implemented.

Images