Audio processing methods, apparatus, devices, storage media, and program products

Abstract

The present disclosure relates to an audio processing method, an apparatus, a device, a storage medium, and a program product, and the method includes: in a process of playing a first audio externally, collecting an external sound to obtain a second audio, determining a length of a playing time of the first audio based on a current system time and a playing time delay, calculating a reference timestamp based on the length of the playing time of the first audio and the collecting time delay, processing the second audio to obtain a third audio based on the reference timestamp, and mixing the first audio and the third audio to obtain a target audio. The technical solution provided in the present disclosure optimizes the effect of time delay on the audio in the process of mixing the first audio and the second audio, synchronizes the played first audio and the recorded second audio, reduces the effect of time delay on the hearing sensation of the audio, and improves the user's karaoke experience.

Description

Technical Field

[0001] This application claims the priority of a Chinese patent application filed on October 19, 2022, with the application number 202211280550.4 and the title "Audio Processing Method, Apparatus, Device, Storage Medium and Program Product". The entire content of the Chinese patent application is incorporated herein by reference.

[0002] The present disclosure relates to the field of audio processing technology, and in particular, to an audio processing method, apparatus, device, storage medium and program product.

Background Art

[0003] With the development of Internet technology, many music players can not only support online playback of a huge number of music pieces, but also provide karaoke services to users. Karaoke service means that the music player plays the accompaniment of a music piece, and the user sings along with the music accompaniment. While singing, the music player can also record the music piece sung by the user so that the user can play it later.

[0004] In related technologies, the audio recording process generally involves turning on the music player on the terminal to play the accompaniment of a music piece. When the user starts singing, the terminal starts recording synchronously to obtain the recorded audio. After the recording is completed, the terminal performs audio mixing based on the music accompaniment and the recorded audio to obtain the completed music piece.

[0005] However, there is a playback time delay when the terminal plays the music accompaniment, and there is a collection time delay in the process of recording audio. Therefore, a large time delay occurs between the music accompaniment and the recorded audio. In the process of mixing the music accompaniment and the recorded audio, due to the large delay, the music accompaniment and the recorded audio are not synchronized, which has a serious impact on the listening experience and deteriorates the karaoke user experience.

Summary of the Invention

[0006] To address the above technical challenges, this disclosure provides an audio processing method, apparatus, device, and storage medium that optimize the approach to recording time delay in background music scenes and solve the problem of excessively long recording time delays affecting listening quality.

[0007] In the first embodiment, the embodiments of the present disclosure provide an audio processing method, the method is In the process of playing the first audio externally, external sounds are collected to obtain the second audio, The length of the playback time of the first audio is determined based on the current system time and the playback time delay, which is the length of time from when the audio player was started to when the audio player outputted the audio. A reference timestamp is calculated based on the length of the playback time of the first audio and a collection time delay that includes the time from when the audio acquisition device was started until the time when the first audio frame of the second audio was acquired. Based on the aforementioned reference timestamp, the second audio is processed to obtain the third audio, This includes mixing the first audio and the third audio to obtain a target audio.

[0008] In a second embodiment, an embodiment of the present disclosure provides an audio processing device, the device is In the process of playing the first audio externally, a second audio acquisition module is used to collect external sound and obtain the second audio, A playback time length determination module for determining the playback time of the first audio based on the current system time and a playback time delay, which is the length of time from when the audio player was started to when the audio player outputted audio. A reference timestamp determination module for calculating a reference timestamp based on the length of the playback time of the first audio and a collection time delay which includes the length of time from the time the audio acquisition device was started to the time the first audio frame of the second audio was acquired, A third audio determination module for processing the second audio to obtain a third audio based on the aforementioned reference timestamp, The system includes a target audio determination module for mixing the first audio and the third audio to obtain a target audio.

[0009] In a third embodiment, the embodiments of the present disclosure provide an electronic device, said electronic device, One or more processors, A memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are made to perform the audio processing method described in any one of the first embodiments.

[0010] In a fourth embodiment, an embodiment of the present disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the audio processing method described in any one of the first embodiments.

[0011] In a fifth embodiment, an embodiment of the present disclosure provides a computer program product that, when executed by a processor, includes a computer program or instructions that implements the audio processing method described in any one of the first embodiments.

[0012] This disclosure relates to an audio processing method, apparatus, device, storage medium, and program product, wherein the method includes, in the process of playing back a first audio externally, collecting external sound to obtain a second audio; determining the length of the playback time of the first audio based on the current system time and a playback time delay which is the length of time from the time the audio player was started to the time the audio player outputted audio; calculating a reference timestamp based on the length of the playback time of the first audio and a collection time delay which includes the length of time from the time the audio collection device was started to the time the first audio frame of the second audio was collected; processing the second audio based on the reference timestamp to obtain a third audio; and mixing the first audio and the third audio to obtain a target audio. The proposed technique provided in this disclosure optimizes the effect of time delay on audio in the process of mixing the first audio and the second audio, synchronizes the played-back first audio and the recorded second audio, reduces the effect of time delay on the perceived sound of the audio, and improves the user's karaoke experience. [Brief explanation of the drawing]

[0013] The above and other features, advantages, and aspects of each embodiment of this disclosure will become more apparent by referring to the embodiments for carrying out the invention described below in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals indicate the same or similar elements. It should be understood that the accompanying drawings are illustrative and the originals and elements are not necessarily drawn to scale. [Figure 1] This is a schematic flowchart of the audio processing method in the embodiments of this disclosure. [Figure 2] This is a schematic diagram illustrating the details of the recording time delay in the embodiments of the present disclosure. [Figure 3] This is a schematic diagram of the alignment method for background music and recording acquisition in the embodiments of the present disclosure. [Figure 4]This is a schematic diagram of the configuration of an audio processing device in an embodiment of the present disclosure. [Figure 5] This is a schematic diagram of the configuration of an electronic device in an embodiment of the present disclosure. [Modes for carrying out the invention]

[0014] The embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While the accompanying drawings show several embodiments of this disclosure, it should be understood that this disclosure can be realized in various forms and should not be construed as being limited to the embodiments described herein. Rather, these embodiments are provided to allow for a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should be understood that the steps described in the embodiments relating to the method of this disclosure may be performed in a different order and / or in parallel. Furthermore, embodiments relating to the method may include additional steps and / or omit the performance of the indicated steps. The scope of this disclosure is not limited in this respect.

[0016] As used herein, the term “including” and its variations mean “including, but not limited to.” The term “based on” means “based at least in part.” The term “one embodiment” means “at least one embodiment,” the term “another embodiment” means “at least one other embodiment,” and the term “several embodiments” means “at least several embodiments.” Related definitions of other terms are provided in the following description.

[0017] It should be noted that the concepts such as “First,” “Second,” etc., as used in this disclosure are used solely to distinguish different devices, modules, or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules, or units.

[0018] It should be noted that the modifiers "one" and "a plurality" referred to in this disclosure are exemplary rather than limiting, and those skilled in the art should understand that, unless otherwise explicitly stated in the context, it should be understood as "one or a plurality".

[0019] In addition, the names of the messages or information for interaction between multiple devices in the embodiments of this disclosure are only used for the purpose of explanation, and are not intended to limit the scope of these messages or information.

[0020] Hereinafter, with reference to the accompanying drawings, the audio processing method provided by the embodiments of this application will be described in detail.

[0021] FIG. 1 is a flowchart of an audio processing method in an embodiment of this disclosure. This embodiment is applicable to a situation where the influence of time delay is reduced during mixing processing. This method can be implemented in the form of software and / or hardware and can be executed by an audio processing device arranged in an electronic device. The electronic device includes a smart terminal with a photo-taking function, including a smartphone, a notebook computer, a tablet computer, a digital camera / video camera, a game device, etc. Optionally, the smart terminal includes a touch screen.

[0022] As described in FIG. 1, the audio processing method provided by the embodiments of this disclosure mainly includes the following steps: S101: In the process of playing the first audio externally, collect external sounds to obtain a second audio.

[0023] In some specific scenarios and applications, it's necessary to mix two types of audio to achieve a particular effect. For example, in short video recording, background music is played to create a certain atmosphere, and then the user's recording device is turned on to collect audio of the user reading aloud, thus achieving the desired effect. Alternatively, in a karaoke application, the song accompaniment is played, the user's singing audio is collected, and finally, the song accompaniment and singing audio are combined to complete a single song.

[0024] Specifically, there is a playback time delay between the time the user terminal receives an audio playback command and the time the user terminal's audio playback device outputs audio, and there is also a collection time delay between the time the user terminal receives an audio recording command and the time the audio collection device collects the first audio frame.

[0025] In embodiments of this disclosure, the first audio may be understood as audio played by the speaker of a user terminal. Optionally, the first audio may refer to background music (BGM) or accompanying music. The second audio refers to audio formed by external sounds collected by the user terminal.

[0026] In one embodiment of the present disclosure, a first audio is played using an audio playback device of the user terminal, and a second audio is collected using an audio collection device of the user terminal, the audio collection device may be one or more of the following: a pickup, a microphone, etc.

[0027] In one specific embodiment of this disclosure, in a karaoke scene, in response to a user's trigger operation on the recording controller in the karaoke application, the audio playback device of the user terminal plays backing music, and the audio acquisition device of the user terminal collects external audio data to obtain a second audio. Alternatively, in a video recording scene, in response to a user's trigger operation on the recording controller in the video recording application, the audio playback device of the user terminal plays backing music, and the audio acquisition device of the user terminal collects external audio data to obtain a second audio.

[0028] S102: Determine the length of the playback time of the first audio based on the current system time and playback time delay, where the playback time delay is the length of time from when the audio player is started to when the audio player outputs the audio.

[0029] In the embodiments of this disclosure, the system time refers to the time when the user terminal's system is operational and displayed, and the current system time can be understood as the time of system operation acquired at the present moment. Here, the current system time refers to a specific point in time.

[0030] In the embodiments of this disclosure, the time delay parameters relating to this disclosure will first be described. As shown in Figure 2, the time delay T1 for the entire process from receiving an audio playback command to collecting audio mainly includes a playback time delay, a transmission time delay Tc, and a collection time delay, where the playback time delay includes a playback software time delay Ta and a playback hardware time delay Tb, and the collection time delay includes a collection software time delay Te and a collection hardware time delay Td. Since the transmission time delay Tc is very small, the effect of the transmission time delay Tc on the mixing of the two audios is not considered in the embodiments of this disclosure.

[0031] In embodiments of this disclosure, the playback time delay is the length of time from the time a playback command for an audio frame is received until the audio player outputs this audio frame. Here, receiving a playback command for an audio frame may mean automatically generating and receiving a playback command for the current audio frame after responding to a playback command for a previous audio frame. For example, after responding to a playback command for the Nth audio frame, a playback command for the N+1th audio frame may be automatically generated and received. Receiving a playback command for an audio frame further includes responding to an audio player activation command and then receiving a playback command for the first audio frame. Here, an audio player activation command is entered by the user via a user terminal, for example, by pressing a physical button set on the user terminal or by touching an activation controller displayed on the user terminal interface.

[0032] In one embodiment of the present disclosure, in a karaoke application, when the user sets background music playback, the system detects the user's trigger operation to the recording controller and then receives a command to start the audio player.

[0033] In one embodiment of the present disclosure, the playback time delay is defined as the difference between the time the audio player outputs the first audio frame and the time the playback command for the first audio frame is received.

[0034] Here, the time when the audio player outputs the first audio frame may be determined by monitoring the state of the audio player using an audio monitoring device, and at the point when it is confirmed that the audio player has output the first audio frame in the first audio, the system time at that point is taken as the time when the audio player outputs the first audio frame.

[0035] In one embodiment of the present disclosure, the playback time delay includes a playback software time delay and a playback hardware time delay, wherein the playback software time delay is one buffer write cycle of the audio player, and the playback hardware time delay is determined by a system time delay function.

[0036] Specifically, the playback software time delay Ta is the length of time it takes for an audio player to receive an audio frame and write this audio frame to the playback buffer. Here, the audio frame in the playback buffer is used to send to the playback hardware device for playback. The playback software time delay Ta is one buffer write cycle of the audio player, and the playback hardware time delay Tb is the length of time from when the audio player's hardware device receives an audio signal until when the audio player's hardware device outputs audio. The playback hardware time delay Tb may be obtained by evaluating the hardware time delay function of the audio system. For example, the algorithm is "AudioSystem:getOutputLatency", but it should be noted that the algorithms provided in this disclosure are merely illustrative and not particularly limited to specific algorithms.

[0037] Specifically, a buffer reserves a certain amount of storage space in the device's memory space, and this storage space is used to buffer data input or output. The purpose of a buffer is to reduce the number of read / write operations to the disk, and since computer operations on the buffer are much faster than operations on the disk, it can improve the computer's operating speed. The playback software time delay can be understood as one buffer write cycle of the audio player, and the time consumed by one computer write operation to the audio player's buffer during the stable operation of the audio playback device.

[0038] In one embodiment of the present disclosure, determining the length of the playback time of the first audio based on the current system time and playback time delay includes calculating a first difference value between the current system time and the time when the audio player stabilizes, and setting a second difference value between the first difference value and the playback time delay as the length of the playback time of the first audio.

[0039] In the embodiments of this disclosure, calculating the first difference value between the current system time and the time when the audio player stabilizes is equivalent to statistically determining the length of time from the time when the audio player stabilizes to the current system time. Specifically, at the time when the audio player stabilizes, the timer starts counting from 0, obtains the time corresponding to the current system time, and takes the length of the timer's counting time as the first difference value. Alternatively, the first difference value is obtained by subtracting the time when the audio player stabilizes from the current system time.

[0040] Furthermore, the second difference value between the first difference value and the playback time delay is defined as the length of the playback time of the first audio. Here, the time at which the audio player stabilizes may be understood as the time at which the audio player stabilizes after being started up and going through a cold start.

[0041] In one embodiment of the present disclosure, the length of time between the current system time and the time when the audio player outputs the first audio frame in the first audio is defined as the length of the first playback time. Specifically, at the time when the audio player outputs the first audio frame, a timer starts counting from zero to obtain the time corresponding to the current system time, and at this time, the length of the timer's counting time is defined as the length of the playback time of the first audio.

[0042] In one embodiment of the present disclosure, in a practical application, since mixing is an online real-time processing process, the current system time may be understood as the time when the system collects the current audio frame in the second audio frame.

[0043] S103: A reference timestamp is calculated based on the playback time of the first audio and the acquisition time delay, where the acquisition time delay includes the time from when the audio acquisition device is started up to when the first audio frame of the second audio is acquired.

[0044] In one embodiment of the present disclosure, the acquisition time delay includes an acquisition software time delay and an acquisition hardware time delay, wherein the acquisition software time delay is the write cycle to the buffer of the audio acquisition device, and the acquisition hardware time delay is determined by a timestamp corresponding to an audio frame included in the second audio.

[0045] Specifically, the acquisition software time delay Te is the length of time from when an audio signal is acquired until it is recorded and stored, and is approximately the buffer write cycle of the audio acquisition device, while the acquisition hardware time delay Td can be understood as the length of time for the audio acquisition device hardware to start up and stabilize, and is obtained by the time extraction function in this disclosure. The acquisition time delay is the sum of the acquisition software time delay Te and the acquisition hardware time delay Td.

[0046] Here, the acquisition software time delay may be understood as the buffer write cycle of the audio acquisition device, and the time consumed by a computer writing to the audio acquisition device's buffer once during the stable operation of the audio acquisition device. The acquisition hardware time delay mainly refers to the length of time from when a hardware device in the audio acquisition device monitors an audio signal until it processes that audio signal to obtain an audio signal that satisfies the write conditions. In the embodiments of this disclosure, this is obtained by a system algorithm, for example, an algorithm called "AudioRecord#getTimestamp," and the algorithms provided in this disclosure are merely illustrative and do not impose any specific limitations on the particular algorithm.

[0047] In one embodiment of the present disclosure, calculating a reference timestamp based on the playback time of the first audio and the collection time delay as described above includes calculating a second difference value between the playback time of the first audio and the collection time delay, and setting the second difference value as the reference timestamp.

[0048] Here, the reference timestamp can be understood as a timestamp that has reference significance when mixing the collected audio.

[0049] In one embodiment of the present disclosure, calculating a reference timestamp based on the playback time of the first audio and the collection time delay as described above includes calculating a second difference value between the playback time of the first audio and the collection time delay, and setting the second difference value as the reference timestamp.

[0050] In the embodiments of this disclosure, the acquisition time delay is subtracted from the length of the playback time of the first audio calculated in the above step to obtain a point in time to be used as a reference timestamp.

[0051] In the embodiments of this disclosure, the impact of the collection time delay on the entire time delay process is optimized by calculating a reference timestamp based on the collection time delay.

[0052] S104: Based on the reference timestamp, the second audio is processed to obtain the third audio.

[0053] In the embodiments of this disclosure, the reference timestamp is data that has reference significance for processing the second audio calculated by the above embodiment.

[0054] In embodiments of the present disclosure, the second audio, which collects external sounds, includes one or more audio frames, and each audio frame is processed in the manner of steps S102 and S103 to obtain a correspondence between each audio frame and a reference timestamp.

[0055] In one embodiment of the present disclosure, processing the second audio based on the aforementioned reference timestamp to obtain a third audio includes determining the correspondence between the reference timestamp and the audio frame, performing packet discard processing on the audio frame corresponding to the reference timestamp if the reference timestamp is less than zero, and designating the audio frame corresponding to the zero or greater reference timestamp as the third audio frame if the reference timestamp is zero or greater.

[0056] In one embodiment of the present disclosure, based on the correspondence between the calculated audio frame and the reference timestamp, if the reference timestamp is less than zero, packet discard processing is performed for the corresponding audio frame, i.e., mixing processing is not performed.

[0057] Specifically, based on the correspondence between the calculated audio frames and the reference timestamp, if the reference timestamp is zero or greater, the audio frame corresponding to a reference timestamp of zero or greater is designated as the third audio frame. Specifically, if the reference timestamp is equal to zero, the corresponding audio frame is designated as the first audio frame of the third audio. If the reference timestamp is greater than zero, the corresponding audio frames are arranged sequentially as audio frames of the third audio in ascending order of reference timestamp.

[0058] S105: The first audio and the third audio are mixed to obtain the target audio.

[0059] Here, the target audio is the audio obtained by mixing the first audio and the third audio. Specifically, the first audio frame of the first audio and the first audio frame of the third audio are aligned, and then the subsequent audio is mixed to obtain the target audio.

[0060] In one specific embodiment of the present disclosure, as shown in Figure 3, when the first audio is played, there is a cold start process after the audio player receives a start command, so that the audio player can reach a stable playback state, and as shown in Figure 3, when the audio player reaches a stable playback state, the time of the system clock is marked as the stable time T1 of the audio player, and the time when the hardware device of the audio player (e.g., horn, speaker, etc.) outputs the first audio frame is marked as T2. Here, the acquisition player, after the cold start, starts acquiring external sound and acquires the second audio, where the system clock corresponding to the first audio frame of the second audio is T0.

[0061] In the embodiments of this disclosure, the timer starts counting from 0 at time T2 when the audio player outputs the first audio frame, and the length of the timer's counting time Tplay is the length of the playback time of the first audio. Tplay=T-T2=T-T1-Tplay_latency (1) Here, T is the current system time, and Tplay_latency is the playback time delay.

[0062] In the embodiments of this disclosure, the time at which the audio acquisition device acquires an audio frame in the second audio is time Tsys_capture. Tsys_capture=T-Tcapture_latency (2) Here, Tcapture_latency is the data collection time delay.

[0063] The reference timestamp Tcapture can be obtained by jointly calculating equations (1) and (2) above. Tcapture=Tplay-Tcapture_latency (3)

[0064] Finally, the reference timestamp Tcapture is used as the timestamp of the second audio. If the reference timestamp Tcapture < 0, it is discarded and not encoded. If the reference timestamp is equal to 0, the corresponding audio frame is used as the first audio frame of the third audio, and it is aligned with the first audio frame of the first audio. Then, the first and third audio are encoded to obtain the target audio.

[0065] Figure 4 is a schematic diagram of the configuration of an audio processing device in an embodiment of the present disclosure, which is applicable to audio processing situations, and which can be implemented in software and / or hardware form and placed in an electronic device. The electronic device includes smart terminals having audio processing capabilities, such as smartphones, notebook computers, tablet computers, digital cameras / camcorders, and game devices. Optionally, the smart terminals are equipped with touchscreens.

[0066] As illustrated in Figure 4, the audio processing device 40 provided by the embodiment of the present disclosure mainly comprises a second audio acquisition module 41, a playback time length determination module 42, a reference timestamp determination module 43, a third audio determination module 44, and a target audio determination module 45.

[0067] Here, the second audio acquisition module 41 is used to acquire the second audio by collecting external sounds during the process of playing the first audio externally.

[0068] The playback time length determination module 42 is used to determine the playback time of the first audio based on the current system time and the playback time delay, where the playback time delay is the length of time from when the audio player is started up to when the audio player outputs the audio.

[0069] The reference timestamp determination module 43 is used to calculate a reference timestamp based on the length of the playback time of the first audio and the collection time delay, where the collection time delay includes the length of time from when the audio acquisition device is started up to when the first audio frame of the second audio is acquired.

[0070] The third audio determination module 44 is used to process the second audio based on the reference timestamp to obtain the third audio.

[0071] The target audio determination module 45 is used to mix the first audio and the third audio to obtain the target audio.

[0072] Embodiments of this disclosure provide an audio processing device for performing the following steps: obtaining a second audio by collecting external sound in the process of playing a first audio externally; determining the length of the playback time of the first audio based on the current system time and a playback time delay which is the length of time from the time the audio player was started to the time the audio player outputted audio; calculating a reference timestamp based on the length of the playback time of the first audio and a collection time delay which includes the length of time from the time the audio collection device was started to the time the first audio frame of the second audio was collected; processing the second audio based on the reference timestamp to obtain a third audio; and mixing the first audio and the third audio to obtain a target audio. The proposed technology provided in this disclosure optimizes the effect of time delay on audio in the process of mixing the first audio and the second audio, synchronizes the played first audio and the recorded second audio, reduces the effect of time delay on the perceived audio, and improves the user's karaoke experience.

[0073] In one possible embodiment, the playback time delay includes a playback software time delay and a playback hardware time delay, wherein the playback software time delay is the buffer write cycle of the audio player, and the playback hardware time delay is determined by a system time delay function.

[0074] In one possible embodiment, the acquisition time delay includes an acquisition software time delay and an acquisition hardware time delay, wherein the acquisition software time delay is the write cycle of the audio acquisition device's buffer, and the acquisition hardware time delay is determined by a timestamp corresponding to an audio frame included in the second audio.

[0075] In one possible embodiment, the second audio includes at least one or more audio frames, and the third audio determination module 44 includes a correspondence determination unit for determining the correspondence between the reference timestamp and the audio frame, a first processing unit for packet discarding the audio frame corresponding to the reference timestamp if the reference timestamp is less than zero, and a second processing unit for making the audio frame corresponding to the zero or greater reference timestamp the third audio if the reference timestamp is zero or greater.

[0076] In one possible embodiment, the playback time length determination module 42 includes a first difference value calculation unit for calculating a first difference value between the current system time and the stable time of the audio player, and a second difference value calculation unit for determining the playback time length of the first audio by determining a second difference value between the first difference value and the playback time delay.

[0077] In one possible embodiment, the reference timestamp determination module 43 specifically calculates a second difference value between the length of the playback time of the first audio and the collection time delay, and uses the second difference value as the reference timestamp.

[0078] The audio processing apparatus provided in the embodiments of this disclosure can perform the steps performed in the audio processing method provided in the embodiments relating to the method of this disclosure, and a description of the specific steps and beneficial effects is not repeated here.

[0079] Figure 5 is a schematic diagram of the configuration of an electronic device in an embodiment of the present disclosure. Referring specifically to Figure 5 below, Figure 5 shows a schematic diagram of a configuration suitable for implementing the electronic device 500 in an embodiment of the present disclosure. The electronic device 500 in an embodiment of the present disclosure includes, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (tablet computers), PMPs (Portable Multimedia Players), in-vehicle terminals (e.g., vehicle navigation terminals), and wearable terminal devices, as well as fixed terminals such as digital televisions, desktop computers, and smart home devices. The electronic device shown in Figure 5 is merely an example and does not limit in any way the functions and scope of use of the embodiment of the present disclosure.

[0080] As shown in Figure 5, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 501 capable of performing various appropriate operations and processes based on a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503 in order to carry out the image rendering method of the embodiment described in this disclosure. The RAM 503 also stores various programs and data necessary for the operation of the terminal device 500. The processing unit 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0081] Typically, the I / O interface 505 may be connected to an input device 506, which may include, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, output device 507, which may include, for example, a liquid crystal display (LCD), speaker, vibrator, storage device 1008, which may include, for example, magnetic tape, hard disk, and communication device 509. The communication device 509 may enable the terminal device 500 to communicate with other devices wirelessly or via a wired connection to exchange data. Figure 5 shows a terminal device 500 with various devices, but it should be understood that it is not necessary to implement or include all of the devices shown. More or fewer devices may be implemented or included as alternatives.

[0082] In particular, according to embodiments of the present disclosure, the process described above with reference to the flowchart may be implemented as a computer software program. For example, an embodiment of the present disclosure has a computer program product which includes a computer program stored on a non-temporary computer-readable medium, and which includes program code for performing the method shown in the flowchart, and thus performs the page jump method described above. In such an embodiment, the computer program may be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing unit 501, it performs the functions described above as defined in the methods of embodiments of the present disclosure.

[0083] In this disclosure, the computer-readable medium described above may be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. The computer-readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage mediums include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, CD-ROMs, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that is used by or can be used in combination with an instruction execution system, apparatus, or device. On the other hand, in this disclosure, the computer-readable signal medium may include data signals propagated in the baseband or as part of a carrier carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium for transmitting, propagating, or transmitting programs used by or in combination with instruction execution systems, apparatus, or devices. Program code contained in a computer-readable medium can be transmitted using any suitable medium, including but not limited to wires, fiber optic cables, RF (radio frequency), or any suitable combination thereof.

[0084] In some embodiments, clients and servers may communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and may interconnect with digital data communications (e.g., communication networks) in any form or medium. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), internetworks (e.g., the Internet), and end-to-end networks (e.g., ad-hoc end-to-end networks), as well as networks currently known or future-developed.

[0085] The computer-readable medium described above may be included in the electronic device, or it may be a separate component and not be assembled into the electronic device.

[0086] The computer-readable medium described above stores one or more programs, and when one or more programs are executed by the terminal device, the terminal device is instructed to perform the following processes: to obtain a second audio by collecting external sounds during the process of playing a first audio externally; to determine the length of the playback time of the first audio based on the current system time and a playback time delay which is the length of time from when the audio player was started to when the audio player outputted audio; to calculate a reference timestamp based on the length of the playback time of the first audio and a collection time delay which includes the length of time from when the audio collection device was started to when the first audio frame of the second audio was collected; to obtain a third audio by processing the second audio based on the reference timestamp; and to obtain a target audio by mixing the first audio and the third audio.

[0087] If one or more of the above programs are selectively executed by this terminal device, the terminal device may also perform the other steps described in the above embodiment.

[0088] Computer program code for performing the operations of this disclosure may be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code may run entirely on a user computer, partially on a user computer, as a standalone software package, partially on a user computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it may be connected to an external computer (for example, connected via the Internet using an Internet service provider).

[0089] The flowcharts and block diagrams in the accompanying drawings illustrate architectures, functions, and operations that may be implemented according to the systems, methods, and computer program products relating to each embodiment of this disclosure. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or part of code, which contains one or more executable instructions for implementing a specified logical function. Note that in some alternative implementations, the functions associated with a box may occur in a different order than those associated with the accompanying drawings. For example, two consecutive boxes may actually be executed substantially in parallel, or in reverse order depending on the functions involved. Also note that each box in the block diagram and / or flowchart, and combinations of boxes in the block diagram and / or flowchart, may be implemented in a dedicated hardware-based system that performs a given function or operation, or in a combination of dedicated hardware and computer instructions.

[0090] The units described in the embodiments of this disclosure may be implemented by software or by hardware. Herein, the names of the units do not constitute an limitation to the units themselves in a given context.

[0091] In this specification, the functions described above may be performed, at least in part, by one or more hardware logic units. For example, non-limiting examples of usable and exemplary hardware logic units include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standards (ASSPs), systems-on-a-chip (SOCs), and composite programmable logic devices (CPLDs).

[0092] In the context of this disclosure, a machine-readable medium may be a tangible medium that contains or can store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium includes, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, CD-ROMs, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0093] According to one or more embodiments of the present disclosure, the present disclosure provides a video processing method which includes: obtaining a second audio by collecting external sound in the process of playing a first audio externally; determining the length of playback time of the first audio based on the current system time and a playback time delay which is the length of time from when the audio player is started to when the audio player outputs audio; calculating a reference timestamp based on the length of playback time of the first audio and a collection time delay which includes the length of time from when the audio collection device is started to when the first audio frame of the second audio is collected; processing the second audio based on the reference timestamp to obtain a third audio; and mixing the first audio and the third audio to obtain a target audio.

[0094] According to one or more embodiments of the present disclosure, the present disclosure provides an audio processing device comprising: a second audio acquisition module for obtaining a second audio by collecting external sound in the process of playing a first audio externally; a playback time length determination module for determining the length of the playback time of the first audio based on the current system time and a playback time delay which is the length of time from the time the audio player was started to the time the audio player outputted audio; a reference timestamp determination module for calculating a reference timestamp based on the length of the playback time of the first audio and a collection time delay which includes the length of time from the time the audio acquisition device was started to the time the first audio frame of the second audio was collected; a third audio determination module for processing the second audio based on the reference timestamp to obtain a third audio; and a target audio determination module for mixing the first audio and the third audio to obtain a target audio.

[0095] According to one or more embodiments of this disclosure, this disclosure provides electronic equipment, and this electronic equipment is One or more processors, A memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are instructed to perform the audio processing method described in any of the embodiments provided in this disclosure.

[0096] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements an audio processing method described in any of the embodiments provided in the present disclosure.

[0097] Embodiments of this disclosure also provide a computer program product that, when executed by a processor, includes a computer program or instructions that implement the audio processing method described above.

[0098] The above description is merely an explanation of preferred embodiments of the present disclosure and the technical principles employed. Those skilled in the art will understand that the scope of the disclosure is not limited to any technical solution formed by a specific combination of the above technical features, but also covers other technical solutions formed by any combination of the above technical features or equivalent features without departing from the disclosed concepts. For example, this includes technical solutions formed by replacing the above features with (but not limited to) similarly functional technical features disclosed in this disclosure.

[0099] Furthermore, although the operations are described using a specific order, these operations should not be interpreted as requiring that they be executed in a specific order or sequentially. Multitasking and parallel processing may be advantageous in certain environments. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of this disclosure. Some features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, each feature described in the context of a single embodiment can also be implemented in multiple embodiments, individually or in any appropriate subcombination.

[0100] While this subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter as defined in the attached claims is not necessarily limited to the specific features or behaviors described above. Rather, the specific features and behaviors described above are merely exemplary forms of implementing the claims.

Claims

1. In the process of playing the first audio externally, external sounds are collected to obtain the second audio, The length of the playback time of the first audio is determined based on the current system time and the playback time delay, which is the length of time from when the audio player was started to when the audio player outputted the audio. A reference timestamp is calculated based on the length of the playback time of the first audio and a collection time delay that includes the time from when the audio acquisition device was started until the time when the first audio frame of the second audio was acquired. Based on the aforementioned reference timestamp, the second audio is processed to obtain the third audio, The first audio and the third audio are mixed to obtain the target audio, Audio processing methods including [specific methods].

2. The method according to claim 1, wherein the playback time delay includes a playback software time delay and a playback hardware time delay, the playback software time delay being the buffer write period of the audio player, and the playback hardware time delay being determined by a system time delay function.

3. The method according to claim 1, wherein the acquisition time delay includes an acquisition software time delay and an acquisition hardware time delay, the acquisition software time delay being the write cycle of the buffer of the audio acquisition device, and the acquisition hardware time delay being the length of time from when a hardware device in the audio acquisition device monitors an audio signal until it obtains an audio signal that satisfies the write condition.

4. The second audio includes at least one or more audio frames, Processing the second audio based on the aforementioned reference timestamp to obtain the third audio is: To determine the correspondence between the aforementioned reference timestamp and the aforementioned audio frame, If the aforementioned reference timestamp is less than zero, the audio frame corresponding to the aforementioned reference timestamp is to be discarded. If the aforementioned reference timestamp is zero or greater, the audio frame corresponding to the zero or greater reference timestamp shall be designated as the third audio. The method according to claim 1, including the method described in claim 1.

5. Determining the length of the playback time of the first audio based on the current system time and playback time delay is: The first difference between the current system time and the time when the audio player has stabilized is calculated, The second difference between the first difference value and the playback time delay is set to be the length of the playback time of the first audio, The method according to claim 1, including the method described in claim 1.

6. Calculating a reference timestamp based on the playback time of the first audio and the collection time delay is: The second difference between the length of the playback time of the first audio and the acquisition time delay is calculated, The aforementioned second difference value is used as the reference timestamp, The method according to claim 1, including the method described in claim 1.

7. In the process of playing the first audio externally, a second audio acquisition module is used to collect external sound and obtain the second audio, A playback time length determination module for determining the playback time of the first audio based on the current system time and a playback time delay, which is the length of time from when the audio player is started to when the audio player outputs audio. A reference timestamp determination module for calculating a reference timestamp based on the length of the playback time of the first audio and a collection time delay which includes the length of time from the time the audio acquisition device was started to the time the first audio frame of the second audio was acquired, A third audio determination module for processing the second audio to obtain a third audio based on the aforementioned reference timestamp, A target audio determination module for mixing the first audio and the third audio to obtain a target audio, An audio processing unit equipped with the following features.

8. One or more processors, A memory for storing one or more programs, An electronic device that, when the one or more programs are executed by the one or more processors, causes the one or more processors to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 6.

10. A computer program product comprising a computer program or instruction that, when executed by a processor, implements the method described in any one of claims 1 to 6.

Images