Audio data processing method, system, device and non-transitory storage medium

Abstract

This application discloses an audio data processing method, system, apparatus, and non-volatile storage medium. The method includes: a master device generating and sending an initial packet sequence number to a slave device based on the playback progress information and a reference timestamp of a multimedia file played on the master device, wherein the reference timestamp indicates the start time of the multimedia file playback; acquiring audio data packets sent by the slave device to the master device, wherein the audio data packets carry a data packet sequence number, which is an incrementing sequence number generated by the slave device based on the initial packet sequence number; and processing the audio data packets according to the data packet sequence number, wherein the processing methods include discarding the audio data packets and writing the audio data packets into a target audio file according to the data packet sequence number. This application solves the technical problem of audio-visual asynchrony in the synthesized result caused by the inability to guarantee synchronization between the recorded audio and the played multimedia data progress in related technologies.

Description

Technical Field

[0001] This application relates to the field of electronic digital data processing, and more specifically, to an audio data processing method, system, apparatus, and non-volatile storage medium. Background Technology

[0002] In related technologies, when synthesizing audio data recorded from a slave device and multimedia data played on a master device, it is impossible to guarantee that the playback progress of the audio data recorded from the slave device and the multimedia data are aligned, resulting in audio-visual asynchrony in the synthesized result.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides an audio data processing method, system, apparatus, and non-volatile storage medium to at least solve the technical problem of audio-visual asynchrony in the synthesized result caused by the inability to guarantee the synchronization of recording and playback of multimedia data in related technologies.

[0005] According to one aspect of the embodiments of this application, an audio data processing method is provided, comprising: a master device generating and sending an initial packet sequence number to a slave device based on playback progress information and a reference timestamp of a multimedia file played on the master device, wherein the reference timestamp is used to indicate the start playback time of the multimedia file, and the playback progress information includes a reference playback progress of the multimedia file when the slave device accesses the master device; acquiring an audio data packet sent by the slave device to the master device, wherein the audio data packet carries a data packet sequence number, and the data packet sequence number is an incrementing sequence number generated by the slave device based on the initial packet sequence number; and processing the audio data packet according to the data packet sequence number, wherein the processing method of the audio data packet includes discarding the audio data packet and writing the audio data packet into a target audio file according to the data packet sequence number.

[0006] Optionally, processing audio data packets according to their sequence numbers includes: detecting the real-time playback progress of the multimedia file; if a change in the real-time playback progress is detected, resetting the data writing start point in the target audio file according to the changed playback progress; starting from the data writing start point, if it is determined that audio data packets will be written to the target audio file, writing the audio data packets to the target audio file according to their sequence numbers.

[0007] Optionally, processing the audio data packet based on the data packet sequence number includes: determining the reference packet sequence number corresponding to the reference timestamp; comparing the data packet sequence number and the reference packet sequence number; discarding the audio data packet if the comparison result shows that the data packet sequence number is less than the reference packet sequence number; determining whether the audio data packet is the first data packet sent by the device if the comparison result shows that the data packet sequence number is greater than the reference packet sequence number; determining the writing position of the audio data packet based on the difference between the data packet sequence number and the reference packet sequence number if the audio data packet is determined to be the first data packet; filling the space between the start position and the writing position of the target audio file with silence data corresponding to the audio data packet; and writing the audio data packet to the writing position after filling with silence data.

[0008] Optionally, the master device generates and sends an initial packet sequence number to the slave device based on the playback progress information and reference timestamp of the multimedia file played on the master device. This includes: when the reference playback progress indicates that the slave device is connected to the master device and the multimedia file is being played, determining the reference packet sequence number corresponding to the reference timestamp, and generating and sending an initial packet sequence number corresponding to the reference playback progress to the slave device based on the reference playback progress and the reference packet sequence number; when the reference playback progress indicates that the slave device is connected to the master device but the multimedia file is not being played, sending the reference packet sequence number as the initial packet sequence number to the slave device after the multimedia file starts playing.

[0009] Optionally, processing the audio data packet based on the data packet sequence number includes: determining the reference data packet corresponding to the audio data packet, wherein the reference data packet is the most recent data packet written to the target audio file before the audio data packet, and the audio data packet and the reference data packet correspond to the same slave device; determining the difference between the sequence number of the audio data packet and the sequence number of the reference data packet, and processing the audio data packet based on the difference.

[0010] Optionally, processing the audio data packet based on the difference includes: discarding the audio data packet if the difference is less than a preset difference; writing the audio data packet into the target audio file if the difference is equal to the preset difference; and padding the reference data packet with silence data according to the difference if the difference is greater than the preset difference, and writing the audio data packet after the silence data after padding the silence data.

[0011] Optionally, the method further includes: determining the file timing length of the target audio file; stopping the reception of audio data packets if the file timing length is greater than a preset timing length, wherein the preset timing length is a timing length determined based on the total playback duration of the multimedia file. According to another aspect of the embodiments of this application, an audio data processing system is also provided, including: a master device and at least one slave device, wherein the master device is configured to generate and send a reference timestamp to the slave device, wherein the reference timestamp corresponds to the start playback time of a multimedia file in the master device; acquiring audio data packets sent by the slave device to the master device, wherein the audio data packets carry a data packet sequence number, the data packet sequence number being an incrementing sequence number generated by the slave device based on the reference timestamp; processing the audio data packets according to the data packet sequence number, wherein the processing method for the audio data packets includes discarding the audio data packets and writing the audio data packets into a target audio file according to the data packet sequence number; and the slave device is configured to generate and send audio data packets to the master device, and generate a data packet sequence number for the audio data packets based on the time difference between the generation time of the audio data packets and the reference timestamp.

[0012] According to another aspect of the embodiments of this application, an audio data processing system is also provided, including a master device and at least one slave device. The master device is configured to generate and send an initial packet sequence number to the slave device based on playback progress information and a reference timestamp of a multimedia file played on the master device. The reference timestamp indicates the start time of playback of the multimedia file, and the playback progress information includes a reference playback progress of the multimedia file when the slave device accesses the master device. The master device also acquires audio data packets sent by the slave device, wherein the audio data packets carry a data packet sequence number, which is an incrementing sequence number generated by the slave device based on the initial packet sequence number. The master device processes the audio data packets according to the data packet sequence number, wherein the processing method includes discarding the audio data packets and writing the audio data packets into a target audio file according to the data packet sequence number. The slave device is configured to generate and send audio data packets to the master device, and generate a data packet sequence number for the audio data packets based on the time difference between the generation time of the audio data packets and the reference timestamp.

[0013] According to another aspect of the embodiments of this application, an audio data processing apparatus is also provided, applicable to a master device, comprising: a first processing module, configured to generate and send an initial packet sequence number to a slave device based on playback progress information and a reference timestamp of a multimedia file played in the master device, wherein the reference timestamp is used to indicate the start playback time of the multimedia file, and the playback progress information includes a reference playback progress of the multimedia file when the slave device accesses the master device; a second processing module, configured to acquire audio data packets sent by the slave device to the master device, wherein the audio data packets carry a data packet sequence number, and the data packet sequence number is an incrementing sequence number generated by the slave device based on the initial packet sequence number; and a third processing module, configured to process the audio data packets according to the data packet sequence number, wherein the processing method for processing the audio data packets includes discarding the audio data packets and writing the audio data packets into a target audio file according to the data packet sequence number.

[0014] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, wherein a program is stored in the non-volatile storage medium, and the program controls the device where the non-volatile storage medium is located to execute an audio data processing method when it runs.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of an audio data processing method.

[0016] In this embodiment, a master device generates and sends a reference timestamp to a slave device, wherein the reference timestamp corresponds to the start playback time of the multimedia file in the master device; audio data packets sent from the slave device to the master device are obtained, wherein the audio data packets carry a data packet sequence number, which is an incrementing sequence number generated by the slave device based on the reference timestamp; the audio data packets are processed according to the data packet sequence number, wherein the processing method of the audio data packets includes discarding the audio data packets and writing the audio data packets into the target audio file according to the data packet sequence number. The data packet sequence number reflects the time difference between the generation time of the audio data packet and the reference timestamp, thereby achieving the purpose of determining the playback progress of the multimedia file corresponding to the audio data packet based on the data packet sequence number. This achieves the technical effect of ensuring that the audio data packets written into the target audio file are synchronized with the playback progress of the multimedia file, and thus solves the technical problem of audio-visual asynchrony in the synthesis result caused by the inability to guarantee the synchronization of the recording audio and the playback multimedia data progress in related technologies. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a schematic diagram of the structure of a computer terminal (or mobile device) according to an embodiment of this application;

[0019] Figure 2 This is a flowchart illustrating an audio data processing method according to an embodiment of this application;

[0020] Figure 3 This is a flowchart illustrating an audio data packet processing procedure based on packet sequence number, according to an embodiment of this application.

[0021] Figure 4 This is a flowchart illustrating an audio data packet writing process according to an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the structure of an audio data processing system according to an embodiment of this application;

[0023] Figure 6 This is a schematic diagram illustrating a synchronization process between a master device and a slave device according to an embodiment of this application;

[0024] Figure 7 This is a schematic diagram of an audio data processing device provided according to an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] In applications such as real-time karaoke, online education, and multi-party video conferencing, it is often necessary to synthesize audio from multiple terminal devices with video played on a single main device. Related technologies typically employ simple buffer management or global timers for audio-visual alignment. Two common approaches are:

[0028] 1) When the master device starts playing a media file, it notifies the slave device to start collecting and sending audio data. The master device then writes the received audio data directly into the recording file.

[0029] 2) Configure high-precision clocks for both master and slave devices and align them using absolute timestamps.

[0030] However, the synchronization methods in related technologies have at least the following unresolved problems:

[0031] Network latency impact: Due to unavoidable network transmission delays, when audio data packets sent from the slave device arrive at the master device: the media may not have started playing yet. Once the media starts playing, the received recording data needs to be saved, resulting in the saving of early invalid audio data due to network transmission delays. When the recording file and the media are played back simultaneously, the overall recording file is lagging behind. Alternatively, the media may have already been playing for some time, causing the beginning of the recording to be missing. When the recording file and the media are played back simultaneously, the recording file is ahead.

[0032] Other issues with devices joining mid-play: Existing technologies struggle to gracefully handle scenarios where devices join mid-play of media. Restarting media playback would interrupt the experience; directly writing audio data to a new device would cause its recording start point to lag behind the media start point, resulting in desynchronization.

[0033] Hardware clock reliance: Solutions that rely on high synchronization of clocks between devices are costly to implement and difficult to guarantee clock accuracy for all consumer devices. They also introduce at least one network round-trip delay, which cannot meet the needs of scenarios with high real-time requirements.

[0034] Therefore, there is an urgent need for an audio and video synchronous recording technology that can overcome the above-mentioned shortcomings, achieve high precision and robustness, and support flexible device addition.

[0035] To address the aforementioned issues, this application provides relevant solutions, which are detailed below.

[0036] According to an embodiment of this application, an embodiment of an audio data processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0037] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing an audio data processing method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0038] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0039] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the audio data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned audio data processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0040] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0041] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0042] Under the above operating environment, embodiments of this application provide an audio data processing method, such as... Figure 2 As shown, the method includes the following steps:

[0043] Step S202: The master device generates and sends an initial packet sequence number to the slave device based on the playback progress information and reference timestamp of the multimedia file being played on the master device. The reference timestamp is used to indicate the start time of the multimedia file playback, and the playback progress information includes the reference playback progress of the multimedia file when the slave device accesses the master device.

[0044] In some embodiments of this application, to achieve synchronization between different devices, the master device can generate a media start timestamp (mvStartTimeStamp), also known as a base timestamp, when the first frame of a media file begins playback. This timestamp is based on the master device's system time. The master device can send the initial packet sequence number generated based on this base timestamp to the slave devices, or it can simultaneously send this timestamp to all slave devices intending to join the recording.

[0045] In some embodiments of this application, if the slave device requests access after the multimedia file has already started playing, the master device, in addition to sending a reference timestamp to the slave device, will also determine the packet sequence number corresponding to the access time based on the playback progress of the multimedia file corresponding to the slave device's access time and the reference timestamp. Subsequently, when generating audio data packets, the slave device can use the packet sequence number corresponding to the access time as the initial packet sequence number, and increment it to generate subsequent data packet sequence numbers.

[0046] Step S204: Obtain the audio data packet sent from the slave device to the master device, wherein the audio data packet carries a data packet sequence number, which is an incrementing sequence number generated by the slave device based on the initial packet sequence number;

[0047] Optionally, after acquiring audio data packets, the slave device adds an incrementally increasing sequence number (timestamp) to each audio data packet based on a reference timestamp, and then sends the audio data packet carrying the sequence number to the master device. Optionally, "based on a reference timestamp" here means generating the packet sequence number based on the reference timestamp and a preset sequence number generation rule. For example, determining the initial sequence number corresponding to the reference timestamp, and then generating the packet sequence number incrementally according to the generation rule. The generation rule can include the interval between generating a new packet sequence number, that is, the preset time interval between two adjacent packet sequence numbers. In this way, the playback progress of the multimedia file corresponding to the audio data packet can be determined by the packet sequence number, thereby achieving synchronization between the audio recorded by the slave device and the multimedia file played on the master device. The playback progress of the multimedia file can be the playback duration of the multimedia file. If the playback progress of the multimedia file has changed, such as from the tenth second to the twentieth second, the skipped ten seconds will also be included in the playback progress of the multimedia file.

[0048] As an optional implementation, for slave devices with a fixed audio duration for each transmission, the packet sequence number can be determined directly by incrementing the base timestamp according to a preset time interval. For slave devices with variable audio durations, the packet sequence number can be determined based on the time interval between the current data packet transmission time and the previous data packet transmission time, and this time interval can be sent to the master device. The master device can then verify the packet sequence number based on the time interval.

[0049] In some embodiments of this application, different slave devices receive different initial packet sequence numbers depending on the time they connect to the master device. Optionally, if the multimedia file is playing when the reference playback progress indicates that the slave device has connected to the master device, a reference packet sequence number corresponding to the reference timestamp is determined, and an initial packet sequence number corresponding to the reference playback progress is generated and sent to the slave device based on the reference playback progress and the reference packet sequence number; if the multimedia file is not playing when the reference playback progress indicates that the slave device has connected to the master device, the reference packet sequence number is sent to the slave device as the initial packet sequence number after the multimedia file starts playing.

[0050] In some embodiments of this application, if the slave device sends audio data packets to the master device before the master device starts playing multimedia files such as music videos, the sequence number of these audio data packets sent before the multimedia files are sent starts from 0 and increments. In this case, the reference packet sequence number corresponding to the reference timestamp will be greater than 0.

[0051] Step S206: Process the audio data packet according to the data packet sequence number. The processing methods for the audio data packet include discarding the audio data packet and writing the audio data packet into the target audio file according to the data packet sequence number.

[0052] In the technical solution provided in step S206, the step of processing audio data packets according to the data packet sequence number includes: detecting the real-time playback progress of the multimedia file; if a jump in the real-time playback progress is detected, resetting the data writing start point in the target audio file according to the changed playback progress; starting from the data writing start point, if it is determined that the audio data packet will be written to the target audio file, writing the audio data packet to the target audio file according to the audio data packet sequence number.

[0053] In some embodiments of this application, the master device can monitor state change events of the multimedia file player to determine whether the real-time playback progress of the multimedia file has changed. These state change events may include Seek operation completion events (such as the onSeekComplete event) and buffering end events (such as the onBufferingEnd event). Once these events are detected, the new real-time playback progress is immediately obtained, and the write pointer of the target audio file is forcibly reset to the file position corresponding to the new real-time playback progress. This mechanism can be implemented using the checkSeekTo() process. After detecting a Seek marker, this process calls the file.seek(mvPos) operation (or an equivalent file pointer setting function) and clears the Seek marker. Afterward, audio data packets received by the master device will be written starting from the new file position, thus achieving rapid resynchronization after a playback progress change.

[0054] As an optional implementation, the steps of processing audio data based on the data packet sequence number include: determining the reference packet sequence number corresponding to the reference timestamp; comparing the data packet sequence number and the reference packet sequence number; discarding the audio data packet if the comparison result shows that the data packet sequence number is less than the reference packet sequence number; determining whether the audio data packet is the first data packet sent by the device if the comparison result shows that the data packet sequence number is greater than the reference packet sequence number; and determining the writing position of the audio data packet based on the difference between the data packet sequence number and the reference packet sequence number, and writing the audio data packet to the target audio file according to the writing position.

[0055] In some embodiments of this application, the step of writing an audio data packet to a target audio file according to the write position includes: filling the space between the start position and the write position of the target audio file with silence data corresponding to the audio data packet; and after filling with silence data, writing the audio data packet to the write position.

[0056] Optionally, when the master device determines that an audio data packet is the first data packet sent by a newly joined slave device, it calculates the time difference interval = timestamp - mvStartTimeStamp. Then, at the current write position in the recording file, it inserts silence data of length interval. The silence data uses PCM all-zero samples with the same sampling rate, bit depth, and number of channels (e.g., 48kHz, 16bit, single channel) as the recording data.

[0057] In some embodiments of this application, the step of processing audio data packets based on data packet sequence numbers includes: determining a reference data packet corresponding to the audio data packet, wherein the reference data packet is the most recent data packet written to the target audio file before the audio data packet, and the audio data packet and the reference data packet correspond to the same slave device; determining the difference between the sequence number of the audio data packet and the sequence number of the reference data packet, and processing the audio data packet based on the difference.

[0058] As an optional implementation, the steps of processing audio data packets based on the difference include: discarding the audio data packet if the difference is less than a preset difference; writing the audio data packet into the target audio file if the difference is equal to the preset difference; and filling the reference data packet with silence data according to the difference if the difference is greater than the preset difference, and writing the audio data packet after the silence data after filling the silence data.

[0059] Optionally, assuming the sequence number of the audio data packet is timestamp and the sequence number of the reference data packet is mLastTimeStamp, the difference delta = timestamp - mLastTimeStamp. If delta <= 0, it is considered a duplicate or out-of-order packet and is discarded; if delta > 1 (e.g., greater than 2, indicating packet loss), it is compensated by filling in silence data of the length corresponding to (delta - 1), and then the audio data packet is written after the compensated silence data.

[0060] In some embodiments of this application, the method further includes: determining the file timing length of the target audio file; and stopping the reception of audio data packets if the file timing length is greater than a preset timing length, wherein the preset timing length is a timing length determined based on the total playback duration of the multimedia file.

[0061] In some embodiments of this application, the file timing length of the target audio file refers to the playback duration corresponding to the target audio file. If it is determined that the file timing length corresponding to the target audio file is greater than a preset timing length, it indicates that the audio data packets sent by the device are sent after the multimedia file playback is completed, and should not be written into the target audio file.

[0062] In some embodiments of this application, the following are also provided: Figure 3 The diagram illustrates the audio data packet processing flow based on packet sequence numbers. From... Figure 3 As can be seen from this, the process includes:

[0063] First frame pre-frame data filtering and judgment stage:

[0064] In this stage, it is necessary to determine whether the packet sequence number is less than the base packet sequence number corresponding to the base timestamp. If so ("Y" branch), it means that this packet is "expired" data that was sent before the multimedia file started playing or was delayed due to network latency, and the system will discard it (needWrite = false). This step ensures that data before the multimedia file starts playing is not written.

[0065] Silent data population decision phase:

[0066] If the packet sequence number is greater than or equal to the base packet sequence number ("N" branch), the system further determines whether this is the first write to the data stream and whether the packet sequence number is greater than the base packet sequence number. If both conditions are met ("Y" branch), the difference between the packet sequence number and the base packet sequence number is calculated (interval = packet sequence number - base packet sequence number). Subsequently, the system fills the current file write position with silence data (fillZeroData) of the same duration as the interval. This mechanism ensures that even if the device adds the audio file midway through multimedia file playback, the beginning of the target audio file can be precisely aligned with the already played duration of the multimedia file through silence filling, thus achieving the requirement of accurately filling silence data.

[0067] Abnormal interval check phase:

[0068] If this is not the first write, since the slave device's packet sequence number increments by 1, the master device can calculate the interval (interval = current packet sequence number - previous packet sequence number) between the current packet sequence number and the sequence number of the previously written packet (reference packet sequence number) to determine if there are any lost, duplicate, or expired packets and take appropriate action.

[0069] If interval <= 0 (“Y” branch), it indicates that the data packet is out of order or duplicated, and it should be discarded.

[0070] If interval exceeds 2 ("Y" branch), it indicates that there is packet loss during the period. To avoid severe packet loss due to the network, it is necessary to fill in (interval - 1) the corresponding data length of silent data.

[0071] If the interval is 1 ("N" branch), the system will update the sequence number record of the last written data packet (the sequence number of the last written data packet = the sequence number of the current data packet) and finally approve the writing of this data packet (needWrite = true).

[0072] Ultimately, only data packets that successfully pass the above three checks (i.e., checkPackageNo outputs needWrite = true) will be executed for the final writeFile operation and written to the target audio file.

[0073] In some embodiments of this application, the process by which the master device writes audio data packets to the target audio file is as follows: Figure 4 As shown, after the process starts, the checkSeekTo verification step is executed first. The core function of this step is to accurately locate the read / write position of the target audio file. By reading the current read / write pointer state of the file and combining it with the playback progress information of the multimedia file, the starting offset of the data write is calibrated. This operation can avoid write position errors caused by device restarts, pauses and resumes, etc., ensuring that subsequent audio data packets can be written to the correct timing segments, laying the positional foundation for overall synchronization.

[0074] Next comes the checkSize verification step, which focuses on controlling the file timing length of the target audio file. The master device calculates the accumulated file timing length of the target audio file in real time, and simultaneously retrieves a preset timing length parameter (this parameter is preset based on the total playback duration of multimedia files in the master device). By comparing this parameter, it determines whether the current file timing length has reached or exceeded the preset timing length. If the preset threshold is reached, it means that the multimedia file playback has ended, and the master device will stop receiving all subsequent audio data packets to avoid invalid data writing and file redundancy. If the threshold is not exceeded, the core packet sequence number verification step is entered to ensure the timing accuracy of data writing.

[0075] Packet sequence number verification (checkPackageNo) is a crucial step in achieving high-precision synchronization, comprising a three-tiered progressive verification logic. The first tier involves filtering data before the first frame. The master device compares the sequence number of the current audio data packet with the reference packet sequence number (corresponding to the sequence number associated with the reference timestamp at the start of multimedia file playback). If the data packet sequence number is less than the reference packet sequence number, it is determined to be "expired" data sent before the multimedia file started playing or due to network latency, and is directly marked as not needing to be written (needWrite = false), preventing such data from interfering with normal recording timing. The second tier involves silent data filling decisions. If the data packet sequence number is greater than or equal to the reference packet sequence number, and this is the first write of that data stream, the master device calculates the difference (interval) between the data packet sequence number and the reference packet sequence number, and fills the target audio file with silent data (fillZeroData) of the duration corresponding to this difference at the current write position. This mechanism effectively solves the timing alignment problem when adding synchronous recording midway, ensuring that the beginning of the recording file accurately matches the already played duration of the multimedia file, guaranteeing synchronization continuity. The third layer is the abnormal interval check. Since the sequence number of the data packets on the slave device increments by 1 according to a fixed rule, the master device identifies and handles abnormal situations by calculating the interval between the current data packet sequence number and the sequence number of the previous written data packet: when the interval is ≤0, it is determined to be an out-of-order or duplicate data packet and is directly discarded; when the interval exceeds 2, it is determined to be a packet loss and fills in (interval-1) the corresponding duration of silent data to compensate for the timing gap and avoid recording interruptions due to network packet loss; when the interval is equal to 1, it is determined to be normal data transmission, updates the sequence number record of the previous written data packet, and marks it as allowed to write (needWrite = true).

[0076] After the three-level verification process described above, only audio data packets that pass all verifications will undergo the final writeFile operation, writing them to the target audio file according to the calibrated read / write positions, thus completing a single data write process. The entire process, through progressive steps of position calibration, length control, and sequence number verification, effectively solves core problems such as position misalignment, timing deviation, and data anomalies in multi-device synchronous recording. Combined with compensation mechanisms such as silence filling, it achieves high-precision synchronization between audio files and multimedia files, ensuring audio data quality in multi-device recording scenarios.

[0077] By employing a master device to generate and send a reference timestamp to a slave device, where the reference timestamp corresponds to the start playback time of the multimedia file in the master device; acquiring audio data packets sent from the slave device to the master device, where each audio data packet carries a data packet sequence number, which is an incrementing sequence number generated by the slave device based on the reference timestamp; and processing the audio data packets according to the data packet sequence number, including discarding the audio data packets and writing them into the target audio file according to the data packet sequence number, the sequence number reflects the time difference between the generation time of the audio data packet and the reference timestamp. This achieves the goal of determining the playback progress of the multimedia file corresponding to the audio data packet based on the data packet sequence number, thereby ensuring that the audio data packets written into the target audio file are synchronized with the playback progress of the multimedia file. This solves the technical problem of audio-visual asynchrony in the synthesized result caused by the inability to guarantee the synchronization of the recorded audio and the played multimedia data in related technologies.

[0078] This application provides an audio data processing system. Figure 5 This is a schematic diagram of the system structure. From Figure 5 As can be seen from the diagram, the system includes: a master device 50 and at least one slave device 52. The master device 50 is used to generate and send an initial packet sequence number to the slave device 52 based on the playback progress information and a reference timestamp of the multimedia file played in the master device 50. The reference timestamp indicates the start time of the multimedia file playback, and the playback progress information includes the reference playback progress of the multimedia file when the slave device 52 accesses the master device 50. The master device 50 also acquires audio data packets sent by the slave device 52 to the master device 50, wherein the audio data packets carry a data packet sequence number, which is an incrementing sequence number generated by the slave device 52 based on the initial packet sequence number. The master device 52 processes the audio data packets according to the data packet sequence number, wherein the processing methods include discarding the audio data packets and writing the audio data packets into a target audio file according to the data packet sequence number. The slave device 52 is used to generate and send audio data packets to the master device 50, and to generate a data packet sequence number for the audio data packets based on the time difference between the generation time of the audio data packets and the reference timestamp.

[0079] In some embodiments of this application, the main device 50 can be a device with audio and video output capabilities, such as a television, tablet, or computer. The slave devices can be devices with audio acquisition devices such as microphones, like mobile phones. Taking a karaoke scenario as an example, a multi-device karaoke system refers to a system where multiple slave devices (often referring to audio acquisition devices such as mobile phones) use microphones as input devices to collect user voices and send their respective collected recording data to the karaoke main device in real time according to an agreed data transmission protocol. The main device (often referring to audio output devices such as televisions, tablets, or TV boxes) receives the multiple audio streams from the slave devices and then mixes and outputs the multiple audio streams with local media (such as music videos or music) in real time. This system typically also supports playback or upload functions, requiring the simultaneous storage of multiple voice recordings during real-time karaoke. The timing of multiple slave devices joining the karaoke system is uncertain; they can join before or during the playback of multimedia files on the main device.

[0080] In some embodiments of this application, the synchronization process between the master device 50 and the slave device 52 is as follows: Figure 6 As shown. From Figure 6 As can be seen, the master device 50, acting as the synchronization core, initializes the reference packet sequence number simultaneously with the start of playback of multimedia files such as MVs. It generates a reference packet sequence number `mvStartTimeStamp` based on the system time at the start of MV playback and stores it as a static variable. Simultaneously, it calculates the corresponding reference packet sequence number by combining it with the current playback progress of the MV (`mvPos`), i.e., `playstate.startTimeStamp = mvStartTimeStamp + mvPos`. After generating this reference packet sequence number, the master device 50 synchronously notifies all slave devices 52 participating in the recording, providing a unified timing reference for all slave devices 52. This ensures that the recording data sequence number of each slave device 52 remains aligned with the MV playback progress, avoiding timing deviations caused by reference differences.

[0081] Before receiving the reference packet sequence number notification from the master device 50, the first slave device 52 has already started recording and generated audio data packet sequence numbers with 0 as the initial value. After successfully receiving the playstate.startTimeStamp synchronized from the master device 50, the first slave device 52 will immediately update its current data packet sequence number to the reference packet sequence number. For each subsequent audio data packet collected, its sequence number is incremented sequentially according to a fixed rule based on the updated reference packet sequence number. This mechanism not only preserves the recording data collected by the first slave device 52 during the early start-up phase, but also achieves timing synchronization with the MV playback progress of the master device 50 through sequence number updates, effectively avoiding timing misalignment problems in the pre-recorded data.

[0082] The second slave device 52 uses a different sequence number initialization method. After receiving the `playstate.startTimeStamp` synchronized from the master device 50, it directly starts the recording operation using this reference packet sequence number as the initial value. The sequence number of each subsequent audio data packet acquired by this device is also generated sequentially according to a fixed auto-incrementing rule. This method is suitable for scenarios where the device adds synchronous recording during MV playback. By directly reusing the reference packet sequence number of the master device 50, it ensures that the recording data of the second slave device 52 is consistent with the MV playback progress from the start, without requiring additional timing calibration operations.

[0083] The synchronization logic of the remaining slave devices 52 follows the same pattern as the first or second slave device 52, selecting the corresponding sequence number initialization and update method based on their own startup timing. The goal of the entire process is to achieve a unified benchmark and dynamic alignment: the master device 50 generates and distributes benchmark packet sequence numbers, providing a unique timing anchor for all slave devices 52; each slave device 52 then chooses to update existing sequence numbers or directly reuse benchmark packet sequence numbers based on its own startup timing, thus binding the recording data packet sequence number to the MV playback progress. Ultimately, the recording data sequence numbers of all slave devices 52 correspond one-to-one with the MV playback timing, laying a solid foundation for the synchronization of subsequent writing processes and ensuring the audio data synchronization accuracy in multi-device recording scenarios.

[0084] This application provides an audio data processing apparatus suitable for use in a main device. Figure 7 This is a schematic diagram of the device. From Figure 7 As can be seen from the diagram, the device includes: a first processing module 70, used to generate and send an initial packet sequence number to the slave device based on the playback progress information and reference timestamp of the multimedia file played on the master device, wherein the reference timestamp is used to indicate the start time of the multimedia file playback, and the playback progress information includes the reference playback progress of the multimedia file when the slave device connects to the master device; a second processing module 72, used to obtain audio data packets sent by the slave device to the master device, wherein the audio data packets carry a data packet sequence number, and the data packet sequence number is an incrementing sequence number generated by the slave device based on the reference timestamp; and a third processing module 74, used to process the audio data packets according to the data packet sequence number, wherein the processing method of the audio data packets includes discarding the audio data packets and writing the audio data packets into a target audio file according to the data packet sequence number.

[0085] In some embodiments of this application, the first processing module 70 is further configured to: determine the reference packet sequence number corresponding to the reference timestamp when the reference playback progress indicates that the slave device is connected to the master device and the multimedia file is being played, and generate and send an initial packet sequence number corresponding to the reference playback progress to the slave device based on the reference playback progress and the reference packet sequence number; and send the reference packet sequence number as the initial packet sequence number to the slave device after the multimedia file starts playing when the reference playback progress indicates that the slave device is connected to the master device and the multimedia file is not being played.

[0086] In some embodiments of this application, the third processing module 74 processes audio data packets according to the data packet sequence number, including: detecting the real-time playback progress of the multimedia file; if a jump in the real-time playback progress is detected, resetting the data writing start point in the target audio file according to the changed playback progress; starting from the data writing start point, if it is determined that the audio data packet will be written to the target audio file, writing the audio data packet to the target audio file according to the audio data packet sequence number.

[0087] In some embodiments of this application, the third processing module 74 processes the audio data packet based on the data packet sequence number, including: determining the reference packet sequence number corresponding to the reference timestamp; comparing the data packet sequence number and the reference packet sequence number; discarding the audio data packet if the comparison result is that the data packet sequence number is less than the reference packet sequence number; determining whether the audio data packet is the first data packet sent by the device if the comparison result is that the data packet sequence number is greater than the reference packet sequence number; determining the writing position of the audio data packet based on the difference between the data packet sequence number and the reference packet sequence number if the audio data packet is determined to be the first data packet; filling the space between the start position and the writing position of the target audio file with silence data corresponding to the audio data packet; and writing the audio data packet to the writing position after filling with silence data.

[0088] In some embodiments of this application, the third processing module 74 processes the audio data packet according to the data packet sequence number, including: determining the reference data packet corresponding to the audio data packet, wherein the reference data packet is the most recent data packet written to the target audio file before the audio data packet, and the audio data packet and the reference data packet correspond to the same slave device; determining the difference between the packet sequence number of the audio data packet and the packet sequence number of the reference data packet, and processing the audio data packet according to the difference.

[0089] In some embodiments of this application, the third processing module 74 processes the audio data packet based on the difference, including: discarding the audio data packet if the difference is less than a preset difference; writing the audio data packet into the target audio file if the difference is equal to the preset difference; and filling the reference data packet with silence data according to the difference if the difference is greater than the preset difference, and writing the audio data packet after the silence data after filling the silence data.

[0090] In some embodiments of this application, the third processing module 74 is further configured to: determine the file timing length of the target audio file; and stop receiving audio data packets if the file timing length is greater than a preset timing length, wherein the preset timing length is a timing length determined based on the total playback duration of the multimedia file.

[0091] It should be noted that each module in the above-mentioned audio data processing device can be a program module (for example, a set of program instructions to implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0092] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, which stores a program. When the program runs, it controls the device where the non-volatile storage medium is located to execute the following audio data processing method: a master device generates and sends a reference timestamp to a slave device, wherein the reference timestamp corresponds to the start playback time of a multimedia file in the master device; the master device obtains an audio data packet sent by the slave device, wherein the audio data packet carries a data packet sequence number, which is an incrementing sequence number generated by the slave device based on the reference timestamp; and the audio data packet is processed according to the data packet sequence number, wherein the processing method for the audio data packet includes discarding the audio data packet and writing the audio data packet into a target audio file according to the data packet sequence number.

[0093] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program executes the following audio data processing method: a master device generates and sends a reference timestamp to a slave device, wherein the reference timestamp corresponds to the start playback time of a multimedia file in the master device; acquires an audio data packet sent by the slave device to the master device, wherein the audio data packet carries a data packet sequence number, the data packet sequence number being an incrementing sequence number generated by the slave device based on the reference timestamp; and processes the audio data packet according to the data packet sequence number, wherein the processing method for the audio data packet includes discarding the audio data packet and writing the audio data packet into a target audio file according to the data packet sequence number.

[0094] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program, wherein the computer program, when executed by a processor, implements the following steps of an audio data processing method: a master device generates and sends a reference timestamp to a slave device, wherein the reference timestamp corresponds to the start playback time of a multimedia file in the master device; acquires an audio data packet sent by the slave device to the master device, wherein the audio data packet carries a data packet sequence number, the data packet sequence number being an incrementing sequence number generated by the slave device based on the reference timestamp; and processes the audio data packet according to the data packet sequence number, wherein the processing method for the audio data packet includes discarding the audio data packet and writing the audio data packet into a target audio file according to the data packet sequence number.

[0095] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0096] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0097] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0098] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0099] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0100] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An audio data processing method, characterized in that, include: The master device generates and sends an initial packet sequence number to the slave device based on the playback progress information and reference timestamp of the multimedia file being played on the master device. The reference timestamp is used to indicate the start time of the multimedia file, and the playback progress information includes the reference playback progress of the multimedia file when the slave device connects to the master device. The audio data packet sent by the slave device to the master device is obtained, wherein the audio data packet carries a data packet sequence number, and the data packet sequence number is an incrementing sequence number generated by the slave device based on the initial packet sequence number; The audio data packet is processed according to the data packet sequence number, wherein the processing method of the audio data packet includes discarding the audio data packet and writing the audio data packet into a target audio file according to the data packet sequence number.

2. The audio data processing method according to claim 1, characterized in that, Processing the audio data packet according to the data packet sequence number includes: Detect the real-time playback progress of the multimedia file; If a change in the real-time playback progress is detected, the data writing start point in the target audio file is reset according to the changed playback progress. Starting from the data writing start point, if it is determined that the audio data packet will be written to the target audio file, the audio data packet will be written to the target audio file according to the audio data packet sequence number.

3. The audio data processing method according to claim 1, characterized in that, Processing the audio data packet according to the data packet sequence number includes: Determine the reference packet sequence number corresponding to the reference timestamp; Compare the data packet sequence number with the base packet sequence number; If the comparison result shows that the sequence number of the data packet is less than the sequence number of the reference packet, the audio data packet is discarded. If the comparison result shows that the sequence number of the data packet is greater than the sequence number of the reference packet, it is determined whether the audio data packet is the first data packet sent by the slave device. If the audio data packet is determined to be the first data packet, the writing position of the audio data packet is determined based on the difference between the data packet sequence number and the reference packet sequence number; Between the start position of the target audio file and the write position, fill with silence data corresponding to the audio data packet; After filling in the silence data, the audio data packet is written to the write location.

4. The audio data processing method according to claim 1, characterized in that, The master device generates and sends an initial packet sequence number to the slave device based on the playback progress information and reference timestamp of the multimedia file being played on the master device, including: When the reference playback progress indicates that the slave device is connected to the master device, and the multimedia file is being played, the reference packet sequence number corresponding to the reference timestamp is determined, and the initial packet sequence number corresponding to the reference playback progress is generated and sent to the slave device based on the reference playback progress and the reference packet sequence number. If the multimedia file is not playing when the reference playback progress indicates that the slave device has connected to the master device, the reference packet sequence number is sent to the slave device as the initial packet sequence number after the multimedia file starts playing.

5. The audio data processing method according to claim 1, characterized in that, Processing the audio data packet according to the data packet sequence number includes: Determine the reference data packet corresponding to the audio data packet, wherein the reference data packet is the most recent data packet written to the target audio file before the audio data packet, and the audio data packet and the reference data packet correspond to the same slave device; Determine the difference between the sequence number of the audio data packet and the sequence number of the reference data packet, and process the audio data packet based on the difference.

6. The audio data processing method according to claim 5, characterized in that, Processing the audio data packet based on the difference includes: If the difference is less than a preset difference, the audio data packet is discarded; If the difference is equal to the preset difference, the audio data packet is written into the target audio file; If the difference is greater than the preset difference, mute data is filled after the reference data packet according to the difference, and after filling the mute data, the audio data packet is written after the mute data.

7. The audio data processing method according to claim 1, characterized in that, The method further includes: Determine the file timing length of the target audio file; If the file timing length is greater than a preset timing length, the reception of the audio data packet is stopped, wherein the preset timing length is a timing length determined based on the total playback duration of the multimedia file.

8. An audio data processing system, characterized in that, Includes a master device and at least one slave device, wherein, The master device is configured to generate and send an initial packet sequence number to the slave device based on the playback progress information and a reference timestamp of the multimedia file being played on the master device. The reference timestamp indicates the start time of the multimedia file's playback, and the playback progress information includes a reference playback progress of the multimedia file when the slave device connects to the master device. The master device also acquires audio data packets sent by the slave device, wherein the audio data packets carry a data packet sequence number, which is an incrementing sequence number generated by the slave device based on the initial packet sequence number. Furthermore, the master device processes the audio data packets according to the data packet sequence number, wherein the processing method includes discarding the audio data packets and writing the audio data packets into a target audio file according to the data packet sequence number. The slave device is configured to generate and send the audio data packet to the master device, and to generate the data packet sequence number of the audio data packet based on the time difference between the generation time of the audio data packet and the reference timestamp.

9. An audio data processing apparatus, suitable for use in a main device, characterized in that, include: The first processing module is used to generate and send an initial packet sequence number to the slave device based on the playback progress information and reference timestamp of the multimedia file played in the master device. The reference timestamp is used to indicate the start time of the multimedia file playback, and the playback progress information includes the reference playback progress of the multimedia file when the slave device accesses the master device. The second processing module is used to obtain the audio data packet sent by the slave device to the master device, wherein the audio data packet carries a data packet sequence number, and the data packet sequence number is an incrementing sequence number generated by the slave device based on the initial packet sequence number; The third processing module is used to process the audio data packet according to the data packet sequence number, wherein the processing method of the audio data packet includes discarding the audio data packet and writing the audio data packet into a target audio file according to the data packet sequence number.

10. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a program, wherein when the program is executed, it controls the device containing the non-volatile storage medium to perform the audio data processing method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the audio data processing method according to any one of claims 1 to 7.

Images