Method for audio frame dynamic recombination and audio quality analysis

Abstract

The invention discloses a method for audio frame dynamic recombination and audio quality analysis, which comprises the following steps of: receiving an audio data packet to be detected, sorting audio data, judging whether the audio data packet has packet loss or not, if the packet loss exists, recovering the packet loss data, and simultaneously, judging the audio frame type of the audio data packet without packet loss so as to realize the audio frame dynamic recombination and the audio quality analysis. And determining an audio signal generated by the output zero-damage audio frame. And if the to-be-detected audio data packet is the interfered audio frame or the damaged audio frame, the audio signal is not generated, so that the normal work of the vocal music equipment is ensured, and meanwhile, better audio quality experience is provided for consumers.

Description

Technical Field

[0001] The invention belongs to the field of loudspeaker signal processing, and in particular relates to a method for dynamic reorganization of audio frames and audio quality analysis. Background Art

[0002] In recent years, with the popularization of broadband and the rapid development of network technology, speaker audio services are no longer limited to traditional broadcasting services, but have gradually expanded to various fields of life, such as entertainment activities such as audio and video on demand, as well as various emerging audio services formed on the Internet and mobile terminals. In order to provide high-quality audio signals, each audio frame needs to be divided into multiple data packets for wireless transmission during the audio transmission process. However, due to weak network signals and other reasons, the received audio data may sometimes have sequence errors, which makes it impossible for the speaker to correctly form audio frames, resulting in the speaker not being able to work properly. Real-time monitoring of audio frames to determine whether there is sound quality damage, while meeting audio quality requirements, providing consumers with a better audio quality experience is a technical problem that needs to be solved urgently. Summary of the invention

[0003] In view of the above defects, the technical problem solved by the present invention is to provide a method for dynamic reorganization of audio frames and audio quality analysis, which receives audio data packets to be detected and sorts the audio data to determine whether the audio data packets have packet loss. If there is packet loss, the packet loss data is restored, and at the same time, the audio frame type of the audio data packets that have not been lost is determined to determine the audio signal generated by the output lossless audio frame. The normal operation of vocal equipment is guaranteed, and consumers are provided with a better audio quality experience.

[0004] A first aspect of the present invention provides a method for dynamic reorganization of audio frames and audio quality analysis, characterized in that the method comprises: S101: receiving an audio data packet to be detected, and sorting the audio data in the audio data packet to be detected, and encapsulating it into a first audio data packet.

[0005] S102: Receive the first audio data packet, re-sort the second audio data packet into a second audio data packet, determine whether the second audio data packet is lost, and if so, restore the lost data.

[0006] S103: Determine the type of audio frame in the audio data packet that has not been lost; if the audio frame in the audio data packet that has not been lost is a damage-free audio frame, output the data packet to be detected and generate an audio signal; if the audio frame in the audio data packet that has not been lost is an interfered audio frame or a damaged audio frame, the data packet to be detected does not meet the playback requirements and does not generate an audio signal.

[0007] According to an embodiment of the present invention, each sub-audio data packet in the audio data packet to be detected The audio frames include at least two audio frames, namely, a first audio sample and a second audio sample, wherein the first audio sample and the second audio sample respectively include: frame header information , frame information and frame end information , the frame header information Includes: the serial number of the audio sample , encoding format, sampling rate, bit rate and number of channels.

[0008] According to an embodiment of the present invention, in S101, the audio data packet to be detected is The audio data in the audio data packet is sorted and encapsulated into the first audio data packet. , including: according to the reorganization function The audio data packets to be detected are sorted, and the reorganization function is shown in formula (1): Formula (1) in, Indicates the set of received audio data packets to be detected. represents a first audio data packet set; According to the serial number Determine each sub-audio data packet The order is as shown in formula (2): Formula (2) in, Indicates audio data packets The position where the packet should be inserted, n represents the number of packets currently received.

[0009] According to an embodiment of the present invention, the step of receiving the first audio data packet and re-sorting the second audio data packet into a second audio data packet in step S102 includes: Sort.

[0010] According to an embodiment of the present invention, the determining in S102 whether the second audio data packet is lost includes: Judgment, including: Determine the serial number Is it continuous? For example, whether formula (3) holds for all audio samples: Formula (3) If the first audio sample and the second audio sample in the second audio data packet satisfy formula (3), then there is no packet loss in the sub-audio data packet in the second audio data packet; If the second audio data packet contains the first audio sample and the second audio sample which do not satisfy equation (3), then there is packet loss in the sub-audio data packet of the second audio data packet.

[0011] According to an embodiment of the present invention, if there is packet loss in S102, recovering the packet loss data includes: calculating the sequence number of the missing audio sample, as shown in formula (4): Formula (4) in, represents the second audio sample, represents a first audio sample; Calculate the number of packet losses according to formula (5): Formula (5) in, Represents an indicator function, if the condition is true, the result is 1, otherwise it is 0; The missing audio data is calculated by linear interpolation method, as shown in formula (6): Formula (6) represents the i-th lost audio data packet, Indicates loss The last received audio packet before, Indicates loss After the first received audio packet, Indicates lost audio packets The serial number, and Respectively The sequence number of the adjacent packet.

[0012] According to an embodiment of the present invention, determining the type of audio frame in the non-lost data packet in S103 includes: determining the type of the audio frame according to the frame header information of the non-lost audio frame, as shown in formula (7): Formula (7) Among them, B is the frame header information, and T is the audio frame type.

[0013] The second aspect of the present invention provides an intelligent device, including a transmitter, a receiver, a memory and a processor; the memory is used to store computer instructions; the processor is used to run the computer instructions stored in the memory to implement the above-mentioned method for dynamic reconstruction of audio frames and audio quality analysis.

[0014] A third aspect of the present invention provides a storage medium, comprising: a readable storage medium and computer instructions, wherein the computer instructions are stored in the readable storage medium; the computer instructions are used to implement the above-mentioned method for dynamic reconstruction of audio frames and audio quality analysis.

[0015] The beneficial effects provided by the present invention are as follows: By sorting the audio data packets twice, it is ensured that even if the network condition is poor and the data packets are out of order, the audio data packets can be correctly sorted by the sequence number, thereby ensuring the integrity of the audio frame and the audio quality played by the speaker. At the same time, by matching the paired audio frames and judging whether the audio data packet is lost by the sequence number, the normal operation of the equipment is ensured, and the complete recovery of the audio data is guaranteed, and better audio quality is obtained for consumers. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0017] Figure 1 The present invention is a flowchart of a method for dynamic reorganization of audio frames and audio quality analysis disclosed in an embodiment of the present invention.

[0018] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0019] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0020] The first aspect of the present invention discloses a method for dynamic reorganization of audio frames and audio quality analysis, such as Figure 1As shown, it includes: S101: receiving an audio data packet to be detected, and sorting the audio data in the audio data packet to be detected, and encapsulating it into a first audio data packet. The audio data packet corresponds to at least two audio frames, and each audio data packet has a sequence number indicating the order of the audio data packet in the multiple audio data packets, sorting the multiple audio data packets through a reorganization algorithm to generate a reordered data packet, and encapsulating the reordered data packet into a data packet to be detected for transmission; (outputting the reordered data packet to regenerate the at least two audio frames).

[0021] Each sub-audio data packet in the audio data packet to be detected The audio frames include at least two audio frames, namely, a first audio sample and a second audio sample, wherein the first audio sample and the second audio sample respectively include: frame header information , frame information and frame end information , the frame header information Includes: the serial number of the audio sample , encoding format, sampling rate, bit rate and number of channels.

[0022] Among them, the reassembly algorithm: first needs to determine the structure of the audio frame, which usually includes the frame header (containing information such as the frame sequence number), the frame body (containing audio data) and the frame tail (which may contain checksum information). During the reassembly process, the algorithm needs to be able to detect whether the data packet is out of order. Usually, the sequence number in the frame header can be used to determine the order of the frames, and the reassembly algorithm can accurately identify and correct the out-of-order frames.

[0023] Specifically, audio data packets are received, each of which contains the following information: audio data packets , the sequence number of the i-th audio data (indicating its order) , the frame header of the audio frame corresponding to the i-th audio data packet ; The frame body of the audio frame corresponding to the i-th audio data packet ; The end of the audio frame corresponding to the i-th audio data packet .

[0024] Assuming that each audio frame is divided into two packets, a complete audio frame consists of two packets, denoted as and .

[0025] Secondly, determine the audio structure, which can be expressed as formula (1): Formula (1) in, Contains control information such as the serial number SN, Contains the main audio data, Contains calibration information.

[0026] Next, detect and sort out-of-order packets and define a reassembly algorithm function , this function receives a collection of out-of-order packets , and returns the sorted data packet set , as shown in formula (2).

[0027] Formula (2) in, Represents the set of received data packets, Represents a sorted set of packets.

[0028] Using a Serial Number Determine the order of data packets. The specific order can be expressed by the following mathematical formula (3): Formula (3) in, Indicates audio data packets The position where the packet should be inserted, n represents the number of packets currently received.

[0029] Assume that four out-of-order packets are received , whose serial numbers are , , , .

[0030] right ,turn up The correct position in The closest serial number is here ,so Should be ranked Behind. , because it has the smallest sequence number, it should be ranked first. , it has the largest sequence number, so it should be at the end. , it should be ranked and between.

[0031] Sorted data packet set After sorting, the data packets are repackaged into detection data packets for the next step of quality detection.

[0032] Through the above steps, it can be ensured that even when the network condition is poor and the data packets are out of order, the audio data packets can be correctly sorted by the sequence number, thereby ensuring the integrity of the audio frame and the audio quality played by the speaker.

[0033] Step S102: receiving the first audio data packet, re-sorting the second audio data packet into a second audio data packet, determining whether the second audio data packet is lost, and if so, restoring the lost data.

[0034] Real-time Transport Protocol (RTP) is used to transmit audio and video in IP networks. The following is the process of receiving audio packets: Determine the RTP transmission address (IP address and port number) of the audio stream. Use Socket programming to create a UDP socket and bind it to the specified IP address and port number. Start listening for audio data packets from the sender.

[0035] The mathematical description is as follows: Formula (4) In the formula, IP represents the IP address of the received data packet, and Port represents the port number of the received data packet.

[0036] Assume that the RTP transport address of the audio stream is 192.168.1.10:5000, then a socket will be created and listen to this address.

[0037] The frame header of the RTP data packet contains information such as sequence number (SN), timestamp (T), and identification bit (M).

[0038] Read the data packet and read the first 12 bytes from the received data packet, which is the fixed header of RTP.

[0039] Extract information: extract the sequence number (2 bytes), timestamp (4 bytes), and identification bit (1 bit) from the header.

[0040] The mathematical description is as follows: Formula (5) SN is the sequence number used to identify the order of the data packet. T is the timestamp, indicating the sampling time of the audio frame in the data packet. M is the flag, marking whether the current data packet is a key frame or an end frame.

[0041] Assume that an RTP data packet is received, and its header information is as follows: Serial number: 0x1234 (hexadecimal), converted to decimal is 4660.

[0042] Timestamp: 0x00000ABC (hexadecimal), converted to decimal is 2748.

[0043] Identification bit: The second bit is 1, indicating that this is a key frame.

[0044] Sort the data packets and regenerate the audio signal. Sort all received data packets according to the sequence number SN.

[0045] Check whether the serial number is continuous. If the serial number is not missing, it meets formula (6): Formula (6) If this holds true for all sequence numbers, it is assumed that there is no packet loss.

[0046] If the sequence number is missing, as shown in formula (7): Formula (7) in, is the current serial number, Is the previous serial number.

[0047] Calculate the number of lost packets based on the missing sequence numbers. Count the missing sequence numbers and determine the location of the lost packets based on the count value. The number of lost packets L is calculated according to formula (8): Formula (8) in, Represents an indicator function, if the condition is true then the result is 1, otherwise it is 0.

[0048] When packet loss is detected, an attempt is made to recover the lost audio data by copying adjacent packets.

[0049] Assuming that each data packet contains k audio samples, we can use linear interpolation to estimate the content of the lost data packet, as shown in equation (9): Formula (9) represents the estimated i-th lost packet.

[0050] is the last received packet before the lost packet.

[0051] Is the first received packet after the lost packet.

[0052] is the sequence number of the lost packet. and are the sequence numbers of adjacent packets respectively.

[0053] Suppose we have a sequence of 5 packets, numbered 1 to 5, packets Lost.

[0054] Before sorting:

[0055] After sorting: (Already in sorting state) Check if the serial number is missing:

[0056] Calculate the number of lost packets:

[0057] Recover the lost audio data as shown in formula (10).

[0058] Formula (10)

[0059] In this way, we can try to recover lost packets, thereby reducing interruptions and impairments in audio playback.

[0060] If there is no packet loss, the sorted data packets are directly used to regenerate the audio signal.

[0061] Assume that a series of data packets are received with sequence numbers [4660, 4662, 4663, 4664]. Checking finds that the sequence numbers are continuous, so there is no packet loss, and these data packets can be directly used to reconstruct the audio signal.

[0062] Please note that the above processes and formulas are simplified examples. In actual applications, more complex error detection and recovery mechanisms may be required to handle problems such as packet loss and network delay.

[0063] S103: Determine the type of audio frame in the audio data packet that has not been lost; if the audio frame in the audio data packet that has not been lost is a damage-free audio frame, output the data packet to be detected and generate an audio signal; if the audio frame in the audio data packet that has not been lost is an interfered audio frame or a damaged audio frame, the data packet to be detected does not meet the playback requirements and does not generate an audio signal.

[0064] First, we need to analyze the frame header information of the unlost audio frame to determine the frame type. Suppose we have a function , it receives the frame header information B and returns the frame type T, as shown in formula (11).

[0065] Formula (11) Among them, B is the frame header information, which contains the metadata of the frame, such as frame number, timestamp, etc. T is the frame type, which can be "impaired audio frame", "interfered audio frame" or "innocent audio frame".

[0066] Using the frame types of neighboring audio frames Since audio signals have short-term correlation, we can use the types of neighboring frames to infer the type of the current frame. is a prediction function based on the previous frame type and the following frame type To predict the type of the current frame , as shown in formula (12).

[0067] Formula (12) By judging the frame type in the packet loss and the predicted frame type, we can determine which category the frame in the packet loss belongs to. is the judgment function, where P is the packet loss situation and the output is J, as shown in formula (13).

[0068] Formula (13) Among them, J is the judgment result, which can be "damaged audio frame", "interfered audio frame" or "undamaged audio frame".

[0069] Processing different types of frames,We adopt different processing methods according to the judgment results.,If J is a "non-destructive audio frame", the data packet to be detected is directly,output to regenerate the audio signal, as shown in equation (14).

[0070] Formula (14) in, is the output audio signal, It is the audio signal reconstructed from the detected packets.

[0071] If J is a “damaged audio frame” or “interfered audio frame”, it needs to be further processed or marked, as shown in formula (15).

[0072] Formula (15) in, It is a marked audio signal, indicating that it does not meet the playback requirements. It is a marking function that receives the audio signal and frame type and performs corresponding processing.

[0073] Assume the frame sequence is known: frame 1 (no damage), frame 2 (packet loss), frame 3 (no damage) The frame header information of frame 1 and frame 3 is analyzed to obtain the type, as shown in equation (16) and equation (17): Formula (16) Formula (17) The type of predicted frame 2 is shown in equation (18): Formula (18) Determine the type of frame 2, as shown in formula (19): Formula (19) According to the judgment result, the data packet to be detected is directly output to regenerate the audio signal, as shown in formula (20): Formula (20) Please note that the examples here are simplified. In actual applications, and Functions such as will contain more complex logic and algorithms to more accurately predict and determine the frame type.

[0074] Calculate the packet loss position. Suppose the missing sequence number is .

[0075] Determine the current frame type based on the adjacent frame types. Suppose the current frame type is , the adjacent frame type is .

[0076] Determine the type of packet loss frame. The mathematical formulas are shown in equations (21), (22) and (23): Formula (21) Formula (22) Formula (23) Assumptions ,but .

[0077] If adjacent frames , , you can output data packets.

[0078] The above processes and formulas are written in LaTeX format compatible with Markdown. In specific applications, these steps and formulas can be adjusted and optimized according to actual needs.

[0079] The second aspect of the present invention provides an intelligent device, including a transmitter, a receiver, a memory and a processor; the memory is used to store computer instructions; the processor is used to run the computer instructions stored in the memory to implement the above-mentioned method for dynamic reconstruction of audio frames and audio quality analysis.

[0080] A third aspect of the present invention provides a storage medium, comprising: a readable storage medium and computer instructions, wherein the computer instructions are stored in the readable storage medium; the computer instructions are used to implement the above-mentioned method for dynamic reconstruction of audio frames and audio quality analysis.

[0081] The beneficial effects provided by the present invention are as follows: by sorting the audio data packets twice, it is ensured that even if the network condition is poor and the data packets are out of order, the audio data packets can be correctly sorted by the sequence number, thereby ensuring the integrity of the audio frame and the audio quality played by the speaker. At the same time, by matching the paired audio frames and judging whether the audio data packet is lost by the sequence number, the normal operation of the equipment is ensured, the complete recovery of the audio data is guaranteed, and better audio quality is obtained for consumers.

[0082] Obviously, the above specific implementation cases are only examples for illustrating the application of the present method, rather than limiting the implementation methods. For those skilled in the art, other different forms of changes and modifications can be made on the basis of the above description to study other related issues. Therefore, the protection scope of the present invention should be the protection scope of the claims.

[0083] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0084] The electronic device and other embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0085] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the embodiments of the present invention have been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

[0087] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0088] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for dynamic reorganization of audio frames and audio quality analysis, characterized in that: The method comprises: S101: receiving an audio data packet to be detected, sorting the audio data in the audio data packet to be detected, and encapsulating the audio data into a first audio data packet; S102: receiving the first audio data packet, re-sorting the second audio data packet into a second audio data packet, determining whether the second audio data packet is lost, and if so, recovering the lost data; S103: Determine the type of audio frame in the audio data packet that has not been lost; if the audio frame in the audio data packet that has not been lost is an intact audio frame, output the data packet to be detected and generate an audio signal; if the audio frame in the audio data packet that has not been lost is an interfered audio frame or a damaged audio frame, the data packet to be detected does not meet the playback requirements and does not generate an audio signal.

2. The method according to claim 1, characterized in that Each sub-audio data packet in the audio data packet to be detected The audio frames include at least two audio frames, namely, a first audio sample and a second audio sample, wherein the first audio sample and the second audio sample respectively include: frame header information , frame information and frame end information , the frame header information Includes: the serial number of the audio sample , encoding format, sampling rate, bit rate and number of channels.

3. The method according to claim 2, characterized in that In S101, the audio data packet to be detected is The audio data in the audio data packet is sorted and encapsulated into the first audio data packet. , including: according to the reorganization function The audio data packets to be detected are sorted, and the reorganization function is shown in formula (1): Formula (1) in, Indicates the set of received audio data packets to be detected. represents a first audio data packet set; According to the serial number Determine each sub-audio data packet The order is as shown in formula (2): Formula (2) in, Indicates audio data packets The position where the packet should be inserted, n represents the number of packets currently received.

4. The method according to claim 3, characterized in that The step S102 of receiving the first audio data packet and reordering the second audio data packet into a second audio data packet includes: Sort.

5. The method according to claim 4, characterized in that The step S102 of determining whether the second audio data packet is lost includes: Judgment, including: Determine the serial number Is it continuous? For example, whether formula (3) holds for all audio samples: Formula (3) If the first audio sample and the second audio sample in the second audio data packet satisfy formula (3), then there is no packet loss in the sub-audio data packet in the second audio data packet; If the second audio data packet contains the first audio sample and the second audio sample which do not satisfy equation (3), then there is packet loss in the sub-audio data packet of the second audio data packet.

6. The method according to claim 5, characterized in that If there is packet loss in S102, the packet loss data is recovered, including: calculating the sequence number of the missing audio sample, as shown in formula (4): Formula (4) in, represents the second audio sample, represents a first audio sample; Calculate the number of packet losses according to formula (5): Formula (5) in, Represents an indicator function, if the condition is true, the result is 1, otherwise it is 0; The missing audio data is calculated by linear interpolation method, as shown in formula (6): Formula (6) represents the i-th lost audio data packet, Indicates loss The last received audio packet before, Indicates loss After the first received audio packet, Indicates lost audio packets The serial number, and Respectively The sequence number of the adjacent packet.

7. The method according to claim 6, characterized in that The step of determining the type of audio frame in the non-lost data packet in S103 includes: determining the type of the audio frame according to the frame header information of the non-lost audio frame, as shown in formula (7): Formula (7) Among them, B is the frame header information, and T is the audio frame type.

8. A smart device, characterized in that: include: transmitter, receiver, memory and processor; The memory is used to store computer instructions; the processor is used to execute the computer instructions stored in the memory to implement the method for dynamic reconstruction of audio frames and audio quality analysis as described in any one of claims 1 to 7.

9. A storage medium, characterized in that: include: A readable storage medium and computer instructions, wherein the computer instructions are stored in the readable storage medium; The computer instructions are used to implement the method for dynamic audio frame reconstruction and audio quality analysis as described in any one of claims 1 to 7.

Images