Audio lag value determination method and device, equipment and storage medium

Abstract

The invention discloses an audio lag value determination method and device, equipment and a storage medium, and belongs to the field of audio analysis. The method comprises the following steps: receiving original audio frames of audio and sampling information of each original audio frame, wherein the sampling information is used for indicating sampling positions of the original audio frames in the audio; according to the sampling position of each original audio frame, the duration of a lost audio frame of the audio is determined, and the lost audio frame is the audio frame lost in the audio receiving process; and according to the duration of the audio and the duration of the lost audio frame, determining a lagging value of the audio, the lagging value being used for reflecting the lagging degree of the received audio. According to the method and the device, the lagging value of the audio is determined based on the duration of the lost audio frames instead of the number of the lost audio frames, so that the accuracy of the determined lagging value of the audio can still be ensured even if the duration of the transmitted audio frames changes.

Description

Technical Field

[0001] The present application relates to the field of audio analysis, and in particular to a method, device, equipment and storage medium for determining an audio stutter value. Background Art

[0002] The quality of audio calls has a great impact on user experience. The audio receiver can measure the degree of audio (audio call) jamming by calculating the audio jamming value, thereby prompting the user.

[0003] Audio frames are usually carried in audio data packets for transmission. Due to network fluctuations and other conditions, audio data packets may be lost during audio transmission, resulting in audio frame loss, which in turn causes audio freezes. In related technologies, the audio freeze value can be calculated by counting the number of lost audio data packets.

[0004] As network conditions change, the length of the audio data packet may also change. The above solution has low accuracy in determining the audio jam value in this scenario. Summary of the invention

[0005] The present application provides a method, apparatus, device and storage medium for determining an audio stutter value, which can improve the accuracy of determining an audio stutter value. The technical solution is as follows:

[0006] According to one aspect of the present application, a method for determining an audio jamming value is provided, the method being applied to an audio receiving end, the method comprising:

[0007] Receive original audio frames of audio and sampling information of each original audio frame, where the sampling information is used to indicate a sampling position of the original audio frame in the audio;

[0008] Determine, according to the sampling position of each original audio frame, the duration of a lost audio frame of the audio, wherein the lost audio frame is an audio frame lost during the process of receiving the audio;

[0009] A jamming value of the audio is determined according to the duration of the audio and the duration of the lost audio frame, where the jamming value is used to reflect the jamming degree of the received audio.

[0010] According to another aspect of the present application, a method for determining an audio jamming value is provided, the method being applied to an audio transmitting end, the method comprising:

[0011] Acquire an audio frame of an audio and sampling information of the audio frame, where the sampling information is used to indicate a sampling position of the audio frame in the audio;

[0012] Sending the audio frame of the audio and the sampling information of the audio frame to an audio receiving end, the audio receiving end is used to determine the duration of the lost audio frame of the audio according to the sampling position of each original audio frame, and determine the jamming value of the audio according to the duration of the audio and the duration of the lost audio frame;

[0013] The original audio frame is an audio frame of the received audio, the lost audio frame is an audio frame lost during the process of receiving the audio, and the jamming value is used to reflect the jamming degree of the received audio.

[0014] According to another aspect of the present application, a device for determining an audio freeze value is provided, the device comprising:

[0015] A receiving module, used for receiving original audio frames of audio and sampling information of each original audio frame, wherein the sampling information is used for indicating a sampling position of the original audio frame in the audio;

[0016] A determination module, configured to determine the duration of a lost audio frame of the audio according to a sampling position of each original audio frame, wherein the lost audio frame is an audio frame lost during the process of receiving the audio;

[0017] The determination module is further used to determine a jamming value of the audio according to the duration of the audio and the duration of the lost audio frame, wherein the jamming value is used to reflect the jamming degree of the received audio.

[0018] In an optional design, the determining module is used to:

[0019] Determine the estimated starting sampling position of the i+1th original audio frame according to the sampling position of the i-th original audio frame, where i is a positive integer;

[0020] When the actual starting sampling position of the i+1th original audio frame is different from the estimated starting sampling position of the i+1th original audio frame, determining the duration of the lost audio frame between the i+1th original audio frame and the i+1th original audio frame according to the difference between the estimated starting sampling position of the i+1th original audio frame and the actual starting sampling position of the i+1th original audio frame;

[0021] The actual starting sampling position of the i+1th original audio frame is determined according to the sampling information of the i+1th original audio frame.

[0022] In an optional design, the sampling information includes an actual starting sampling position and a sampling interval of the original audio frame in the audio, and the sampling interval is used to reflect the interval between the original audio frame and the actual starting sampling position of the next audio frame of the original audio frame in the audio; the determining module is used to:

[0023] Determine an estimated starting sampling position of the (i+1)th original audio frame according to a first sum of an actual starting sampling position of the (i)th original audio frame and a sampling interval of the (i)th original audio frame;

[0024] The duration of the lost audio frame between the i-th original audio frame and the i+1-th original audio frame is determined according to the difference between the actual starting sampling position of the i+1-th original audio frame and the first sum.

[0025] In an optional design, the determining module is used to:

[0026] According to the proportion of the duration of the lost audio frame in the duration of the audio, a first jamming value of the audio is determined as the jamming value of the audio, and the proportion is positively correlated with the jamming degree of the audio.

[0027] In an optional design, the determining module is used to:

[0028] Determine a second jamming value of the audio according to an occurrence interval of the lost audio frames, wherein the occurrence interval is negatively correlated with the jamming degree of the audio;

[0029] A freeze value of the audio is determined according to a first freeze value of the audio and a second freeze value of the audio.

[0030] In an optional design, the determining module is used to:

[0031] Determine the estimated starting sampling position of the i+1th original audio frame according to the sampling position of the i-th original audio frame, where i is a positive integer;

[0032] When the actual starting sampling position of the i+1th original audio frame is different from the estimated starting sampling position of the i+1th original audio frame, determining an occurrence interval of lost audio frames between the i-th original audio frame and the i+1th original audio frame according to a position difference between a sampling position of the i-th original audio frame and a sampling position of a target original audio frame;

[0033] The target original audio frame is the next audio frame of the most recent lost audio frame before the i-th original audio frame.

[0034] In an optional design, the sampling information includes an actual starting sampling position and a sampling interval of the original audio frame in the audio; and the determining module is used to:

[0035] Determine, according to the position difference between the second sum and the actual starting sampling position of the target original audio frame, an occurrence interval of lost audio frames between the i-th original audio frame and the i+1-th original audio frame;

[0036] The second sum is the sum of the actual starting sampling position and the sampling interval of the i-th original audio frame.

[0037] In an optional design, the device further includes:

[0038] An acquisition module, configured to acquire an energy value of the original audio frame, wherein the energy value of the original audio frame is determined according to a maximum value of each sampling point of the original audio frame;

[0039] The determination module is used to determine that the jamming value of the audio is the jamming value with the minimum jamming degree when the energy values ​​of each original audio frame of the audio are lower than the energy value threshold.

[0040] In an optional design, the audio is one channel of multiple channels; and the determining module is used to:

[0041] Determine the jam value of the multiple audio channels according to the weighted average value of the jam value of each audio channel in the multiple audio channels;

[0042] The weight of the jamming value of the n-th audio channel is determined according to the energy value of the n-th audio channel, the energy value of the n-th audio channel is determined according to the energy value of the original audio frame of the n-th audio channel, and n is a positive integer.

[0043] In an optional design, the receiving module is used to:

[0044] An audio data packet of the audio is received, where the audio data packet carries an audio frame of the audio, and a packet header of the audio data packet carries the sampling information.

[0045] According to another aspect of the present application, a device for determining an audio freeze value is provided, the device comprising:

[0046] An acquisition module, used to acquire an audio frame of an audio and sampling information of the audio frame, wherein the sampling information is used to indicate a sampling position of the audio frame in the audio;

[0047] A sending module, used to send the audio frame of the audio and the sampling information of the audio frame to an audio receiving end, wherein the audio receiving end is used to determine the duration of the lost audio frame of the audio according to the sampling position of each original audio frame, and determine the jamming value of the audio according to the duration of the audio and the duration of the lost audio frame;

[0048] The original audio frame is an audio frame of the received audio, the lost audio frame is an audio frame lost during the process of receiving the audio, and the jamming value is used to reflect the jamming degree of the received audio.

[0049] According to another aspect of the present application, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the method for determining the audio stutter value as described above.

[0050] According to another aspect of the present application, a computer-readable storage medium is provided, in which at least one program is stored. The at least one program is loaded and executed by a processor to implement the method for determining the audio stutter value as described above.

[0051] According to another aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method for determining the audio stutter value provided in various optional implementations of the above aspects.

[0052] The beneficial effects of the technical solution provided by this application include at least:

[0053] In the process of determining the audio jam value, the duration of the lost audio frame lost during the audio transmission is analyzed according to the sampling information of the received original audio frame, and the audio jam value is determined accordingly. Since the audio jam value is determined based on the duration of the lost audio frame, rather than the number of lost audio frames, the accuracy of the determined audio jam value can still be guaranteed even if the duration of the transmitted audio frame changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 is a structural block diagram of a computer system provided by an exemplary embodiment of the present application;

[0056] Figure 2 is a schematic diagram of a process of determining an audio stutter value provided by an exemplary embodiment of the present application;

[0057] Figure 3 is a flowchart of a method for determining an audio stutter value provided by an exemplary embodiment of the present application;

[0058] Figure 4 is a flowchart of a method for determining an audio stutter value provided by an exemplary embodiment of the present application;

[0059] Figure 5 is a flowchart of a method for determining an audio stutter value provided by an exemplary embodiment of the present application;

[0060] Figure 6 is a schematic diagram of an original audio frame provided by an exemplary embodiment of the present application;

[0061] Figure 7 It is a schematic diagram of a process of determining a jamming value of an audio channel provided by an exemplary embodiment of the present application;

[0062] Figure 8 is a schematic diagram of a process of determining a jam value of multi-channel audio provided by an exemplary embodiment of the present application;

[0063] Fig. 9 is a schematic diagram of a process of playing audio provided by an exemplary embodiment of the present application;

[0064] Fig.10 is a schematic diagram of a process of playing audio provided by an exemplary embodiment of the present application;

[0065] Fig.11 is a structural schematic diagram of a device for determining an audio stutter value provided by an exemplary embodiment of the present application;

[0066] Fig.12 is a structural schematic diagram of a device for determining an audio stutter value provided by an exemplary embodiment of the present application;

[0067] Fig.13is a structural schematic diagram of a device for determining an audio stutter value provided by an exemplary embodiment of the present application;

[0068] Fig.14 It is a schematic diagram of the structure of a terminal provided by an exemplary embodiment of the present application.

[0069] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application. DETAILED DESCRIPTION

[0070] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0071] First, the relevant terms involved in this application are introduced.

[0072] 3A: is a general term for the three core technologies of audio processing, including acoustic echo cancellation (AEC), automatic gain control (AGC) and noise suppression (AutomaticNoiseSuppression).

[0073] Jitter buffer: used to store remote audio frames (received audio frames) and provide acceleration and deceleration capabilities when playing decoded audio frames to smooth out the uneven audio data caused by network jitter. For example, if the time interval between received audio frames is large and cannot be played at the original speed of the audio, the decoding and playback of audio frames can be slowed down to ensure the user's audio playback experience. Jitter buffer is also a core technology in audio and plays an important role in anti-jitter and weak network conditions.

[0074] Figure 1 1 is a block diagram of a computer system provided by an exemplary embodiment of the present application. The computer system includes a first terminal 110 and a second terminal 120, and the first terminal 110 and the second terminal 120 are connected via a communication network 130. In some embodiments, the communication network 130 between the first terminal 110 and the second terminal 120 is a direct communication network. In some embodiments, the communication network 130 between the first terminal 110 and the second terminal 120 is established through a server 140, and the first terminal 110 and the second terminal 120 are respectively connected to the server 140.

[0075] In some embodiments, a client supporting audio transmission is deployed in the first terminal 110. The client in the first terminal 110 is an independent client, or a small program that depends on the host program to run, or a web page, which is not limited in the embodiments of the present application. In some embodiments, the client includes an instant messaging client, a video client, a social client, a financial client, an online shopping client, a music client, a takeaway client, an office client, a game client, a map client, a traffic client, a navigation client, etc., and the first terminal 110 is connected to the communication network 130 through the client.

[0076] In some embodiments, a client supporting audio transmission is deployed in the second terminal 120. The client in the second terminal 120 is an independent client, or a small program that depends on the host program to run, or a web page, which is not limited in the embodiments of the present application. In some embodiments, the client includes an instant messaging client, a video client, a social client, a financial client, an online shopping client, a music client, a takeaway client, an office client, a game client, a map client, a traffic client, a navigation client, etc., and the second terminal 120 is connected to the communication network 130 through the client.

[0077] It should be noted that Figure 1 The number of the first terminal 110, the second terminal 120 and the server 140 is only used as an example and is not intended to limit the structure of the computer system provided in the embodiment of the present application. It is understandable that there may be more terminals that establish a connection through the communication network 130 for audio transmission. The above-mentioned clients in the first terminal 110 and the second terminal 120 are the same or different, for example, they are versions corresponding to the same client under different operating systems, or they are different versions of the same client, and the embodiment of the present application does not limit this. The embodiment of the present application is introduced by taking the first terminal 110 and the second terminal 120 performing audio transmission through the server 140 as an example.

[0078] In some embodiments, when the first terminal 110 sends audio to the second terminal 120, it will send an audio frame of the audio to the server 140. The server 140 will send the audio frame of the audio sent by the first terminal 110 to the second terminal 120. In some embodiments, the audio frame is carried in an audio data packet for transmission. Among them, the first terminal 110 can be called an audio transmitter (far-end / far-end user), and the second terminal 120 can be called an audio receiver (near-end / near-end user). It should be noted that when the second terminal 120 sends audio to the first terminal 110, the identities of the two will also change accordingly. At this time, the second terminal 120 is the audio transmitter and the first terminal 110 is the audio receiver.

[0079] When sending an audio frame of audio, the first terminal 110 also obtains sampling information of the audio frame and sends it to the second terminal 120. The sampling information is used to indicate the sampling position of the audio frame in the audio. The second terminal 120 receives the original audio frame of the audio sent by the first terminal 110 and the sampling information of each original audio frame. Then, the second terminal 120 can determine the jamming value of the audio according to the sampling information. The original audio frame is the audio frame that the second terminal 120 can receive among the audio frames sent by the first terminal 110. In some embodiments, the second terminal 120 can determine the duration of the lost audio frame of the audio according to the sampling position of each original audio frame. The lost audio frame is the audio frame lost in the process of the second terminal 120 receiving the audio. Then, the second terminal 120 can determine the first jamming value of the audio as the jamming value of the audio according to the duration of the audio and the duration of the lost audio frame. The jamming value is used to reflect the jamming degree of the received audio. In some embodiments, the second terminal 120 can also determine the second jamming value of the audio according to the occurrence interval of the lost audio frame, and determine the jamming value of the audio according to the first jamming value of the audio and the second jamming value of the audio. In some embodiments, the second terminal 120 can also obtain the energy value of the original audio frame, and when the energy values ​​of each original audio frame of the audio are lower than the energy value threshold, determine the jamming value of the audio as the jamming value with the minimum jamming degree, wherein the energy value of the original audio frame is determined according to the maximum value of each sampling point of the original audio frame. In some embodiments, the first terminal 110 can send multiple audio channels to the second terminal 120, and the second terminal 120 can determine the jamming value of each audio channel in the above manner, and then determine the jamming value of the multiple audio channels according to the weighted average of the jamming value of each audio channel. Among them, the weight of the jamming value of one audio channel is determined according to the energy value of the audio channel, and the energy value of the audio channel is determined according to the energy value of the original audio frame of the audio channel.

[0080] It is worth noting that the above-mentioned terminals include but are not limited to mobile terminals such as mobile phones, tablet computers, portable laptops, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc., and can also be implemented as desktop computers, etc.; the above-mentioned servers can be independent physical servers, or they can be server clusters or distributed systems composed of multiple physical servers, or they can be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0081] Among them, cloud technology refers to a hosting technology that unifies hardware, application programs, networks and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient.

[0082] In some embodiments, the above server can also be implemented as a node in a blockchain system.

[0083] Figure 2 FIG. 1 is a schematic diagram of a process for determining an audio freeze value provided by an exemplary embodiment of the present application. Figure 2 As shown, in step S1, the audio transmitting end (remote end) obtains the audio frames of the audio and the actual starting sampling position and sampling interval of each audio frame in the audio, and the sampling interval is used to reflect the interval between the actual starting sampling position of the audio frame and the next audio frame of the audio frame in the audio. In step S2, the audio transmitting end adds a header to the audio data packet of the audio frame, for example, adding the data packet type, audio coding format, data packet size, audio frame duration (sampling interval), etc. in the header. In step S3, the audio transmitting end supplements the actual starting sampling position of the audio frame carried by it in the header of the audio data packet. In step S4, the audio transmitting end sends the audio data packet to the audio receiving end (near end) (uplink process), and the transmission process can be realized through server transfer.

[0084] In step S5, the audio receiving end receives the audio data packet sent by the audio sending end (downlink process), and parses the audio data packet to obtain the original audio frame of the audio and the actual starting sampling position and sampling interval of each original audio frame. The original audio frame is the audio frame received by the audio receiving end among the audio frames sent by the audio sending end.

[0085] In step S6, the audio receiving end determines the estimated starting sampling position of the i+1th original audio frame according to the first sum of the actual starting sampling position of the i+1th original audio frame and the sampling interval of the i+1th original audio frame. And when the actual starting sampling position of the i+1th original audio frame is different from the estimated starting sampling position of the i+1th original audio frame (there is a lost audio frame), the duration of the lost audio frame (one or more continuous audio frames) between the i+1th original audio frame and the i+1th original audio frame is determined according to the difference between the actual starting sampling position of the i+1th original audio frame and the first sum. In some embodiments, the audio receiving end also determines the occurrence interval of the lost audio frame between the i+1th original audio frame and the i+1th original audio frame according to the position difference between the second sum and the actual starting sampling position of the target original audio frame when the actual starting sampling position of the i+1th original audio frame is different from the estimated starting sampling position of the i+1th original audio frame. The second sum is the sum of the actual starting sampling position and the sampling interval of the i-th original audio frame, and the target original audio frame is the next audio frame of the nearest lost audio frame before the i-th original audio frame. By executing the above process, the audio receiving end can determine the duration and occurrence interval of the lost audio frame lost during the audio transmission process. Afterwards, the audio receiving end can determine the first jamming value of the audio according to the proportion of the duration of the lost audio frame in the duration of the audio, and the proportion is positively correlated with the jamming degree of the audio. According to the occurrence interval of the lost audio frame, the second jamming value of the audio can be determined, and the occurrence interval is negatively correlated with the jamming degree of the audio. Afterwards, according to at least one of the first jamming value of the audio and the second jamming value of the audio, the jamming value of the audio can be determined, and the jamming value is used to reflect the jamming degree of the received audio. In some embodiments, the audio receiving end can also obtain the energy value of the original audio frame, and when the energy values ​​of each original audio frame of the audio are lower than the energy value threshold, the jamming value of the audio is determined to be the jamming value with the smallest jamming degree. The energy value of the original audio frame is determined according to the maximum value of each sampling point of the original audio frame.

[0086] The above process of determining the stuttering value of the audio is for a single-channel audio. For multi-channel audio, the audio receiving end can determine the stuttering value of the multi-channel audio according to the weighted average of the stuttering value of each audio in the multi-channel audio. The process of determining the stuttering value of each audio channel can refer to the above content. The weight of the stuttering value of the n-th audio channel in the multi-channel audio is determined according to the energy value of the n-th audio channel, and the energy value of the n-th audio channel is determined according to the energy value of the original audio frame of the n-th audio channel, and the energy value of the original audio frame is determined according to the maximum value of each sampling point of the original audio frame.

[0087] In the process of determining the audio jam value, the duration of the lost audio frame lost during the audio transmission is analyzed according to the sampling information of the received original audio frame, and the audio jam value is determined accordingly. Since the audio jam value is determined based on the duration of the lost audio frame, rather than the number of lost audio frames, the accuracy of the determined audio jam value can still be guaranteed even if the duration of the transmitted audio frame changes.

[0088] Figure 3 FIG. 1 is a flow chart of a method for determining an audio jam value provided by an exemplary embodiment of the present application. The method can be used in an audio receiving terminal or a client on the audio receiving terminal, for example, Figure 1 The second terminal shown in Figure 3 As shown, the method includes:

[0089] Step 302: Receive original audio frames and sampling information of each original audio frame.

[0090] In some embodiments, the audio is the audio corresponding to the voice. In some embodiments, the audio can also be the audio of a song, the audio of background music, or the audio of other sounds, and the embodiments of the present application do not limit the specific content of the audio. The audio is acquired by the audio transmitting end and sent to the audio receiving end. Optionally, the audio is recorded by the audio transmitting end, or is locally acquired by the audio transmitting end, or is acquired by the audio transmitting end through other devices, and the embodiments of the present application do not limit the source of the audio.

[0091] In the process of sending audio, the audio transmitter will send multiple audio frames of the audio, some of which may be lost during the transmission process, for example, due to network fluctuations, the transmitted audio frames are lost. The original audio frame is the audio frame that the audio receiver can receive in the audio frame sent by the audio transmitter. The sampling information is used to indicate the sampling position of the original audio frame in the audio, and the sampling position is used to reflect the time domain position occupied by the original audio frame in the audio. Optionally, the sampling information includes at least one of the actual starting sampling position, the actual ending sampling position and the sampling interval of the original audio frame. Optionally, the sampling interval is used to reflect the interval between the actual starting sampling position of the original audio frame and the next audio frame of the original audio frame in the audio. The sampling interval can also be understood as the audio frame duration of the audio frame, and the audio frame duration is the duration occupied by the audio frame in the audio. Optionally, the actual starting sampling position or the actual ending sampling position is an absolute position or a relative position in the audio. For example, the actual starting sampling position is a position relative to the starting point of the audio. For example, the actual starting sampling position (the starting point of the audio) of the first audio frame of the audio is 0, and the sampling interval is 20. The actual starting sampling position of the second audio frame of the audio is 20, and the sampling interval is 40. The actual starting sampling position of the third audio frame of the audio is 60, and the sampling interval is 20.

[0092] In some embodiments, audio is transmitted via audio data packets, and the audio data packets carry audio frames of audio, for example, each audio data packet carries an audio frame of audio. In some embodiments, the header of the audio data packet carries the above-mentioned sampling information. In some embodiments, the sampling information includes the actual starting sampling position and sampling interval of the original audio frame, and the sampling information is added to the header of the audio data packet by the audio transmitter, and the timing when the audio transmitter adds the actual starting sampling position and sampling interval in the header of the audio data packet is the same or different. For example, the audio transmitter first adds basic information related to the audio frame in the header, including the sampling interval, and then additionally adds the actual starting sampling position of the audio frame. The actual starting sampling position is obtained by the audio transmitter by analyzing the audio frame it sends.

[0093] Step 304: Determine the duration of the lost audio frame of the audio according to the sampling position of each original audio frame.

[0094] The lost audio frame is an audio frame lost in the process of receiving audio. For example, when the audio frame is carried in an audio data packet for transmission, if the audio data packet is lost, the audio frame carried by the audio data packet is lost.

[0095] Optionally, the audio receiving end can estimate the estimated starting sampling position of the next audio frame of the original audio frame in the audio according to the sampling position of the received i-th original audio frame, where i is a positive integer. For example, the audio receiving end determines the actual ending sampling position of the i-th original audio frame as the estimated starting sampling position of the next audio frame, and the actual ending sampling position of the i-th original audio frame can be calculated by the sum of the actual starting sampling position of the i-th original audio frame and the sampling interval of the i-th original audio frame. The audio receiving end compares the estimated starting sampling position with the actual starting sampling position of the next original audio frame of the actually received original audio frame, that is, the i+1-th original audio frame. If the two are different (the estimated starting sampling position is less than the actual starting sampling position), it can be determined that there are one or more lost audio frames between the received i-th original audio frame and the i+1-th original audio frame. The audio receiving end can determine the duration of one or more lost audio frames between the lost i-th original audio frame and the i+1-th original audio frame according to the difference between the estimated starting sampling position and the actual starting sampling position of the i+1-th original audio frame. The actual starting sampling position of the i+1th original audio frame may be determined according to the sampling information of the i+1th original audio frame.

[0096] Step 306: Determine the audio freeze value according to the duration of the audio and the duration of the lost audio frame.

[0097] The jamming value is used to reflect the jamming degree of the audio received by the audio receiving end. The jamming degree is positively correlated with the duration of the lost audio frame, that is, the longer the duration of the lost audio frame, the higher the jamming degree. Optionally, the jamming value of the audio is positively correlated or negatively correlated with the jamming degree. For example, the higher the jamming degree, the higher the jamming value score. Alternatively, the higher the jamming degree, the lower the jamming value score. Optionally, the audio receiving end determines the first jamming value of the audio based on the proportion of the duration of the lost audio frame in the duration of the audio, and uses it as the jamming value of the audio.

[0098] In some embodiments, the audio receiving end can also determine the second audio jam value according to the occurrence interval of the lost audio frame, and determine the audio jam value according to the first audio jam value and the second audio jam value, for example, the value reflecting the highest degree of jamming between the first jam value and the second jam value is used as the audio jam value. The method for determining the occurrence interval of the lost audio frame can refer to the above process of determining the lost audio frame.

[0099] In some embodiments, the audio receiving end can also obtain the energy value of the original audio frame, and when the energy values ​​of each original audio frame of the audio are lower than the energy value threshold, determine the audio jam value as the jam value with the least jamming degree. The energy value of the original audio frame is determined according to the maximum value of each sampling point of the original audio frame. Optionally, the audio receiving end obtains the energy value of the original audio frame by analyzing the received original audio frame. Alternatively, the energy value of the original audio frame is sent by the audio sending end, for example, carried in the header of the audio data packet of the original audio frame. In some embodiments, the timing when the audio sending end adds the actual starting sampling position of the original audio frame in the header of the audio data packet is the same as the timing when the energy value of the original audio frame is added.

[0100] In some embodiments, the audio transmitter can send multiple audio channels to the audio receiver. For example, in a multi-person conference scenario, each of the multiple audio transmitters will send one audio channel (for example, representing the voice of a participant) to the audio receiver. The audio receiver can determine the jamming value of each audio channel in the above manner, and then determine the jamming value of the multiple audio channels based on the weighted average of the jamming values ​​of each audio channel. The weight of the jamming value of one audio channel is determined based on the energy value of the audio channel, and the energy value of the audio channel is determined based on the energy value of the original audio frame of the audio channel.

[0101] Optionally, the audio receiving end determines the audio jam value according to the statistical period T, and the duration of the statistical period T is not greater than the duration of the audio. In each statistical period T, the audio receiving end uses the statistical period T as the duration of the audio, and combines the duration of the lost audio frame in the statistical period T and / or the occurrence interval of the lost audio frame to determine the audio jam value in the statistical period T in the above manner.

[0102] In summary, the method provided in this embodiment analyzes the duration of the lost audio frames lost during audio transmission according to the sampling information of the received original audio frames during the process of determining the audio stuttering value, and determines the audio stuttering value based on this. Since the audio stuttering value is determined based on the duration of the lost audio frames, rather than the number of lost audio frames, the audio stuttering value is determined based on the duration of the lost audio frames, so even if the duration of the transmitted audio frames changes, the accuracy of the determined audio stuttering value can still be guaranteed.

[0103] Figure 4 FIG. 1 is a flow chart of a method for determining an audio jam value provided by an exemplary embodiment of the present application. The method can be used in an audio transmitter or a client on the audio transmitter, for example, Figure 1 The first terminal shown in Figure 4 As shown, the method includes:

[0104] Step 402: Acquire audio frames of audio and sampling information of the audio frames.

[0105] In some embodiments, the audio is the audio corresponding to the voice. In some embodiments, the audio can also be the audio of a song, the audio of background music, or the audio of other sounds, and the embodiments of the present application do not limit the specific content of the audio. The audio is acquired by the audio transmitting end and sent to the audio receiving end. Optionally, the audio is recorded by the audio transmitting end, or is locally acquired by the audio transmitting end, or is acquired by the audio transmitting end through other devices, and the embodiments of the present application do not limit the source of the audio.

[0106] The sampling information is used to indicate the sampling position of the original audio frame in the audio, and the sampling position is used to reflect the time domain position occupied by the original audio frame in the audio. Optionally, the sampling information includes at least one of the actual starting sampling position, the actual ending sampling position and the sampling interval of the original audio frame. Optionally, the sampling interval is used to reflect the interval between the actual starting sampling position of the original audio frame and the next audio frame of the original audio frame in the audio. The sampling interval can also be understood as the audio frame duration of the audio frame, and the audio frame duration is the duration occupied by the audio frame in the audio. Optionally, the audio transmitter obtains the sampling information of the audio frame by analyzing the audio frame of the audio.

[0107] Step 404: Send the audio frame and the sampling information of the audio frame to the audio receiving end.

[0108] In some embodiments, audio is transmitted via audio data packets, and the audio data packets carry audio frames of the audio, for example, each audio data packet carries an audio frame of the audio. In some embodiments, the packet header of the audio data packet carries the above-mentioned sampling information. In some embodiments, the sampling information includes the actual starting sampling position and sampling interval of the original audio frame, and the sampling information is added to the packet header of the audio data packet by the audio transmitter, and the timing when the audio transmitter adds the actual starting sampling position and sampling interval in the packet header of the audio data packet is the same or different. For example, the audio transmitter first adds basic information related to the audio frame in the packet header, including the sampling interval, and then additionally adds the actual starting sampling position of the audio frame.

[0109] When sending audio, the audio transmitter sends multiple audio frames of the audio, some of which may be lost during transmission, for example, due to network fluctuations. Among the audio frames sent by the audio transmitter, the audio frames that can be received by the audio receiver can be called original audio frames, and the audio frames lost during the audio receiving process can be called lost audio frames. The audio receiver is used to determine the duration of the lost audio frames of the audio based on the sampling position of each original audio frame, and to determine the audio jam value based on the duration of the audio and the duration of the lost audio frame. The jam value is used to reflect the jam degree of the received audio.

[0110] In some embodiments, the audio receiving end may also determine the first stutter value according to the duration of the lost audio frame, and determine the second stutter value according to the interval of occurrence of the lost audio frame, and determine the stutter value of the audio according to the first stutter value of the audio and the second stutter value of the audio. In some embodiments, when determining the stutter value of the audio, the audio receiving end may also refer to the energy value of the original audio frame, and the energy value of the original audio frame is determined according to the maximum value of each sampling point of the original audio frame. In some embodiments, multiple audio transmitting ends can send multiple audio channels to the audio receiving end, and the audio receiving end can determine the stutter value of each audio channel, and then determine the stutter value of the multiple audio channels according to the stutter value of each audio channel. For the specific implementation process of the above scheme, reference may be made to the relevant contents in other embodiments, and the embodiments of the present application will not be repeated here.

[0111] In summary, the method provided in this embodiment, by sending the sampling information of the audio frame when sending the audio frame, enables the audio receiving end to analyze the duration of the lost audio frame lost during the audio transmission process according to the sampling information of the received original audio frame in the process of determining the audio stuttering value, and thereby determine the audio stuttering value. Since the audio stuttering value is determined based on the duration of the lost audio frame, rather than the number of lost audio frames, the audio stuttering value is determined, so even if the duration of the transmitted audio frame changes, the accuracy of the determined audio stuttering value can still be guaranteed.

[0112] Figure 5 FIG. 1 is a flow chart of a method for determining an audio stutter value provided by an exemplary embodiment of the present application. The method can be used for Figure 1 The system shown in Figure 5 As shown, the method includes:

[0113] Step 502: The audio transmitter obtains audio frames and sampling information of the audio frames.

[0114] In some embodiments, the audio is the audio corresponding to the voice. In some embodiments, the audio can also be the audio of a song, the audio of background music, or the audio of other sounds, and the embodiments of the present application do not limit the specific content of the audio. The audio is acquired by the audio transmitting end and sent to the audio receiving end. Optionally, the audio is recorded by the audio transmitting end, or is locally acquired by the audio transmitting end, or is acquired by the audio transmitting end through other devices, and the embodiments of the present application do not limit the source of the audio.

[0115] The sampling information is used to indicate the sampling position of the original audio frame in the audio, and the sampling position is used to reflect the time domain position occupied by the original audio frame in the audio. Optionally, the sampling information includes the actual starting sampling position and sampling interval of the original audio frame. Optionally, the sampling interval is used to reflect the interval between the actual starting sampling position of the original audio frame and the next audio frame of the original audio frame in the audio. The sampling interval can also be understood as the audio frame duration of the audio frame, and the audio frame duration is the duration occupied by the audio frame in the audio. Optionally, the audio transmitter obtains the sampling information of the audio frame by analyzing the audio frame of the audio.

[0116] Step 504: the audio transmitting end transmits the audio frame and the sampling information of the audio frame to the audio receiving end.

[0117] In some embodiments, audio is transmitted via audio data packets, and the audio data packets carry audio frames of audio, for example, each audio data packet carries an audio frame of audio. In some embodiments, the header of the audio data packet carries the above-mentioned sampling information. In some embodiments, the sampling information includes the actual starting sampling position and sampling interval of the original audio frame, and the sampling information is added to the header of the audio data packet by the audio transmitter, and the timing of adding the actual starting sampling position and sampling interval in the header of the audio data packet by the audio transmitter is the same or different. For example, the audio transmitter first adds basic information related to the audio frame in the header, and the basic information includes the data packet type, the timestamp of the data packet, the sequence number of the data packet, the encoding grid of the data packet, the size of the data packet, and the sampling interval (data duration), and then the audio transmitter adds the actual starting sampling position of the audio frame.

[0118] Step 506: The audio receiving end obtains the original audio frames of the audio and the sampling information of each original audio frame.

[0119] When sending audio, the audio transmitter sends multiple audio frames of the audio, some of which may be lost during the transmission process, for example, due to network fluctuations. Among the audio frames sent by the audio transmitter, the audio frames that can be received by the audio receiver can be called original audio frames, and the audio frames that are lost during the audio receiving process can be called lost audio frames.

[0120] Optionally, the audio receiving end receives an audio data packet sent by the audio sending end, the audio data packet carries audio frames of the audio, the audio frames carried by the audio data packets received by the audio receiving end are the original audio frames, the packet header of the audio data packet carries sampling information, and the audio frames carried by the audio data packets lost during the transmission process are the lost audio frames.

[0121] Step 508: The audio receiving end determines the duration of the lost audio frame according to the sampling position of each original audio frame.

[0122] After receiving each original audio frame and the sampling position corresponding to the original audio frame, the audio receiving end determines the estimated starting sampling position of the i+1th original audio frame based on the sampling position of the i-th original audio frame, where i is a positive integer. Optionally, the sampling information includes the actual starting sampling position and sampling interval of the original audio frame in the audio, and the sampling interval is used to reflect the interval between the actual starting sampling position of the original audio frame and the next audio frame of the original audio frame in the audio. The audio receiving end can determine the estimated starting sampling position of the i+1th original audio frame based on the first sum of the actual starting sampling position of the i-th original audio frame and the sampling interval of the i-th original audio frame. That is, the audio receiving end determines the actual ending sampling position of the i-th original audio frame as the estimated starting sampling position of the i+1th original audio frame.

[0123] In the case where no audio frames are lost during audio transmission, the audio frames should be continuous, that is, the actual end sampling position of the previous audio frame is the actual start sampling position of the next audio frame. If two consecutive original audio frames received by the audio receiving end do not meet the above conditions, it can be considered that there is a lost audio frame between the two original audio frames. Optionally, when the actual start sampling position of the i+1th original audio frame is different from the estimated start sampling position of the i+1th original audio frame, the audio receiving end determines that there is a lost audio frame between the i-th original audio frame and the i+1th original audio frame, and the audio receiving end can determine the duration of the lost audio frame between the i-th original audio frame and the i+1th original audio frame based on the difference between the estimated start sampling position of the i+1th original audio frame and the actual start sampling position of the i+1th original audio frame. Among them, the actual start sampling position of the i+1th original audio frame is determined based on the sampling information of the i+1th original audio frame, and there are one or more lost audio frames between the i-th original audio frame and the i+1th original audio frame. Optionally, the audio receiving end may determine the duration of the lost audio frame between the i-th original audio frame and the i+1-th original audio frame according to the difference between the actual starting sampling position of the i+1-th original audio frame and the above-mentioned first sum.

[0124] For example, Figure 6FIG. 1 is a schematic diagram of an original audio frame provided by an exemplary embodiment of the present application. Figure 6 As shown, the original audio frames received by the audio receiving end include audio frames 601-607, the numbers in the upper left corner of the original audio frame box represent the actual starting sampling position of the original audio frame, and the numbers in the original audio frame box represent the sampling interval of the original audio frame. The calculation rule of the actual starting sampling position of the audio frame is to start from a certain reference time, such as starting from 0, indicating the starting sampling position of the audio, and use the audio frame duration of the current audio frame as the incremental value to determine the actual starting sampling position of the audio frame. For example, for audio frame 601, the current reference position is 0. When the audio frame duration of audio frame 601 is 20ms, the actual starting sampling position of audio frame 601 is 0, and then the reference position will become 20ms. When the audio frame duration of the next audio frame 602 sent is 40ms, then the actual starting sampling position of the audio frame 602 is 20ms, and the reference position will become the sum of the current reference position and the audio frame duration of audio frame 602, that is, 20ms+40ms=60ms. In this way, if there is no packet loss, the actual starting sampling position of a certain audio frame and the audio frame duration can be used to predict the actual starting sampling position of the next audio frame, that is, the sum of the reference position (actual starting sampling position) and the audio frame duration of the audio frame is the actual starting sampling position of the next audio frame. According to the above process, it can be seen that the actual starting sampling position of the next audio frame is determined to be 80ms according to audio frame 603, and the actual starting sampling position of audio frame 604 is 100ms, so it can be seen that 20ms of audio frames are lost between audio frames 603 and 604. The actual starting sampling position of the next audio frame is determined to be 160ms according to audio frame 606, and the actual starting sampling position of audio frame 607 is 200ms, so it can be seen that 40ms of audio frames are lost between audio frames 606 and 607.

[0125] Step 510: The audio receiving end determines a first audio freeze value according to the duration of the lost audio frame.

[0126] The audio receiving end may determine the first audio jam value according to the proportion of the duration of the lost audio frame in the duration of the audio. The audio receiving end may determine the first audio jam value as the audio jam value. The jam value is used to reflect the jam degree of the audio received by the audio receiving end. The above proportion is positively correlated with the audio jam degree, and the audio jam degree is positively correlated or negatively correlated with the jam value.

[0127] Optionally, the audio receiving end determines the audio jam value according to the statistical period T, and the duration of the statistical period T is not greater than the duration of the audio. In each statistical period T, the audio receiving end uses the statistical period T as the duration of the audio, and combines the duration of the lost audio frame in the statistical period T to determine the first jam value of the audio in the statistical period T in the above manner.

[0128] For example, the audio receiving end converts the duration of the lost audio frame in the statistical period T into a value between 0 and 100. The larger the value, the smoother the audio and the lower the degree of stuttering. The calculation definition function for the first stuttering value is: y1=1-x / T, x is the duration of the lost audio frame, T is the statistical period, and y1 is the first stuttering value in the statistical period. For example, if the duration of the lost audio frames in a statistical period is 20ms, 40ms, and 20ms, respectively, then the corresponding first stuttering value is (1-20 / T)×(1-40 / T)×(1-20 / T), and then multiplied by 100, the first stuttering value of the audio in the statistical period can be obtained.

[0129] Step 512: The audio receiving end determines a second audio freeze value according to an occurrence interval of lost audio frames.

[0130] The occurrence interval of lost audio frames is negatively correlated with the audio jam level. That is, the larger the occurrence interval of lost audio frames, the lower the audio jam level. This is because in this case, continuous and smooth audio frames account for a larger proportion of the audio, and the user's hearing experience is better. Optionally, the audio receiving end can determine the second audio jam value based on the proportion of the occurrence interval of lost audio frames in the duration of the audio.

[0131] Optionally, the audio receiving end determines the audio jam value according to the statistical period T, and the duration of the statistical period T is not greater than the duration of the audio. In each statistical period T, the audio receiving end uses the statistical period T as the duration of the audio, and combines the occurrence interval of the lost audio frame in the statistical period T to determine the second jam value of the audio in the statistical period T in the above manner.

[0132] For example, the audio receiving end converts the occurrence interval of lost audio frames in the statistical period T into a value between 0 and 100. The larger the value, the smoother the audio and the lower the degree of stuttering. The calculation definition function for the second stuttering value is: y2=z / T, z is the occurrence interval of lost audio frames, T is the statistical period, and y2 is the second stuttering value in the statistical period. For example, if the occurrence intervals of lost audio frames in a statistical period are 100ms, 20ms, and 50ms, respectively, then the corresponding second stuttering value is (100 / T)×(20 / T)×(50 / T), and then multiplied by 100, you can get the second stuttering value of the audio in the statistical period.

[0133] For determining the occurrence interval of lost audio frames, the audio receiving end can determine the estimated starting sampling position of the i+1th original audio frame based on the sampling position of the i-th original audio frame, and then determine whether the actual starting sampling position of the i+1th original audio frame is different from the estimated starting sampling position of the i+1th original audio frame. This step can be specifically referred to the relevant description in the previous text. This step is used to determine whether a lost audio frame occurs and the position of the lost audio frame. Among them, in the case where there is a lost audio frame (one or more consecutive ones, which can be regarded as one) between the i-th original audio frame and the i+1th original audio frame, the starting position of this part of the lost audio frame determined by the audio receiving end in the audio is the actual ending sampling position of the i-th original audio frame, and the ending position in the audio is the actual starting sampling position of the i+1th original audio frame. Through the above process, the audio receiving end can determine the time domain position of each lost audio frame in the audio.

[0134] When the actual starting sampling position of the i+1th original audio frame is different from the estimated starting sampling position of the i+1th original audio frame, there is a lost audio frame between the i-th original audio frame and the i+1th original audio frame. The audio receiving end can determine the occurrence interval of the lost audio frames between the i-th original audio frame and the i+1th original audio frame according to the position difference between the sampling position of the i-th original audio frame and the sampling position of the target original audio frame. The target original audio frame is the next audio frame of the most recent lost audio frame before the i-th original audio frame, and the next audio frame is the original audio frame. Optionally, if there is no lost audio frame before the i-th original audio frame, the first audio frame of the audio is determined as the target original audio frame at this time. Optionally, the audio receiving end determines the occurrence interval of the lost audio frames between the i-th original audio frame and the i+1th original audio frame according to the position difference between the second sum and the actual starting sampling position of the target original audio frame. The second sum is the sum of the actual starting sampling position and the sampling interval of the i-th original audio frame.

[0135] Step 514: The audio receiving end determines the audio freeze value according to the first audio freeze value and the second audio freeze value.

[0136] The audio receiving end determines the value reflecting the maximum audio jamming degree between the first jamming value and the second jamming value as the audio jamming value, or determines the value reflecting the minimum audio jamming degree as the audio jamming value, or determines the average of the first jamming value and the second jamming value as the audio jamming value. The audio receiving end can also determine the second jamming value as the audio jamming value.

[0137] Optionally, the audio receiving end also obtains the energy value of the original audio frame, which is determined based on the maximum value of each sampling point of the original audio frame. The numerical value of the sampling point of the audio frame represents the sound information of the audio sampled through the sampling point. The energy value of the audio frame can measure the loudness of the audio frame when it is played. Generally, the larger the energy value, the more effective information the audio frame contains, and the easier it is for the user to perceive it after it is lost. Optionally, the audio receiving end obtains the energy value of the original audio frame by analyzing the received original audio frame. Alternatively, the energy value of the original audio frame is sent by the audio sending end, for example, carried in the packet header of the audio data packet of the original audio frame.

[0138] When the energy values ​​of the original audio frames of the audio are all lower than the energy value threshold, the audio receiving end will determine the audio jam value as the jam value with the least jamming degree. The reason is that if the energy values ​​of the original audio frames of the audio are all lower than the energy value threshold, it can be considered that the audio contains less effective information and is not easily perceived by the user. At this time, even if the second audio frame is lost, the audio can be considered to be smooth. Among them, the above energy value threshold is set manually.

[0139] Optionally, the audio received by the audio receiving end is one channel of audio in a multi-channel audio, and the multi-channel audio can be transmitted simultaneously. For example, in a multi-person conference, each participant corresponds to one voice (audio), and each participant will receive the voice of other participants. The audio receiving end can determine the stuttering value of each audio channel through the above steps, and then determine the stuttering value of the multi-channel audio according to the weighted average of the stuttering value of each audio channel in the multi-channel audio. Among them, the weight of the stuttering value of the n-th audio channel is determined according to the energy value of the n-th audio channel, and the energy value of the n-th audio channel is determined according to the energy value of the original audio frame of the n-th audio channel, for example, the average value of the energy values ​​of the original audio frames of the n-th audio channel, and n is a positive integer.

[0140] For example, Figure 7 FIG. 1 is a schematic diagram of a process for determining a jamming value of an audio channel provided by an exemplary embodiment of the present application. Figure 7 As shown, in step A1, after receiving multiple channels of audio, the audio receiving end starts to determine the jam value for one channel of audio. In step A2, the audio receiving end determines the duration of the lost audio (data loss) lost in the one channel of audio. In step A3, the audio receiving end determines the occurrence interval of the lost audio lost in the one channel of audio (data discontinuity occurs). In step A4, the audio receiving end determines the first jam value according to the duration of the lost audio, determines the second jam value according to the occurrence interval of the lost audio, and determines the jam value of the audio according to the first jam value and the second jam value.

[0141] For example, Figure 8FIG. 1 is a schematic diagram of a process for determining a jam value of multi-channel audio provided by an exemplary embodiment of the present application. Figure 8 As shown, in step B1, the audio receiving end determines the jamming value of the first audio channel through the above steps. In step B2, the audio receiving end determines the jamming values ​​of the remaining audio channels through the above steps. In step B3, the audio receiving end calculates the weighted average of the jamming values ​​of each downlink audio channel, thereby obtaining the jamming values ​​of multiple audio channels.

[0142] In some embodiments, after determining the audio jamming value, the audio receiving end can display the determined audio jamming value.

[0143] In some embodiments, when the audio receiving end determines that there are lost audio frames in the audio through the above method, it can determine the duration and time domain position of the lost audio frame, and add the duration to the first duration of the first o original audio frames of the lost audio frame and the second duration of the last p original audio frames of the lost audio frame to obtain the target duration and send it to the server, and send the o original audio frames and the p original audio frames to the server. After that, the server will intercept the audio segments of the target duration from each audio in the audio database, and calculate the first similarity between the segment of the first duration of the head of each audio segment and the above o original audio frames, and calculate the second similarity between the segment of the second duration of the tail of the audio segment and the above p original audio frames. When the first similarity and the second similarity are both greater than the similarity threshold, the server determines that the segment between the segment of the first duration of the head and the segment of the second duration of the tail in the audio segment is the target audio segment, and sends it to the audio receiving end. When playing the audio, the audio receiving end will play the target audio segment between the o original audio frames and the p original audio frames. Among them, o and p are set manually, the similarity threshold is set manually, and the audio database is established by manually collecting various audios. The target audio segment is obtained through the server. Since the target audio segment is obtained through similarity calculation, it may be the segment corresponding to the lost audio frame. Inserting the target audio segment when playing the audio can ensure smooth audio playback and the content is consistent with the lost audio frame.

[0144] In some embodiments, the audio received by the audio receiving end is speech. When the audio receiving end determines that there is a lost audio frame in the audio through the above method, the duration and time domain position of the lost audio frame can be determined, and the first u original audio frames of the lost audio frame and the last v original audio frames of the lost audio frame can be obtained. Then, speech recognition is performed on the u original audio frames to obtain the first text content of the u original audio frames, and speech recognition is performed on the v original audio frames to obtain the second text content of the v original audio frames. Among them, u and v are manually set. After that, the audio receiving end inputs the duration of the lost audio frame, the first text content, the second text content and the guide information (prompt) into the large language model (LargeLanguage Model, LLM) to obtain the predicted text content predicted by the large language model. Among them, the guide information is used to guide the large language model to predict the text content between the first text content and the second text content, and the pronunciation duration meets the duration of the lost audio frame. For example, the duration of the lost audio frame is 2s, the first text content is "tomorrow", and the second text content is "how is it, is it hotter than today", and the guidance information is "please predict what the text content between "tomorrow" and "how is it, is it hotter than today" may be that requires "2s" to pronounce", and the predicted text content output by the large language model is "weather". After obtaining the predicted text content, the audio receiving end will display the predicted text content and prompt information when playing the audio, and the prompt information is used to prompt the role and source of the predicted text content. By way of example, the prompt information is "the content displayed here is used to predict the content lost when the audio is stuck, and the model prediction is for reference only". Through the above scheme, in the voice call scenario, the text content of the lost audio frame can be predicted according to the context of the lost audio frame and displayed to the user, thereby avoiding the situation where the user cannot know the complete content of the voice call when the audio frame is lost, thereby improving the user experience.

[0145] The above-mentioned large language model is an artificial intelligence model designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc. LLM is characterized by its large scale and billions of parameters, which help them learn complex patterns in language data. In some embodiments, the disease prediction model in this application is a large language model that supports natural language generation.

[0146] In summary, the method provided in this embodiment analyzes the duration of the lost audio frames lost during audio transmission according to the sampling information of the received original audio frames during the process of determining the audio stuttering value, and determines the audio stuttering value based on this. Since the audio stuttering value is determined based on the duration of the lost audio frames, rather than the number of lost audio frames, the audio stuttering value is determined based on the duration of the lost audio frames, so even if the duration of the transmitted audio frames changes, the accuracy of the determined audio stuttering value can still be guaranteed.

[0147] The method provided in this embodiment can also accurately estimate the duration of the lost audio by inferring the estimated starting sampling position of the i+1th original audio frame based on the sampling position of the i+1th original audio frame, and determining the duration of the corresponding lost audio frame when the actual starting sampling position of the i+1th original audio frame is different from the estimated starting sampling position.

[0148] The method provided in this embodiment also determines the duration of the lost audio frame according to the actual starting sampling position and sampling interval of the audio frame sent by the audio sending end. The audio receiving end can simply and conveniently determine the duration of the lost audio frame through the received information.

[0149] The method provided in this embodiment also provides a way to measure the degree of audio stuttering based on the duration of lost audio frames by determining the audio stuttering value based on the proportion of the duration of lost audio frames in the duration of the audio.

[0150] The method provided in this embodiment also determines the audio stuttering value based on the proportion of the duration of the lost audio frames in the duration of the audio and the interval at which the lost audio frames appear in the audio, providing a way to measure the degree of audio stuttering based on both the frame loss duration and the frame loss interval. The accuracy of measuring the degree of audio stuttering can be improved through more measurement dimensions.

[0151] The method provided in this embodiment can also accurately determine the occurrence interval of lost audio by determining the occurrence interval of lost audio frames based on the position difference between the sampling position of the i-th original audio frame and the sampling position of the target original audio frame when determining the occurrence of lost audio frames.

[0152] The method provided in this embodiment also determines the occurrence interval of lost audio frames according to the actual starting sampling position and sampling interval of the audio frame sent by the audio sending end. The audio receiving end can simply and conveniently determine the occurrence interval of lost audio frames through the received information.

[0153] The method provided in this embodiment also considers the influence of the energy value of the original audio frame when determining the audio stuttering value. Therefore, when the audio frame is lost but contains less valid information, the stuttering value can be determined as the stuttering value with the minimum degree of stuttering, thereby avoiding the deviation between the user's actual hearing experience and the determined stuttering value.

[0154] The method provided in this embodiment also determines the jamming value of multiple audio channels according to the weighted average of the jamming value of each audio channel in the multiple audio channels, thereby providing an implementation method that can measure the jamming degree of multiple audio channels.

[0155] In a specific example, Fig. 9 FIG. 1 is a schematic diagram of a process of playing audio provided by an exemplary embodiment of the present application. Fig. 9 As shown, in the related art, in steps C1-C5, the audio collected by the remote user is first processed by 3A to filter out the echo and noise in the collected data, and the sound gain is adjusted, and then the audio is encoded, and the packet header is added before it is uploaded to the server. In steps C6-C7, the audio data is first put into the jitter buffer after being pulled down from the server by the near-end user. The jitter buffer will be used to estimate the jitter to decide the acceleration and deceleration of the audio data packet. In steps C8-C10, the near-end user reads the remote audio frame from the jitter buffer, then decodes it, and performs the acceleration and deceleration processing of the playback according to the current estimation of the jitter buffer, and finally sends it to the speaker for playback.

[0156] Fig.10 FIG. 1 is a schematic diagram of a process of playing audio provided by an exemplary embodiment of the present application. Fig.10 As shown, compared to Fig. 9 In the process, steps D5 and D9 are added. In step D5, when the far-end audio frame is uplinked, the far-end user will supplement the fields in the packet header and add the data timestamp (actual starting sampling position) and energy value field. In step D9, when the audio is downlinked, the near-end user performs audio jam statistics before decoding the audio frame, mainly identifying the jam scene through the data timestamp and energy value, that is, determining the audio jam value.

[0157] The supplementary header part in the above step D5 is used to add the above two fields to the audio data packet header, which is used as a reference for audio jam statistics. After the audio data packet is encoded, a data packet header is added to the data packet to describe the data information of the current data packet. Table 1 shows the common header format.

[0158] Table 1

[0159] Audio packet header Package Type Timestamp Serial Number Encoding format Packet size Data duration 。。。

[0160] The calculation rule of the data timestamp is to start from a certain reference time, such as starting from 0, and use the data duration of the current audio data packet as the incremental value for calculation. For example, if the current reference timestamp is 0, when a 20ms audio data packet is sent, the data timestamp of the current audio data packet is 0, and then the reference timestamp will become 20ms. When the next audio data packet sent is 40ms, the data timestamp of the audio data packet will be 20ms, and the reference timestamp will become the sum of the current reference timestamp and the duration of the current audio data packet, that is, 20ms+40ms=60ms. In this way, if there is no packet loss, the data timestamp of the current audio data packet can be used to predict the data timestamp of the next audio data packet, that is, the sum of the current data timestamp and the length of the current data packet is the data timestamp of the next packet.

[0161] The energy value of an audio data packet is used to measure the loudness of an audio data packet. Generally, the larger the energy value, the more effective information the audio data packet contains, and the easier it is for the user to perceive it after it is lost. The energy value is calculated by obtaining the maximum data value in the audio frame. For example, an audio frame is regarded as a string of 16-bit values, and the maximum value among them is taken as the energy value of the audio frame.

[0162] For jamming statistics, the near-end user will first perform jamming statistics for each audio channel, and then perform jamming value statistics for all downlinks based on the jamming value of each channel to obtain the audio jamming level of the current user.

[0163] The jamming evaluation of single-channel audio mainly refers to the data timestamp and energy value. The method for the near-end user to determine the first jamming value through the data timestamp is to use the data timestamp and data duration of the current audio data packet to calculate the data timestamp of the next audio data packet without losing the audio data packet. For example, the data timestamp of the current audio data packet is N1, and the data duration is M1, then the data timestamp of the next audio data packet should be N1+M1. When the next audio data packet is received, if the data timestamp is N1+M1, it is considered that there is no data loss at present. If the data timestamp of the next audio data packet is N2, which is not equal to N1+M1, then the duration of the lost audio data packet is N2-(N1+M1), which means jamming occurs. At this time, it is necessary to record the duration of the lost audio data packet and the interval between the loss of the audio data packet. For example, an audio data packet such as Figure 6As shown. Then the data duration of the lost audio data packets is 100–60–20=20ms, 200–140–20=40ms. The data discontinuity interval duration (the interval of the lost audio frame) is 60+20(–0)=80ms, 140+20–100=60ms. The values ​​are recorded in a list respectively. At this time, the results of data loss statistics are 20ms and 40ms, and the results of data discontinuity statistics are 80ms and 60ms. The discontinuity interval of the statistics is because even if the audio data packets of the same length are lost, if the time interval is different, the user's hearing experience will be different. For example, if 20ms of data is lost every 100ms and 160ms of data is lost every 960ms, and if 20ms of data is lost every 20ms and 160ms of data is lost every 320ms, the latter scenario will feel more stuck. The calculation process of the stuck value for single-channel audio is as follows.

[0164] (1) Obtain data loss information, discontinuity information, and audio energy value of the audio channel according to a fixed statistical period T.

[0165] (2) The data loss information of the statistical period is converted into a data value between 0 and 100. The larger the value, the smoother the video. A jamming function (first jamming value) is defined for the duration of the lost data. For example, y = 1-x / T, where x is the duration of the lost data and T is the statistical period. For example, if the data lost in a statistical period are 20ms, 40ms, and 20ms respectively, then the corresponding first jamming value is (1-20 / T)×(1-40 / T)×(1-20 / T). Then multiply it by 100 to obtain the first jamming value of the data loss dimension.

[0166] (3) Convert the result of data discontinuity statistics into a data value between 0 and 100, and define a jamming function (second jamming value) for the duration of data discontinuity, such as y2=z / T, where z is the duration of discontinuity and T is the statistical period. For example, if the duration of discontinuous data in a statistical period is 100ms, 20ms, and 50ms, respectively, then the corresponding second jamming value is (100 / T)×(20 / T)×(50 / T), and then multiply it by 100 to get the second jamming value of the data discontinuity dimension.

[0167] (4) The minimum value between (2) and (3) is taken as the jamming value of the current statistical period.

[0168] (5) The energy value of the statistical period is counted. If the audio frame with low energy value, especially the audio frame with 0 energy value, is lost, the impact on the user's hearing experience is small. Therefore, an energy value threshold can be defined. When the energy value of the audio data packet in the statistical period is lower than the threshold, it is considered that there is no jamming in the statistical period, and the jamming value is 100 points.

[0169] The statistical rules for the jamming values ​​of all audio channels are as follows.

[0170] (1) Calculate the energy value and jamming value of all audio channels according to the statistical period. The statistical period needs to be consistent with the statistical period of a single channel, which is also T.

[0171] (2) The energy value of each audio channel is used as the weight of the jamming value, and the average jamming value of all audio channels is calculated. For example, there is a data sequence e1, s1, e2, s2, ..., en, sn, e represents the energy value, s represents the jamming value of an audio channel, then the jamming value of all audio channels s = (e1*s1+e2*s2+...+en*sn) / (e1+e2+...+en). In addition, if the energy values ​​of all audio channels are currently 0, then the jamming values ​​of all audio channels are 100.

[0172] The above method provided in the embodiment of the present application can obtain the audio jam value concisely and accurately, and can present it in numerical form. In addition, the jam value in the case of multi-channel audio can also be accurately counted.

[0173] It should be noted that before collecting relevant user data (such as audio in this application) and during the process of collecting relevant user data, this application can display a prompt interface, pop-up window or output voice prompt information. The prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining user-related data after obtaining the user's confirmation operation on the prompt interface or pop-up window, otherwise (that is, when the user's confirmation operation on the prompt interface or pop-up window is not obtained), the relevant steps of obtaining user-related data are terminated, that is, the user's relevant data is not obtained. In other words, all user data collected by this application are collected with the user's consent and authorization, and the collection, use and processing of relevant user data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0174] It should be noted that the sequence of the method steps provided in the embodiments of the present application can be appropriately adjusted, and the steps can be increased or decreased accordingly according to the circumstances. Any technician familiar with the technical field can easily think of the changed methods within the technical scope disclosed in the present application, and they should be included in the protection scope of the present application, so they will not be repeated here.

[0175] Fig.11 FIG. 1 is a schematic diagram of a structure of a device for determining an audio freeze value provided by an exemplary embodiment of the present application. Fig.11 As shown, the device comprises:

[0176] The receiving module 1101 is used to receive original audio frames of audio and sampling information of each original audio frame, where the sampling information is used to indicate a sampling position of the original audio frame in the audio;

[0177] A determination module 1102 is used to determine the duration of a lost audio frame of the audio according to a sampling position of each original audio frame, where the lost audio frame is an audio frame lost during the process of receiving the audio;

[0178] The determination module 1102 is further used to determine a jamming value of the audio according to the duration of the audio and the duration of the lost audio frame, wherein the jamming value is used to reflect the jamming degree of the received audio.

[0179] In an optional design, the determining module 1102 is used to:

[0180] Determine the estimated starting sampling position of the i+1th original audio frame according to the sampling position of the i-th original audio frame, where i is a positive integer;

[0181] When the actual starting sampling position of the i+1th original audio frame is different from the estimated starting sampling position of the i+1th original audio frame, determining the duration of the lost audio frame between the i+1th original audio frame and the i+1th original audio frame according to the difference between the estimated starting sampling position of the i+1th original audio frame and the actual starting sampling position of the i+1th original audio frame;

[0182] The actual starting sampling position of the i+1th original audio frame is determined according to the sampling information of the i+1th original audio frame.

[0183] In an optional design, the sampling information includes an actual starting sampling position and a sampling interval of the original audio frame in the audio, and the sampling interval is used to reflect the interval between the original audio frame and the actual starting sampling position of the next audio frame of the original audio frame in the audio; the determining module 1102 is used to:

[0184] Determine an estimated starting sampling position of the (i+1)th original audio frame according to a first sum of an actual starting sampling position of the (i)th original audio frame and a sampling interval of the (i)th original audio frame;

[0185] The duration of the lost audio frame between the i-th original audio frame and the i+1-th original audio frame is determined according to the difference between the actual starting sampling position of the i+1-th original audio frame and the first sum.

[0186] In an optional design, the determining module 1102 is used to:

[0187] According to the proportion of the duration of the lost audio frame in the duration of the audio, a first jamming value of the audio is determined as the jamming value of the audio, and the proportion is positively correlated with the jamming degree of the audio.

[0188] In an optional design, the determining module 1102 is used to:

[0189] Determine a second jamming value of the audio according to an occurrence interval of the lost audio frames, wherein the occurrence interval is negatively correlated with the jamming degree of the audio;

[0190] A freeze value of the audio is determined according to a first freeze value of the audio and a second freeze value of the audio.

[0191] In an optional design, the determining module 1102 is used to:

[0192] Determine the estimated starting sampling position of the i+1th original audio frame according to the sampling position of the i-th original audio frame, where i is a positive integer;

[0193] When the actual starting sampling position of the i+1th original audio frame is different from the estimated starting sampling position of the i+1th original audio frame, determining an occurrence interval of lost audio frames between the i-th original audio frame and the i+1th original audio frame according to a position difference between a sampling position of the i-th original audio frame and a sampling position of a target original audio frame;

[0194] The target original audio frame is the next audio frame of the most recent lost audio frame before the i-th original audio frame.

[0195] In an optional design, the sampling information includes an actual starting sampling position and a sampling interval of the original audio frame in the audio; the determining module 1102 is used to:

[0196] Determine, according to the position difference between the second sum and the actual starting sampling position of the target original audio frame, an occurrence interval of lost audio frames between the i-th original audio frame and the i+1-th original audio frame;

[0197] The second sum is the sum of the actual starting sampling position and the sampling interval of the i-th original audio frame.

[0198] In an alternative design, such as Fig.12 As shown, the device also includes:

[0199] An acquisition module 1103 is used to acquire an energy value of the original audio frame, where the energy value of the original audio frame is determined according to a maximum value of each sampling point of the original audio frame;

[0200] The determining module 1102 is configured to determine, when the energy values ​​of the original audio frames of the audio are all lower than the energy value threshold, that the stuttering value of the audio is the stuttering value with the minimum stuttering degree.

[0201] In an optional design, the audio is one of multiple audio channels; the determining module 1102 is used to:

[0202] Determine the jam value of the multiple audio channels according to the weighted average value of the jam value of each audio channel in the multiple audio channels;

[0203] The weight of the jamming value of the n-th audio channel is determined according to the energy value of the n-th audio channel, the energy value of the n-th audio channel is determined according to the energy value of the original audio frame of the n-th audio channel, and n is a positive integer.

[0204] In an optional design, the receiving module 1101 is used to:

[0205] An audio data packet of the audio is received, where the audio data packet carries an audio frame of the audio, and a packet header of the audio data packet carries the sampling information.

[0206] Fig.13 FIG. 1 is a schematic diagram of a structure of a device for determining an audio freeze value provided by an exemplary embodiment of the present application. Fig.13 As shown, the device comprises:

[0207] An acquisition module 1301 is used to acquire an audio frame of an audio and sampling information of the audio frame, where the sampling information is used to indicate a sampling position of the audio frame in the audio;

[0208] A sending module 1302 is used to send the audio frame of the audio and the sampling information of the audio frame to the audio receiving end, and the audio receiving end is used to determine the duration of the lost audio frame of the audio according to the sampling position of each original audio frame, and determine the jamming value of the audio according to the duration of the audio and the duration of the lost audio frame;

[0209] The original audio frame is an audio frame of the received audio, the lost audio frame is an audio frame lost during the process of receiving the audio, and the jamming value is used to reflect the jamming degree of the received audio.

[0210] It should be noted that the device for determining the audio stutter value provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device for determining the audio stutter value provided in the above embodiment and the method for determining the audio stutter value belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0211] An embodiment of the present application also provides a computer device, which includes: a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the method for determining the audio stutter value provided in the above-mentioned method embodiments.

[0212] In some embodiments, the computer device is a terminal. Fig.14 It is a schematic diagram of the structure of a terminal provided by an exemplary embodiment of the present application.

[0213] Typically, the terminal 1400 includes a processor 1401 and a memory 1402 .

[0214] The processor 1401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1401 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0215] The memory 1402 may include one or more computer-readable storage media, which may be non-transitory. The memory 1402 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1402 is used to store at least one instruction, which is used to be executed by the processor 1401 to implement the method for determining the audio stutter value provided in the method embodiment of the present application.

[0216] In some embodiments, the terminal 1400 may further optionally include: a peripheral device interface 1403 and at least one peripheral device. The processor 1401, the memory 1402 and the peripheral device interface 1403 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1403 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1404, a display screen 1405, a camera assembly 1406, an audio circuit 1407 and a power supply 1408.

[0217] The peripheral device interface 1403 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1401 and the memory 1402. In some embodiments, the processor 1401, the memory 1402, and the peripheral device interface 1403 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1401, the memory 1402, and the peripheral device interface 1403 may be implemented on a separate chip or circuit board, which is not limited in the embodiments of the present application.

[0218] The radio frequency circuit 1404 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1404 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1404 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1404 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1404 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1404 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0219] The display screen 1405 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1405 is a touch display screen, the display screen 1405 also has the ability to collect touch signals on the surface or above the surface of the display screen 1405. The touch signal can be input to the processor 1401 as a control signal for processing. At this time, the display screen 1405 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1405 can be one, and the front panel of the terminal 1400 is set; in other embodiments, the display screen 1405 can be at least two, which are respectively set on different surfaces of the terminal 1400 or are folded; in some other embodiments, the display screen 1405 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 1400. Even, the display screen 1405 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1405 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0220] The camera assembly 1406 is used to capture images or videos. Optionally, the camera assembly 1406 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal 1400, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1406 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0221] The audio circuit 1407 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1401 for processing, or input them into the radio frequency circuit 1404 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1400. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1401 or the radio frequency circuit 1404 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1407 may also include a headphone jack.

[0222] The power supply 1408 is used to power various components in the terminal 1400. The power supply 1408 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1408 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0223] In some embodiments, the terminal 1400 further includes one or more sensors 1409 . The one or more sensors 1409 include, but are not limited to: an acceleration sensor 1410 , a gyroscope sensor 1411 , a pressure sensor 1412 , an optical sensor 1413 , and a proximity sensor 1414 .

[0224] The acceleration sensor 1410 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 1400. For example, the acceleration sensor 1410 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1401 can control the touch display screen 1405 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1410. The acceleration sensor 1410 can also be used for collecting game or user motion data.

[0225] The gyro sensor 1411 can detect the body direction and rotation angle of the terminal 1400, and the gyro sensor 1411 can cooperate with the acceleration sensor 1410 to collect the user's 3D actions on the terminal 1400. The processor 1401 can implement the following functions based on the data collected by the gyro sensor 1411: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0226] The pressure sensor 1412 can be set on the side frame of the terminal 1400 and / or the lower layer of the touch display screen 1405. When the pressure sensor 1412 is set on the side frame of the terminal 1400, it can detect the user's holding signal of the terminal 1400, and the processor 1401 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 1412. When the pressure sensor 1412 is set on the lower layer of the touch display screen 1405, the processor 1401 controls the operability controls on the UI interface according to the user's pressure operation on the touch display screen 1405. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0227] The optical sensor 1413 is used to collect the ambient light intensity. In one embodiment, the processor 1401 can control the display brightness of the touch display screen 1405 according to the ambient light intensity collected by the optical sensor 1413. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 1405 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 1405 is reduced. In another embodiment, the processor 1401 can also dynamically adjust the shooting parameters of the camera assembly 1406 according to the ambient light intensity collected by the optical sensor 1413.

[0228] The proximity sensor 1414, also called a distance sensor, is usually arranged on the front panel of the terminal 1400. The proximity sensor 1414 is used to collect the distance between the user and the front of the terminal 1400. In one embodiment, when the proximity sensor 1414 detects that the distance between the user and the front of the terminal 1400 is gradually decreasing, the processor 1401 controls the touch display screen 1405 to switch from the screen-on state to the screen-off state; when the proximity sensor 1414 detects that the distance between the user and the front of the terminal 1400 is gradually increasing, the processor 1401 controls the touch display screen 1405 to switch from the screen-off state to the screen-on state.

[0229] Those skilled in the art will understand that Fig.14 The structure shown in the figure does not constitute a limitation on the terminal 1400, and the terminal 1400 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0230] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores at least one instruction, at least one program, code set or instruction set. When the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor of a computer device, the method for determining the audio stutter value provided in the above-mentioned method embodiments is implemented.

[0231] The present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method for determining the audio stutter value provided in the above-mentioned method embodiments.

[0232] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned readable storage medium may be a read-only memory, a disk or an optical disk, etc.

[0233] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent switching, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining an audio stutter value, It is characterized in that The method is applied to an audio receiving end, and the method comprises: Receive original audio frames of audio and sampling information of each original audio frame, where the sampling information is used to indicate a sampling position of the original audio frame in the audio; Determine, according to the sampling position of each original audio frame, the duration of a lost audio frame of the audio, wherein the lost audio frame is an audio frame lost during the process of receiving the audio; A jamming value of the audio is determined according to the duration of the audio and the duration of the lost audio frame, where the jamming value is used to reflect the jamming degree of the received audio.

2. The method according to claim 1, It is characterized in that The determining, according to the sampling position of each original audio frame, the duration of the lost audio frame of the audio comprises: Determine the estimated starting sampling position of the i+1th original audio frame according to the sampling position of the i-th original audio frame, where i is a positive integer; When the actual starting sampling position of the i+1th original audio frame is different from the estimated starting sampling position of the i+1th original audio frame, determining the duration of the lost audio frame between the i+1th original audio frame and the i+1th original audio frame according to the difference between the estimated starting sampling position of the i+1th original audio frame and the actual starting sampling position of the i+1th original audio frame; The actual starting sampling position of the i+1th original audio frame is determined according to the sampling information of the i+1th original audio frame.

3. The method according to claim 2, It is characterized in that The sampling information includes an actual starting sampling position and a sampling interval of the original audio frame in the audio, wherein the sampling interval is used to reflect the interval between the actual starting sampling position of the original audio frame and the next audio frame of the original audio frame in the audio; The step of determining the estimated starting sampling position of the (i+1)th original audio frame according to the sampling position of the i-th original audio frame comprises: Determine an estimated starting sampling position of the (i+1)th original audio frame according to a first sum of an actual starting sampling position of the (i)th original audio frame and a sampling interval of the (i)th original audio frame; The determining, according to a difference between an estimated starting sampling position of the i+1th original audio frame and an actual starting sampling position of the i+1th original audio frame, a duration of a lost audio frame between the i-th original audio frame and the i+1th original audio frame comprises: The duration of the lost audio frame between the i-th original audio frame and the i+1-th original audio frame is determined according to the difference between the actual starting sampling position of the i+1-th original audio frame and the first sum.

4. The method according to any one of claims 1 to 3, It is characterized in that The determining the audio jam value according to the duration of the audio and the duration of the lost audio frame includes: According to the proportion of the duration of the lost audio frame in the duration of the audio, a first jamming value of the audio is determined as the jamming value of the audio, and the proportion is positively correlated with the jamming degree of the audio.

5. The method according to claim 4, It is characterized in that The method further comprises: Determine a second jamming value of the audio according to an occurrence interval of the lost audio frames, wherein the occurrence interval is negatively correlated with the jamming degree of the audio; A freeze value of the audio is determined according to the first freeze value of the audio and the second freeze value of the audio.

6. The method according to claim 5, It is characterized in that The method further comprises: Determine the estimated starting sampling position of the i+1th original audio frame according to the sampling position of the i-th original audio frame, where i is a positive integer; When the actual starting sampling position of the i+1th original audio frame is different from the estimated starting sampling position of the i+1th original audio frame, determining an occurrence interval of lost audio frames between the i-th original audio frame and the i+1th original audio frame according to a position difference between a sampling position of the i-th original audio frame and a sampling position of a target original audio frame; The target original audio frame is the next audio frame of the most recent lost audio frame before the i-th original audio frame.

7. The method according to claim 6, It is characterized in that The sampling information includes the actual starting sampling position and sampling interval of the original audio frame in the audio; The determining, according to the position difference between the sampling position of the i-th original audio frame and the sampling position of the target original audio frame, an occurrence interval of lost audio frames between the i-th original audio frame and the i+1-th original audio frame comprises: Determine, according to the position difference between the second sum and the actual starting sampling position of the target original audio frame, an occurrence interval of lost audio frames between the i-th original audio frame and the i+1-th original audio frame; The second sum is the sum of the actual starting sampling position and the sampling interval of the i-th original audio frame.

8. The method according to any one of claims 1 to 3, It is characterized in that The method further comprises: Acquire an energy value of the original audio frame, where the energy value of the original audio frame is determined according to a maximum value of each sampling point of the original audio frame; When the energy values ​​of the original audio frames of the audio are all lower than the energy value threshold, the jamming value of the audio is determined to be the jamming value with the minimum jamming degree.

9. The method according to any one of claims 1 to 3, It is characterized in that The audio is one of multiple audio channels; the method further includes: Determine the jam value of the multiple audio channels according to the weighted average value of the jam value of each audio channel in the multiple audio channels; The weight of the jamming value of the n-th audio channel is determined according to the energy value of the n-th audio channel, the energy value of the n-th audio channel is determined according to the energy value of the original audio frame of the n-th audio channel, and n is a positive integer.

10. A method for determining an audio stutter value, It is characterized in that The method is applied to an audio transmitting end, and the method comprises: Acquire an audio frame of an audio and sampling information of the audio frame, where the sampling information is used to indicate a sampling position of the audio frame in the audio; Sending the audio frame of the audio and the sampling information of the audio frame to an audio receiving end, the audio receiving end is used to determine the duration of the lost audio frame of the audio according to the sampling position of each original audio frame, and determine the jamming value of the audio according to the duration of the audio and the duration of the lost audio frame; The original audio frame is an audio frame of the received audio, the lost audio frame is an audio frame lost during the process of receiving the audio, and the jamming value is used to reflect the jamming degree of the received audio.

11. A device for determining an audio stutter value, It is characterized in that The device comprises: A receiving module, used for receiving original audio frames of audio and sampling information of each original audio frame, wherein the sampling information is used for indicating a sampling position of the original audio frame in the audio; A determination module, configured to determine the duration of a lost audio frame of the audio according to a sampling position of each original audio frame, wherein the lost audio frame is an audio frame lost during the process of receiving the audio; The determination module is further used to determine a jamming value of the audio according to the duration of the audio and the duration of the lost audio frame, wherein the jamming value is used to reflect the jamming degree of the received audio.

12. A device for determining an audio stutter value, It is characterized in that The device comprises: An acquisition module, used to acquire an audio frame of an audio and sampling information of the audio frame, wherein the sampling information is used to indicate a sampling position of the audio frame in the audio; A sending module, used to send the audio frame of the audio and the sampling information of the audio frame to an audio receiving end, wherein the audio receiving end is used to determine the duration of the lost audio frame of the audio according to the sampling position of each original audio frame, and determine the jamming value of the audio according to the duration of the audio and the duration of the lost audio frame; The original audio frame is an audio frame of the received audio, the lost audio frame is an audio frame lost during the process of receiving the audio, and the jamming value is used to reflect the jamming degree of the received audio.

13. A computer device, It is characterized in that The computer device includes a processor and a memory, wherein at least one program is stored in the memory, and the at least one program is loaded and executed by the processor to implement the method for determining the audio stuttering value according to any one of claims 1 to 9, or the method for determining the audio stuttering value according to claim 10.

14. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the method for determining the audio stutter value according to any one of claims 1 to 9, or the method for determining the audio stutter value according to claim 10.

15. A computer program product, It is characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method for determining an audio stutter value as described in any one of claims 1 to 9, or the method for determining an audio stutter value as described in claim 10.

Images