Speech Quality Assessment System

Abstract

The present invention provides a new audio quality assessment system, which includes an assessment system running on a receiving system in a VoIP communication system. The new audio quality assessment system can accurately estimate the MOS value of a VoIP call in a time window. The audio quality assessment system determines the effective PLC count, PLC impact factor, effective AS count, AS impact factor, network impact factor, codec type of the received voice packet, bit rate of the received voice packet, and initial MOS value extracted from a configured codec-bit rate MOS table, and accurately estimates the MOS value based on this data. Based on the effectively collected statistical data of the receiving system module and the pre-configured codec-bit rate MOS table, the MOS estimation value can be determined more accurately and efficiently.

Description

[0001] Cross - reference to related applications

[0002] This application claims priority to U.S. Patent Application No. 17211968, filed on March 25, 2021. Technical field

[0003] The present invention generally relates to the field of real - time communication, and more specifically to the field of VoIP systems. More particularly, the present invention relates to an audio quality assessment system that can accurately estimate the mean opinion score (MOS) of VoIP calls in a VoIP communication system. Background art

[0004] Voice over Internet Protocol (VoIP) refers to a series of technologies for voice communication and multimedia sessions over an Internet Protocol (IP) network, such as the Internet. The data of VoIP session participants is sent to the electronic devices of other participants, such as smartphones, tablets, laptops, or other types of mobile or portable electronic devices. Participants can also use desktop computers to join VoIP sessions. The receiving - end electronic device usually receives voice data (also referred to as audio data in this article) in the form of data packets and plays the voice data to the corresponding participants.

[0005] Due to various reasons such as packet jitter and packet loss, the received voice data may not be the same as the voice data at the sending end. Playing back the received voice data will make the listener feel that the listening experience is damaged. Characterizing the degree of listening experience damage at the receiving end without any reference has always been an unresolved problem in the industry. A common method for characterizing listening experience damage is to monitor the state of the current network, such as packet loss rate and jitter duration, within a given time period (or time window). Then, the state of the underlying connection network is used to estimate the mean opinion score (MOS). MOS is a commonly used metric in the field of quality of experience (QoE) and telecommunications engineering. It represents the overall quality of the system, especially the quality of audio communication. It is usually a subjective quality evaluation method. People usually adjust the underlying technology or system to make the estimated MOS value as close as possible to the reference MOS obtained from monitors or observers. In this method, the estimated MOS is not reliable or accurate because there is no direct relationship between the network state and human hearing. In addition, this traditional method is invasive. Summary of the invention

[0006] Therefore, if you need to accurately know the audio quality in a real-time voice communication system (such as a VoIP system), a new system and method are required. The new system proposed by the present invention is a parameter-based non-intrusive audio quality assessment system that evaluates the audio quality by analyzing the audio data at each time interval (also referred to as a time window in this article) at the receiving end, so as to more accurately estimate the MOS value. The estimated MOS value can be used to monitor the quality of each voice call and diagnose problems with the quality of experience (QoE) in the real-time voice communication system.

[0007] Generally speaking, the present invention provides a computer-implemented method for determining the MOS of a VoIP call in a VoIP communication system based on various embodiments. The VoIP communication system has a sending system and a receiving system. The method includes starting a MOS estimation program on the receiving system at the beginning stage of a VoIP call time window. The receiving system includes a processor; a memory adapted to the processor; an audio output interface adapted to the processor; a network interface adapted to the processor; an audio input interface adapted to the processor; and an operating system run by the processor. The method further includes: determining a set of VoIP call statistics for the VoIP call; determining the PLC impact factor for the VoIP call; determining a first MOS impact factor based on the PLC impact factor and the PLC count in the set of VoIP call statistics; determining the AS impact factor for the VoIP call; determining a second MOS impact factor based on the AS impact factor and the AS count in the set of VoIP call statistics; determining the network impact factor; determining a third MOS impact factor based on the network impact factor; determining an initial MOS based on the codec type and bit rate in the set of VoIP call statistics, and a set of codec-bit rate MOS reference values; and determining an MOS estimated value for the VoIP call based on the initial MOS, the first MOS impact factor, the second MOS impact factor, and the third MOS impact factor. In one embodiment, the first MOS impact factor is the product of the PLC impact factor and the PLC count; the second MOS impact factor is the product of the AS impact factor and the AS count. The method may further include: determining whether a data packet received from the sending system is a voice data packet; when it is determined that the received data packet is a voice data packet, setting the voice flag to "on"; determining the bit rate after the voice flag is set to "on"; increasing the PLC count when the voice flag is "on" after a PLC operation; and increasing the AS count when the voice flag is "on" after an AS operation.

[0008] In addition, according to the present invention, an audio quality evaluation system is also provided for determining the MOS value of a VoIP call on a VoIP communication system having a sending system and a receiving system. The audio quality evaluation system starts the estimation process of MOS in the receiving system at the beginning stage of the time window of the VoIP call. The receiving system includes a processor; a memory adapted to the processor; an audio output interface adapted to the processor; a network interface adapted to the processor; an audio input interface adapted to the processor; and an operating system run by the processor. The audio quality evaluation system can also perform the following operations: determining a set of VoIP call statistics of the VoIP call; determining the PLC impact factor of the VoIP call; determining the first MOS impact factor according to the PLC impact factor and the PLC count in the set of VoIP call statistics; determining the AS impact factor of the VoIP call; determining the second MOS impact factor according to the AS impact factor and the AS count in the set of VoIP call statistics; determining the network impact factor; determining the third MOS impact factor according to the network impact factor; determining the initial MOS according to the codec type and bit rate in the set of VoIP call statistics and a set of codec-bit rate MOS reference values; and determining the MOS estimated value of the VoIP call according to the initial MOS, the first MOS impact factor, the second MOS impact factor, and the third MOS impact factor. In one embodiment, the first MOS impact factor is the product of the PLC impact factor and the PLC count; the second MOS impact factor is the product of the AS impact factor and the AS count. The audio quality evaluation system can also determine whether the data packet received from the sending system is a voice data packet; when it is determined that the received data packet is a voice data packet, set the voice identifier to "on"; determine the bit rate after setting the voice identifier to "on"; increase the PLC count when the voice identifier is "on" after PLC operation; and increase the AS count when the voice identifier is "on" after AS operation. Description of the Drawings

[0009] This patent or application document contains at least one color drawing. The Patent Office will provide a copy of this patent or patent application publication with color drawings upon request and payment of the relevant fees.

[0010] The technical features of the present invention will be specifically pointed out in the claims. At the same time, the present invention and its constitution and usage can be better understood by referring to the following description and the drawings of the description that form a part of the description. In all the drawings of the present invention, the same reference numerals represent the same components:

[0011] Figure 1 is a schematic diagram of an existing VoIP communication system.

[0012] Figure 2 It is a schematic diagram of a receiving device in an existing VoIP communication system.

[0013] Figure 3 It is a schematic diagram of an improved receiving device in a VoIP communication system according to the present invention.

[0014] Figure 4 It is a flowchart according to the present invention, which shows the process by which a VoIP receiving system with a new audio quality assessment system determines the statistical data of a VoIP communication system based on various factors affecting the listening feeling.

[0015] Figure 5 It is a schematic diagram of the operation of a receiving device in a VoIP communication system according to the present invention.

[0016] Figure 6 It is a flowchart showing the process by which an audio quality assessment system accurately estimates the MOS value of a VoIP call in a no-reference state according to the present invention.

[0017] Figure 7 It is a flowchart showing the process of determining a codec-bitrate MOS table according to the present invention.

[0018] Figure 8 It is a flowchart showing the process by which an evaluation configuration system determines the influencing factors of a module according to the present invention.

[0019] Figure 9 It is a flowchart showing the process by which an evaluation configuration system determines the network influencing factors according to the present invention.

[0020] Figure 10 It is a matrix diagram according to the present invention, which shows the relationship between network influencing factors, packet loss probability and packet recovery rate in a VoIP communication system.

[0021] Those of ordinary skill in the art should understand that, in order to simply and clearly show the above-mentioned drawings, the components in the drawings are not necessarily drawn to scale. The sizes of some components in the drawings may be enlarged relative to other components to help understand the present invention. In addition, the specific order of certain elements, parts, components, modules, steps, operations, events and / or processes described or illustrated herein may not be necessary in actual applications. Those of ordinary skill in the art should understand that, for the sake of simplicity and clarity, those well-known and easily understandable useful and / or necessary elements in commercially available and feasible implementation schemes may not be described herein in order to clearly present the various implementation schemes of the present invention. Detailed implementation manners

[0022] Figure 1FIG. 0 is a schematic diagram of a VoIP communication system in the prior art, which is generally denoted by 100. The system 100 includes electronic devices 120 and 122 participating in the communication, both of which can communicate via a network (such as the Internet) 110. Each electronic device participating in the communication can send and receive audio and / or video data. When it sends data outward, it is called a sending device or a sending end in the present invention. Similarly, when it receives data, it is called a receiving device or a receiving end in the present invention. The electronic devices 120-122 can be desktop computers, laptop computers, tablet computers, or smart phones with network connection functions, etc.

[0023] The sending device 120 includes an audio recording (such as a microphone) device 102, an audio processing module 104, an audio encoding module 106, and a network data packet sending module 108 that sends audio and / or video data packets to the receiving device 122 through a network interface. The audio processing module 104 can implement functions such as echo cancellation and noise reduction at the sending end. The receiving device 122 includes a network data packet receiving module 112, a NetEQ module 114, and an audio playback module 116. The audio playback module 116 can play the received audio data through an audio output interface (such as a speaker controller and a speaker). The NetEQ module 114 can maintain a dynamic jitter buffer and run an error concealment algorithm to eliminate the negative impacts brought by network jitter and packet loss. Thus, it can maintain a relatively high voice quality as much as possible while minimizing the delay. Figure 2 The receiving device 122 will be further elaborated. The NetEQ module 114 includes an audio data packet decision module 202, a decoding module 204, an acceleration module 206, a deceleration module 208, and an audio data packet loss concealment (PLC) module 210. Modules 206-208 are collectively referred to as the AS module in the present invention, where A represents acceleration (Accelerate), and S represents deceleration (Slow down or Slow).

[0024] The quality of a VoIP call is mainly affected by five major factors. The first type of factor is the voice volume recorded by the audio recording device 102. If the volume is not large enough, then the listener at the receiving end can hardly perceive the coding loss and / or network loss of the voice. The second type of factor affecting the VoIP call quality at the receiving end is the audio data coding loss caused by the codec. The audio data is recorded by the audio recording device 102 and can be processed by the audio processing module 104. There are two main codec factors for evaluating the VoIP call quality loss caused by the codec. One is the type of the codec, and the other is the coding bit rate of the voice data (such as 16 kbps). Different codecs (such as Opus) have different coding efficiencies. For the same codec, the higher the bit rate, the smaller the coding distortion of the voice, and the higher the MOS. The MOS value of each codec at each different bit rate can be obtained through an offline configuration program.

[0025] The third type of factor affecting the VoIP call quality at the receiving end is the network condition, which can corrupt the voice data sent by the audio data packet sending device 108 (also referred to as network corruption in the present invention). The communication network 110 often has the possibility of becoming unstable. In this case, data packets may be lost or delayed during transmission through the network 110. Theoretically, if the data packets arrive at the receiving end 122 without any loss or delay, it can be considered that the audio quality of the VoIP call is only affected by encoding. Once network problems occur, the data packets may fail to reach the receiving end, arrive late, or experience jitter. In this case, the audio quality at the receiving end will be impaired.

[0026] The fourth type of factor affecting the VoIP call quality at the receiving end is the network problem coping strategy executed by the receiving device 122. The VoIP system usually adopts some modules (such as modules 206 - 210) to reduce the impact of network problems on the receiving end. If there are no network problems, the acceleration module 206, the deceleration module 208, and the packet loss concealment module 210 are not run. Conversely, one or more of the three modules 206 - 210 will be run to accelerate the audio, slow down the audio, and / or perform packet loss compensation to reduce the impact of packet delay and / or loss. Therefore, the output of these modules will directly affect the final voice signal quality. Correspondingly, by directly monitoring the working states of each module, the MOS estimation value can more accurately reflect the listening experience of the audio receiver at the receiving device 122 for the VoIP system. The fifth type of factor affecting the VoIP call quality at the receiving end includes operations such as echo cancellation and noise reduction performed by the audio processing module 104 on the sending device 120.

[0027] The non-intrusive evaluation method based on VoIP audio parameters for determining the MOS of a VoIP communication system generally estimates the MOS value according to the third type of factor among the above five types of influencing factors. Such methods include the P.1201 standard, a parametric non-intrusive evaluation for the quality of audiovisual media streams. The P.1201 standard is maintained by the International Telecommunication Union (“ITU”). However, the network-related parameters relied on by the P.1201 method generally do not affect the listening experience of the listener for the VoIP system. Therefore, the audio MOS value estimated by the P.1201 method is accurate only in a few network cases. In addition, in addition to packet loss, out-of-order packets also have a great impact on the MOS estimation value.

[0028] When the packet delay changes, the VoIP system at the receiving end calls and runs the acceleration and / or deceleration module to increase and / or decrease the consumption rate of voice packets. This operation will have a negative impact on the listening experience of the receiving party. The more frequently the PLC and AS modules are run, the worse the output voice quality at the receiving end. The actual impact of packet out-of-order on the listening experience of the receiving end is difficult to reflect at the parameter level. This means that traditional general VoIP audio non-intrusive evaluation methods (such as P.1201) are difficult to accurately estimate the MOS value of a VoIP call.

[0029] The present invention overcomes the disadvantages and deficiencies of traditional methods for estimating the MOS value of a VoIP communication system. In particular, the non-intrusive audio quality evaluation method and system provided by the present invention can monitor a series of parameters that are more closely related to audio quality, thereby improving the accuracy of estimation. The new system and method provided by the present invention are based on the internal working state of the VoIP communication system for evaluation. Compared with traditional methods, the new system and method provided by the present invention consider almost all factors affecting voice quality, so that the MOS value can be estimated more accurately. The VoIP working state includes codec type, the bit rate received by the decoder, the volume of the decoded frame, the number of frames generated by the PLC module, the number of frames affected by the AS module, the packet loss model, etc. The new system and method provided by the present invention monitor the states of various modules (such as modules 204 - 210) in the VoIP communication system, rather than just the state of the communication network 110.

[0030] Figures 3 - 10 The new system and method provided by the present invention will be further described. Figure 3 FIG. shows a schematic diagram of the new receiving system of the present invention, which is generally designated as 300. The new receiving system 300 includes a processor, a memory adapted to the processor, an audio output interface adapted to the processor; a network interface adapted to the processor, an audio input interface adapted to the processor, and an operating system run by the processor. The new receiving system 300 may further include a video output interface adapted to the processor and a video input interface adapted to the processor.

[0031] The improved new VoIP receiving system further includes a non-intrusive audio quality evaluation system 302. In one embodiment, the evaluation system 302 is a computer software application program that can be run by the processor. The evaluation system 302 evaluates the audio quality of a VoIP call according to various statistical data of the VoIP system modules at the receiving end. In another embodiment, the evaluation system 302 communicates with a server (such as a cloud server) system 304 through the Internet 110. In this case, the evaluation system 302 provides evaluation data and results to the server system 304. Figure 4 The new receiving system 300 and the evaluation system 302 will be further described.

[0032] Figure 4 is a flowchart according to the present invention, which shows the process of a VoIP receiving system with a new audio quality assessment system determining VoIP communication system statistics based on various factors affecting the listening experience. The overall process is represented as 400. At 402, the packet receiving device 112 receives one or more packets. For ease of reference, this will be briefly described as "receiving packets at 402" herein. Codec information can be obtained from the received packets. The packet receiving device 112 obtains the codec type and other codec information. Then, the decoding module 204 uses the codec data to decode the received packets at 412.

[0033] At 404, for the received packets, the decision module 202 will decide what operation to perform on the packets. The operations can be the packet decoding operation performed by the decoding module 204 at 404, the PLC operation performed by the PLC module 210 at 406, the acceleration operation performed by the acceleration module 206 at 422, or the deceleration operation performed by the deceleration module 208 at 422. After the PLC operation, at 408, the evaluation system 302 determines whether the voice flag is "on". For example, the binary value TRUE represents "yes" (on), and FALSE represents "no" (off). If the voice flag is "on", then at 410, the evaluation system 302 will increment the count of the effective PLC module. Otherwise, no operation will be performed (also referred to as NOP in the present invention).

[0034] After decoding the packets at 412, at 414, the evaluation system 302 determines whether the packet is a voice packet. If so, the evaluation system 302 sets the voice flag to "on" at 416. Otherwise, the evaluation system 302 performs NOP, that is, the evaluation system 302 does not perform any operation on the packet. To determine whether a packet is a voice packet, one criterion is whether the volume meets the standard. The volume is compared with a predetermined voice volume threshold to determine whether the volume is sufficient. In one embodiment, the volume of a constant voice signal is adjusted upward from zero. When the received voice is clearly audible, the corresponding volume is set as the threshold.

[0035] At 418, the evaluation system 302 estimates the bit rate of the received voice data packet. For example, the evaluation system 302 uses the bit stream length of the data packet to determine the encoding bit rate of the voice data packet. After the acceleration module 206 or the deceleration module 208 operates on the received data packet, at 424, the evaluation system 302 determines whether the voice identifier for the current time window is "on". If the status is "on", then at 426, the evaluation system 302 will increment the acceleration count or the deceleration count respectively. Otherwise, the evaluation system 302 executes a NOP. In one embodiment, the acceleration count and the deceleration count are the same count, and when either the acceleration module 206 or the deceleration module 208 is executed, it is incremented. In other words, if the AS module is executed on the data packet at 422, then at 426, the valid AS module count will be incremented.

[0036] It should be noted that the audio quality evaluation is performed in a given time window. For example, the time window can be five seconds or two minutes. At the start of each evaluation time window, the voice identifier is initialized to the "off" state, the PLC count is initialized to 0, the acceleration count is initialized to 0, and the deceleration count is initialized to 0. When the acceleration count and the deceleration count are combined into the same AS count, the AS count is initialized to 0 at the start of each evaluation time window. The PLC count, the acceleration count, the deceleration count, and the AS count are referred to herein as the NetEQ state. The NetEQ state is also referred to herein as VoIP module statistics, NetEQ module status, and NetEQ module statistics.

[0037] Figure 5 The operating steps of the new receiving system 300 are further elaborated, and all the steps are collectively represented as 500. 502 represents the time line in the time window. Column 504 indicates a series of operations of the NetEQ modules 204 - 210 along the time line 502. Columns 506 and 508 indicate the corresponding operations performed by the evaluation system 302. Along the time line 502, the NetEQ modules 204 - 210 perform operations 510 - 532. For each such operation, the evaluation system 302 performs the corresponding operations shown in columns 506 - 508.

[0038] In operations 510 - 512, 516, 528 - 530, the voice data packet is decoded. Correspondingly, the evaluation system 302 sets the voice identifier to on and updates the bit rate at 418. For operation 514, the acceleration is valid acceleration. Therefore, it will be indicated that the acceleration is valid and the acceleration count (or the AS count) is incremented. In one embodiment, when the voice identifier is "on", then in the present invention, operation 514 is referred to as an effective operation or a valid operation that affects the listening sense; otherwise, it is referred to as an ineffective operation or a useless operation that does not affect the listening sense.

[0039] In operations 518 and 522, non-voice data packets are decoded. Correspondingly, the evaluation system 302 sets the voice flag to "off". For operation 520, the deceleration operation is exactly an ineffective deceleration operation. In this case, the evaluation system 302 indicates that the deceleration operation is ineffective and does not perform additional operations on this data packet. In one embodiment, when the voice flag is set to "on", operation 520 is called an effective deceleration operation or an effective deceleration operation that affects the listening perception; otherwise, it is called an ineffective deceleration operation or a useless deceleration operation that affects the listening perception.

[0040] For operations 524 - 526, they are ineffective PLC operations. In this case, the evaluation system 302 indicates ineffectiveness and no longer performs operations related to the corresponding data packet. In one embodiment, when the voice flag is "on", operations 524 - 526 are called effective PLC operations or effective PLC operations that affect the listening perception; otherwise, they are called ineffective PLC operations or useless PLC operations that do not affect the listening perception. In contrast, operation 532 is an effective PLC operation. In this case, the evaluation system 302 indicates that it is effective and increments the PLC count.

[0041] Figure 6 The no-reference audio quality assessment is further elaborated. Figure 6 A flowchart showing the process by which the audio quality assessment system 302 accurately estimates the MOS value of a VoIP call is shown, and this process is generally represented as 600. At 602, the evaluation system 302 starts the estimation process at the beginning of the time window. For example, the evaluation system 302 initializes the effective PLC count to 0, the effective AS module count to 0, sets the voice flag to "off", and sets the estimated bit rate to 0, and starts the estimation process. At 604, the evaluation system 302 determines a set of VoIP call statistics. For example, at 604, the evaluation system 302 runs process 400 to obtain VoIP statistics such as codec type, bit rate, PLC count, and AS count data. At 606, the evaluation system 302 determines the first influencing factor. In one embodiment, the first influencing factor is the PLC influencing factor. At 608, the evaluation system 302 determines the first MOS influencing factor based on the first influencing factor and the first statistic in the set of VoIP statistics (such as the effective PLC count). The effective PLC count is the final value at the end of the time window. In one embodiment, the first MOS influencing factor IF1 is determined according to the following formula:

[0042] IF1 = m * g1

[0043] Where m is the PLC count and g1 is the PLC impact factor. At 610, the evaluation system 302 determines a second impact factor. In one embodiment, the second impact factor is the AS impact factor. At 612, the evaluation system 302 determines a second MOS impact factor based on the second impact factor and the second VoIP statistic (such as the valid AS count) in the set of VoIP statistics. The valid AS count is the final value at the end of the time window. In one embodiment, the second MOS impact factor IF2 is determined by the following formula:

[0044] IF2 = n * g2

[0045] Where n is the AS count and g2 is the AS impact factor. At 614, the evaluation system 302 determines a network impact factor. At 616, the evaluation system 302 derives a third MOS impact factor (denoted as IF3 herein) based on the network impact factor. At 618, the evaluation system 302 determines a set of codec-bitrate MOS reference values. For example, this set of values can be a table of values, where the codec type and bitrate are respectively the dimensions or number axes in the table. For a specific codec type and a specific bitrate, the corresponding MOS reference value is located within the codec-bitrate MOS table. It should be noted that step 618 is typically performed outside the current time window. It is a configuration process that is usually carried out separately.

[0046] At 620, based on the codec type and bitrate determined by process 400, the evaluation system 302 determines an initial MOS value, i.e., MOS initial . At 622, the evaluation system 302 determines a MOS estimate based on the initial MOS and the first, second, and third MOS impact factors. In one embodiment, a more accurate MOS estimate can be determined by the following formula:

[0047] MOS estimate = MOS initial * (1 – IF1 – IF2) * IF3

[0048] Figure 7 The process of determining the codec-bitrate MOS table at 618 is further illustrated. Figure 7A flowchart showing the process of determining the codec-bitrate MOS table is presented, and the overall process is denoted as 700. At 702, the network of the VoIP call does not cause network impairment to the voice quality. At 704, the evaluation configuration system selects a codec from a set of codec types. In one implementation, the evaluation configuration system is a separate configuration or test system for determining the codec-bitrate MOS table as well as the PLC, AS, and network impact factors. The evaluation configuration system includes a software application. In the present invention, the evaluation configuration system and the evaluation system 302 are collectively referred to as the audio quality evaluation system.

[0049] At 706, the evaluation configuration system selects a bitrate from a set of bitrates. At 708, the evaluation configuration system determines the MOS value of the VoIP call using the selected codec type and bitrate. In one implementation, the MOS value is obtained based on the POLQA standard, which is a standard for evaluating the voice quality over IP networks. The International Telecommunication Union standardized POLQA as Recommendation P.863. Alternatively, the MOS value can also be obtained using a subjective voice quality assessment method.

[0050] For all combinations of a set of codec types and a set of bitrates, step 708 is repeatedly executed. At 710, in the codec-bitrate MOS table, each MOS value is associated with the corresponding codec type and bitrate.

[0051] Figure 8 A process for determining the PLC impact factor and the AS impact factor is shown. Figure 8 A flowchart showing the process by which the evaluation configuration system determines the PLC impact factor and the AS impact factor is presented, and the overall process is denoted as 800. At 802, the evaluation configuration system selects a codec as the transmitting-end codec. At 804, the bitrate used for encoding the audio data is selected. At 806, the evaluation configuration system uses a specific model (such as the second-order Gilbert-Elliot model) to simulate network problems. In this model, the packet loss probability p is randomly set, while the recovery rate r is set to 1. At 808, the evaluation configuration system sends test audio to the receiving device. At 810, the evaluation configuration system records the effective PLC count m x and the effective AS count n x . At 812, the evaluation configuration system obtains the MOS value (MOS test ) of the test according to the P.863 standard. At 814, the evaluation configuration system locates the MOS reference value (MOS max ) from the codec-bitrate MOS table using the selected codec and the selected bitrate. At 816, the evaluation configuration system obtains the distortion rate of the test. For example, the following formula can be used to determine the distortion rate:

[0052] DR x = MOS test / MOS max

[0053] The distortion rate and the influence factors conform to the following relationship:

[0054] (1 - m x * g1 - n x * g2) = DR x

[0055] Where x represents the test. At 818, the evaluation configuration system repeatedly applies this test for different parameters (such as different code types and bitrates). After two or more tests, at 820, the evaluation configuration system calculates the PLC influence factor and the AS influence factor by means of operations (such as algebraic operations).

[0056] Figure 9 The process of determining the network factor is shown. Figure 9 The flowchart showing the process by which the evaluation configuration system obtains the network influence factor is shown, and this process is represented as 900 as a whole. To obtain the network influence factor, the evaluation configuration system first needs to determine whether the current network is mainly burst packet loss or mainly random packet loss. Burst packet loss means consecutive packet loss, while random loss means discontinuous frame loss. With the same effective PLC count, different packet loss types will bring different auditory sensations to the listeners at the receiving end. Therefore, it is necessary to adjust the predicted MOS value according to the packet loss state. In the present invention, p represents the packet loss probability and r represents the packet recovery rate.

[0057] In the test mentioned in process 800, after determining the influence factors g1 and g2, at 902, the evaluation configuration system gradually sets the simulated network problem to burst packet loss, that is, randomly sets p, and at the same time reduces r in the Gilbert - Elliot model to approach 0, such as 0.01. At 904, the evaluation configuration system sends test audio materials. At 906, the evaluation configuration system records the effective PLC module count m1 and the AS module count n1. At 908, the evaluation configuration system determines the MOS value of the test according to the P.863 standard (MOS test ). At 910, the evaluation configuration system locates the MOS reference value (MOS max ) from the codec - bitrate MOS table according to the selected codec and the selected bitrate. At 912, for the given p and r values, the evaluation configuration system determines the network influence factor according to the following formula:

[0058] IF3 = MOS max * (1 – m1 * g1 – n1 * g2) / MOS test

[0059] Figure 10 The relationship between IF3 and p and r is further shown. Figure 10 A schematic diagram illustrating the relationship between IF3 and p and r is shown, and is generally designated as 1000. The r-axis, p-axis, and IF3-axis are represented by 1002, 1004, and 1006 respectively.

[0060] Based on the above description, it is obvious that the present invention can have many other modifications and variations. Therefore, it should be noted that within the scope of the appended claims, the present invention can be implemented in a manner different from the above specific description. For example, some VoIP communication systems may not have some of the above modules, or may include additional modules. When a module does not exist in the VoIP call system, the influence of the module on the final output voice quality will not exist. When the VoIP system has additional modules, the additional modules will affect the final output voice quality. Using the present invention, the influencing factors of the additional modules can also be quantified and integrated into the reference-free audio quality assessment system. In this case, the additional modules will appear as paired elements (such as 406 - 408, 410 - 412).

[0061] The above description of the present invention is for better illustration and explanation, and is not intended to be exclusive or to limit the present invention to the above specific form. The above description is for better explaining the principles of the present invention and the practical applications of these principles, so that those skilled in the relevant art can best utilize the present invention to implement various embodiments and make various modifications in the intended appropriate uses. It should be recognized that words such as "a" or "an" in this article include both singular and plural forms. On the contrary, in appropriate cases, the situation of multiple elements mentioned in this article should also include their singular forms.

[0062] The scope of the present invention is not limited only to the content of the above specification, but is determined by the claims. In addition, although the claims proposed may have a relatively narrow scope, it should be recognized that the scope of the present invention is much broader than the scope proposed by the claims. We will propose claims with a broader scope in one or more applications claiming the priority of this application. If the part of the content disclosed in the above specification and drawings is not included within the scope of the claims, then the said invention content is not publicly disclosed, and we reserve the right to file one or more patent applications for the above additional invention content in the future.

Claims

1. A computer-implemented method for determining the MOS estimation value of a VoIP call in a VoIP communication system, the VoIP communication system including a sending system and a receiving system, the method comprising: 1) At the start stage of the VoIP call time window, start a MOS estimation program on the receiving system, the receiving system including: (a) A processor; (b) A memory adapted to the processor; (c) An audio output interface adapted to the processor; (d) A network interface adapted to the processor; (e) An audio input interface adapted to the processor; and (f) An operating system run by the processor; 2) Determine a set of VoIP call statistics for the VoIP call; 3) Determine the PLC impact factor of the VoIP call; 4) Determine a first MOS impact factor based on the PLC impact factor and the PLC count in the set of VoIP call statistics; 5) Determine the AS impact factor of the VoIP call; 6) Determine a second MOS impact factor based on the AS impact factor and the AS count in the set of VoIP call statistics; 7) Determine the network impact factor; 8) Determine a third MOS impact factor based on the network impact factor; 9) Determine an initial MOS value based on the codec type and bit rate in the set of VoIP call statistics, and a set of codec-bit rate MOS reference values; and 10) Determine the MOS estimation value of the VoIP call based on the initial MOS value, the first MOS impact factor, the second MOS impact factor, and the third MOS impact factor.

2. The computer-implemented method for determining the MOS estimation value of a VoIP call in a VoIP communication system according to claim 1, wherein: The first MOS impact factor is the product of the PLC impact factor and the PLC count; and The second MOS impact factor is the product of the AS impact factor and the AS count.

3. The computer-implemented method for determining the MOS estimation value of a VoIP call in a VoIP communication system according to claim 1, wherein the MOS estimation value is determined by the following formula: MOS estimated value = MOS initial *(1 – IF1 – IF2)*IF3 where MOS initial represents the initial MOS value described above, IF1 represents the first MOS impact factor, IF2 represents the second MOS impact factor, and IF3 represents the third MOS impact factor.

4. The computer-implemented method for determining the MOS estimation value of a VoIP call in a VoIP communication system according to claim 1, further comprising: 1) Determine whether the data packet received from the sending system is a voice data packet; 2) When it is determined that the received data packet is a voice data packet, set the voice flag to "on"; 3) Determine the bit rate after setting the voice flag to "on"; 4) After PLC operation, when the voice flag is "on", increment the PLC count; and 5) After AS operation, when the voice flag is "on", increment the AS count.

5. An audio quality evaluation system for determining the MOS estimation value of a VoIP call in a VoIP communication system, wherein the audio quality evaluation system includes a sending system and a receiving system, and the audio quality evaluation system can perform the following operations: 1) At the start stage of the time window of the VoIP call, start the MOS estimation process in the receiving system, and the receiving system includes: (a) A processor; (b) A memory adapted to the processor; (c) An audio output interface adapted to the processor; (d) A network interface adapted to the processor; (e) An audio input interface adapted to the processor; And (f) An operating system run by the processor; 2) Determine a set of VoIP call statistics of the VoIP call; 3) Determine the PLC impact factor of the VoIP call; 4) Determine the first MOS impact factor according to the PLC impact factor and the PLC count in the set of VoIP call statistics; 5) Determine the AS impact factor of the VoIP call; 6) Determine the second MOS impact factor according to the AS impact factor and the AS count in the set of VoIP call statistics; 7) Determine the network impact factor; 8) Determine the third MOS impact factor according to the network impact factor; 9) Determine the initial MOS value according to the codec type and bit rate in the set of VoIP call statistics, and a set of codec-bit rate MOS reference values; And 10) Determine the MOS estimation value of the VoIP call according to the initial MOS value, the first MOS impact factor, the second MOS impact factor, and the third MOS impact factor.

6. The audio quality evaluation system according to claim 5, wherein: 1) The first MOS impact factor is the product of the PLC impact factor and the PLC count; and 2) The second MOS impact factor is the product of the AS impact factor and the AS count.

7. The audio quality evaluation system according to claim 5, wherein the MOS estimation value is determined by the following formula: MOS Estimated Value = MOS initial *(1 – IF1 – IF2)*IF3 Among them, MOS initial represents the initial MOS value, IF1 represents the first MOS influence factor, IF2 represents the second MOS influence factor, and IF3 represents the third MOS influence factor.

8. The audio quality evaluation system according to claim 5, wherein the audio quality evaluation system can also perform the following operations: 1) Determine whether the data packet received from the sending system is a voice data packet; 2) When it is determined that the received data packet is a voice data packet, set the voice flag to "on"; 3) Determine the bit rate after setting the voice flag to "on"; 4) After the PLC operation, when the voice flag is "on", increment the PLC count; and 5) After the AS operation, when the voice flag is "on", increment the AS count.

Images