Method for transmitting and receiving voice data and related apparatus

Abstract

The application discloses a voice data sending and receiving method and related equipment; a to-be-encoded data sequence can be acquired; for each voice data in the to-be-encoded data sequence, associated voice data of the voice data is determined; the voice data and the corresponding associated voice data are redundantly encoded to obtain an encoded data sequence, the encoded data sequence comprises a plurality of encoded voice data packets having a time sequence relationship, each encoded voice data packet comprises a payload field and a redundancy field, the payload field contains the voice data, and the redundancy field contains compressed associated voice data; and the voice data packets in the encoded data sequence are sent to a network side in a predetermined order. The application can guarantee the anti-packet loss capability of a transmission network and effectively improve the utilization rate of network bandwidth.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a method for transmitting and receiving voice data and related equipment. Background Technology

[0002] Voice calls are a common communication method in people's daily lives, and the quality of voice calls directly affects the user's communication experience. Among them, packet loss is a significant factor affecting voice call quality. Due to interference from factors such as network fluctuations, data packets, including voice data, may be lost during transmission.

[0003] In current related technologies, this problem is generally solved by packet loss retransmission technology. Packet loss retransmission is a technical means that when the data receiver detects that the target data packet has timed out and has not been received or finds that the target received packet is erroneous, it sends a request packet to the data sender to request the data sender to retransmit the erroneous data packet.

[0004] Existing packet loss retransmission mechanisms generally require each data packet to receive a status confirmation, which means sending and receiving confirmation messages. This operation consumes a considerable amount of network bandwidth resources. If the data receiver still fails to receive the correct data packet after the first retransmission, the data sender needs to continue retransmitting until it succeeds. In cases of weak network capabilities, each data packet may go through this process, which can lead to excessive consumption of network bandwidth resources and hinder data transmission. Summary of the Invention

[0005] This application provides a method for sending and receiving voice data and related equipment. The related equipment may include a voice data sending and receiving device, electronic equipment, and storage medium. This method can ensure the transmission network's ability to withstand packet loss, and at the same time, it can effectively improve the utilization rate of network bandwidth by compressing the associated voice data.

[0006] This application provides a method for transmitting voice data, including:

[0007] Obtain a sequence of data to be encoded, the sequence of data to be encoded including multiple speech data that are to be encoded and have a temporal relationship;

[0008] For each piece of speech data, determine the associated speech data, which is the speech data in the data sequence to be encoded whose temporal relationship with the speech data satisfies a preset association condition;

[0009] For each voice data, the voice data and its corresponding associated voice data are redundantly encoded to obtain an encoded data sequence, the encoded data sequence comprising a plurality of encoded voice data packets having a time sequence relationship, the encoded voice data packet comprising a payload field and a redundancy field, the payload field containing the voice data, and the redundancy field containing compressed associated voice data;

[0010] The voice data packets in the encoded data sequence are sent to the network side in a predetermined order.

[0011] Embodiments of the present application provide a voice data receiving method, comprising:

[0012] The encoded voice data packet in the encoded data sequence is received, the encoded voice data packet comprising a payload field and a redundancy field, the payload field containing voice data, and the redundancy field containing compressed associated voice data, the associated voice data being voice data that meets a preset association condition in terms of time sequence relationship with the voice data;

[0013] The encoded voice data packet is decoded, and data packet loss detection is performed based on the decoding result;

[0014] When it is detected that there is data packet loss, a target associated voice data packet associated with the lost voice data packet in the encoded data sequence is determined;

[0015] The target associated voice data packet is redundantly decoded to extract compressed associated voice data from the redundancy field of the target associated voice data packet;

[0016] According to the compressed associated voice data, data recovery is performed on the voice data in the payload field of the lost voice data packet.

[0017] Correspondingly, embodiments of the present application provide a voice data sending device, comprising:

[0018] An obtaining unit is configured to obtain a to-be-encoded data sequence, the to-be-encoded data sequence comprising a plurality of to-be-encoded voice data having a time sequence relationship;

[0019] A first determining unit is configured to determine, for each voice data, associated voice data of the voice data, the associated voice data being voice data in the to-be-encoded data sequence that meets a preset association condition in terms of time sequence relationship with the voice data;

[0020] The encoding unit is configured to redundantly encode, for each piece of voice data, the voice data and its corresponding associated voice data to obtain an encoded data sequence, wherein the encoded data sequence comprises a plurality of encoded voice data packets having a time sequence relationship, and each encoded voice data packet comprises a payload field and a redundancy field, wherein the payload field contains the voice data, and the redundancy field contains compressed associated voice data.

[0021] The sending unit is configured to send the voice data packets in the encoded data sequence to the network side in a predetermined order.

[0022] Correspondingly, the application provides a voice data receiving device, which comprises:

[0023] The receiving unit is configured to receive the encoded voice data packets in the encoded data sequence, wherein each encoded voice data packet comprises a payload field and a redundancy field, wherein the payload field contains voice data, and the redundancy field contains compressed associated voice data, and the associated voice data is voice data that satisfies a preset association condition in terms of time sequence relationship with the voice data.

[0024] The decoding unit is configured to decode the encoded voice data packets and perform packet loss detection of data transmission based on the decoding result.

[0025] The second determining unit is configured to determine a target associated voice data packet associated with the lost voice data packet in the encoded data sequence when it is detected that there is packet loss.

[0026] The redundancy decoding unit is configured to redundantly decode the target associated voice data packet to extract the compressed associated voice data from the redundancy field of the target associated voice data packet.

[0027] The recovery unit is configured to perform data recovery of the voice data in the payload field of the lost voice data packet according to the compressed associated voice data.

[0028] Optionally, in some embodiments of the application, the voice data sending device further comprises a generating unit and a fusion unit, which are configured as follows:

[0029] The generating unit is configured to generate redundancy encoding information according to the packet loss information sent by the network side.

[0030] The fusion unit is configured to fuse the redundancy encoding information and the voice data to obtain at least one redundancy data packet, wherein the redundancy data packet comprises a redundancy fusion field, and the redundancy fusion field contains fused information of the redundancy encoding information and the voice data.

[0031] The sending unit can be specifically configured to add the redundant data packet to the encoded data sequence, and send the voice data packet in the encoded data sequence to the network side in a predetermined order.

[0032] Optionally, in some embodiments of the present application, the generating unit can include an obtaining subunit and a constructing subunit, as follows:

[0033] The obtaining subunit is configured to obtain redundant data structure information according to the packet loss information sent by the network side.

[0034] The constructing subunit is configured to construct redundant encoding information according to the redundant data structure information.

[0035] Optionally, in some embodiments of the present application, the encoding unit can include an encoding subunit, a setting subunit and a compressing subunit, as follows:

[0036] The encoding subunit is configured to encode the voice data to obtain an initial encoded voice data packet, wherein the initial encoded voice data packet includes a payload field, and the payload field includes encoded voice data.

[0037] The setting subunit is configured to set a redundant field in the initial encoded voice data packet.

[0038] The compressing subunit is configured to compress associated voice data corresponding to the voice data, and add the compressed associated voice data to the redundant field to generate an encoded voice data packet.

[0039] Optionally, in some embodiments of the present application, the encoded data sequence further includes a redundant data packet; the redundant data packet includes a redundant fusion field, and the redundant fusion field contains redundant encoding information and fused information of voice data.

[0040] The voice data receiving device can further include a redundant data packet decoding unit and a data recovery unit, as follows:

[0041] The redundant data packet decoding unit is configured to decode the redundant data packet to extract fused information from a redundant fusion field of the redundant data packet when the target associated voice data packet fails in redundant decoding.

[0042] The data recovery unit is configured to perform data recovery on voice data in a payload field of a lost voice data packet according to the fused information.

[0043] Optionally, in some embodiments of the present application, the redundant data packet decoding unit can be specifically configured to decode the redundant data packet when the target associated voice data packet redundancy decoding fails and the number of voice data packets in the decoded data sequence received is not less than a preset number.

[0044] Optionally, in some embodiments of the present application, the data recovery unit can include an inverse operator unit and a recovery subunit, as follows:

[0045] The inverse operator unit is configured to perform inverse operation on the fused information to obtain an inverse encoding matrix corresponding to the fused information.

[0046] The recovery subunit is configured to perform data recovery on voice data in a payload field of a lost voice data packet according to the inverse encoding matrix.

[0047] Optionally, in some embodiments of the present application, the voice data receiving device can further include a discarding unit, as follows:

[0048] The discarding unit is configured to discard the target associated voice data packet when the target associated voice data packet redundancy decoding fails and the decoding time exceeds a preset time.

[0049] An electronic device provided in an embodiment of the present application includes a processor and a memory. The memory stores a plurality of instructions. The processor loads the instructions to perform steps in a voice data sending method or a voice data receiving method provided in an embodiment of the present application.

[0050] In addition, an embodiment of the present application further provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, steps in a voice data sending method or a voice data receiving method provided in an embodiment of the present application are implemented.

[0051] The embodiment of the present application provides a voice data sending and receiving method and related equipment, which can acquire a to-be-encoded data sequence, the to-be-encoded data sequence comprising a plurality of to-be-encoded voice data with a time sequence relationship; for each voice data, determining associated voice data of the voice data, the associated voice data being voice data in the to-be-encoded data sequence and satisfying a preset association condition with the voice data in the time sequence relationship; for each voice data, performing redundancy encoding on the voice data and the corresponding associated voice data to obtain an encoded data sequence, the encoded data sequence comprising a plurality of encoded voice data packets with a time sequence relationship, the encoded voice data packet comprising a payload field and a redundancy field, the payload field containing the voice data, and the redundancy field containing compressed associated voice data; and sending the voice data packet in the encoded data sequence to a network side in a predetermined order. The embodiment of the present application can solve the loss problem of the voice data packet corresponding to the associated voice data by carrying compressed associated voice data in the encoded voice data packet, which can guarantee the anti-packet loss capability of a transmission network, and at the same time, can effectively improve the utilization rate of network bandwidth by compressing the associated voice data. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0053] Figure 1a is a scene schematic diagram of the voice data sending and receiving method provided by the embodiment of the present application;

[0054] Figure 1b is a flowchart of the voice data sending and receiving method provided by the embodiment of the present application;

[0055] Figure 1c is another flowchart of the voice data sending and receiving method provided by the embodiment of the present application;

[0056] Figure 1d is an explanatory schematic diagram of the voice data sending and receiving method provided by the embodiment of the present application;

[0057] Figure 1e is an explanatory diagram of the voice data sending and receiving method provided by the embodiment of the present application;

[0058] Figure 2a is another flowchart of the voice data sending and receiving method provided by the embodiment of the present application;

[0059] Figure 2b is another flowchart of a voice data sending and receiving method provided by an embodiment of the present application;

[0060] Figure 2c is a schematic diagram of a voice data sending and receiving method provided by an embodiment of the present application;

[0061] Figure 2d is a page schematic diagram of a voice data sending and receiving method provided by an embodiment of the present application;

[0062] Figure 3a is a structural schematic diagram of a voice data sending device provided by an embodiment of the present application;

[0063] Figure 3b is another structural schematic diagram of a voice data receiving device provided by an embodiment of the present application;

[0064] Figure 4 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0066] The embodiments of the present application provide a voice data sending and receiving method and related devices.

[0067] Specifically, the embodiments of the present application provide a voice data sending device suitable for a first electronic device, which can be a terminal or a server, etc. The embodiments of the present application also provide a voice data receiving device suitable for a second electronic device, which can be a terminal or a server, etc. The terminal can be a mobile phone, a tablet computer, a notebook computer, or a personal computer (PC, Personal Computer), etc. The server can be a single server, or a server cluster composed of multiple servers.

[0068] The voice data sending and receiving method provided by the present application can be applied to, for example, Figure 1aIn the application environment shown in scenario (a), the voice data transmission and reception system includes a transmitter 110 and a receiver 120, which are connected via a network. In one embodiment, the transmitter 110 can acquire a data sequence to be encoded, which includes multiple voice data to be encoded and having a temporal relationship; for each voice data, it determines associated voice data, which is voice data in the data sequence to be encoded whose temporal relationship with the voice data satisfies a preset association condition; for each voice data, it performs redundant encoding on the voice data and its corresponding associated voice data to obtain an encoded data sequence, which includes multiple encoded voice data packets with a temporal relationship, and the encoded voice data packets include a payload field and a redundancy field, where the payload field contains the voice data and the redundancy field contains compressed associated voice data; and it sends the voice data packets in the encoded data sequence to the network side (specifically, the receiver 120) in a predetermined order.

[0069] The receiver 120 can be used to: receive encoded voice data packets from the encoded data sequence sent by the transmitter 110, wherein the encoded voice data packets include a payload field and a redundancy field, the payload field contains voice data, the redundancy field contains compressed associated voice data, and the associated voice data is voice data whose temporal relationship with the voice data satisfies a preset association condition; decode the encoded voice data packets and perform data packet loss detection based on the decoding result; when data packet loss is detected, determine the target associated voice data packet in the encoded data sequence that is associated with the lost voice data packet; perform redundancy decoding on the target associated voice data packet to extract compressed associated voice data from the redundancy field of the target associated voice data packet; and recover the voice data in the payload field of the lost voice data packet based on the compressed associated voice data.

[0070] In another embodiment, the sending end 110 can transmit the voice data packets in the encoded data sequence to a cloud server, which then forwards them to the receiving end 120. For example... Figure 1a As shown in scenario (b), in a specific application scenario, both the sending end 110 and the receiving end 120 run applications that support voice transmission, such as instant voice communication clients. The cloud server 130 can provide computing and storage capabilities for the application. Both the sending end 110 and the receiving end 120 can connect to the cloud server 130 via the network, thereby enabling voice transmission between the two instant voice communication clients based on the cloud server 130.

[0071] The sending end 110 and the receiving end 120 can be terminals, which can be, but are not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices, and can also be vehicle-mounted devices such as vehicle-mounted audio and video devices, vehicle-mounted display devices, and vehicle-mounted detection devices, and the like. The sending end 110 and the receiving end 120 can also be servers or server clusters. The cloud server 130 can be a standalone physical server, or a server cluster or distributed system formed by multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like, but is not limited thereto. The voice data sending and receiving method or device disclosed in the present application can be composed of a plurality of servers as a blockchain, and the servers are nodes on the blockchain. For example, in some scenarios requiring voiceprint recognition, the voice data sent by the sending end can be stored on the blockchain node for subsequent voiceprint voice to verify the identity of the user. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application.

[0072] The following will be described in detail. It should be noted that the order of the following embodiments is not limited as the preferred order of the embodiments.

[0073] The embodiments of the present application will be described from the perspective of a voice data sending device, which can be integrated in a first electronic device, which can be a terminal or a server or the like.

[0074] As shown in Figure 1b , the specific flow of the voice data sending method is as follows:

[0075] 101, obtaining a to-be-encoded data sequence, the to-be-encoded data sequence comprising a plurality of to-be-encoded voice data having a time sequence relationship.

[0076] The to-be-encoded data sequence can be a to-be-encoded voice data sequence. Specifically, in some embodiments, the to-be-encoded data sequence can include a plurality of voice data arranged in chronological order, and the time length of each voice data is not limited. The voice data can be an original analog voice signal or a digital voice signal.

[0077] Specifically, each speech data in the data sequence to be encoded can be obtained by dividing a certain speech segment. The division method can be equal duration division, that is, each speech data has the same duration, or it can be divided according to different durations. This embodiment does not limit this.

[0078] like Figure 1c As shown, in a VoIP (Voice over Internet Protocol) call system, voice data undergoes multiple stages, including acquisition, preprocessing, encoding, flow control, transmission, decoding, post-processing, and playback. Each stage affects the final voice quality. From an end-to-end perspective, factors affecting the call experience can be divided into two parts: the source and the channel (link). For the source, the main interference factors are acoustic noise, echo, and other physical characteristics. For the channel, the main interference factors are packet loss, latency, and jitter caused by the network environment. Packet loss, in particular, can cause voice discontinuity and stuttering.

[0079] In this embodiment, a joint source and channel coding method can be used to combat packet loss, thereby improving the audio quality of VoIP. Optionally, source coding can be implemented using in-band FEC coding, and channel coding can be implemented using RSFEC coding. In-band FEC coding adds associated voice data as redundant information to the corresponding voice data packet, while RSFEC coding generates a certain number of redundant data packets based on the packet loss rate of the voice data. When data packet loss occurs, the lost data packets can be recovered from the redundant data packets.

[0080] FEC, short for Forward Error Correction, is a widely used error correction technique in communication systems. The sending end adds redundant error correction codes to the payload data before transmission. The receiving end uses these codes to detect errors, and if errors are found, they are corrected using the error correction codes. When FEC operates at the transport or application layer, it processes data packets and can be used to recover lost packets in the network.

[0081] Wherein, RS (Reed-Solomon) coding is a forward error correction channel coding, which is effective for polynomials generated by correcting oversampling data. When the receiver receives enough correct points (data), it can recover the original polynomial, even if many points on the received polynomial are distorted by noise interference. Specifically, RS coding is a function of error correction code using the characteristics of Vandermonde matrix or Cauchy matrix, which can generate m redundant packets based on RS algorithm by constructing n original load packets (to be protected data) and m redundant packets, and send n original load packets and m redundant packets to the receiving end. As long as the receiving end loses less than m packets, it can recover the lost packets through the redundant packets, that is, the lost data packets are recovered through the received data packets, which can resist continuous packet loss to a certain extent.

[0082] 102. For each voice data, determine the associated voice data of the voice data, which is the voice data in the to-be-encoded data sequence that satisfies a preset association condition with the voice data in terms of time sequence.

[0083] Wherein, the preset association condition can be set according to actual conditions. For example, the preset association condition can be that the time corresponding to the associated voice data is adjacent to the time corresponding to the voice data. For example, a certain voice data is a voice segment corresponding to the 11th to 15th second, and the associated voice data of the voice data can be a voice segment corresponding to the 6th to 10th second.

[0084] Specifically, the voice data in the to-be-encoded data sequence is arranged in chronological order. In some embodiments, the associated voice data of a certain voice data can be the voice data adjacent to the position of the voice data in the to-be-encoded data sequence; for example, the associated voice data can be the voice data of the previous time period of the voice data in the to-be-encoded data sequence.

[0085] 103. For each voice data, redundantly encode the voice data and its corresponding associated voice data to obtain an encoded data sequence, the encoded data sequence comprising a plurality of encoded voice data packets having a time sequence relationship, the encoded voice data packet comprising: a payload field and a redundancy field, the payload field containing the voice data, and the redundancy field containing compressed associated voice data.

[0086] Optionally, in the embodiment, the arrangement order of the encoded voice data packets in the encoded data sequence can correspond to the arrangement order of the voice data in the to-be-encoded data sequence; that is, the encoded voice data packets in the encoded data sequence can be arranged based on the time sequence of the voice data corresponding to the payload field of the encoded voice data packet. In some embodiments, the encoded voice data packet can also be numbered according to the time sequence of the voice data corresponding to the payload field of the encoded voice data packet, and the number information can be stored in the packet header, and the position of the encoded voice data packet in the encoded data sequence can be determined according to the encoding information.

[0087] For example, the voice data in the payload field of the encoded voice data packet is the voice segment from the 16th second to the 20th second, and the time sequence corresponding to the encoded voice data packet is from the 16th second to the 20th second. The encoded voice data packet in the encoded data sequence can be sorted based on the time sequence.

[0088] Optionally, in the embodiment, the step of "redundantly encoding the voice data and the associated voice data corresponding to the voice data" can include:

[0089] encoding the voice data to obtain an initial encoded voice data packet, the initial encoded voice data packet including a payload field, and the payload field including encoded voice data;

[0090] setting a redundancy field in the initial encoded voice data packet;

[0091] compressing the associated voice data corresponding to the voice data, and adding the compressed associated voice data to the redundancy field to generate an encoded voice data packet.

[0092] In this way, when the voice data packet corresponding to the associated voice data is lost, the associated voice data compressed and added to the redundancy field of the encoded voice data packet can be used to restore a "quality acceptable" voice, and the compressed associated voice data will not increase the bandwidth too much.

[0093] In a specific embodiment, as shown in Figure 1d The to-be-encoded data sequence includes a plurality of voice data having a time sequence relationship, which can be denoted as payload 1, payload 2, and payload 3, and the like. The voice data is redundantly encoded to obtain an encoded data sequence, and the encoded data sequence includes a plurality of encoded voice data packets. Except for the first encoded voice data packet, the encoded voice data packet can include a three-part structure, i.e., a packet header, a payload field, and a redundancy field. The payload field of the encoded voice data packet can carry corresponding voice data, and the redundancy field can carry compressed associated voice data, as shown in Figure 1dThe second coded speech data packet shown has a payload 2 of the speech data corresponding to the stored speech data in the payload field itself, and a redundancy 1 of the compressed payload 1, so that if the first coded speech data packet is lost, the data of the payload 1 can be recovered by the redundancy 1 in the second coded speech data packet.

[0094] In this embodiment, the redundancy information of the previous frame of speech data can be carried in the current frame of speech coding data by in-band FEC coding (i.e. source coding), so that when a packet is lost, the speech data at the lost position can be recovered by the redundancy information of the previous frame carried in the speech coding data of the next frame at the lost position. In the case of a fixed coding rate, there is a competitive relationship between the redundancy information of the previous frame and the bits occupied by the current frame of speech coding data, and a higher number of bits occupied by the redundancy information of the previous frame will result in a significant decrease in speech coding quality, so the application can improve the utilization of network bandwidth by compressing the redundancy information of the previous frame.

[0095] In this embodiment, the in-band FEC mechanism can be supported by adding a redundancy packet to the coding payload based on Opus [RFC6716] and AMR [RFC4867]. Opus is a lossy sound coding format, and AMR is an audio file format mainly used on mobile devices. RFC (Request For Comments) means a series of files arranged in order of number.

[0096] 104. transmitting the speech data packets in the coded data sequence to the network side in a predetermined order.

[0097] The network side is specifically the receiving end of the speech data packets. The predetermined order can be the time sequence of the speech data, and the transmission order of the speech data packets in the coded data sequence is determined based on the time sequence of the speech data corresponding to the payload field of the speech data packets in the coded data sequence. Alternatively, the packet header of the speech data packets can store a packet number representing the transmission order of the speech data packets.

[0098] Optionally, before the step of transmitting the speech data packets in the coded data sequence to the network side in a predetermined order, the method can further include:

[0099] generating redundancy coding information according to the packet loss information transmitted by the network side;

[0100] fusing the redundancy coding information and the speech data to obtain at least one redundancy data packet, the redundancy data packet including a redundancy fusion field containing the fused information of the redundancy coding information and the speech data;

[0101] The step of "sending the voice data packets in the encoded data sequence to the network side in a predetermined order" comprises:

[0102] The redundant data packets are added to the encoded data sequence, and the voice data packets in the encoded data sequence are sent to the network side in a predetermined order.

[0103] The packet loss information indicates a packet loss rate when data transmission is performed under the current network state.

[0104] The fusion manner of the redundant encoding information and the voice data can be various, and the embodiment is not limited in this aspect. For example, the fusion manner can be multiplication, etc. The fusion of the redundant encoding information and the voice data can obtain fused information of the two. In the embodiment, the fused information can be taken as data information of the redundant data packet, and stored in the redundant fusion field of the redundant data packet.

[0105] It should be noted that the redundant data packet can also be regarded as a voice data packet. Specifically, the redundant data packet is a data packet in which information of the redundant encoding information and the voice data is fused.

[0106] Optionally, in the embodiment, the step of "generating the redundant encoding information according to the packet loss information sent by the network side" can comprise:

[0107] Obtaining redundant data structure information according to the packet loss information sent by the network side;

[0108] Constructing the redundant encoding information according to the redundant data structure information.

[0109] The number of the redundant data packets to be sent to the network side can be determined according to the packet loss information (specifically, the packet loss rate) under the current network state, and the number of the redundant data packets is the product of the number of the encoded voice data packets to be sent to the network side and the packet loss rate under the current network state. For example, the data transmission packet loss rate under the current network state is 50%, and the number of the encoded voice data packets to be sent to the network side is 10, and then the number of the redundant data packets to be sent to the network side is 5.

[0110] When network data transmission is performed between the network sending end and the network receiving end under the current network state, the network receiving end (i.e., the network side) can count the packet loss rate when network data transmission is performed under the current network state according to the network data packets sent by the network sending end.

[0111] The number of the redundant data packets to be sent to the network side can be determined according to the packet loss information sent by the network side; and the redundant data structure information is generated based on the number of the data packets and the number of the encoded voice data packets in the encoded data sequence.

[0112] The redundant data structure information may include the number of rows and columns of the encoding matrix corresponding to the generated redundant encoding information. In some embodiments, the number of rows of the encoding matrix corresponding to the redundant encoding information is the sum of the number of encoded voice data packets and the number of redundant data packets, and the number of columns of the encoding matrix corresponding to the redundant encoding information is the same as the number of encoded voice data packets. For convenient data storage, the upper part of the encoding matrix corresponding to the redundant encoding information can be the encoding matrix corresponding to the encoded voice data packets, and the lower part can be the encoding matrix corresponding to the redundant data packets.

[0113] The encoding matrix corresponding to the redundant encoding information is generated based on the RS encoding algorithm, and the encoding matrix corresponding to the redundant encoding information satisfies the condition that any submatrix is ​​invertible.

[0114] In one specific embodiment, assume that encoded voice data packets D1-D5 and redundant data packets C1-C3 need to be sent to the network side; such as Figure 1e As shown, the encoded voice data packets D1 to D5 and the redundant data packets C1 to C3 are obtained by multiplying the encoding matrix B and the data matrix D corresponding to the redundant encoding information. Specifically, the encoded voice data packets D1 to D5 represent the voice data D1 to D5 in the payload field of the encoded voice data packets, and the redundant data packets C1 to C3 represent the fused information C1 to C3 in the redundant fusion field of the redundant data packets.

[0115] In this matrix, the number of rows in data matrix D is the same as the number of encoded speech data packets, and the number of columns in the data matrix is ​​1. Specifically, the elements in data matrix D can be speech data D1 to D5 from the payload field of the encoded speech data packets.

[0116] Optionally, in this embodiment, the encoded voice data packets can be regarded as vectors D = (D1, D2, ..., Dn), where n is the number of encoded voice data packets, and the redundant data packets can be regarded as vectors C = (C1, C2, ..., Cm), where m is the number of redundant data packets. According to the RS encoding principle, n encoded voice data packets and m redundant data packets are sent to the network side. When the number of lost data packets is less than m, the lost data packets can be recovered from the redundant data packets, which can combat continuous packet loss to a certain extent.

[0117] It can be learned from the above that the embodiment can acquire a to-be-encoded data sequence, the to-be-encoded data sequence comprising a plurality of to-be-encoded voice data having a time sequence relationship; for each voice data, determining associated voice data of the voice data, the associated voice data being voice data in the to-be-encoded data sequence and having a time sequence relationship with the voice data satisfying a preset association condition; for each voice data, redundantly encoding the voice data and the corresponding associated voice data to obtain an encoded data sequence, the encoded data sequence comprising a plurality of encoded voice data packets having a time sequence relationship, the encoded voice data packet comprising a payload field and a redundancy field, the payload field containing the voice data, and the redundancy field containing compressed associated voice data; and sending the voice data packet in the encoded data sequence to a network side in a predetermined order. The embodiment of the application can solve the problem of loss of the voice data packet corresponding to the associated voice data by carrying compressed associated voice data in the encoded voice data packet, which can not only ensure the anti-packet loss capability of the transmission network, but also effectively improve the utilization rate of network bandwidth.

[0118] The embodiment will be described from the perspective of a voice data receiving device, which can be integrated in a second electronic device, which can be a terminal or a server or the like.

[0119] As shown in Figure 2a , the specific process of the voice data receiving method is as follows:

[0120] 201. Receive an encoded voice data packet in an encoded data sequence, the encoded voice data packet comprising a payload field and a redundancy field, the payload field containing voice data, and the redundancy field containing compressed associated voice data, the associated voice data being voice data having a time sequence relationship with the voice data satisfying a preset association condition.

[0121] The encoded voice data packet in the encoded data sequence has a time sequence relationship.

[0122] The preset association condition can be set according to actual conditions, and the embodiment does not limit this. For example, the preset association condition can be that the time corresponding to the associated voice data is adjacent to the time corresponding to the voice data. For example, a certain voice data is a voice segment corresponding to the 11th to 15th second, and the associated voice data of the voice data can be a voice segment corresponding to the 6th to 10th second.

[0123] Specifically, the associated voice data in the redundancy field of the encoded voice data packet can be the previous frame of voice segment of the voice data in the payload field.

[0124] 202. Decode the encoded voice data packet and perform data packet loss detection based on the decoding result.

[0125] Optionally, in this embodiment, the header of the encoded voice data packet may include the packet's serial number, which may indicate the timing of the encoded voice data packet. Specifically, the serial number indicates the timing of the voice data in the payload field of the encoded voice data packet. Decoding the encoded voice data packet may specifically involve extracting the serial number from the header of the encoded voice data packet.

[0126] For example, if the decoding result of the previous encoded voice data packet is number 1, and the decoding result of the current encoded voice data packet is number 3, it indicates that a data packet was lost during the data transmission process, and the lost data packet is number 2.

[0127] 203. When a data packet loss is detected, determine the target associated voice data packet in the encoded data sequence that is associated with the lost voice data packet.

[0128] The target associated voice data packet contains relevant information about the voice data in the payload field of the lost voice data packet. Specifically, the compressed associated voice data in the redundancy field of the target associated voice data packet is obtained by compressing the voice data in the payload field of the lost voice data packet; it can be understood that the target associated voice data packet is an encoded voice data packet whose temporal relationship with the lost voice data packet in the encoded data sequence satisfies the preset association condition.

[0129] 204. Perform redundant decoding on the target associated voice data packet to extract compressed associated voice data from the redundant fields of the target associated voice data packet.

[0130] 205. Based on the compressed associated voice data, recover the voice data in the payload field of the lost voice data packet.

[0131] In some embodiments, the compressed associated speech data can be directly used as the speech data in the payload field of the lost speech data packet. In other embodiments, the compressed associated speech data can be upsampled and the upsampled speech data can be used as the speech data in the payload field of the lost speech data packet.

[0132] After data recovery and obtaining the voice data from the payload field of the lost voice data packet, the voice data can be stored in the voice playback cache.

[0133] Optionally, in the embodiment, the coded data sequence further comprises a redundancy data packet; the redundancy data packet comprises a redundancy fusion field, the redundancy fusion field containing fused information of redundancy coding information and voice data; the voice data receiving method can further comprise:

[0134] when the target associated voice data packet fails in redundancy decoding, decoding the redundancy data packet to extract the fused information from the redundancy fusion field of the redundancy data packet;

[0135] recovering the voice data in the payload field of the lost voice data packet according to the fused information.

[0136] wherein, the target associated voice data packet fails in redundancy decoding, specifically, the compressed associated voice data cannot be extracted from the redundancy field of the target associated voice data packet, that is, the in-band FEC decoding fails, then the RS decoding can be used, that is, the redundancy data packet is decoded.

[0137] Optionally, in the embodiment, the step of "when the target associated voice data packet fails in redundancy decoding, decoding the redundancy data packet" can comprise:

[0138] when the target associated voice data packet fails in redundancy decoding, and the number of received voice data packets in the coded data sequence is not less than a preset number, decoding the redundancy data packet.

[0139] Optionally, the preset number can be the number of coded voice data packets in the coded data sequence sent by the sending end.

[0140] In the embodiment, the fused information in the redundancy fusion field of the redundancy data packet can be obtained based on the fusion of the redundancy coding information and the voice data, and the redundancy coding information is generated based on the RS coding algorithm. For RS coding, if the coded data sequence contains n coded voice data packets and m redundancy data packets, when the number of lost data packets is less than m, that is, when the number of received data packets is not less than n, the voice data in the payload field of the lost coded voice data packet can be recovered according to the redundancy data packet.

[0141] wherein, the received coded data sequence can comprise any one or combination of coded voice data packets and redundancy data packets. The lost voice data packet can be any one or combination of coded voice data packets and redundancy data packets. The number of lost voice data packets is not more than (less than or equal to) the number of redundancy data packets, so that the data recovery of the lost voice data packet can be performed according to the received data packet, that is, the RS code forward error correction method of the embodiment can recover at most the same number of randomly lost data packets as the number of redundancy data packets.

[0142] Optionally, in the embodiment, the step of "recovering the speech data in the payload field of the lost speech data packet according to the fused information" can include:

[0143] performing inverse operation on the fused information to obtain an inverse encoding matrix corresponding to the fused information;

[0144] recovering the speech data in the payload field of the lost speech data packet according to the inverse encoding matrix.

[0145] For RS encoding and decoding, the relationship between the data matrix of the original encoded speech data packet and the actually received encoded speech data packet and the redundant data packet and the inverse encoding matrix corresponding thereto is that the data matrix of the original encoded speech data packet is equal to the product of the actually received encoded speech data packet and the redundant data packet and the inverse matrix of the encoding matrix corresponding to the actually received encoded speech data packet and the redundant data packet; therefore, the speech data in the payload field of the lost speech data packet can be recovered based on the inverse encoding matrix.

[0146] The RS code forward error correction method provided in the embodiment can construct a redundant data packet by using a data matrix and redundant encoding information generated by using an RS encoding algorithm, and then transmit the encoded speech data packet and the redundant data packet to a network receiving end, so as to facilitate recovering the encoded speech data packet to the maximum extent according to the redundant data packet. The method can not only better solve the problem of burst packet loss and random packet loss of network data packets caused by network transmission, but also guarantee the decoding integrity and real-time performance of the audio stream and the quality of the audio call.

[0147] Optionally, in the embodiment, the speech data receiving method can further include:

[0148] when the target associated speech data packet fails in the redundant decoding and the decoding time exceeds the preset time, discarding the target associated speech data packet.

[0149] The preset time can be set according to actual conditions, and the embodiment does not limit this. For example, the preset time can be set to 60 ms (milliseconds) or the like.

[0150] After discarding the target associated speech data packet, the network receiving end can continue to wait for receiving a new speech data packet.

[0151] In a specific embodiment, as shown in FIG. 2, the sending and receiving process of the speech data is shown, and the specific process is as follows: Figure 2b

[0152] (1) the receiving end occurs network packet loss, and sends the packet loss information to the sending end;

[0153] ​(2) the sending end receives the packet loss information sent by the receiving end;

[0154] (3) the sending end adds redundancy to the voice data in the to-be-encoded data sequence based on the packet loss condition through in-band FEC (i.e., source encoding) and RS FEC (i.e., channel encoding), to obtain encoded voice data packets and redundant data packets;

[0155] (4) the sending end transmits the encoded voice data packets and the redundant data packets to the receiving end through a network;

[0156] (5) the receiving end detects whether there is packet loss at present;

[0157] (6) if there is no packet loss, directly put the voice data packets into a play buffer for audio equipment to play;

[0158] (7) if there is packet loss, first perform in-band FEC decoding, specifically, extract the compressed associated voice data from the redundancy field of the target associated voice data packet associated with the lost voice data packet;

[0159] (8) judge whether the in-band FEC decoding (i.e., source decoding) is successful, i.e., whether the compressed associated voice data is obtained;

[0160] (9) if the in-band FEC decoding is successful, put the recovered voice packet into the play buffer;

[0161] (10) if the in-band FEC decoding fails, detect whether it is greater than a preset waiting timeout;

[0162] (11) if it is greater than the preset waiting timeout, discard the target associated voice data packet, and return to step (5) to continue to wait for receiving new voice data packets;

[0163] (12) if it is less than or equal to the waiting timeout, detect whether the number of the currently received voice data packets is greater than or equal to n (n is the number of the encoded voice data packets sent by the sending end);

[0164] (13) if the number of the currently received voice data packets is greater than or equal to n, RS decoding (i.e., channel decoding) can be performed according to the redundant data packets to recover the lost voice data packet and put it into the play buffer;

[0165] (14) otherwise, return to step (5) to continue to wait for receiving new voice data packets.

[0166] The specific process of the in-band FEC encoding is as follows: for each voice data, the associated voice data of the voice data in the to-be-encoded data sequence is determined, and the voice data and the associated voice data are redundantly encoded to obtain an encoded voice data packet, the encoded voice data packet comprising a payload field and a redundancy field, the payload field comprising the voice data, and the redundancy field comprising compressed associated voice data.

[0167] The specific process of the RS FEC encoding is as follows: redundant encoding information is generated from the packet loss information through RS encoding, and the redundant encoding information and the voice data are fused to obtain a redundant data packet.

[0168] The voice data sending and receiving method provided in the application can adopt source and channel joint encoding, in the case of packet loss, if recovery fails through channel encoding, there is a certain probability of recovery through source encoding, the packet loss resistance is increased, the network bandwidth utilization is improved, the delay is reduced, and the voice call quality is improved.

[0169] In a specific embodiment, as shown in Figure 2c the sending end encodes three voice data packets through in-band FEC and encodes through RS FEC to obtain three encoded voice data packets and redundant data packets, and sends the three encoded voice data packets and redundant data packets to the receiving end. In the receiving end, assuming that the encoded voice data packet where the payload 2 is located is lost in the transmission process, if only channel encoding is used, the payload 2 needs to be recovered through RS decoding after the redundant data packet arrives, and the system delay is increased by 2 voice frames (assuming that the voice playing time of 1 voice frame is 20 ms, the delay is 40 ms); if source and channel joint encoding is used, after receiving the encoded voice data packet of the payload 3, the low-rate redundant information (i.e. associated voice data) of the payload 2 carried by the encoded voice data packet of the payload 3 through in-band FEC can be directly recovered to obtain the payload 2, so the system delay is increased by 1 audio packet (20 ms), and the time delay is reduced.

[0170] The voice data sending and receiving method and device of the application can be applied in various scenarios, such as Figure 2d as shown in FIG. 8, a multi-person voice conference of a certain instant messaging application program, in the conference session, each avatar corresponds to a voice call side client, each client can send an uplink voice stream, and then transmits through a media server, and receives multiple downlink voice streams. In the voice call process, the voice data sending and receiving can be performed through the source and channel joint encoding method provided in the application, which can not only ensure the anti-packet loss capability of the transmission network, but also effectively improve the network bandwidth utilization.

[0171] As can be seen, the embodiment can receive encoded voice data packets in an encoded data sequence, the encoded voice data packets comprising: a payload field containing voice data and a redundancy field containing compressed associated voice data, the associated voice data being voice data that meets a preset association condition in terms of timing relationship with the voice data; decoding the encoded voice data packets and performing data packet loss detection based on the decoding result; when detecting that there is data packet loss, determining a target associated voice data packet associated with the lost voice data packet in the encoded data sequence; performing redundancy decoding on the target associated voice data packet to extract compressed associated voice data from the redundancy field of the target associated voice data packet; and performing data recovery on the voice data in the payload field of the lost voice data packet according to the compressed associated voice data. The embodiment can solve the problem of loss of voice data packets corresponding to the associated voice data by carrying compressed associated voice data in the encoded voice data packets, which can not only ensure the anti-packet loss capability of the transmission network, but also effectively improve the utilization rate of network bandwidth by compressing the associated voice data.

[0172] To better implement the above method, the embodiment of the present application further provides a voice data sending and receiving system. The voice data sending and receiving system comprises a voice data sending device 31 and a voice data receiving device 32. As shown in Figure 3a The voice data sending device 31 can comprise an acquisition unit 3101, a first determination unit 3102, an encoding unit 3103 and a sending unit 3104; as shown in Figure 3b The voice data receiving device 32 can comprise a receiving unit 3201, a decoding unit 3202, a second determination unit 3203, a redundancy decoding unit 3204 and a recovery unit 3205, as follows:

[0173] A. Voice data sending device 31

[0174] (1) Acquisition unit 3101;

[0175] The acquisition unit 3101 is configured to acquire an encoded data sequence, the encoded data sequence comprising a plurality of voice data to be encoded and having a timing relationship.

[0176] (2) First determination unit 3102;

[0177] The first determination unit 3102 is configured to determine, for each voice data, associated voice data of the voice data, the associated voice data being voice data in the encoded data sequence that meets a preset association condition in terms of timing relationship with the voice data.

[0178] (3) Encoding unit 3103;

[0179] The encoding unit 3103 is configured to redundantly encode, for each voice data, the voice data and the corresponding associated voice data to obtain an encoded data sequence, wherein the encoded data sequence comprises a plurality of encoded voice data packets having a time sequence relationship, and each encoded voice data packet comprises a payload field and a redundancy field, wherein the payload field contains the voice data, and the redundancy field contains compressed associated voice data.

[0180] Optionally, in some embodiments of the present application, the encoding unit can comprise an encoding subunit, a setting subunit and a compression subunit, as follows:

[0181] The encoding subunit is configured to encode the voice data to obtain an initial encoded voice data packet, wherein the initial encoded voice data packet comprises a payload field, and the payload field comprises encoded voice data.

[0182] The setting subunit is configured to set a redundancy field in the initial encoded voice data packet.

[0183] The compression subunit is configured to compress the associated voice data corresponding to the voice data, and add the compressed associated voice data into the redundancy field to generate an encoded voice data packet.

[0184] (4) The sending unit 3104;

[0185] The sending unit 3104 is configured to send the voice data packets in the encoded data sequence to the network side in a predetermined order.

[0186] Optionally, in some embodiments of the present application, the voice data sending apparatus further comprises a generating unit and a fusion unit, as follows:

[0187] The generating unit is configured to generate redundancy encoding information according to the packet loss information sent by the network side.

[0188] The fusion unit is configured to fuse the redundancy encoding information and the voice data to obtain at least one redundancy data packet, wherein the redundancy data packet comprises a redundancy fusion field, and the redundancy fusion field contains fused information of the redundancy encoding information and the voice data.

[0189] The sending unit can be specifically configured to add the redundancy data packet to the encoded data sequence, and send the voice data packets in the encoded data sequence to the network side in a predetermined order.

[0190] Optionally, in some embodiments of the present application, the generating unit can comprise an obtaining subunit and a constructing subunit, as follows:

[0191] The obtaining subunit is configured to obtain redundancy data structure information according to the packet loss information sent by the network side.

[0192] The constructing subunit is configured to construct redundancy encoding information according to the redundancy data structure information.

[0193] B. The receiving device 32 of the voice data

[0194] (5) The receiving unit 3201;

[0195] The receiving unit 3201 is configured to receive encoded voice data packets in the encoded data sequence, wherein the encoded voice data packets comprise a payload field and a redundancy field, the payload field contains voice data, and the redundancy field contains compressed associated voice data, the associated voice data being voice data that meets a preset association condition in terms of a time sequence relationship with the voice data.

[0196] (6) The decoding unit 3202;

[0197] The decoding unit 3202 is configured to decode the encoded voice data packets and perform packet loss detection for data transmission based on a decoding result.

[0198] (7) The second determining unit 3203;

[0199] The second determining unit 3203 is configured to determine target associated voice data packets associated with lost voice data packets in the encoded data sequence when it is detected that there is packet loss.

[0200] (8) The redundancy decoding unit 3204;

[0201] The redundancy decoding unit 3204 is configured to perform redundancy decoding on the target associated voice data packets to extract compressed associated voice data from a redundancy field of the target associated voice data packets.

[0202] (9) The restoring unit 3205;

[0203] The restoring unit 3205 is configured to perform data restoration on voice data in a payload field of the lost voice data packets according to the compressed associated voice data.

[0204] Optionally, in some embodiments of the present application, the encoded data sequence further comprises a redundancy data packet; the redundancy data packet comprises a redundancy fusion field, and the redundancy fusion field contains redundancy encoding information and fused information of the voice data.

[0205] The receiving device of the voice data can further comprise a redundancy data packet decoding unit and a data restoring unit, as follows:

[0206] The redundancy data packet decoding unit is configured to decode the redundancy data packet to extract fused information from a redundancy fusion field of the redundancy data packet when the target associated voice data packet fails in redundancy decoding.

[0207] The data recovery unit is configured to recover voice data in a payload field of a lost voice data packet according to the fused information.

[0208] Optionally, in some embodiments of the present application, the redundancy data packet decoding unit can be specifically configured to decode the redundancy data packet when the target associated voice data packet fails in redundancy decoding and the number of voice data packets in the encoded data sequence is not less than a preset number.

[0209] Optionally, in some embodiments of the present application, the data recovery unit can include an inverse operator unit and a recovery subunit, as follows:

[0210] The inverse operator unit is configured to perform inverse operation on the fused information to obtain an inverse encoding matrix corresponding to the fused information.

[0211] The recovery subunit is configured to recover voice data in a payload field of a lost voice data packet according to the inverse encoding matrix.

[0212] Optionally, in some embodiments of the present application, the voice data receiving device can further include a discarding unit, as follows:

[0213] The discarding unit is configured to discard the target associated voice data packet when the target associated voice data packet fails in redundancy decoding and the decoding time exceeds a preset time.

[0214] As can be seen from the above, in the present embodiment, the obtaining unit 3101 is configured to obtain a to-be-encoded data sequence, the to-be-encoded data sequence including a plurality of to-be-encoded voice data having a time sequence relationship; the first determining unit 3102 is configured to determine, for each voice data, associated voice data of the voice data, the associated voice data being voice data in the to-be-encoded data sequence that satisfies a preset association condition with the voice data in terms of time sequence relationship; the encoding unit 3103 is configured to perform redundancy encoding on the voice data and the associated voice data corresponding to the voice data to obtain an encoded data sequence, the encoded data sequence including a plurality of encoded voice data packets having a time sequence relationship, each encoded voice data packet including a payload field and a redundancy field, the payload field containing the voice data, and the redundancy field containing compressed associated voice data; and the sending unit 3104 is configured to send the voice data packets in the encoded data sequence to a network side in a predetermined order.

[0215] Or, receiving the encoded speech data packet in the encoded data sequence through the receiving unit 3201, the encoded speech data packet includes: a payload field containing speech data and a redundancy field containing compressed associated speech data, the associated speech data is the speech data which satisfies a preset association condition with the speech data in the timing relationship; decoding the encoded speech data packet by the decoding unit 3202, and detecting the data packet loss based on the decoding result; when detecting that there is a data packet loss, determining the target associated speech data packet associated with the lost speech data packet in the encoded data sequence by the second determining unit 3203; performing redundancy decoding on the target associated speech data packet by the redundancy decoding unit 3204, to extract the compressed associated speech data from the redundancy field of the target associated speech data packet; recovering the speech data in the payload field of the lost speech data packet according to the compressed associated speech data by the recovery unit 3205;

[0216] The embodiment of the present application can carry the compressed associated speech data in the encoded speech data packet to solve the problem of loss of the speech data packet corresponding to the associated speech data, which can not only ensure the anti-packet loss capability of the transmission network, but also effectively improve the utilization rate of network bandwidth by compressing the associated speech data.

[0217] The embodiment of the present application also provides an electronic device, as shown in the figure, which shows a structural schematic diagram of the electronic device related to the embodiment of the present application, which can be a terminal or a server, etc. Specifically: Figure 4 As shown in the figure, which shows a structural schematic diagram of the electronic device related to the embodiment of the present application, which can be a terminal or a server, etc. Specifically:

[0218] The electronic device can include a processor 401 with one or more processing cores, a memory 402 with one or more computer readable storage media, a power supply 403 and an input unit 404, etc. Those skilled in the art can understand that, Figure 4 The structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and can include more or fewer components than the figure, or combine some components, or different component arrangements. Among them:

[0219] The processor 401 is the control center of the electronic device, connects each part of the entire electronic device by various interfaces and lines, and performs various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 402 and calling data stored in the memory 402. Optionally, the processor 401 can include one or more processing cores; preferably, the processor 401 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 401.

[0220] The memory 402 can be used to store software programs and modules, and the processor 401 executes various function applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc.; and the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 402 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 402 can also include a memory controller to provide access for the processor 401 to the memory 402.

[0221] The electronic device also includes a power supply 403 for supplying power to each component, and preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 403 can also include one or more than one direct current or alternating current power supply, a recharging system, a power failure detection circuit, a power converter or inverter, a power state indicator, etc. any component.

[0222] The electronic device can also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0223] Although not shown, the electronic device can also include a display unit, etc., which will not be described here. Specifically, in the present embodiment, the processor 401 in the electronic device will load the executable file corresponding to the process of one or more than one application program into the memory 402 according to the following instructions, and run the application program stored in the memory 402 by the processor 401, so as to realize various functions, as follows:

[0224] Obtaining a to-be-encoded data sequence, the to-be-encoded data sequence comprising a plurality of to-be-encoded voice data having a time sequence relationship; for each voice data, determining associated voice data of the voice data, the associated voice data being voice data in the to-be-encoded data sequence that satisfies a preset association condition with the voice data in terms of time sequence relationship; for each voice data, redundantly encoding the voice data and its corresponding associated voice data to obtain an encoded data sequence, the encoded data sequence comprising a plurality of encoded voice data packets having a time sequence relationship, each encoded voice data packet comprising a payload field and a redundancy field, the payload field containing the voice data, and the redundancy field containing compressed associated voice data; and sending the voice data packets in the encoded data sequence to a network side in a predetermined order.

[0225] Or,

[0226] Receiving an encoded voice data packet in an encoded data sequence, the encoded voice data packet comprising a payload field and a redundancy field, the payload field containing voice data, and the redundancy field containing compressed associated voice data, the associated voice data being voice data that satisfies a preset association condition with the voice data in terms of time sequence relationship; decoding the encoded voice data packet and detecting packet loss of data transmission based on a decoding result; when it is detected that there is packet loss, determining a target associated voice data packet associated with the lost voice data packet in the encoded data sequence; redundantly decoding the target associated voice data packet to extract compressed associated voice data from the redundancy field of the target associated voice data packet; and recovering the voice data in the payload field of the lost voice data packet based on the compressed associated voice data.

[0227] The specific implementation of each operation can refer to the foregoing embodiments, which will not be described here again.

[0228] It can be learned from the above that the embodiment can acquire a to-be-encoded data sequence, the to-be-encoded data sequence comprising a plurality of to-be-encoded voice data having a time sequence relationship; for each voice data, determining associated voice data of the voice data, the associated voice data being voice data in the to-be-encoded data sequence and having a time sequence relationship with the voice data satisfying a preset association condition; for each voice data, performing redundancy encoding on the voice data and the corresponding associated voice data to obtain an encoded data sequence, the encoded data sequence comprising a plurality of encoded voice data packets having a time sequence relationship, the encoded voice data packet comprising a payload field and a redundancy field, the payload field containing the voice data, and the redundancy field containing compressed associated voice data; and sending the voice data packet in the encoded data sequence to a network side in a predetermined order. The embodiment of the application can solve the problem of loss of the voice data packet corresponding to the associated voice data by carrying compressed associated voice data in the encoded voice data packet, which can not only ensure the anti-packet loss capability of the transmission network, but also effectively improve the utilization rate of network bandwidth.

[0229] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions or controlled by instructions related to hardware, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0230] Therefore, the embodiment of the application provides a storage medium, which stores a plurality of instructions, the instructions being capable of being loaded by a processor to execute the steps in any of the voice data sending and receiving methods provided by the embodiment of the application. For example, the instructions can execute the following steps:

[0231] acquiring a to-be-encoded data sequence, the to-be-encoded data sequence comprising a plurality of to-be-encoded voice data having a time sequence relationship; for each voice data, determining associated voice data of the voice data, the associated voice data being voice data in the to-be-encoded data sequence and having a time sequence relationship with the voice data satisfying a preset association condition; for each voice data, performing redundancy encoding on the voice data and the corresponding associated voice data to obtain an encoded data sequence, the encoded data sequence comprising a plurality of encoded voice data packets having a time sequence relationship, the encoded voice data packet comprising a payload field and a redundancy field, the payload field containing the voice data, and the redundancy field containing compressed associated voice data; and sending the voice data packet in the encoded data sequence to a network side in a predetermined order;

[0232] or

[0233] Receiving encoded speech data packets in an encoded data sequence, the encoded speech data packets comprising a payload field containing speech data and a redundancy field containing compressed associated speech data, the associated speech data being speech data that meets a preset association condition in terms of timing relationship with the speech data; decoding the encoded speech data packets and performing packet loss detection based on a decoding result for data transmission; when detecting that there is packet loss, determining a target associated speech data packet associated with the lost speech data packet in the encoded data sequence; performing redundancy decoding on the target associated speech data packet to extract compressed associated speech data from a redundancy field of the target associated speech data packet; and performing data recovery on speech data in a payload field of the lost speech data packet according to the compressed associated speech data.

[0234] The specific implementation of each operation can refer to the foregoing embodiments, which will not be described here again.

[0235] The storage medium can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, or the like.

[0236] Due to the instructions stored in the storage medium, the steps in any of the speech data sending and receiving methods provided in the embodiments of the present application can be performed, and thus the beneficial effects of any of the speech data sending and receiving methods provided in the embodiments of the present application can be achieved. Details are described in the foregoing embodiments, which will not be described here again.

[0237] According to an aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in any of the various optional implementation manners of the speech data sending and receiving aspects.

[0238] The speech data sending and receiving method and related device provided in the embodiments of the present application are described in detail above, and the principles and implementation manners of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method and core idea of the present application; meanwhile, for those skilled in the art, the specific implementation manner and application range can be changed according to the idea of the present application, and the above description of the embodiments should not be understood as limiting the present application.

Claims

1. A voice data transmission method characterized by comprising: The method comprises the following steps: acquiring a to-be-encoded data sequence, wherein the to-be-encoded data sequence comprises a plurality of to-be-encoded voice data having a time sequence relationship; for each voice data, determining associated voice data of the voice data, wherein the associated voice data is voice data in the to-be-encoded data sequence, and a time sequence relationship between the voice data and the associated voice data satisfies a preset association condition; for each voice data, redundantly encoding the voice data and the associated voice data corresponding to the voice data to obtain an encoded data sequence, wherein the encoded data sequence comprises a plurality of encoded voice data packets having a time sequence relationship, and each encoded voice data packet comprises a payload field carrying the voice data and a redundancy field carrying compressed associated voice data; acquiring redundancy data structure information according to packet loss information sent by a network side, and constructing redundancy encoding information according to the redundancy data structure information, wherein the redundancy data structure information comprises a number of rows and a number of columns of an encoding matrix corresponding to the redundancy encoding information, the number of rows is a sum of a number of encoded voice data packets and a number of redundancy data packets, and the number of columns is the same as the number of encoded voice data packets; fusing the redundancy encoding information and the voice data to obtain at least one redundancy data packet, wherein the redundancy data packet comprises a redundancy fusion field, and the redundancy fusion field comprises fused information of the redundancy encoding information and the voice data; adding the redundancy data packet to the encoded data sequence, and sending voice data packets in the encoded data sequence to the network side in a predetermined order; the encoded voice data packet is used to extract compressed associated voice data in a case where a data packet is lost, to recover voice data in a payload field of a lost voice data packet based on the extracted compressed associated voice data, and the extracted compressed associated voice data and the recovered voice data satisfy the preset association condition; the redundancy data packet is used to recover voice data in a payload field of a lost voice data packet in a case where a data packet is lost, if decoding of an encoded voice data packet associated with the lost voice data packet fails, and a receiving end of the voice data receives a number of voice data packets in the encoded data sequence that is not less than a preset number, and the preset number comprises a number of encoded voice data packets in an encoded data sequence sent by a sending end of the voice data.

2. The method of claim 1, wherein, The method comprises the following steps: encoding the voice data to obtain an initial encoded voice data packet, wherein the initial encoded voice data packet comprises a payload field, and the payload field comprises encoded voice data; setting a redundancy field in the initial encoded voice data packet; compressing the associated voice data corresponding to the voice data, and adding the compressed associated voice data to the redundancy field to generate the encoded voice data packet.

3. A receiving method of voice data, characterized by, The method comprises the following steps: receive encoded voice data packets in an encoded data sequence sent in a predetermined order, the encoded data sequence comprising a plurality of encoded voice data packets having a time sequence relationship, the encoded voice data packet comprising a payload field carrying voice data and a redundancy field carrying compressed associated voice data, the associated voice data being voice data that meets a preset association condition with the voice data in terms of time sequence relationship; the encoded data sequence further comprising a redundancy data packet; the redundancy data packet comprising a redundancy fusion field containing fused information of redundancy encoding information and voice data, the redundancy encoding information being constructed according to redundancy data structure information, the redundancy data structure information comprising a number of rows and a number of columns of an encoding matrix corresponding to generated redundancy encoding information, the number of rows being a sum of the number of encoded voice data packets and the number of redundancy data packets, the number of columns being the same as the number of encoded voice data packets, the redundancy data structure information being determined according to packet loss information; perform packet loss detection on the encoded voice data packet based on a decoding result of the encoded voice data packet; when it is detected that there is packet loss, determine a target associated voice data packet associated with the lost voice data packet in the encoded data sequence; perform redundancy decoding on the target associated voice data packet to extract compressed associated voice data from the redundancy field of the target associated voice data packet; perform data recovery on the voice data in the payload field of the lost voice data packet according to the extracted compressed associated voice data; when the redundancy decoding of the target associated voice data packet fails and the number of voice data packets in the received encoded data sequence is not less than a preset number, decode the redundancy data packet to recover the voice data in the payload field of the lost voice data packet, the preset number comprising the number of encoded voice data packets in the encoded data sequence sent by a sending end of the voice data.

4. The method of claim 3, wherein, The decoding of the redundancy data packet to recover the voice data in the payload field of the lost voice data packet comprises: decoding the redundancy data packet to extract fused information from the redundancy fusion field of the redundancy data packet; performing data recovery on the voice data in the payload field of the lost voice data packet according to the fused information.

5. The method of claim 4, wherein, The data recovery on the voice data in the payload field of the lost voice data packet according to the fused information comprises: performing inverse operation on the fused information to obtain an inverse encoding matrix corresponding to the fused information; and performing data recovery on the voice data in the payload field of the lost voice data packet according to the inverse encoding matrix.

6. The method of claim 3, wherein, The method further comprises: when the redundancy decoding of the target associated voice data packet fails and a decoding time exceeds a preset time, discarding the target associated voice data packet.

7. A voice data transmitting apparatus characterized by comprising: comprise: an acquisition unit configured to acquire an encoded data sequence, the encoded data sequence comprising a plurality of voice data to be encoded and having a time sequence relationship; The first determining unit is configured to determine, for each piece of voice data, associated voice data of the voice data, the associated voice data being voice data in the to-be-encoded data sequence that meets a preset association condition in terms of a time sequence relationship with the voice data. The encoding unit is configured to, for each piece of voice data, redundantly encode the voice data and the associated voice data corresponding to the voice data to obtain an encoded data sequence, the encoded data sequence comprising a plurality of encoded voice data packets having a time sequence relationship, each encoded voice data packet comprising a payload field carrying the voice data and a redundancy field carrying compressed associated voice data. The obtaining unit is configured to obtain redundancy data structure information according to packet loss information sent by a network side. The constructing unit is configured to construct redundancy encoding information according to the redundancy data structure information, the redundancy data structure information comprising a number of rows and a number of columns of an encoding matrix corresponding to the redundancy encoding information, the number of rows being a sum of a number of encoded voice data packets and a number of redundancy data packets, and the number of columns being the same as the number of encoded voice data packets. The fusing unit is configured to fuse the redundancy encoding information and the voice data to obtain at least one redundancy data packet, the redundancy data packet comprising a redundancy fusion field, the redundancy fusion field containing fused information of the redundancy encoding information and the voice data. The sending unit is configured to add the redundancy data packet to the encoded data sequence and send, to the network side, voice data packets in the encoded data sequence in a predetermined order. The encoded voice data packet is used to, in a case where a data packet is lost, be decoded to extract compressed associated voice data, to perform data recovery on voice data in a payload field of a lost voice data packet based on the extracted compressed associated voice data, the extracted compressed associated voice data and the data-recovered voice data meeting the preset association condition. The redundancy data packet is used to, in a case where a data packet is lost, be decoded to recover voice data in a payload field of a lost voice data packet, if decoding of an encoded voice data packet associated with the lost voice data packet fails and a receiving end of the voice data receives a number of voice data packets in the encoded data sequence that is not less than a preset number, the preset number comprising a number of encoded voice data packets in the encoded data sequence sent by a sending end of the voice data.

8. A voice data receiving apparatus characterized by comprising: The method comprises the following steps: The receiving unit is configured to receive encoded voice data packets in an encoded data sequence sent in a predetermined order, the encoded data sequence comprising a plurality of encoded voice data packets having a time sequence relationship, each of the encoded voice data packets comprising a payload field carrying voice data and a redundancy field carrying compressed associated voice data, the associated voice data being voice data that meets a preset association condition in terms of the time sequence relationship with the voice data; the encoded data sequence further comprising a redundancy data packet; the redundancy data packet comprising a redundancy fusion field, the redundancy fusion field containing redundancy encoding information and fused information of the voice data, the redundancy encoding information being constructed according to redundancy data structure information, the redundancy data structure information comprising a number of rows and a number of columns of an encoding matrix corresponding to the generated redundancy encoding information, the number of rows being a sum of the number of the encoded voice data packets and the number of the redundancy data packet, the number of columns being the same as the number of the encoded voice data packets, and the redundancy data structure information being determined according to packet loss information. The decoding unit is configured to decode the encoded voice data packets and perform packet loss detection of data transmission based on a decoding result. The second determining unit is configured to, when it is detected that there is packet loss, determine a target associated voice data packet associated with the lost voice data packet in the encoded data sequence. The redundancy decoding unit is configured to perform redundancy decoding on the target associated voice data packet to extract compressed associated voice data from a redundancy field of the target associated voice data packet. The recovery unit is configured to perform data recovery of voice data in a payload field of the lost voice data packet according to the extracted compressed associated voice data. The redundancy data packet decoding unit is configured to, when the redundancy decoding of the target associated voice data packet fails and the number of voice data packets in the received encoded data sequence is not less than a preset number, decode the redundancy data packet, the preset number comprising the number of encoded voice data packets in the encoded data sequence sent by a sending end of the voice data. The data recovery unit is configured to recover the voice data in the payload field of the lost voice data packet according to a decoding result of the redundancy data packet decoding unit.

9. An electronic device, comprising: The memory stores an application, and the processor is configured to execute the application in the memory to perform operations in the voice data sending method in any one of claims 1 to 2 or the voice data receiving method in any one of claims 3 to 6.

10. A storage medium, characterized by The storage medium stores a plurality of instructions, and the instructions are adapted to be loaded by the processor to perform steps in the voice data sending method in any one of claims 1 to 2 or the voice data receiving method in any one of claims 3 to 6.

Images