Method and device for restoring multi-channel real-time media stream data

Abstract

The embodiment of the invention provides a method and a device for restoring multi-path real-time media stream data. The method comprises the following steps: acquiring real-time media stream data; shunting the real-time media stream data according to the shunting identifier to obtain each single-path data stream; judging the data type of the data in each single-path data stream; if the data type is judged to be an SDP message, analyzing the SDP message to obtain media parameter information, constructing a media type mapping relation based on the media parameter information, and updating the media type mapping relation to a pre-constructed hash table, if the data type is judged to be an RTP data frame, analyzing the RTP data frame to obtain a load type, and matching the hash table based on the load type, obtaining an encoder matched with the media type encoding numerical value stored in the media type mapping relation, and obtaining a corresponding decoder based on the encoder; and the decoder is used for processing the RTP data frame to obtain the target media voice stream, so that the aim of accurately restoring the multiple paths of media voice streams is fulfilled.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of multi-channel voice decoding, and in particular to a method and device for restoring multi-channel real-time media stream data. Background Art

[0002] With the continuous evolution and development of mobile network communication technology, more and more voice media coding formats are transmitted in the network, the media quality is getting higher and higher, and the data volume is getting larger and larger. How to decode and restore real-time voice media data streams with multiple conversations and dynamic coding formats is a complex and challenging problem.

[0003] The current media voice stream restoration algorithm is mainly optimized for single-channel real-time media streams. The main process is to convert the media data carried by RTP into media voice streams frame by frame according to the media transmission format based on the SDP message, thereby restoring the voice media file in the specified encoding format. The problems with this method are as follows: 1. It can only extract and restore media voice data in a single format. If the encoding format of the media stream data changes, the restoration of the media stream data may fail. 2. It can only support single-channel real-time media stream decoding and cannot be applied to multi-channel session real-time media stream decoding and restoration scenarios. 3. In the event that a new media format cannot be decoded, data is not retained, resulting in data loss. Summary of the invention

[0004] In view of this, the embodiments of the present application propose a method and device for restoring multi-channel real-time media stream data, aiming to accurately restore multi-channel media voice streams.

[0005] To achieve the above-mentioned purpose, an embodiment of the present application provides a method for restoring multi-channel real-time media stream data, including: obtaining real-time media stream data; diverting the real-time media stream data according to a diversion identifier to obtain each single data stream; judging the data type of the data frame in each single data stream frame by frame; if the data type is judged to be an SDP message, parsing the SDP message to obtain media parameter information, constructing a media type mapping relationship based on the media parameter information, and updating the media type mapping relationship to a pre-constructed hash table, wherein the media type mapping relationship includes a media type encoding value and an encoder; if the data type is judged to be an RTP data frame, parsing the RTP data frame to obtain a load type, matching the hash table based on the load type, obtaining the encoder adapted to the media type encoding value stored in the media type mapping relationship, and obtaining a corresponding decoder based on the encoder; using the decoder to process the RTP data frame to obtain a target media voice stream.

[0006] Optionally, the real-time media stream data is diverted according to the diversion identifier to obtain each single data stream, including: obtaining a first diversion identifier and a second diversion identifier, wherein the first diversion identifier is determined based on the source IP address and source port of the data packet, connection information and a media description name address are obtained based on an SDP message, and the second diversion identifier is determined based on the connection information and the media description name address; based on the first diversion identifier and the second diversion identifier, the real-time media stream data is diverted to obtain each single data stream.

[0007] Optionally, parsing the SDP message to obtain media parameter information, and constructing a media type mapping relationship based on the media parameter information, includes: parsing the SDP message to obtain SDP description information; obtaining SDP attribute information based on the SDP description information, wherein the SDP attribute information includes a media type encoding value, a media type encoding name, a coding rate, and an encoder name; and constructing the media type mapping relationship based on the media type encoding value, the media type encoding name, the coding rate, and the encoder.

[0008] Optionally, matching the hash table based on the load type to obtain the encoder adapted to the media type encoding value stored in the media type mapping relationship, and obtaining the corresponding decoder based on the encoder, includes: obtaining the load type stored in the RTP data frame, wherein the load type includes multiple audio media type encoding values ​​and multiple video media type encoding values; matching the media type mapping relationship stored in the hash table based on the multiple audio media type encoding values ​​and / or multiple video media type encoding values ​​to obtain adapted media type encoding values; determining the media type encoding name and the encoder based on the adapted media type encoding values; and determining the corresponding decoder based on the media type encoding name and the encoder.

[0009] Optionally, after matching the media type mapping relationship stored in the hash table based on the multiple audio media type encoding values ​​and / or the multiple video media type encoding values, the method for restoring multi-channel real-time media stream data also includes: if the adapted media type encoding value cannot be obtained, caching the RTP data frame to a temporary file.

[0010] Optionally, before parsing the SDP message to obtain media parameter information, the method for restoring multi-channel real-time media stream data also includes: judging whether the RTP data frame is damaged according to the frame header attributes in the RTP data frame, and if the RTP data frame is not damaged, filling each of the data frames with a silent frame.

[0011] Optionally, after updating the media type mapping relationship to the pre-built hash table, the method for restoring multi-channel real-time media stream data also includes: deduplicating and sorting the cached RTP data frames and the real-time RTP data frames based on the timestamp and sequence number identifier of the RTP data frames.

[0012] Optionally, after deduplication and sorting operations are performed on the cached RTP data frames and the real-time RTP data frames, the method for restoring multi-channel real-time media stream data also includes: calculating the relative timestamp difference between two adjacent RTP data frames to obtain a frame difference; dividing the frame difference by a standard time interval to obtain the number of silent frames, and inserting silent frames equal to the number of silent frames between two adjacent RTP data frames, wherein the standard time interval is determined according to the media type coding value.

[0013] Optionally, the use of the decoder to process the RTP data frame to obtain the target media voice stream includes: obtaining the media type encoding name of the RTP data frame; initializing the extraction interface of the decoder based on the initialization interface of the decoder and the media type encoding name; using the extraction interface of the decoder to extract the RTP data frame, and calling the decoding interface of the decoder to decode the RTP data frame; using the decoder to decode the RTP data to obtain the target media voice stream.

[0014] To achieve the above-mentioned purpose, an embodiment of the present application also provides a method for restoring multi-channel real-time media stream data, including: an acquisition module, used to acquire real-time media stream data; a diversion module, used to divert the real-time media stream data according to the diversion identifier to obtain each single data stream; a judgment module, used to judge the data type of the data frame in each single data stream frame by frame; a first execution module, used to parse the SDP message to obtain media parameter information if the data type is determined to be an SDP message, construct a media type mapping relationship based on the media parameter information, and update the media type mapping relationship to a pre-constructed hash table, wherein the media type mapping relationship includes a media type encoding value and an encoder; a second execution module, used to parse the RTP data frame to obtain a load type if the data type is determined to be an RTP data frame, match the hash table based on the load type, obtain the encoder adapted to the media type encoding value stored in the media type mapping relationship, and obtain the corresponding decoder based on the encoder; an output module, used to use the decoder to process the RTP data frame to obtain a target media voice stream.

[0015] The embodiments of the present application provide a method and device for restoring multi-channel real-time media stream data, which obtains real-time media stream data; shunts the real-time media stream data according to a shunt identifier to obtain each single-channel data stream; determines the data type of the data in each single-channel data stream; if the data type is determined to be an SDP message, parses the SDP message to obtain media parameter information, builds a media type mapping relationship based on the media parameter information, and updates the media type mapping relationship to a pre-built hash table, wherein the media type mapping relationship includes a media type encoding value and an encoder; if the data type is determined to be an RTP data frame, parses the RTP data frame to obtain a load type, matches the hash table based on the load type, and obtains a media type mapping relationship corresponding to the media type. An encoder adapted to the media type encoding value stored in the media type mapping relationship is used, and a corresponding decoder is obtained based on the encoder; the decoder is used to process the RTP data frame to obtain the target media voice stream, thereby obtaining a single data stream based on the real-time media stream data, and then the media type mapping relationship stored in the hash table is used to match the load type stored in each RTP data frame. During the matching, the media type encoding value is used to obtain the corresponding encoder, and the corresponding decoder is obtained according to the encoder. Finally, the corresponding decoder is used to restore the media voice stream stored in the RTP data frame to obtain the target media voice stream, thereby ultimately achieving the purpose of accurately restoring multiple media voice streams. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flowchart of a method for restoring multi-channel real-time media stream data provided in one embodiment of the present application;

[0017] Figure 2 is a flowchart of a method for restoring multi-channel real-time media stream data provided in one embodiment of the present application;

[0018] Figure 3 This is an example diagram of the RTP header message format of multiple real-time media stream data provided in one embodiment of the present application;

[0019] Figure 4 It is a structural block diagram of a device for restoring multi-channel real-time media stream data provided in one embodiment of the present application. DETAILED DESCRIPTION

[0020] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not be construed as any limitation, and the various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0021] In order to solve the above-mentioned problems, the method for restoring multi-channel real-time media stream data proposed in this method is based on real-time media stream data. In the scenario of multi-channel conversational voice media stream, for multiple complete sessions, after diversion according to the session identifier, it is written into the streaming media message queue in chronological order, and the real-time media stream decoding module consumes and decodes from the message queue. The real-time media stream decoding module first diverts the data stream according to the session identifier, and then decodes and processes it according to the single-channel media stream. In the process of single-channel media stream processing, the data is cached before receiving the SDP message. After receiving the SDP message, the media type mapping relationship is first parsed through the SDP description information and saved in the hash table, and then the cached media stream and the real-time media stream are locally deduplicated and sorted according to the timestamp and sequence number identifier in the RTP. Finally, the corresponding decoder is called according to the media type in the RTP data packet, and the corresponding media format data is restored and output through data frame verification, replacement, supplementation and other operations. In the process of restoring the voice media stream, after receiving the new SDP description information, the hash table should be parsed and updated in time. For newly emerged media data that cannot be decoded, offline backup is used, and offline restoration is performed again after the corresponding decoder is developed to ensure that data is not lost.

[0022] In summary, the method for restoring multi-channel real-time media stream data proposed in this application mainly includes the steps of data diversion, dynamic construction and update of media mapping relationship, RTP data packet parsing, dynamic mapping, abnormal data frame detection and replacement, silent data filling, and voice extraction and restoration. The method can be applied to electronic devices, wherein the electronic device can be a terminal or a server. This embodiment and the following embodiments are all described by taking the server as an example. The implementation details of the method for restoring multi-channel real-time media stream data proposed in this embodiment are specifically described below. The following content is only the implementation details provided for the convenience of understanding and is not necessary for the implementation of this solution.

[0023] The specific process of the method for restoring multiple channels of real-time media stream data proposed in this embodiment can be as follows: Figure 1 and Figure 2 As shown, reference Figure 1 and Figure 2, the method for restoring multiple channels of real-time media stream data may include:

[0024] S10, obtaining real-time media stream data;

[0025] In this step, the server first obtains real-time media stream data in a multi-channel session voice media stream scenario, which includes voice data and video data in various encoding modes.

[0026] S20, splitting the real-time media stream data according to the splitting identifier to obtain each single-channel data stream;

[0027] In an embodiment of the present application, step S20 may include the following execution process:

[0028] S21, obtaining a first shunt identifier and a second shunt identifier, wherein the first shunt identifier is determined based on the source IP address and the source port of the data packet, connection information and a media description name address are obtained based on the SDP message, and the second shunt identifier is determined based on the connection information and the media description name address;

[0029] S22: Based on the first diversion identifier and the second diversion identifier, divert the real-time media stream data to obtain the single-channel data streams.

[0030] Among them, the server generates a first diversion identifier according to the transmission source IP address and source port of the real-time streaming media data packet, and writes the media stream data distribution into different media stream message queues. Each message queue only contains single-channel media stream data and SDP messages carrying SDP description information. The media stream can use the two-tuple generated by the source IP address of the IP layer and the source port of the transport layer as the first diversion identifier, and the media stream can use the connection information (ConnectionInformation, abbreviated as c) and the media description name address (Media Description, name and address abbreviated as m) contained in the SDP message as the second diversion identifier. The server diverts the single-channel data stream according to the first diversion identifier and the second diversion identifier, so that each voice stream can be correctly processed.

[0031] S30, determining the data type of the data in each of the single-channel data streams;

[0032] S40: If it is determined that the data type is an SDP message, parse the SDP message to obtain media parameter information, construct a media type mapping relationship based on the media parameter information, and update the media type mapping relationship to a pre-constructed hash table, wherein the media type mapping relationship includes a media type encoding value and an encoder;

[0033] In this embodiment, the hash table can store media type mapping relationships. Each time the server receives an SDP message, it must construct a media type mapping relationship based on the SDP attribute information carried in the SDP message and update the hash table using the media type mapping relationship.

[0034] Specifically, after receiving a new SDP message, the server must promptly parse the media parameter information, construct a media type mapping relationship based on the media parameter information, and update the media type mapping relationship to the pre-constructed hash table to prevent the inability to timely identify the media type encoding value of the data frame in each RTP data packet when a media type switch occurs.

[0035] In an embodiment of the present application, step S40 may include the following execution process:

[0036] S41, parsing the SDP message to obtain SDP description information;

[0037] The basic structure of the SDP description information consists of a series of description fields, each of which has specific semantics. The following are some common key fields of the SDP description information:

[0038] v (version): specifies the version of the SDP description information.

[0039] m (media description name address): describes a media stream, including a list of media type values ​​(such as audio, video), media transmission port address, etc.

[0040] c (connection information): specifies the connection information of the media stream, including the network protocol and address.

[0041] a (attributes): defines various attributes of a session or media stream, such as media type encoding value, media type encoding name, sampling rate, and encoder name.

[0042] S42. Obtain SDP attribute information based on the SDP description information, wherein the SDP attribute information includes a media type encoding value, a media type encoding name, a coding rate, and an encoder name;

[0043] S43: construct the media type mapping relationship based on the media type encoding value, the media type encoding name, the encoding rate and the encoder name.

[0044] Among them, the main task of step S41-step S43 is to extract the mapping relationship between the media type coding value, the media type coding name, and supplementary information such as the sampling rate and the duration of each frame of data from the SDP description information and construct a media type mapping relationship, which is crucial for the dynamic mapping and decoding of real-time media streams.

[0045] It should be noted that a complete SDP description may correspond to multiple single-channel data streams, each of which is described using independent m, c, and a fields. SDP description information is usually exchanged during session establishment via SIP (Session Initiation Protocol) or other protocols.

[0046] In the embodiment of the present application, after step S40, the method for restoring multiple channels of real-time media stream data may further include the following execution process:

[0047] S50, if it is determined that the data type is an RTP data packet, parsing the RTP data frame in the RTP data packet to obtain a payload type, matching the hash table based on the payload type, obtaining the encoder adapted to the media type encoding value stored in the media type mapping relationship, and obtaining a corresponding decoder based on the encoder;

[0048] In an embodiment of the present application, step S50 may include the following execution process:

[0049] S51, obtaining the load type stored in the RTP data frame, wherein the load type includes a plurality of audio media type encoding values ​​and a plurality of video media type encoding values;

[0050] For example, real-time media stream data is encapsulated into RTP data frames of relatively fixed length through the RTP transmission protocol, and each RTP data frame includes an RTP header and a payload. The RTP header consists of 12 bytes, such as Figure 3 As shown in FIG, the RTP header contains various control information, such as version number, load type, sequence number, timestamp, and synchronization signal, etc. RTP can maintain synchronization between different single-channel data streams through a synchronization signal (Synchronization Source Identifier, SSRC).

[0051] The steps of obtaining the load type stored in the RTP data frame are as follows:

[0052] a. Strip the RTP packet header: Extract the first 12 bytes from the received packet, which is the length of the RTP header. The RTP header contains various control information.

[0053] b. Parsing the RTP header: The structure of the RTP header is as follows:

[0054] - Version: occupies 2 bits and is used to indicate the version of the RTP protocol.

[0055] -Padding: 1 bit, used to indicate whether there are additional padding bytes.

[0056] - Extension bit: 1 bit, used to indicate whether there is an extension part of the RTP header.

[0057] -CSRC Count: occupies 4 bits and indicates the number of CSRC identifiers.

[0058] -Marker: occupies 1 bit and can be customized by the application.

[0059] -Payload Type: occupies 7 bits and is used to indicate the payload type, for example, the media type encoding value of different audio codecs or the media type encoding value of video encoders.

[0060] -Sequence Number: 16 bits, used to identify the order of data packets.

[0061] -Timestamp: occupies 32 bits and is used to synchronize the playback speed of the receiving end.

[0062] -Synchronization Source Identifier (SSRC): occupies 32 bits and is used to identify the sender.

[0063] -CSRC identifier (Contributing Source Identifiers): occupies 32 bits and can be used to identify other sources participating in synchronization.

[0064] c. Get the load type.

[0065] S52: Match the media type mapping relationship stored in the hash table based on the multiple audio media type encoding values ​​and / or the multiple video media type encoding values ​​to obtain the adapted media type encoding values;

[0066] In this step, after parsing the RTP header of the RTP data frame, the server extracts the payload type (PayloadType), and then associates it with the media type mapping relationship stored in the hash table to obtain the adapted media type encoding value.

[0067] In the embodiment of the present application, after step S52, the method for restoring multiple channels of real-time media stream data may further include the following execution steps:

[0068] S70: If the adapted media type encoding value cannot be obtained, cache the RTP data frame in a temporary file.

[0069] Specifically, for newly appearing RTP data frames that cannot be decoded, offline backup is used, and offline restoration is performed again after the corresponding decoder is developed to ensure that data is not lost. This solves the problem of data loss caused by not retaining data when new media formats cannot be decoded.

[0070] S53, determining the media type encoding name and the encoder based on the adapted media type encoding value;

[0071] Specifically, the media type mapping relationship stores the encoder corresponding to the media type encoding name. After determining the media type encoding name, the server can determine the encoder, and the encoder and the decoder appear in pairs.

[0072] S54: Determine the corresponding decoder based on the media type encoding name and the encoder.

[0073] S60: Utilize the decoder to process the RTP data frame to obtain a target media voice stream.

[0074] In an embodiment of the present application, step S60 may include the following execution process:

[0075] S61, obtaining the media type encoding name of the RTP data frame;

[0076] S62, initializing the extraction interface of the decoder based on the initialization interface of the decoder and the media type encoding name;

[0077] S63: Utilize the extraction interface of the decoder to extract the RTP data frame, and call the decoding interface of the decoder to decode the RTP data frame to obtain the target media voice stream.

[0078] The server can restore the decoded media data by executing steps S61 to S63.

[0079] It should be noted that the current voice media data coding formats in the existing network mainly include AMR-NB, AMR-WB, EVS, PCMA, PCMU, and G729. The data extraction and decoding methods of each format are very different. Therefore, it is necessary to design four interfaces for initialization, extraction, decoding, and release for each media extraction and decoding module according to the relevant transmission protocol, and encapsulate them into unified input and output parameters.

[0080] When processing a single data stream, the server uses the identified media type encoding name as a parameter to initialize the encoder's extraction interface through the initialization interface. When the extraction interface extracts subsequent voice data, the server calls the decoding interface to restore the voice data. After the session ends, the release interface is called and the extraction interface is closed, thereby completing the restoration process of each single data stream of the real-time media stream data.

[0081] In an embodiment of the present invention, after step S63, the method for restoring multiple channels of real-time media stream data may further include the following execution steps:

[0082] S80: De-duplicate and sort the cached RTP data frames and the real-time RTP data frames based on the timestamps and sequence number identifiers of the RTP data frames.

[0083] In an embodiment of the present invention, after step S80, the method for restoring multiple channels of real-time media stream data may further include the following execution steps:

[0084] S90. Determine whether the RTP data frame is damaged according to a frame header attribute in the RTP data frame. If the RTP data frame is not damaged, fill each of the data frames with a silent frame.

[0085] Specifically, before extracting the media frame data carried by RTP, it will first determine whether the frame data is damaged based on the frame header attributes. If damaged, the frame data will be replaced with a silent frame. For example, when the load format is AMR or AMRWB, the complete payload includes a payload header, a payload directory, and voice data representing one or more voice frame blocks. There is a bit Q (1 bit) that indicates the quality of the data frame. If the bit value is 0, it means that the frame data is severely damaged, and the decoder should treat the frame voice as a silent frame.

[0086] In the embodiment of the present invention, after step S90, the method for restoring multiple channels of real-time media stream data may further include the following execution process:

[0087] S101, calculating the relative timestamp difference between two adjacent RTP data frames to obtain a frame difference;

[0088] S102. Divide the frame difference by a standard time interval to obtain a number of silent frames, and insert silent frames equal to the number of silent frames between two adjacent RTP data frames, wherein the standard time interval is determined according to the media type coding value.

[0089] It should be noted that in a voice call, there may be problems with discontinuous data transmission or data packet loss. If one party to the call remains silent for a long time, no valid voice media data will be transmitted in the transmission channel. Instead, the decoder is prompted to output comfort noise by transmitting a silence indication frame (SID). If the server only extracts the media data in the RTP data packet, it may affect the call duration, resulting in the inability to synchronize the local and remote voices during playback. Therefore, during the voice restoration process, this application adds corresponding silent data packets between two adjacent RTP data frames as needed.

[0090] For example, two packets of continuous RTP data streams are given below:

[0091] Real-Time Transport Protocol

[0092] [Stream setup by SDP(frame 7185)]

[0093] 10. .... = Version: RFC 1889 Version (2)

[0094] .0.....=Padding:False

[0095] ...0....=Extension:False

[0096] ....0000=Contributing source identifiers count: 0

[0097] 0.......=Marker:False

[0098] Payload type: ITU-T G.711PCMA(8)

[0099] Sequence number: 60154

[0100] [Extended sequence number: 60154]

[0101] Timestamp: 3915211712

[0102] Synchronization Source identifier: 0xf160eaf9(4049660665)

[0103] Real-Time Transport Protocol

[0104] [Stream setup by SDP(frame 7185)]

[0105] 10. .... = Version: RFC 1889 Version (2)

[0106] .0.....=Padding:False

[0107] ...0....=Extension:False

[0108] ....0000=Contributing source identifiers count: 0

[0109] 0.......=Marker:False

[0110] Payload type: ITU-T G.711PCMA(8)

[0111] Sequence number: 60155

[0112] [Extended sequence number: 60155]

[0113] Timestamp: 3915211872

[0114] Synchronization Source identifier: 0xf160eaf9(4049660665)

[0115] After sorting according to the Seq in the RTP header, the server calculates the relative timestamps of two adjacent RTP data frames, and divides the timestamp difference by the standard time interval to determine how many frames of silent data need to be added. Under normal circumstances, the length of a frame is 20 milliseconds, and the standard time interval between two consecutive frames is 160 (the standard time intervals of different media formats are different, generally 160 or 320). If the timestamp difference is n times 160, n-1 silent frames are inserted.

[0116] In summary, the method for restoring multi-channel real-time media stream data provided by the present application adopts the RTP-SDP real-time dynamic mapping algorithm to complete voice extraction and restoration. The multi-channel media voice stream parsing algorithm first diverts the real-time media voice stream according to the diversion identifier, and then parses and restores it according to the single data stream. By receiving and parsing the SDP message in real time and using the dynamic mapping algorithm to build a media type hash table, the media type value, media type name, sampling rate and encoder parameters parsed by the SDP are stored in the hash table, and then the load type of each packet of RTP data is associated with the hash table to match the corresponding media decoder, and finally the RTP load data after abnormal data replacement and silence completion is input into the decoder for decoding and restoration. Therefore, even if media type switching occurs at this stage, it can be handled in a timely manner.

[0117] It should be noted that the step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0118] Based on the above embodiments, the present application also proposes a device for restoring multi-channel real-time media stream data, referring to Figure 4 The apparatus 200 for restoring multi-channel real-time media stream data may include an acquisition module 201, a diversion module 202, a judgment module 203, a first execution module 204, a second execution module 205 and an output module 206, wherein the acquisition module 201 is used to acquire real-time media stream data; the diversion module 202 is used to divert the real-time media stream data according to the diversion identifier to obtain each single-channel data stream; the judgment module 203 is used to judge the data type of the data frame in each single-channel data stream frame by frame; the first execution module 204 is used to parse the SDP message to obtain media parameter information if the data type is judged to be an SDP message, and the media parameter information is obtained based on the media The parameter information constructs a media type mapping relationship, and updates the media type mapping relationship to a pre-constructed hash table, wherein the media type mapping relationship includes a media type encoding value and an encoder; the second execution module 205 is used to parse the RTP data frame to obtain the load type if the data type is determined to be an RTP data frame, match the hash table based on the load type, obtain the encoder adapted to the media type encoding value stored in the media type mapping relationship, and obtain the corresponding decoder based on the encoder; the output module 206 is used to use the decoder to process the RTP data frame to obtain the target media voice stream.

[0119] It is not difficult to find that this embodiment is a system embodiment corresponding to the above method embodiment, and this embodiment can be implemented in conjunction with the above method embodiment. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiments.

[0120] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0121] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A method for restoring multi-channel real-time media stream data, characterized in that: include: Get real-time streaming data; Dividing the real-time stream data according to the diversion identifier to obtain each single-channel data stream; Determining the data type of the data in each of the single-channel data streams; If it is determined that the data type is an SDP message, the SDP message is parsed to obtain media parameter information, a media type mapping relationship is constructed based on the media parameter information, and the media type mapping relationship is updated to a pre-constructed hash table, wherein the media type mapping relationship includes a media type encoding value and an encoder; If it is determined that the data type is an RTP data frame, the RTP data frame is parsed to obtain a payload type, the hash table is matched based on the payload type, the encoder adapted to the media type encoding value stored in the media type mapping relationship is obtained, and a corresponding decoder is obtained based on the encoder; The RTP data frame is processed by the decoder to obtain a target media voice stream.

2. The method for restoring multi-channel real-time media stream data according to claim 1, characterized in that: The real-time stream data is split according to the split identifier to obtain each single-channel data stream, including: Obtain a first diversion identifier and a second diversion identifier, wherein the first diversion identifier is determined based on the source IP address and source port of the data packet, connection information and a media description name address are obtained based on the SDP description information carried by the SDP message, and the second diversion identifier is determined based on the connection information and the media description name address; Based on the first diversion identifier and the second diversion identifier, the real-time stream data is diverted to obtain the single-channel data streams.

3. The method for restoring multi-channel real-time media stream data according to claim 2, characterized in that: The parsing of the SDP message to obtain media parameter information, and constructing a media type mapping relationship based on the media parameter information, includes: Parsing the SDP message to obtain SDP description information; Obtaining SDP attribute information based on the SDP description information, wherein the SDP attribute information includes a media type encoding value, a media type encoding name, a coding rate, and an encoder name; The media type mapping relationship is constructed based on the media type encoding value, the media type encoding name, the encoding rate and the encoder name.

4. The method for restoring multi-channel real-time media stream data according to claim 3, characterized in that: The step of matching the hash table based on the payload type to obtain the encoder adapted to the media type encoding value stored in the media type mapping relationship, and obtaining a corresponding decoder based on the encoder includes: Acquire the load type stored in the RTP data frame, wherein the load type includes a plurality of audio media type encoding values ​​and a plurality of video media type encoding values; Based on the multiple audio media type encoding values ​​and / or the multiple video media type encoding values, matching the media type mapping relationship stored in the hash table, obtaining the adapted media type encoding value, the media type encoding name and the encoder name; The corresponding decoder is determined based on the media type encoding name and the encoder.

5. The method for restoring multi-channel real-time media stream data according to claim 4, characterized in that: After matching the media type mapping relationship stored in the hash table based on the multiple audio media type encoding values ​​and / or the multiple video media type encoding values, the method for restoring multi-channel real-time media stream data further includes: If the adapted media type encoding value cannot be obtained, the RTP data frame is cached in a temporary file.

6. The method for restoring multi-channel real-time media stream data according to claim 1, characterized in that: The method of using the decoder to process the RTP data frame to obtain a target media voice stream includes: Obtain the media type encoding name of the RTP data frame; Initializing the extraction interface of the decoder based on the initialization interface of the decoder and the media type encoding name; Utilizing the extraction interface of the decoder to extract the RTP data frame, and calling the decoding interface of the decoder to decode the RTP data frame; The RTP data is decoded by using the decoder to obtain the target media voice stream.

7. The method for restoring multi-channel real-time media stream data according to claim 1, characterized in that: After the media type mapping relationship is updated in the pre-built hash table, the method for restoring the multi-channel real-time media stream data further includes: The cached RTP data frames and the real-time RTP data frames are deduplicated and sorted based on the timestamp and sequence number identifier of the RTP data frames.

8. The method for restoring multi-channel real-time media stream data according to claim 7, characterized in that: After performing deduplication and sorting operations on the cached RTP data frames and the real-time RTP data frames based on the timestamp and sequence number identifier of the RTP data frames, the method for restoring multi-channel real-time media stream data further includes: It is determined whether the RTP data frame is damaged according to the frame header attribute in the RTP data frame. If the RTP data frame is not damaged, each data frame is filled with a silent frame.

9. The method for restoring multi-channel real-time media stream data according to claim 7, characterized in that: After performing deduplication and sorting operations on the cached RTP data frames and the real-time RTP data frames based on the timestamp and sequence number identifier of the RTP data frames, the method for restoring the multi-channel real-time media stream data further includes: Calculate the relative timestamp difference between two adjacent RTP data frames to obtain a frame difference; The frame difference is divided by a standard time interval to obtain the number of silent frames, and silent frames equal to the number of silent frames are inserted between two adjacent RTP data frames, wherein the standard time interval is determined according to the media type coding value.

10. A device for restoring multi-channel real-time media stream data, characterized in that: include: An acquisition module is used to acquire real-time media stream data; A shunting module, used for shunting the real-time media stream data according to the shunting identifier to obtain each single-channel data stream; A determination module, used for determining the data type of the data in each of the single-channel data streams; A first execution module is configured to parse the SDP message to obtain media parameter information if the data type is determined to be an SDP message, construct a media type mapping relationship based on the media parameter information, and update the media type mapping relationship to a pre-constructed hash table, wherein the media type mapping relationship includes a media type encoding value and an encoder; A second execution module is used for parsing the RTP data frame to obtain a load type if it is determined that the data type is an RTP data frame, matching the hash table based on the load type, obtaining the encoder adapted to the media type encoding value stored in the media type mapping relationship, and obtaining a corresponding decoder based on the encoder; The output module is used to process the RTP data frame using the decoder to obtain a target media voice stream.

Images