Recording method and computer program product

Abstract

The invention discloses a recording method and a computer program product. The method is applied to a recording system, the recording system comprises a media recording module and a media sound combining module, and the method comprises the steps that under the condition that the media recording module receives that a signaling server initiates session establishment, a native file is initialized, sound combining information is written into the native file, and the session establishment carries the sound combining information; the media recording module monitors real-time transport protocol (RTP) streams of the calling terminal and the called terminal to obtain call information, and writes the call information into a native file; the media recording module encapsulates the RTP stream data of the calling terminal and the called terminal, sorts the encapsulated RTP stream data in a buffer area, and writes the sorted RTP stream data into a native file; and under the condition that the call is ended, the media sound combining module carries out sound combining based on the information in the native file to obtain a sound recording file.

Description

Technical Field

[0001] The present application relates to the field of telephone calls, and in particular to a recording method and a computer program product. Background Art

[0002] In the virtual number service, after the signaling completes the media negotiation, the Real-time Transport Protocol (RTP) stream of the calling and called parties is transmitted to the other party through the intermediary media service to realize the call. During this process, media processing such as recording will be completed in the intermediary media service.

[0003] In the related art, there are generally two solutions for recording calls in the virtual number scenario: 1. Parse the RTP streams of the caller and the called party to obtain the corresponding audio data respectively, and then perform real-time decoding, encoding, and harmonization operations according to the harmonization information of different companies to obtain a result file. However, the success rate of this real-time harmonization recording is low, and the service operation and maintenance costs are high. 2. Parse the audio data of the caller and the called party in the RTP stream and save them as audio files respectively, and then transmit the harmonization information and the recording file to the harmonization service through an asynchronous message mechanism to synthesize the recording into one file. However, this method has a complicated harmonization process, and there are problems such as sound asynchrony and partial audio data loss after harmonization. Summary of the invention

[0004] The embodiments of the present application provide a recording method and a computer program product to solve the problem of low success rate of chorus recording in the related art.

[0005] In a first aspect, an embodiment of the present application provides a recording method, which is applied to a recording system, wherein the recording system includes a media recording module and a media sound mixing module, and the method includes: When the media recording module receives a signaling server to initiate a session establishment, the native file is initialized, and the chorus information is written into the native file, and the chorus information is carried in the session establishment; The media recording module monitors the real-time transport protocol RTP streams of the calling and called terminals, obtains call information, and writes the call information into the native file; The media recording module encapsulates the RTP stream data of the calling and called terminals, sorts the encapsulated RTP stream data in a buffer, and writes the sorted RTP stream data into the native file; When the call is ended, the media audio-resonance module performs audio-resonance based on the information in the native file to obtain a recording file.

[0006] In a second aspect, an embodiment of the present application provides a recording system, including a media recording module and a media audio compositing module, wherein: The media recording module is used to initialize the native file and write the chorus information into the native file when receiving the signaling server to initiate the session establishment, and the chorus information is carried in the session establishment; monitor the real-time transport protocol RTP stream of both the calling terminal and the called terminal to obtain the call information, and write the call information into the native file; encapsulate the RTP stream data of both the calling terminal and the called terminal, sort the encapsulated RTP stream data in the buffer, and write the sorted RTP stream data into the native file; The media audio-resonance module is used to perform audio-resonance based on the information in the native file to obtain a recording file when the call is ended.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions, which, when executed by a computer, implement the steps of the method described in the first aspect.

[0010] In an embodiment of the present application, a native file is set during the recording process, and relevant files of the recording, such as harmony information, call information, and encapsulated RTP data, are written into the native file. During the harmony process, harmony is performed based on the above-mentioned native file. Since the RTP information of the caller and the called party is saved in the native file, it is convenient for positioning and repair, thereby improving the success rate of harmony recording. In addition, a buffer is set in the process of storing the recording file, and the encapsulated RTP stream data is sorted in the buffer. The buffer can be used to solve the problems of disorder or packet loss, thereby further improving the success rate of harmony recording. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a flow chart of the recording method provided in an embodiment of the present application; Figure 2It is a schematic diagram of the RTP stream data encapsulation structure provided by an embodiment of the present application; Figure 3 It is a schematic diagram of sorting the encapsulated RTP stream data in the buffer provided by the embodiment of the present application; Figure 4 It is a schematic diagram of aligning RTP stream data of a calling terminal and a called terminal in a time slot provided by an embodiment of the present application; Figure 5 is an overall flow chart of the recording method provided in the embodiment of the present application; Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0013] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here. In addition, the "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated with each other are in an "or" relationship.

[0014] The following is combined with Figures 1 to 6 , a recording method and a computer program product provided in an embodiment of the present application are described in detail through specific embodiments and their application scenarios.

[0015] like Figure 1 As shown, it is a flow chart of a recording method provided by an embodiment of the present application. Figure 1 As shown, the recording method is applied to a recording system, and the recording system includes a media recording module and a media sound mixing module. The recording method may include: the contents shown in S101 to S104.

[0016] In S101, when the media recording module receives a signaling server to initiate a session establishment, the native file is initialized and the chorus information is written into the native file, and the chorus information is carried in the session establishment.

[0017] It is worth noting that the recording system may include a media recording module and a media synthesis module, wherein the media recording module implements recording according to signaling information and RTP stream to record native files; the media synthesis module processes the native files to obtain call information, synthesis information and RTP stream data of call voice to implement synthesis.

[0018] Among them, the signaling server is used to provide signaling notifications for the media recording module.

[0019] In this example, when the signaling server receives the call signaling from the calling terminal, the signaling module of the signaling server goes to transfer and query the chorus information corresponding to the session. The signaling server initiates the session establishment to the media recording module. During the session establishment between the signaling server and the media recording module, the chorus information is passed to the media recording module through the network interface. The session start module of the media recording module will store the corresponding chorus information, initialize the native file, and complete the creation of the native file (RawFile) of the call record in the memory file system. Then, the chorus information is written into the native file. Writing files in the memory file system is equivalent to writing directly into the memory. Reading and writing are completed instantly, which improves storage efficiency.

[0020] It should be noted that in the virtual number service, different enterprises may have differently configured chorus information, and the same enterprise may also configure chorus information based on the bound number relationship. The chorus information is based on the actual information obtained, and this embodiment does not limit it.

[0021] In S102, the media recording module monitors the real-time transport protocol RTP streams of the calling and called terminals, obtains call information, and writes the call information into a native file.

[0022] That is to say, the signaling server will replace the media addresses of the RTP streams of the calling and called terminals with the address and port of the media recording module during media negotiation. After completing the media negotiation, the signaling server notifies the media recording module through the media negotiation interface that the call has started. At this time, the relevant information of the call has been obtained. The media recording module monitors the RTP streams of the calling and called terminals according to the address and port transmitted by the signaling, and writes the call information in the native file. The call information can include the unique ID of the call bill, enterprise ID, calling terminal number, called terminal number, call start time, etc.

[0023] In S103, the media recording module encapsulates the RTP stream data of both the calling terminal and the called terminal, sorts the encapsulated RTP stream data in the buffer, and writes the sorted RTP stream data into the native file.

[0024] In other words, the media recording module can encapsulate the RTP stream data of both the calling and called terminals to record the RTP status, early identification, calling and called parties, encoding type, relative timestamp and other information of the RTP stream data, so as to facilitate subsequent sound synthesis, and then sort the encapsulated RTP stream data in the buffer to facilitate determining whether there is packet loss, and finally write it into the native file, thereby improving the accuracy of the sound synthesis.

[0025] In S104, when the call is ended, the media audio synthesis module performs audio synthesis based on the information in the native file to obtain a recording file.

[0026] In the embodiment of the present application, first, when the media recording module receives the signaling server to initiate the session establishment, the native file is initialized and the chorus information is written into the native file. The chorus information is carried in the session establishment. Then, the media recording module monitors the real-time transport protocol RTP stream of both the calling and called terminals to obtain the call information and write the call information into the native file. The media recording module encapsulates the RTP stream data of both the calling and called terminals, sorts the encapsulated RTP stream data in the buffer, writes the sorted RTP stream data into the native file, and finally, when the call ends, the media chorus module performs chorus based on the information in the native file to obtain the recording file. In this embodiment, the chorus information, call information and encapsulated RTP data are written into the native file to facilitate the chorus to locate and repair, and the encapsulated RTP data is cached into the buffer based on the synchronization source identifier and the sequence number of the message, so that the RTP data is sorted regularly, further reducing abnormal situations and improving the success rate of chorus.

[0027] In an embodiment of the present application, a native file is set during the recording process, and relevant files of the recording, such as harmony information, call information and encapsulated RTP data, are written into the native file. During the harmony process, harmony is performed based on the above-mentioned native file. Since the RTP information of the caller and the called party is saved in the native file, it is convenient to locate and repair, thereby improving the success rate of harmony recording. In addition, a buffer is set in the process of storing the recording file, and the encapsulated RTP stream data is sorted in the buffer. The buffer can be used to solve the problems of disorder or packet loss, thereby further improving the success rate of harmony recording.

[0028] In a possible implementation manner of the present application, the RTP stream data of both the calling terminal and the called terminal include early RTP stream data and call RTP stream data.

[0029] That is to say, after the signaling server completes the media negotiation with the media recording module, the RTP stream starts to be transmitted. After receiving the media negotiation notification from the signaling server, the media service module starts to monitor the RTP streams of the calling party and the called party, and transparently transmits the calling party's RTP stream to the called party, and transparently transmits the called party's RTP stream to the calling party, so that the call voice can reach the other party.

[0030] It is worth noting that when the called terminal is not connected, the media stream is early media RTP, and the calling terminal user can hear the beep tone or the ringback tone set by the called user. When the signaling server notifies the media recording module that the call has started, the transmitted media stream is call RTP, and the calling and called terminals can hear the other party's voice.

[0031] Encapsulating the RTP stream data of both the calling and called terminals may include: adding a header identifier to the RTP stream data, the header identifier is used to record the RTP state, early identifier, calling and called terminal identifier, encoding type and relative timestamp; combining the header identifier with the RTP stream data to obtain the encapsulated RTP stream data.

[0032] In this embodiment, in the recording native file, it is necessary to identify the early RTP, call RTP record, etc., and a header identifier can be added to the RTP to record the RTP status, early identifier, calling and called terminal identifiers, encoding type, relative timestamp and other information, and then add the RTP data itself to form a new RTP encapsulation structure [RTP header identifier] [RTP data], specifically as follows Figure 2 shown.

[0033] like Figure 2 As shown, the first line is the information in the native file, wherein each RTP encapsulation structure is used to store the encapsulated RTP stream data of the calling and called terminals, and each RTP encapsulation structure includes two parts: an RTP header identifier and original RTP data, wherein the RTP header identifier part may include field information such as RTP data type, length, RTP status, early identifier, calling and called terminal identifier, encoding type, relative timestamp, etc.

[0034] The relative timestamp records the relative time from the start of a call to the present. This field is used to solve the problem of frame filling later.

[0035] In a possible implementation of the present application, sorting the encapsulated RTP stream data in the buffer may include: the media recording module parses the encapsulated RTP stream data to obtain the synchronization source identifier (Synchronization Source Identifer, SSRC) and the sequence number (Sequence number, SEQ) of the message in the RTP stream data corresponding to each SSRC; the media recording module writes the encapsulated RTP stream data into different buffers in sequence according to the calling and called terminals according to the order in which the SSRC enters the buffer and the value of SEQ, wherein the terminal corresponding to the SSRC that enters the buffer first is written into the previous buffer, and the RTP stream data with a smaller SEQ value in the buffer is cached to the front column of the buffer.

[0036] In this embodiment, after encapsulating the RTP stream data of both the calling and called terminals, the SSRC and SEQ of RTP are first parsed, and then the data is written into the buffer according to <SSRC, SEQ, RTP encapsulation structure>. Different buffers (buffers) are used for the calling and called terminals respectively.

[0037] As Figure 3 shown, the SSRC of the RTP stream data of the calling terminal is ssrc1, and the RTP stream data of the calling terminal is written into the left buffer, that is, the calling buffer. The SSRC of the RTP stream data of the called terminal is ssrc2, and the RTP stream data of the called terminal is written into the right buffer, that is, the called buffer. Among them, the data is written into the main and called buffers in sequence according to the value of SEQ. The one with a smaller SEQ value is written first, that is, at the front of the queue, and the one with a smaller SEQ value is written later, that is, at the back of the queue, that is, behind the RTP stream data with a smaller SEQ value. Finally, under qualified conditions, it is written into the raw file (RawFile).

[0038] In this embodiment, the buffer is a queue with a sorting rule. The order of entry of SSRC in the three-dimensional structure <SSRC, SEQ, RTP encapsulation structure> into the buffer is used as the first basis for sorting, and the size of the SEQ value is used as the second basis for sorting. The one with a smaller SEQ value is in front of the queue in the buffer.

[0039] It should be noted that the SSRC may change due to multiple media negotiations of signaling, resulting in its change following the change of the IP or port of the RTP stream. The SSRC may also change due to subscribing to or customizing the voice playback or ringback tone service. When a network element sends audio and video RTP to the calling terminal, the SSRC in this case marks the RTP source of this network element. When the called terminal continues to communicate with the calling terminal, the SSRC marks the RTP source of the called terminal.

[0040] In a possible implementation manner of this application, the recording method may further include: during a call, each time the terminal sends an RTP packet, the value of SEQ increases by one; the media recording module determines the RTP status of the RTP stream data according to whether the value of SEQ in the received RTP stream data is continuous. The RTP status includes a normal status, a pending status, and a discarded status.

[0041] That is to say, during a call, each time the terminal sends an RTP packet, the value of SEQ corresponding to this terminal increases by one. When the network condition is not good, SEQ can be used to check for packet loss, and at the same time, SEQ can also be used to reorder the data in case of network jitter.

[0042] In the buffer, the packets are first sorted according to the order of arrival of SSRC, and then sorted in ascending order according to SEQ, which can ensure that the packets from different SSRC sources are in order in the buffer. The packet loss delay time of the communication network is guaranteed to be within a certain time, which can be two seconds, three seconds, or several seconds. If the buffer can store voice data of more than a few seconds, it can ensure that the voice data is in order during this period after passing through the buffer. In other words, the buffer can buffer data for a certain period of time, ensuring that there is no packet loss in the case of a delay within a certain period of time.

[0043] It should be noted that in extreme cases, if the network packet loss exceeds ten seconds, the data is also equivalent to packet loss. Real-time calls handle voice in this delayed situation as packet loss, that is, there is no need to sort and it is directly discarded.

[0044] In the above RTP encapsulation structure, there is a field called RTP status field. RTP status includes pending status, discarded status and normal status. The sorting in the buffer described above is only for the sorting of normal status. For data in abnormal status, sorting is triggered when entering the buffer. The current RTP data is identified by setting the status because the current information is not enough to determine whether it is normal, and it is necessary to wait for the subsequent RTP stream information for further confirmation.

[0045] In a possible implementation of the present application, the media recording module determines the RTP state according to whether the value of SEQ in the received RTP stream data is continuous, which may include: when the media recording module receives the first RTP stream data, comparing the value of the first SEQ of the first RTP stream data with the value of the second SEQ in the previous RTP stream data; when the difference between the value of the first SEQ and the value of the second SEQ is greater than the first difference, setting the RTP state of the first RTP stream data to a pending state; when the values ​​of SEQ of a first number of RTP stream data after the media recording module receives the first RTP stream data are all less than the value of the first SEQ, setting the RTP state of the first RTP stream data to a discarded state, and the first number is greater than a first number threshold; when the values ​​of SEQ of the RTP stream data after the media recording module receives the first RTP stream data are all greater than the value of the first SEQ and gradually increase, setting the RTP state of the first RTP stream data to a normal state.

[0046] That is to say, during a call, if the SEQ value in a certain RTP stream data is S1, and the S1 value is much larger than the SEQ value in the previous RTP stream data, it is not possible to determine whether the SEQ is abnormal. You can first set the RTP state to pending. If the subsequent multiple SEQs are less than this value, the SEQ is considered abnormal, and the RTP state is set to discarded, and it is not parsed as voice data in the subsequent parsing link. If the subsequent SEQ is increasing, the pending state is set to normal, which can prevent the abnormal RTP stream data in the network from affecting the stability of the media recording module.

[0047] In a possible implementation of the present application, the recording method may also include: when the value of the first SEQ is less than the value of the second SEQ, determining whether the first RTP stream data is the RTP stream data for the next cycle; when the first RTP stream data is the RTP stream data for the next cycle, setting the RTP state of the first RTP stream data to a normal state; when the first RTP stream data is not the RTP stream data for the next cycle, determining the difference between the value of the first SEQ and the value of the second SEQ; when the difference between the value of the first SEQ and the value of the second SEQ is less than or equal to the second difference, setting the RTP state of the first RTP stream data to a normal state; when the difference between the value of the first SEQ and the value of the second SEQ is greater than the second difference, setting the RTP state of the first RTP stream data to a pending state, and determining whether the RTP state of the first RTP stream data is a normal state or a discarded state based on the subsequently received RTP stream data.

[0048] That is to say, if the SEQ value in the received RTP stream data is smaller than the SEQ value in the previous RTP stream data, it is necessary to consider whether it enters the next cycle. If so, the RTP state of the RTP stream data is set to a normal state. If it does not enter the next cycle, further compare whether the difference between the two SEQ values ​​is within the length range that the buffer can receive. If the difference between the two SEQ values ​​is within the length range that the buffer can receive, it can be cached in the buffer first, and the RTP state of the RTP stream data is set to a normal state. If the difference between the two SEQ values ​​is not within the length range that the buffer can receive, that is, it exceeds the length range that the buffer can receive, the RTP state of the RTP stream data is set to a pending state, and the RTP state of the RTP stream data is determined to be a normal state or a discarded state based on the subsequently received RTP stream data.

[0049] Among them, determining whether the RTP state of the RTP stream data is normal or discarded based on the subsequently received RTP stream data is the same as the judgment method in the above-mentioned embodiment where the S1 value is much larger than the SEQ value in the previous RTP stream data. This embodiment will not be repeated here, and the above-mentioned embodiment shall prevail.

[0050] In this embodiment, a new structure, namely a buffer, is defined to save data. The buffer can be used to solve the problem of disorder or packet loss. Abnormal RTP data can be avoided according to the state of RTP, so that the data finally written into RawFile is normal data.

[0051] In a possible implementation of the present application, writing the sorted RTP stream data into a native file may include: when it is determined that the RTP state of the RTP stream data is a normal state, the media recording module writes the sorted RTP stream data into the native file.

[0052] That is to say, the RTP state of the RTP stream data written into the native file is normal, which avoids sending abnormal data to the media synthesis module, so that the RTP stream data used by the media synthesis module for synthesis are all normal data, ensuring the quality of the synthesis.

[0053] In a possible implementation of the present application, writing the sorted RTP stream data to the native file may include: when the sorting time of the RTP stream data in the buffer exceeds a time threshold, writing the RTP stream data to the native file; or when the RTP stream data in the buffer exceeds a data threshold, writing the RTP stream data in the buffer to the native file; or setting a timer, and writing the RTP stream data in the buffer to the native file based on the timer.

[0054] In other words, you can set a timer to write the data in the buffer area into the RawFile based on the timer, and you can also set a threshold for the buffer value in the buffer. When the data in the buffer exceeds the threshold, the data in the buffer area is written into the RawFile to ensure that the buffer can always buffer a certain amount of data. You can also write the RTP stream data into the RawFile based on the sorting time of the RTP stream data in the buffer when the sorting time exceeds the time threshold. This allows the buffer to buffer a certain amount or a certain length of data, ensuring that the network does not lose packets when there is a certain delay, making the data used by the subsequent chorus more complete.

[0055] In a possible implementation of the present application, the recording method may further include: when the call ends, the media recording module stops receiving the RTP stream data, and writes the RTP stream data in the buffer with a normal RTP status into a native file.

[0056] That is to say, when the call ends, the signaling server will send a session stop notification to the media recording module, the media recording module will stop receiving RTP stream data, and write the RTP stream data with normal RTP status in the buffer into the native file to ensure that all data received by the media recording module is written into the native file.

[0057] In a possible implementation of the present application, when a call ends, the media synthesis module performs synthesis based on the information in the native file to obtain a recording file, which may include: when the call ends, the media recording module sends asynchronous information to the media synthesis module to notify the media synthesis module to perform synthesis; the media synthesis module obtains the native file from the media synthesis module based on the asynchronous information, the native file including the synthesis information, call information and encapsulated RTP stream data of the calling and called terminals; the media synthesis module performs synthesis based on the native file to obtain a recording file.

[0058] That is to say, when the call ends, the media recording module sends an asynchronous message to the media resonant module to notify the media resonant module that it can start resonant. After the call ends, the information written into the RawFile in the media recording module records the call information, resonant information, and RTP stream information of the caller and the called party in the call. This file can be used to restore the voice stream information in the call scenario. The message that the media recording module notifies the media resonant module only needs to tell how to obtain the original file. The media resonant module can obtain it from the media recording module, which can reduce database queries and interface queries. The call information and resonant information can be parsed by getting the recording file, without the need to query the database or use the interface to transfer the query, thus reducing unnecessary query operations.

[0059] In a possible implementation of the present application, sound synthesis based on the native file to obtain a recording file may include: parsing the native file to obtain sound synthesis information, call information and encapsulated RTP stream data of the calling and called terminals; performing frame insertion processing on the parsed RTP stream data of the calling and called terminals to obtain a frame insertion file, the frame insertion file including an early call file of the calling terminal, an early call file of the called terminal, a calling call file and a called call file; synthesizing the frame insertion file, the sound synthesis information and the call information to obtain a recording file.

[0060] In this embodiment, the parser of the media synchrotron module can parse the synchrotron information, call information and the encapsulated RTP stream data of both the calling and called terminals according to the native file, and then perform frame interpolation processing on the parsed RTP stream data of both the calling and called terminals to obtain a frame interpolation file, which includes the early call file of the calling terminal, the early call file of the called terminal, the calling call file and the called call file; the frame interpolation file, the synchrotron information and the call information are synchrotroned to obtain a recording file. The problem of sound asynchrony can be avoided by frame interpolation. Since the synchrotron information is written in the native file, the transfer query in the synchrotron re-push link can be reduced, which is more conducive to achieving synchrotron synchronization of the calling and called terminals.

[0061] In a possible implementation of the present application, the parsed RTP stream data of the calling and called terminals are interpolated to obtain an interpolated frame file, which may include: setting time slots at intervals of preset duration according to the duration of the call; synchronizing the RTP stream data of the calling and called terminals in each time slot according to the relative timestamp in the native file; when the RTP stream data of the calling and called terminals are synchronized, interpolating the RTP stream data in a time slot according to the RTP timestamp, where the RTP timestamp is obtained by the media synthesis module based on the parsing of the RTP stream data.

[0062] That is to say, after parsing the original file, it is necessary to add frames to avoid the situation where direct synthesis may cause the caller and the called party's voice to be out of sync.

[0063] In this embodiment, synchronization is performed according to the relative timestamp in the native file, and the calling and called terminals in the original file can be synchronized, that is, synchronization is performed between groups. The time slot can be set according to the total duration of the call, and the inter-group synchronization is performed through the time slot, that is, the RTP stream data of the calling and called terminals in the same time slot are placed in the corresponding time slot, and for the same time slot, different RTP stream data are further supplemented according to the RTP timestamp, that is, supplemented according to the difference between the timestamps of the two RTPs, until the data meets the time slot, that is, the duration of the supplemented frame data and the data in the original slot does not exceed the time set by the timestamp. The timestamp of RTP is a step value. When the sampling rate is determined, the increment of this value is also determined, so that the supplementary frame can be completed between two RTP data according to the timestamp of RTP. Among them, the RTP timestamp can be parsed from the RTP stream.

[0064] like Figure 4As shown, the time slot can be set at intervals of two seconds, that is, the time intervals between 0-t1, t1-t2, etc. are all two seconds. At this time, the parsed calling and called data can be placed in the corresponding time slots according to the recorded relative timestamps. For example, the calling data with a relative timestamp less than t1 is placed in the LeftGroup-1 group, and the called data with a relative timestamp less than t1 is placed in the RightGroup-1 group. This is equivalent to the calling and called parties performing a group of frame supplementation according to the time reference method to ensure that the data between the groups is synchronized.

[0065] The interpolation of frames within a group refers to whether interpolation is needed between Ri, Ri+1, Ri+2, etc., which can be determined based on the difference between timestamps. For example, at a sampling rate of 8000, the timestamp increment is 160, the timestamp of RTP1 is 333000, and the timestamp of RTP2 is 333320. The difference between RTP2 and RTP1 is 320, and the timestamp difference is 2 160s, so 2-1 silent frames need to be interpolated.

[0066] In this embodiment, relative time slots are used to reference each other between the caller and the called party to supplement frames, and RTP timestamps are used within a group of RTP data to further supplement frames, restore the voice data of the call scene, and achieve voice synchronization between the caller and the called party.

[0067] In a possible implementation of the present application, the recording method may also include: in the case where there are multiple negotiations between the media recording module and the signaling server, real-time decoding is performed according to the encoding type and load type in the RTP stream data to obtain decoded pulse code modulation data; and the bytes corresponding to the decoded pulse code modulation data are spliced ​​to obtain a frame supplement file.

[0068] In this embodiment, since there may be multiple media negotiations between the signaling server and the media recording module, this will lead to the existence of multiple encoded data in the native file. In the frame supplementation link, the pulse code modulation (PCM) data after decoding can be directly supplemented. That is to say, decoding is performed in the frame supplementation, and real-time decoding is performed according to the recorded encoding type and the load type (PayloadType) in RTP. The final PCM stream after audio decoding, after different encodings and decoding PCM, directly splicing the corresponding bytes can assemble a file. The final delivery to the synthesis is the calling file stream and the called file stream. Here, early and call are also distinguished, so two pairs of file streams will be obtained here. After finally obtaining the early calling and called PCM file streams and the calling and called call streams, the synthesis file information is constructed according to the call information, and then according to the configuration items of the synthesis information, it is decided to synthesize the synthesis file into a customized style to complete the entire synthesis process.

[0069] In this embodiment, native files are stored in the media recording module. During the storage process, the call data buffer is used to solve the RTP disorder, the RTP status is used to identify abnormal RTP data, and the call data of the caller and the called party are written into a native file for storage according to the defined structure. In the sound-joining module, after parsing the caller and the called party data, a secondary frame filling operation is performed. First, the caller and the called party data are grouped in different time slots according to the relative timestamp, and the inter-group frame filling for mutual reference between the caller and the called party is completed. Then, the frames are filled again between the RTPs in the group according to the RTP timestamp, so as to further reduce the asynchrony between the caller and the called party. In the sound-joining module, the voice data can be parsed in a customized strategy. In the case of multiple encodings, they are uniformly decoded into PCM streams and then joined.

[0070] Figure 5 The specific process has been described in detail in the above embodiments, and will not be repeated in this embodiment. The above embodiments shall prevail.

[0071] The embodiment of the present application also provides a recording system, including a media recording module and a media sound-resonance module, wherein the media recording module is used to initialize a native file and write sound-resonance information into the native file when a signaling server initiates session establishment, and the sound-resonance information is carried in the session establishment; monitor the real-time transport protocol RTP stream of both the calling and called terminals to obtain call information, and write the call information into the native file; encapsulate the RTP stream data of both the calling and called terminals, sort the encapsulated RTP stream data in a buffer, and write the sorted RTP stream data into the native file; the media sound-resonance module is used to perform sound resonance based on the information in the native file to obtain a recording file when the call ends.

[0072] In an embodiment of the present application, a native file is set during the recording process, and relevant files of the recording, such as harmony information, call information and encapsulated RTP data, are written into the native file. During the harmony process, harmony is performed based on the above-mentioned native file. Since the RTP information of the caller and the called party is saved in the native file, it is convenient to locate and repair, thereby improving the success rate of harmony recording. In addition, a buffer is set in the process of storing the recording file, and the encapsulated RTP stream data is sorted in the buffer. The buffer can be used to solve the problems of disorder or packet loss, thereby further improving the success rate of harmony recording.

[0073] like Figure 6 As shown, the embodiment of the present application also provides an electronic device 600, including a processor 601, a memory 602, and a program or instruction stored in the memory 602 and executable on the processor 601. When the program or instruction is executed by the processor 601, each process of the above-mentioned recording processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0074] Optionally, the embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above recording method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0075] Optionally, an embodiment of the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the various processes of the recording method embodiment described above are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be repeated here.

[0076] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0077] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0078] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A recording method, characterized in that: Applied to a recording system, the recording system includes a media recording module and a media sound mixing module, and the method includes: When the media recording module receives a signaling server to initiate a session establishment, the native file is initialized, and the chorus information is written into the native file, and the chorus information is carried in the session establishment; The media recording module monitors the real-time transport protocol RTP streams of the calling and called terminals, obtains call information, and writes the call information into the native file; The media recording module encapsulates the RTP stream data of the calling and called terminals, sorts the encapsulated RTP stream data in a buffer, and writes the sorted RTP stream data into the native file; When the call is ended, the media audio synthesis module performs audio synthesis based on the information in the native file to obtain a recording file.

2. The method according to claim 1, characterized in that The RTP stream data of the calling and called terminals include early RTP stream data and call RTP stream data; The step of encapsulating the RTP stream data of the calling and called terminals comprises: Adding a header identifier to the RTP stream data, wherein the header identifier is used to record the RTP state, early identifier, calling and called terminal identifiers, encoding type and relative timestamp; The header identifier is combined with the RTP stream data to obtain encapsulated RTP stream data.

3. The method according to claim 1, characterized in that The step of sorting the encapsulated RTP stream data in the buffer includes: The media recording module parses the encapsulated RTP stream data to obtain a synchronization source identifier SSRC and a sequence number SEQ of a message in the RTP stream data corresponding to each SSRC; The media recording module writes the encapsulated RTP stream data into different buffers in sequence according to the order in which the SSRC enters the buffer and the value of the SEQ, wherein the terminal corresponding to the SSRC that enters the buffer first is written into the previous buffer, and the RTP stream data with a smaller SEQ value in the buffer is cached to the front row of the buffer.

4. The method according to claim 3, characterized in that The method further comprises: During a call, each time the terminal sends an RTP message, the value of SEQ increases by one; The media recording module determines the RTP state of the RTP stream data according to whether the value of the SEQ in the received RTP stream data is continuous, and the RTP state includes a normal state, a pending state and a discarded state.

5. The method according to claim 4, characterized in that The media recording module determines the RTP state of the RTP stream data according to whether the value of the SEQ in the received RTP stream data is continuous, including: When the media recording module receives the first RTP stream data, compare the value of the first SEQ of the first RTP stream data with the value of the second SEQ in the previous RTP stream data; When the difference between the value of the first SEQ and the value of the second SEQ is greater than a first difference, setting the RTP state of the first RTP stream data to a pending state; In a case where the SEQ values ​​of a first number of RTP stream data after the media recording module receives the first RTP stream data are all smaller than the first SEQ value, setting the RTP state of the first RTP stream data to a discard state, and the first number is greater than a first number threshold; When the SEQ values ​​of the RTP stream data after the media recording module receives the first RTP stream data are all greater than the first SEQ value and gradually increase, the RTP state of the first RTP stream data is set to a normal state.

6. The method according to claim 5, characterized in that The method further comprises: When the value of the first SEQ is less than the value of the second SEQ, determining whether the first RTP stream data is the RTP stream data of the next cycle; In a case where the first RTP stream data is the RTP stream data of the next cycle, setting the RTP state of the first RTP stream data to a normal state; In a case where the first RTP stream data is not the RTP stream data of the next cycle, determining a difference between a value of the first SEQ and a value of the second SEQ; When the difference between the value of the first SEQ and the value of the second SEQ is less than or equal to the second difference, setting the RTP state of the first RTP stream data to a normal state; When the difference between the value of the first SEQ and the value of the second SEQ is greater than the second difference, the RTP state of the first RTP stream data is set to a pending state, and based on the subsequently received RTP stream data, the RTP state of the first RTP stream data is determined to be a normal state or a discarded state.

7. The method according to claim 4, characterized in that The step of writing the sorted RTP stream data into the native file comprises: When it is determined that the RTP state of the RTP stream data is a normal state, the media recording module writes the sorted RTP stream data into the native file.

8. The method according to claim 1, characterized in that The step of writing the sorted RTP stream data into the native file comprises: When the sorting duration of the RTP stream data in the buffer exceeds a duration threshold, writing the RTP stream data into the native file; or When the RTP stream data in the buffer exceeds a data threshold, writing the RTP stream data in the buffer into the native file; or A timer is set, and the RTP stream data in the buffer is written into the native file based on the timer.

9. The method according to claim 1, characterized in that: The method further comprises: When the call ends, the media recording module stops receiving the RTP stream data, and writes the RTP stream data in the buffer with the RTP status being a normal state into the native file.

10. The method according to claim 1, characterized in that When the call is ended, the media audio-resonance module performs audio-resonance based on the information in the native file to obtain a recording file, including: When the call is ended, the media recording module sends an asynchronous message to the media audio-mixing module to notify the media audio-mixing module to perform audio-mixing; The media audio-mixing module obtains the native file from the media audio-mixing module based on the asynchronous information, wherein the native file includes audio-mixing information, call information, and encapsulated RTP stream data of both the calling and called terminals; The media sound-mixing module performs sound-mixing based on the native file to obtain a recording file.

11. The method according to claim 10, characterized in that Performing a harmonization based on the original file to obtain a recording file includes: Parsing the native file to obtain the audio information, call information and encapsulated RTP stream data of both the calling and called terminals; Performing frame interpolation processing on the parsed RTP stream data of both the calling and called terminals to obtain an interpolation file, wherein the interpolation file includes an early call file of the calling terminal, an early call file of the called terminal, a calling call file, and a called call file; The frame supplementation file, the sound fusion information and the call information are fused to obtain a recording file.

12. The method according to claim 11, characterized in that The frame-filling process is performed on the parsed RTP stream data of the calling and called terminals to obtain a frame-filling file, including: According to the duration of the call, time slots are set at intervals of preset duration; Synchronizing the RTP stream data of the calling and called terminals in each time slot according to the relative timestamp in the native file; In the case where the RTP stream data of the calling and called terminals are synchronized, the RTP stream data is frame-added according to the RTP timestamp within a time slot, and the RTP timestamp is obtained by the media synthesis module based on the analysis of the RTP stream data.

13. The method according to claim 12, characterized in that The method further comprises: In the case where there are multiple negotiations between the media recording module and the signaling server, real-time decoding is performed according to the coding type and the load type in the RTP stream data to obtain decoded pulse code modulation data; The bytes corresponding to the decoded pulse code modulation data are spliced ​​to obtain a frame-filling file.

14. A recording system, characterized in that: It includes a media recording module and a media synchro module, wherein: The media recording module is used to initialize the native file and write the chorus information into the native file when receiving the signaling server to initiate the session establishment, and the chorus information is carried in the session establishment; monitor the real-time transport protocol RTP stream of both the calling and called terminals to obtain the call information, and write the call information into the native file; encapsulate the RTP stream data of both the calling and called terminals, sort the encapsulated RTP stream data in the buffer, and write the sorted RTP stream data into the native file; The media audio-resonance module is used to perform audio-resonance based on the information in the native file to obtain a recording file when the call is ended.

15. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

17. A computer program product, characterized in that The computer program product comprises a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the steps of the method according to any one of claims 1 to 13 are implemented.

Images