Audio recording method and computer program product

Abstract

The application discloses a recording method and a computer program product. The method is applied to a recording system, the recording system comprises a media recording module and a media mixing module, and the method comprises the following steps: in the case that the media recording module receives a signaling server to initiate session establishment, initializing a native file, and writing mixing information into the native file, the mixing information being carried in the session establishment; the media recording module listens to real-time transport protocol (RTP) streams of both a calling terminal and a called terminal, obtains call information, and writes the call information into the native file; the media recording module encapsulates the RTP stream data of both the calling terminal and the called terminal, sorts the encapsulated RTP stream data in a buffer, and writes the sorted RTP stream data into the native file; and in the case that the call is ended, the media mixing module mixes based on the information in the native file to obtain a recording file.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of call, in particular to a recording method and a computer program product. BACKGROUND

[0002] In a virtual number service, after media negotiation is completed through signaling, real-time transport protocol (RTP) streams of a calling terminal and a called terminal are transmitted to each other through media services of a mediation server to realize a call, and in this process, media processing such as recording is completed in the media services of the mediation server.

[0003] In related technologies, there are generally two schemes for recording a call in a virtual number scenario: 1. Audio data corresponding to RTP streams of a calling terminal and a called terminal are respectively parsed, and then a result file is obtained through real-time decoding, encoding, and mixing of different enterprise mixing information, but the real-time mixing recording success rate is low, and the service operation and maintenance cost is high; 2. Audio data of the calling terminal and the called terminal are respectively saved as audio files, and then the mixing information and the recording files are transmitted to a mixing service through an asynchronous message mechanism to synthesize the recording into one file, but the mixing process is complex, and there are situations such as asynchronous sound and missing of part of the audio data after mixing. SUMMARY

[0004] Embodiments of the present application provide a recording method and a computer program product to solve the problem of low mixing recording success rate in related technologies.

[0005] In a first aspect, embodiments of the present application provide a recording method applied to a recording system, the recording system comprising a media recording module and a media mixing module, and the method comprising:

[0006] In a case where the media recording module receives a session establishment initiated by a signaling service end, initializing a native file and writing mixing information into the native file, the session establishment carrying the mixing information;

[0007] The media recording module listens to real-time transport protocol (RTP) streams of a calling terminal and a called terminal, obtains call information, and writes the call information into the native file;

[0008] The media recording module encapsulates RTP stream data of the calling terminal and the called terminal, sorts the encapsulated RTP stream data in a buffer, and writes the sorted RTP stream data into the native file;

[0009] In a case where the call ends, the media mixing module mixes based on information in the native file to obtain a recording file.

[0010] In a second aspect, the embodiments of the present application provide a recording system, comprising a media recording module and a media mixing module, wherein

[0011] The media recording module is configured to initialize a native file in the case of receiving a signaling server initiating session establishment, write mixing information into the native file, and carry the mixing information in the session establishment; listen to real-time transport protocol (RTP) streams of a calling terminal and a called terminal to obtain call information, and write the call information into the native file; encapsulate the RTP stream data of the calling terminal and the called terminal, sort the encapsulated RTP stream data in a buffer, and write the sorted RTP stream data into the native file.

[0012] The media mixing module is configured to mix based on information in the native file in the case of ending a call to obtain a recording file.

[0013] In a third aspect, the embodiments of the present application provide an electronic device, comprising a processor, a memory, and a program or instructions stored in the memory and executable on the processor, and the program or instructions are executed by the processor to implement the steps of the method in the first aspect.

[0014] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the method in the first aspect.

[0015] In a fifth aspect, the embodiments of the present application provide a computer program product, and the computer program product comprises a computer program stored in a non-transitory computer readable storage medium, and the computer program comprises program instructions, and the program instructions are executed by a computer to implement the steps of the method in the first aspect.

[0016] In the embodiments of the present application, a native file is set in a recording process, and related files of recording, such as mixing information, call information, and encapsulated RTP data, are written into the native file, and mixing is performed based on the native file in a mixing process. Since the RTP information of a calling terminal and a called terminal is saved in the native file, positioning and repair are facilitated, the success rate of mixing recording is improved, and a buffer is set in the process of storing a recording file, and encapsulated RTP stream data is sorted in the buffer, the out-of-order or packet loss situation can be solved by means of the buffer, and the success rate of mixing is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are included to provide a further understanding of the present application and constitute a part of this application, illustrate certain illustrative embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0018] Figure 1 This is a flowchart of the recording method provided in the embodiments of this application;

[0019] Figure 2 This is a schematic diagram of the RTP stream data encapsulation structure provided in the embodiments of this application;

[0020] Figure 3 This is a schematic diagram illustrating the sorting of encapsulated RTP stream data in a buffer according to an embodiment of this application;

[0021] Figure 4 This is a schematic diagram illustrating the alignment of RTP stream data of the calling and called terminals within a time slot, as provided in an embodiment of this application.

[0022] Figure 5 This is an overall flowchart of the recording method provided in the embodiments of this application;

[0023] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0026] The following is in conjunction with the appendix Figures 1 to 6 The present application provides a detailed description of a recording method and computer program product through specific embodiments and application scenarios.

[0027] like Figure 1 The diagram shown is a flowchart of a recording method provided in an embodiment of this application. Figure 1 As shown, the recording method is applied to a recording system, which includes a media recording module and a media mixing module. The recording method may include the contents shown in S101 to S104.

[0028] In S101, in the case that the media recording module receives signaling service end to initiate session establishment, initialize the raw file, and write the mixed voice information into the raw file, the mixed voice information is carried in the session establishment.

[0029] It is worth noting that the recording system can include a media recording module and a media mixed voice module, wherein the media recording module implements recording according to signaling information and RTP stream to record a raw file; the media mixed voice module processes the raw file to obtain the RTP stream data of the call information, the mixed voice information and the call voice to implement mixed voice.

[0030] The signaling service end is used to provide signaling notification for the media recording module.

[0031] In the present example, when the signaling service end receives the call signaling of the calling terminal, the signaling module of the signaling service end goes to the transfer query of the mixed voice information corresponding to the session, the signaling service end initiates session establishment to the media recording module, and the mixed voice information is transmitted to the media recording module through the network interface in the session establishment between the signaling service end and the media recording module. The session start module of the media recording module stores the corresponding mixed voice information, initializes the raw file, completes the creation of the raw file (RawFile) of the call sheet in the memory file system, and then writes the mixed voice information in the raw file. Writing a file in the memory file system is equivalent to writing directly into memory, and reading and writing are completed instantaneously, improving storage efficiency.

[0032] It should be noted that different enterprises can have different configurations of mixed voice information in virtual number services, and the same enterprise can also configure mixed voice information according to the binding number relationship, and the actual obtained mixed voice information is used as the standard, and the present embodiment is not limited.

[0033] In S102, the media recording module listens to the real-time transport protocol (RTP) stream of the calling and called terminals to obtain the call information, and writes the call information into the raw file.

[0034] That is, the signaling service end replaces the media addresses of the calling and called terminals in the media negotiation with the address and port of the media recording module. After completing the media negotiation, the signaling service end notifies the media recording module that the call has started through the media negotiation interface. At this time, the related information of the call has been obtained. The media recording module listens to the RTP stream of the calling and called terminals according to the address and port delivered by the signaling, and writes the call information into the raw file. The call information can include the call sheet identity (ID), enterprise ID, calling terminal number, called terminal number, call call time, etc.

[0035] In S103, the media recording module encapsulates the RTP stream data of the calling terminal and the called terminal, sorts the encapsulated RTP stream data in the buffer, and writes the sorted RTP stream data into the original file.

[0036] That is, the media recording module can encapsulate the RTP stream data of the calling terminal and the called terminal to record the RTP state, early identification, calling and called, encoding type, relative timestamp and other information of the RTP stream data, facilitate subsequent sound mixing, and then sort the encapsulated RTP stream data in the buffer to facilitate determining whether there is a packet loss phenomenon, and finally write into the original file, thereby improving the accuracy of sound mixing.

[0037] In S104, in the case that the call ends, the media sound mixing module mixes based on the information in the original file to obtain a recording file.

[0038] In the embodiment of the application, first, in the case that the media recording module receives a signaling server to initiate session establishment, the original file is initialized, and the sound mixing information is written into the original file, the sound mixing information is carried in the session establishment, then the media recording module listens to the real-time transport protocol (RTP) stream of the calling terminal and the called terminal to obtain call information, and writes the call information into the original file; the media recording module encapsulates the RTP stream data of the calling terminal and the called terminal, sorts the encapsulated RTP stream data in the buffer, writes the sorted RTP stream data into the original file, and finally, in the case that the call ends, the media sound mixing module mixes based on the information in the original file to obtain a recording file. In the embodiment, the sound mixing information, the call information and the encapsulated RTP data are written into the original file, facilitating sound mixing positioning and repair, and the encapsulated RTP data is cached to the buffer based on the synchronization source identifier and the sequence number of the packet, so that the RTP data is regularly sorted, further reducing abnormal conditions and improving the sound mixing success rate.

[0039] In the embodiment of the application, a original file is set in the recording process, and the related files of the recording, such as the sound mixing information, the call information and the encapsulated RTP data, are written into the original file, and the sound mixing is performed based on the original file in the sound mixing process. Since the RTP information of the calling terminal and the called terminal is saved in the original file, positioning and repair are facilitated, the sound mixing recording success rate is improved, and a buffer is set in the process of storing the recording file, the encapsulated RTP stream data is sorted in the buffer, the out-of-order or packet loss condition can be solved by means of the buffer, and the sound mixing success rate is further improved.

[0040] In one possible implementation of the application, the RTP stream data of the calling terminal and the called terminal includes early RTP stream data and call RTP stream data.

[0041] That is, after the signaling server and the media recording module complete media negotiation, the RTP stream starts to be transmitted, the media service module starts to listen to the RTP stream of the calling terminal and the called terminal after receiving the media negotiation notification from the signaling server, and transmits the calling terminal RTP stream to the called terminal and the called terminal RTP stream to the calling terminal, so that the voice of the calling terminal reaches the called terminal.

[0042] It is worth noting that when the called terminal is not connected, the media stream is early media RTP, and the calling terminal user can hear the information such as the beep sound or the color ring set by the called user. When the signaling server notifies the media recording module that the call starts, the transmitted media stream is call RTP, and at this time the calling terminal and the called terminal can hear the voice of the other party.

[0043] The RTP stream data of the calling terminal and the called terminal is encapsulated, which can include adding a header identifier in the RTP stream data, the header identifier is used to record the RTP state, early identification, calling terminal and called terminal identifier, encoding type and relative timestamp; the header identifier is combined with the RTP stream data to obtain encapsulated RTP stream data.

[0044] In the embodiment, in the recording of the original file, the early RTP and call RTP recording need to be identified, and the RTP state, early identification, calling terminal and called terminal identifier, encoding type, relative timestamp and other information can be recorded by adding a header identifier to the RTP, and then the data of the RTP itself is added to constitute a new RTP encapsulation structure [RTP header identifier] [RTP data], as shown in Figure 2 .

[0045] As shown in Figure 2 , the first behavior is the information in the original file, wherein each RTP encapsulation structure is used to store the encapsulated RTP stream data of the calling terminal and the called terminal, and each RTP encapsulation structure includes two parts of the RTP header identifier and the original RTP data, wherein the RTP header identifier part can include the RTP data type, length, RTP state, early identification, calling terminal and called terminal identifier, encoding type, relative timestamp and other field information.

[0046] The relative timestamp records the relative time since the call is initiated, and this field is used to solve the frame supplementing in the future.

[0047] In a possible implementation of the present application, sorting the encapsulated RTP stream data in a buffer may include: the media recording module parsing the encapsulated RTP stream data to obtain the Synchronization Source Identifier (SSRC) and the sequence number (SEQ) of the packets in the RTP stream data corresponding to each SSRC; the media recording module writing the encapsulated RTP stream data into different buffers in sequence according to the master and slave terminals based on the order in which the SSRC enters the buffer and the value of SEQ, where the terminal corresponding to the SSRC that enters the buffer first is written into the previous buffer, and in the buffer, the RTP stream data with a smaller SEQ value is cached at the front of the buffer.

[0048] In this embodiment, after encapsulating the RTP stream data of both the master and slave terminals, first parse the SSRC and SEQ of the RTP, and then write the data into the buffer according to <SSRC, SEQ, RTP encapsulation structure>. The master and slave terminals use different buffers respectively.

[0049] As Figure 3 shown, the SSRC of the RTP stream data of the calling terminal is ssrc1, and the RTP stream data of the calling terminal is written into the left buffer, that is, the calling buffer. The SSRC of the RTP stream data of the called terminal is ssrc2, and the RTP stream data of the called terminal is written into the right buffer, that is, the called buffer. Among them, in the master and slave buffers, they are written in sequence according to the size of the SEQ value. The one with a smaller SEQ value is written first, that is, in the front of the queue, and the one with a smaller SEQ value is written later, that is, in the back of the queue, that is, behind the RTP stream data with a smaller SEQ value. Finally, under qualified conditions, it is written into the RawFile.

[0050] In this embodiment, the buffer is a queue with sorting rules. The order in which the SSRC in the three-dimensional structure <SSRC, SEQ, RTP encapsulation structure> enters the buffer is used as the first basis for sorting, and the size of the SEQ value is used as the second basis for sorting. The one with a smaller SEQ value is in the front of the queue in the buffer.

[0051] It should be noted that the SSRC may change due to multiple media negotiations of signaling, resulting in its change following the change of the IP or port of the RTP stream. The SSRC may also change due to subscribing or customizing the voice playback or ringback tone service. When a network element sends audio and video RTP to the calling terminal, the SSRC in this case marks the RTP source of this network element. When the called terminal continues to communicate with the calling terminal, the SSRC marks the RTP source of the called terminal.

[0052] In a possible implementation of the application, the recording method can further include: during the call, the value of the SEQ is increased by one each time the terminal sends an RTP packet; and the media recording module determines the RTP state of the RTP stream data according to whether the value of the SEQ in the received RTP stream data is continuous, wherein the RTP state includes a normal state, a pending state and a discard state.

[0053] That is, during the call, the value of the SEQ corresponding to the terminal is increased by one each time the terminal sends an RTP packet, and the SEQ can be used to check packet loss when the network condition is poor, and the SEQ can also be used to reorder data in the case of network jitter.

[0054] In the buffer, the data packets of different SSRC sources are sorted in order according to the arrival order of the SSRCs and then sorted in ascending order according to the SEQs, so that the data packets of different SSRC sources are sorted in order in the buffer. The packet loss delay time of the communication network is guaranteed within a certain time, which can be two seconds, three seconds or several seconds. The buffer can store voice data for more than several seconds, so that the voice data is sorted in order after passing through the buffer within the time. That is, the buffer can buffer data for a certain time, so that the voice data is sorted in order within a certain time without packet loss.

[0055] It should be noted that in an extreme case, the network packet loss exceeds several seconds, and the data is also equivalent to being lost. The voice in the case of real-time call processing is processed in the form of packet loss, that is, there is no need to sort, and the data is discarded directly.

[0056] In the above RTP encapsulation structure, there is an RTP state field, and the RTP state includes a pending state, a discard state and a normal state. The above-described sorting in the buffer is only for the normal state, and for data in a non-normal state, sorting is triggered when the data enters the buffer. The state is set to indicate that the current RTP data is not sufficient to determine whether it is normal, and further information of the subsequent RTP stream is needed to further confirm.

[0057] In a possible implementation of the present application, the media recording module determines the RTP state according to whether the value of SEQ in the received RTP stream data is continuous, which can include: in the case that the media recording module receives first RTP stream data, comparing the value of the first SEQ of the first RTP stream data with the value of the second SEQ of the previous RTP stream data; in the case that the difference between the value of the first SEQ and the value of the second SEQ is greater than the first difference, setting the RTP state of the first RTP stream data as the pending state; in the case that the values of the SEQ of the first quantity of RTP stream data received after the media recording module receives the first RTP stream data are all less than the value of the first SEQ, setting the RTP state of the first RTP stream data as the discard state, and the first quantity is greater than the first quantity threshold; in the case that the values of the SEQ of the RTP stream data received after the media recording module receives the first RTP stream data are all greater than the value of the first SEQ and gradually increase, setting the RTP state of the first RTP stream data as the normal state.

[0058] That is, in the process of the call, the value of SEQ in a certain RTP stream data received is S1, and the value of S1 is much greater than the value of SEQ in the previous RTP stream data. At this time, it cannot be determined whether the SEQ is abnormal, and the RTP state can be set as pending first. If the values of the subsequent multiple SEQs are less than the value, it is determined that the SEQ is abnormal, the state of the RTP is set as the discard state, and the subsequent analysis link is not analyzed as voice data. If the subsequent SEQs gradually increase, the pending state is set as the normal state, so that the stability of the media recording module can be avoided from being affected by the abnormal RTP stream data in the network.

[0059] In a possible implementation of the present application, the recording method can further include: in the case that the value of the first SEQ is less than the value of the second SEQ, determining whether the first RTP stream data is the RTP stream data of the next round of cycle; in the case that the first RTP stream data is the RTP stream data of the next round of cycle, setting the RTP state of the first RTP stream data as the normal state; in the case that the first RTP stream data is not the RTP stream data of the next round of cycle, determining the difference between the value of the first SEQ and the value of the second SEQ; in the case that the difference between the value of the first SEQ and the value of the second SEQ is less than or equal to the second difference, setting the RTP state of the first RTP stream data as the normal state; in the case that the difference between the value of the first SEQ and the value of the second SEQ is greater than the second difference, setting the RTP state of the first RTP stream data as the pending state, and determining the RTP state of the first RTP stream data as the normal state or the discard state based on the subsequently received RTP stream data.

[0060] That is, if the SEQ value in the received RTP stream data is less than the SEQ value in the previous RTP stream data, it needs to be considered whether it enters the next round of circulation, if yes, the RTP state of the RTP stream data is set to normal state, if no, it further compares whether the difference between the two SEQ values is within the length range that the buffer can receive, if yes, it can be cached to the buffer first, and the RTP state of the RTP stream data is set to normal state, if no, the difference between the two SEQ values is not within the length range that the buffer can receive, that is, it exceeds the length range that the buffer can receive, the RTP state of the RTP stream data is set to pending state, and the RTP state of the RTP stream data is determined to be normal state or discarded state according to the subsequent received RTP stream data.

[0061] Wherein, the determination of the RTP state of the RTP stream data to be normal state or discarded state according to the subsequent received RTP stream data is the same as the determination in the above embodiment S1, the SEQ value in the previous RTP stream data is much larger, and the embodiment will not be repeated here, and the above embodiment is preferred.

[0062] In this embodiment, a new structure, buffer, is defined to save data, and the buffer can be used to solve the out-of-order or packet loss situation, and the abnormal RTP data can be avoided according to the state of the RTP, so that the data finally written into the RawFile are all normal data.

[0063] In one possible implementation of the present application, writing the sorted RTP stream data into the native file can include: in the case that the RTP state of the RTP stream data is determined to be normal state, the media recording module writes the sorted RTP stream data into the native file.

[0064] That is, the RTP state of the RTP stream data written into the native file is normal state, which avoids sending abnormal data to the media mixing module, so that the RTP stream data used by the media mixing module for mixing are all normal data, and the mixing quality is guaranteed.

[0065] In one possible implementation of the present application, writing the sorted RTP stream data into the native file can include: in the case that the sorting duration of the RTP stream data in the buffer exceeds the duration threshold, writing the RTP stream data into the native file; or in the case that the RTP stream data in the buffer exceeds the data threshold, writing the RTP stream data in the buffer into the native file; or setting a timer, and writing the RTP stream data in the buffer into the native file based on the timer.

[0066] That is, the data in the buffer area can be written into the RawFile based on the timer by setting the timer, and a threshold of the number of buffered values in the buffer area can be set, and when the data in the buffer area exceeds the threshold, the data in the buffer area is written into the RawFile, so as to ensure that the buffer area can always buffer a certain amount of data, and the network can not lose packets in a certain time delay, so that the data used by the subsequent sound mixing is more complete.

[0067] In a possible implementation of the application, the recording method can further include: in the case where the call ends, the media recording module stops receiving the RTP stream data, and writes the RTP stream data in the buffer area in the normal state into the RawFile.

[0068] That is, in the case where the call ends, the signaling server sends a session stop notification to the media recording module, the media recording module stops receiving the RTP stream data, and writes the RTP stream data in the normal state in the buffer area into the RawFile, so as to ensure that all the data received by the media recording module is written into the RawFile.

[0069] In a possible implementation of the application, in the case where the call ends, the media sound mixing module performs sound mixing based on the information in the RawFile to obtain a recording file, which can include: in the case where the call ends, the media recording module sends an asynchronous message to the media sound mixing module to inform the media sound mixing module to perform sound mixing; the media sound mixing module obtains the RawFile from the media sound mixing module based on the asynchronous message, and the RawFile includes sound mixing information, call information, and encapsulated RTP stream data of the calling terminal and the called terminal; and the media sound mixing module performs sound mixing based on the RawFile to obtain a recording file.

[0070] That is, when the call ends, the media recording module sends an asynchronous message to the media sound mixing module to inform the media sound mixing module to start sound mixing. After the call ends, the information recorded in the RawFile in the media recording module records the call information, sound mixing information, and RTP stream information of the calling terminal and the called terminal in the call, and the information of the voice stream in the call scene can be restored by taking the file. The message of the media recording module informing the media sound mixing module only needs to inform how to obtain the RawFile, and the media sound mixing module can obtain the RawFile from the media recording module, so as to reduce the database query and interface query, and the recording file can be taken to parse the call information and sound mixing information, without the need to query the database or use the interface to transfer and query, thereby reducing unnecessary query operations.

[0071] In a possible implementation of the present application, the mixing based on the native file to obtain the recording file can include: parsing the native file to obtain mixing information, call information and encapsulated RTP stream data of the calling terminal and the called terminal; performing frame filling processing on the parsed RTP stream data of the calling terminal and the called terminal to obtain a frame filling file, the frame filling file including early call files of the calling terminal and the called terminal, calling files of the calling terminal and the called terminal; mixing the frame filling file, the mixing information and the call information to obtain the recording file.

[0072] In the present embodiment, the parser of the media mixing module can parse the mixing information, the call information and the encapsulated RTP stream data of the calling terminal and the called terminal according to the native file, and then perform frame filling processing on the parsed RTP stream data of the calling terminal and the called terminal to obtain a frame filling file, the frame filling file including early call files of the calling terminal and the called terminal, calling files of the calling terminal and the called terminal; mixing the frame filling file, the mixing information and the call information to obtain the recording file. The frame filling can avoid the problem of asynchronous voice, and since the mixing information is written in the native file, the transfer query of the mixing re-push link can be reduced, which is more conducive to realizing the mixing synchronization of the calling terminal and the called terminal.

[0073] In a possible implementation of the present application, the frame filling processing on the parsed RTP stream data of the calling terminal and the called terminal to obtain a frame filling file can include: setting time slots at a preset time interval according to the length of the call; synchronizing the RTP stream data of the calling terminal and the called terminal in each time slot according to the relative time stamp in the native file; in the case of synchronization of the RTP stream data of the calling terminal and the called terminal, performing frame filling on the RTP stream data according to the RTP time stamp in a time slot, the RTP time stamp being obtained by the media mixing module based on the RTP stream data.

[0074] That is, after the native file is parsed, frame filling is still needed to avoid the situation that the voice of the calling terminal and the called terminal can be out of synchronization if directly mixed.

[0075] In this embodiment, synchronization is performed according to the relative time stamp in the original file, the calling terminal and the called terminal in the original file can be synchronized, that is, groups are synchronized, a time slot can be set according to the total duration of the call, and the groups are synchronized through the time slot, that is, the RTP stream data of the calling terminal and the called terminal in the same time slot is put into the corresponding time slot, and the different RTP stream data in the same time slot is further frame filled according to the RTP time stamp, that is, the frame filling is performed according to the difference between the time stamps of two RTPs until the data satisfies the time slot, that is, the duration of the frame filled data and the original slot data does not exceed the time set by the time stamp. The time stamp of RTP is a step value, and when the sampling rate is determined, the increment of the value is also determined, so that the frame filling between two RTP data can be completed according to the time stamp of RTP. The RTP time stamp can be parsed from the RTP stream.

[0076] As shown in Figure 4 , a time slot can be set at an interval of two seconds, that is, the time interval between 0-t1, t1-t2… is two seconds, at this time, the calling terminal and the called terminal can be put into the corresponding time slot according to the recorded relative time stamp according to the parsed calling terminal and called terminal data. For example, the calling data with a relative time stamp less than t1 is placed in the LeftGroup-1 group, and the called data with a relative time stamp less than t1 is placed in the RightGroup-1 group. This is equivalent to one group of frame filling according to the time reference mode of the calling terminal and the called terminal, which ensures that the data between the groups is synchronized.

[0077] The frame filling in the group refers to whether Ri, Ri+1, Ri+2… needs to be frame filled, which can be determined according to the difference between the time stamps. For example, under the condition of 8000 sampling rate, the increment of the time stamp is 160, the time stamp of RTP1 is 333000, and the time stamp of RTP2 is 333320. The difference between RTP2 and RTP1 is 320, and the difference between the time stamps is 2*160, so 2-1 silent frames need to be filled.

[0078] In this embodiment, the relative time slot is used to mutually reference the frame filling between the calling terminal and the called terminal, and the RTP time stamp is used to further realize the frame filling in the RTP data in a group, restore the voice data of the call scene, and realize the voice synchronization of the calling terminal and the called terminal.

[0079] In a possible implementation of the present application, the recording method can further include: in the case that the media recording module and the signaling server exist multiple negotiations, decoding the pulse code modulation data obtained by real-time decoding according to the encoding type and the load type in the RTP stream data.

[0080] In this embodiment, due to the possibility of multiple media negotiations between the signaling server and the media recording module, multiple encoding data may exist in the original file. In the frame filling link, the pulse code modulation (PCM) data after decoding can be directly filled. That is, decoding is performed in the frame filling, and real-time decoding is performed according to the recorded encoding type and the payload type (PayloadType) in the RTP in the decoding, and the PCM stream after audio decoding is obtained. Different encoding decoding PCM is directly spliced to assemble a file. The calling file stream and the called file stream are finally delivered to the sound mixing. Here, early and conversation are also distinguished, so two pairs of file streams are obtained here. After obtaining the early calling PCM file stream and the called PCM file stream, the sound mixing file information is constructed according to the conversation information, and the sound mixing file is synthesized into the customer's customized style according to the configuration item of the sound mixing information, and the whole sound mixing process is completed.

[0081] In this embodiment, the original file is stored in the media recording module. In the storage process, the RTP disorder is solved by using the conversation data buffer, the abnormal RTP data is identified by using the RTP state, and the conversation data of the calling and called parties is written into an original file according to the defined structure for storage. In the sound mixing module, after the calling and called data is parsed, a secondary frame filling operation is performed. First, the calling and called data is grouped in different time slots according to the relative time stamp, the inter-group frame filling of the calling and called parties is completed, and the inter-group frame filling of the calling and called parties is further reduced. In the sound mixing module, the voice data can be parsed in a customized strategy mode. For the case where multiple encodings exist, the PCM stream is uniformly decoded and then sound mixed.

[0082] Figure 5 The whole flowchart of the recording method is shown in the figure. The specific flowchart has been described in detail in the above embodiments, and will not be described here again. The above embodiments are preferred.

[0083] The recording system provided in the embodiments of the present application comprises a media recording module and a media sound mixing module. The media recording module is configured to initialize an original file when a signaling server initiates a session establishment, write sound mixing information into the original file, and carry the sound mixing information in the session establishment; listen to real-time transport protocol (RTP) streams of calling and called terminals to obtain conversation information, and write the conversation information into the original file; encapsulate the RTP stream data of the calling and called terminals, sort the encapsulated RTP stream data in a buffer, and write the sorted RTP stream data into the original file. The media sound mixing module is configured to perform sound mixing based on the information in the original file when a conversation ends, and obtain a recording file.

[0084] In the embodiment of the present application, a native file is set in the recording process, and the related files of the recording, such as the mixed sound information, the call information and the encapsulated RTP data are written into the native file, and the mixing is performed based on the native file in the mixing process. Since the RTP information of the calling party and the called party is saved in the native file, the positioning and repairing are facilitated, the success rate of the mixed recording is improved, and a buffer is set in the process of storing the recording file, and the encapsulated RTP stream data is sorted in the buffer. The out-of-order or packet loss situation can be solved by means of the buffer, and the success rate of the mixing is further improved.

[0085] As shown in Figure 6 The embodiment of the present application also provides an electronic device 600, which comprises a processor 601, a memory 602, a program or instruction stored in the memory 602 and executable on the processor 601. When the program or instruction is executed by the processor 601, the processes of the above-mentioned recording processing method embodiment are realized, and the same technical effects can be achieved. To avoid repetition, details are not described herein.

[0086] Optionally, the embodiment of the present application also provides a computer readable storage medium, and a computer program is stored on the computer readable storage medium. When the computer program is executed by the processor, the processes of the above-mentioned recording method embodiment are realized, and the same technical effects can be achieved. To avoid repetition, details are not described herein. The computer readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0087] Optionally, the embodiment of the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer readable storage medium. The computer program comprises program instructions, and the program instructions are executed by a computer to realize the processes of the above-mentioned recording method embodiment, and the same technical effects can be achieved. To avoid repetition, details are not described herein.

[0088] It should be noted that, in this document, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed, or further include elements inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0089] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, also can be through hardware, but many cases the former is the better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the contribution to the prior art can be embodied in the form of software products, the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), including a number of instructions to make a terminal (may be a mobile phone, computer, server, air conditioner, or network equipment, etc.) executes the method described in various embodiments of the present application.

[0090] The embodiments of the present application are described above in conjunction with the drawings, but the present application is not limited to the above-mentioned specific embodiments, the above-mentioned specific embodiments are only illustrative, but not limited, those skilled in the art can make many forms without departing from the purpose of the present application and the scope of the claims under the inspiration of the present application, all belong to the protection of the present application.

Claims

1. A recording method characterized by, The method is applied to a recording system, and the recording system comprises a media recording module and a media mixing module, and the method comprises the following steps: In the case that the media recording module receives a signaling server to initiate session establishment, initializing a native file and writing mixing information into the native file, wherein the mixing information is carried in the session establishment; The media recording module listens to real-time transport protocol (RTP) streams of both a calling terminal and a called terminal, obtains call information, and writes the call information into the native file; The media recording module encapsulates the RTP stream data of both the calling terminal and the called terminal, sorts the encapsulated RTP stream data in a buffer, and writes the sorted RTP stream data into the native file; In the case that the call ends, the media mixing module mixes based on the information in the native file to obtain a recording file.

2. The method of claim 1, wherein, The RTP stream data of both the calling terminal and the called terminal comprises early RTP stream data and call RTP stream data; The encapsulation of the RTP stream data of both the calling terminal and the called terminal comprises the following steps: Adding a header identifier in the RTP stream data, wherein the header identifier is used to record RTP status, early identification, calling terminal and called terminal identification, encoding type and relative time stamp; Combining the header identifier with the RTP stream data to obtain encapsulated RTP stream data.

3. The method of claim 1, wherein, The sorting of the encapsulated RTP stream data in the buffer comprises the following steps: The media recording module analyzes the encapsulated RTP stream data to obtain a synchronization source identifier (SSRC) and a sequence number (SEQ) of a message in the RTP stream data corresponding to each SSRC; The media recording module writes the encapsulated RTP stream data into different buffers according to the order of the SSRC entering the buffer and the value of the SEQ, wherein the terminal corresponding to the SSRC entering the buffer first is written into a previous buffer, and the RTP stream data with a smaller SEQ value is cached in the front row of the buffer.

4. The method of claim 3, wherein, The method further comprises the following steps: During the call, the value of the SEQ increases by one each time the terminal sends an RTP message; The media recording module judges the RTP status of the RTP stream data according to whether the value of the SEQ in the received RTP stream data is continuous, wherein the RTP status comprises a normal state, a pending state and a discard state.

5. The method of claim 4, wherein, The media recording module judges the RTP status of the RTP stream data according to whether the value of the SEQ in the received RTP stream data is continuous, and the judgment comprises the following steps: In the case that the media recording module receives first RTP stream data, comparing the value of a first SEQ of the first RTP stream data with the value of a second SEQ in previous RTP stream data; In the case that the difference between the value of the first SEQ and the value of the second SEQ is greater than a first difference value, setting the RTP status of the first RTP stream data as the pending state; In the case that the difference between the value of the first SEQ and the value of the second SEQ is greater than a first difference value, setting the RTP status of the first RTP stream data as the pending state; In a case where values of SEQs of RTP stream data after the media recording module receives first RTP stream data are all less than a value of the first SEQ, the first RTP stream data is set to a discard state; In a case where values of SEQs of RTP stream data after the media recording module receives first RTP stream data are all greater than the value of the first SEQ and gradually increase, the first RTP stream data is set to a normal state.

6. The method of claim 5, wherein, The method further comprises: In a case where the value of the first SEQ is less than the value of the second SEQ, it is determined whether the first RTP stream data is next round of RTP stream data; In a case where the first RTP stream data is next round of RTP stream data, the RTP state of the first RTP stream data is set to a normal state; In a case where the first RTP stream data is not next round of RTP stream data, it is determined a difference between the value of the first SEQ and the value of the second SEQ; In a case where the difference between the value of the first SEQ and the value of the second SEQ is less than or equal to a second difference value, the RTP state of the first RTP stream data is set to a normal state; In a case where the difference between the value of the first SEQ and the value of the second SEQ is greater than the second difference value, the RTP state of the first RTP stream data is set to a pending state, and the RTP state of the first RTP stream data is determined to be a normal state or a discard state based on subsequently received RTP stream data.

7. The method of claim 4, wherein, The writing of the sorted RTP stream data into the native file comprises: In a case where the RTP state of the RTP stream data is determined to be a normal state, the media recording module writes the sorted RTP stream data into the native file.

8. The method of claim 1, wherein, The writing of the sorted RTP stream data into the native file comprises: In a case where a sorting duration of the RTP stream data in the buffer exceeds a duration threshold, the RTP stream data is written into the native file; or In a case where the RTP stream data in the buffer exceeds a data threshold, the RTP stream data in the buffer is written into the native file; or A timer is set, and the RTP stream data in the buffer is written into the native file based on the timer.

9. The method of claim 1, wherein, The method further comprises: In a case where the call ends, the media recording module stops receiving RTP stream data, and writes the RTP stream data with a normal state in the buffer into the native file.

10. The method of claim 1, wherein, The mixing of the media recording module based on information in the native file to obtain a recording file in a case where the call ends comprises: In a case where the call ends, the media recording module sends asynchronous information to the media mixing module to inform the media mixing module to mix; The media mixing module acquires the native file from the media mixing module based on the asynchronous information, and the native file comprises mixing information, call information and encapsulated RTP stream data of the calling terminal and the called terminal; The media mixing module acquires the native file from the media mixing module based on the asynchronous information, and the native file comprises mixing information, call information and encapsulated RTP stream data of the calling terminal and the called terminal; The media mixing module mixes based on the native file to obtain a recording file.

11. The method of claim 10, wherein, Mixing based on the native file to obtain a recording file, comprising: analyzing the native file to obtain mixing information, call information and encapsulated RTP stream data of the calling terminal and the called terminal; performing frame filling processing on the analyzed RTP stream data of the calling terminal and the called terminal to obtain a frame filling file, the frame filling file comprising early call files of the calling terminal and the called terminal, calling terminal call files and called terminal call files; mixing the frame filling file, the mixing information and the call information to obtain a recording file.

12. The method of claim 11, wherein, The frame filling processing on the analyzed RTP stream data of the calling terminal and the called terminal to obtain a frame filling file, comprising: setting time slots at preset time intervals according to the length of the call; synchronizing the RTP stream data of the calling terminal and the called terminal in each time slot according to the relative time stamp in the native file; in the case of synchronization of the RTP stream data of the calling terminal and the called terminal, performing frame filling on the RTP stream data in a time slot according to the RTP time stamp, the RTP time stamp being obtained by the media mixing module based on the RTP stream data analysis.

13. The method of claim 12, wherein, The method further comprises: in the case of multiple negotiations between the media recording module and the signaling server, performing real-time decoding according to the encoding type and the load type in the RTP stream data to obtain decoded pulse code modulation data; splicing the bytes corresponding to the decoded pulse code modulation data to obtain a frame filling file.

14. A sound recording system characterized by comprising: comprising a media recording module and a media mixing module, wherein the media recording module is configured to, in the case of receiving a session establishment initiated by a signaling server, initialize a native file, write mixing information into the native file, the session establishment carrying the mixing information, listen to real-time transmission protocol (RTP) stream of the calling terminal and the called terminal to obtain call information and write the call information into the native file, encapsulate the RTP stream data of the calling terminal and the called terminal and sort the encapsulated RTP stream data in a buffer, and write the sorted RTP stream data into the native file; the media mixing module is configured to, in the case of call termination, mix based on the information in the native file to obtain a recording file.

15. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 13.

17. A computer program product, characterised in that, The computer program product comprises a computer program stored on a non-transitory computer readable storage medium, and the computer program comprises program instructions, which are executed by a computer to implement the steps of the method of any one of claims 1 to 13.

Images