Asymmetric audio and video data collaboration method

Abstract

The invention relates to an asymmetric audio and video data collaboration method, which belongs to the field of audio and video calls, and comprises the following steps: establishing an audio call between a calling side enterprise user and a called terminal, and establishing a video collaboration channel; obtaining service data, performing format preprocessing to convert the service data into a private format video file, reading audio frame data in the private format video file at a fixed frequency in a playing process, and caching the audio frame data in an audio buffer area; obtaining decoded enterprise RTP audio data in the calling call; re-sampling the enterprise RTP audio data and the first frame of audio data in the audio buffer area, and then merging to realize an audio mixing function; and the RTP audio packet after sound mixing is sent to the called mobile phone, and a user of the called mobile phone hears the sound in the calling call and the sound in the video file. According to the method provided by the invention, the one-way video asymmetric negotiation can be superposed in the voice call, a picture and voice synchronization mode is adopted, the voice of the calling side and the voice in the video file are mixed and then sent to the called mobile phone, and the user experience is improved.

Description

Technical Field

[0001] The present invention belongs to the field of voice and video calls, and in particular relates to an asymmetric audio and video data collaboration method. Background Art

[0002] Existing voice and video calls are an end-to-end communication mode that cannot provide data capabilities such as business introductions or remote verification such as PPT / WORD / PDF during calls, which makes the business expansion of enterprises face technical bottlenecks and cost pressures.

[0003] Inserting asymmetric audio and video negotiation into the voice platform to realize data collaborative service and converting business data into video stream to send to the called party's mobile terminal is a feasible solution. However, the integration of the sound of business data video and the voice of the calling party's customer service personnel faces a huge challenge.

[0004] When enterprise users promote services to end-users' mobile phones, simply playing the audio and video in the video file cannot meet the real-time interaction needs of both parties. That is, while playing the video, it is also necessary to communicate with the customer in real time, such as introducing business details. During a video call, audio and video are generally forwarded on a channel respectively, which is called a socket in network communication. Therefore, how to synthesize the audio in the video into the RTP audio of the ordinary call in real time during the call, and then forward it to the video receiver, is one of the technical problems that need to be solved at present. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the purpose of the invention is to provide an asymmetric audio and video data collaboration method, which superimposes a one-way video asymmetric negotiation in a voice call and adopts a sound and picture synchronization method, mixes the sound on the calling side and the sound in the video file and sends them to the called mobile phone, thereby improving the user experience.

[0006] The present invention proposes an asymmetric audio and video data collaboration method, comprising:

[0007] S1: The calling enterprise user establishes an audio call with the called terminal, and the call platform establishes a video collaboration channel;

[0008] S2, obtaining the service data of the enterprise user on the calling side, converting the service data into a private format video file by format preprocessing, parsing the private format video file, reading the audio frame data in the private format video file at a fixed frequency during playback, and caching it in an audio buffer; wherein the service data includes static files without sound and audio and video files with sound;

[0009] S3, obtaining the enterprise RTP audio data decoded by the enterprise user on the calling side during the calling call;

[0010] S4, resampling the enterprise RTP audio data and the first frame of audio data in the audio buffer and merging them to generate a new RTP audio packet to achieve a mixing function;

[0011] S5, sending the mixed RTP audio packet to the called mobile phone, and the user of the called mobile phone can hear the sound of the calling phone and the sound in the video file.

[0012] Further, in S2, the static file without sound includes: PPT file, WORD file, PDF file, JPEG file; the audio and video file with sound is an MP4 video format file.

[0013] Further, in S2, the soundless static file is converted into a private format video file, including: converting the static file into a 25FPS MP4 video format file, inserting two silent frames before each video frame in the converted video format file to achieve synchronization with the calling side voice, and performing format preprocessing on the converted MP4 video format file to obtain a private format video file.

[0014] Furthermore, the length of the two silence frames is 20 milliseconds.

[0015] Furthermore, in S2, the audio and video file with sound is converted into a private format video file, including: performing format preprocessing on the MP4 video format file with sound to obtain the private format video file.

[0016] Furthermore, the MP4 video format file is format preprocessed to separate the audio data and the video data, the separated audio data is converted into an audio format, the separated video data is converted into a video stream, and then the converted data is saved to generate a private format file.

[0017] Furthermore, in S2, during the file playback process, the audio data in the private format video file is read at a fixed rate and saved to the end of the audio buffer.

[0018] Furthermore, the fixed rate is 20 milliseconds.

[0019] Further, in S4, the enterprise RTP audio data is communication, presentation, and advertising audio associated with the business data.

[0020] Furthermore, in S5, the calling platform pushes the mixed RTP audio packet to the called mobile phone in H264 video stream format.

[0021] The beneficial effects of the present invention are as follows:

[0022] (1) For static files without sound, such as PPT, WORD, and PDF, they are converted into video files. At the same time, in order to achieve voice synchronization on the caller side, two silent frames are inserted before each video frame in the converted video format file. For business data with sound, the conversion to the private video format is completed after format preprocessing.

[0023] (2) Based on the video collaborative channel, the RTP audio data in the caller's call is resampled and merged with the first frame of audio data in the audio buffer, and the generated RTP audio packet is sent to the receiving end to achieve a mixing effect. Finally, the called end can hear the caller's voice and the sound in the video file.

[0024] (3) By transforming the call platform, the voice stream of the enterprise user and the voice stream in the video file are synthesized and forwarded to the called party in real time, so as to achieve the data coordination purpose in asymmetric audio and video calls. That is, the static data file is forwarded to the called user in the form of video through the voice call platform, and a silent frame is inserted into the static data video file to achieve voice synchronization with the calling side. The video synchronization voice method is used to improve the customer service experience.

[0025] (4) By superimposing one-way video asymmetric negotiation on traditional voice calls and adopting a synchronized audio and video method, the sound of the calling party and the sound in the video file are sent to the called party's mobile phone, achieving functions such as business promotion and advertising, and bringing new economic growth points for corporate users. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are only used to illustrate specific embodiments and are not considered to limit the present invention. In the entire drawings, the same reference symbols represent the same components. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0027] Figure 1 A flowchart of an asymmetric audio and video data collaboration method according to an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of asymmetric audio and video data collaborative mixing according to an embodiment of the present invention;

[0029] Figure 3 A logic diagram of business data video preprocessing according to an embodiment of the present invention;

[0030] Figure 4 A schematic diagram of converting a static file without sound into a file format with audio and video synchronization according to an embodiment of the present invention;

[0031] Figure 5The figure is a schematic diagram of mixing an RTP audio packet and audio buffer data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present invention.

[0033] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0034] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. The terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0035] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0036] The technical terms involved in the present invention are explained below:

[0037] SIP (Session Initiation Protocol): Session Initiation Protocol

[0038] SDP (Session Description Protocol): Session Description Protocol

[0039] RTP (Real-time Transport Protocol): Real-time Transport Protocol

[0040] VoLTE (Voice over LTE): A 4G IP transmission technology that allows high-definition audio and video calls on 4G networks.

[0041] The present invention proposes an asymmetric audio and video data collaboration method, which achieves the data collaboration goal by interacting with a call platform and performing audio mixing.

[0042] like Figure 1 and Figure 2 As shown, the present invention proposes an asymmetric audio and video data collaboration method, including:

[0043] S1: The calling enterprise user establishes an audio call with the called terminal, and the call platform establishes a video collaboration channel.

[0044] Specifically, in this step, after the audio call is established, the calling platform establishes a video collaboration channel.

[0045] S2, obtains the business data of the enterprise user on the calling side, pre-processes the business data into a private format video file, parses the private format video file, reads the audio frame data in the private format video file at a fixed frequency during playback, and caches it in the audio buffer.

[0046] In this step, during the file playback process, the audio data in the private format video file is read at a fixed rate and saved to the end of the audio buffer. Preferably, the fixed rate is 20 milliseconds. That is, the audio frame data is read at a fixed interval of 20 milliseconds and cached in the audio buffer.

[0047] It should be noted that the fixed rate is generally defined by the ptime parameter of the SIP SDP, such as the above-mentioned 20 milliseconds.

[0048] The following describes the conversion method of private format video files according to different types of business data.

[0049] In the embodiment of the present invention, the business data comes from the enterprise user on the calling side. For example, the customer service staff makes a voice call to the called party, and after the call is connected, they want to convert the content of a PPT or PDF file into a video and push it to the customer for display and viewing. The PPT or PDF here is the business data.

[0050] Specifically, business data includes documents, pictures or videos. According to the presence or absence of sound, business data includes static files without sound and audio and video files with sound. Among them, static files without sound include: PPT files, WORD files, PDF files, JPEG files, etc.; audio and video files with sound are MP4 video format files. That is, ordinary video files have sound, but documents do not.

[0051] The original files with or without sound need to be preprocessed and silent audio frames added, and finally converted into a private video format file.

[0052] (1) Video preprocessing of business data without sound

[0053] Converting a silent static file into a private format video file includes: converting the static file into a 25FPS MP4 video format file, inserting two silent frames before each video frame in the converted video format file to achieve synchronization with the caller side voice, and performing format preprocessing on the converted MP4 video format file to obtain a private format video file.

[0054] In the embodiment of the present invention, the length of the two silent frames is 20 milliseconds. That is, when the static data file is converted into a video file, it is transcoded at 25FPS and two 20ms long silent frames are inserted to achieve picture-sound synchronization.

[0055] Specifically, the business data of static files such as PPT, WORD, and PDF are usually converted into video files without any sound, in order to achieve the purpose of video to audio synchronization. Before preprocessing into a private format video file, the ordinary document data file is converted into a 25FPS mp4 video format file, and silent audio data is added. Because the corresponding frame of the static data file converted into a 25FPS video file is 40ms, and the ordinary voice frame is 20ms, it is necessary to insert two silent frames before each video frame in the converted video format file to achieve synchronization with the caller side voice. Figure 4 The resulting file format is shown.

[0056] (2) Video preprocessing of business data with sound

[0057] Specifically, converting the audio and video file with sound into the private format video file includes: performing format preprocessing on the MP4 video format file with sound to obtain the private format video file.

[0058] like Figure 3 As shown, the MP4 video format file obtained by the above conversion is format preprocessed, the audio data and the video data are separated, the separated audio data is converted into audio, the separated video data is converted into video stream, and then the converted data is saved to generate a private format file.

[0059] According to the technical method described above, the business data video without sound / without audio is pre-processed and converted into a private video format, and then the subsequent processes are performed in ascending order.

[0060] S3, obtaining the enterprise RTP audio data decoded by the enterprise user on the calling side during the calling call.

[0061] Specifically, in this step, the decoded audio data of the calling channel is obtained.

[0062] S4, resamples the enterprise RTP audio data and the first frame of audio data in the audio buffer and merges them to generate a new RTP audio packet to achieve a mixing function.

[0063] In this step, the enterprise RTP audio data obtained in S3 is resampled and merged with the first audio frame in the audio buffer. The merging method is to add them byte by byte in sequence to obtain new audio data.

[0064] In an embodiment of the present invention, the enterprise RTP audio data is communication, presentation, and advertising audio associated with the business data. The present invention synthesizes the audio of the call in real time during the playback of the video file, thereby meeting the needs of the enterprise customers for presentations, remote guidance, content sharing, and other services.

[0065] S5, the mixed RTP audio packet is sent to the called mobile phone, and the user of the called mobile phone can hear the sound of the calling party and the sound in the video file.

[0066] like Figure 5 As shown, in step S4, the RTP audio data in the caller's call is combined with the first frame of audio data in the audio buffer to generate a new RTP audio packet, which is sent to the receiving end to achieve the mixing function. Finally, in step S5, the receiving end mobile phone user can hear the caller's voice and the sound in the video file.

[0067] The video file format is not the common MP4 format, but a converted private format. The audio data is decoded data, and the video data is H264 encoded using RTP payload.

[0068] In an embodiment of the present invention, the calling platform pushes the mixed RTP audio packets to the called mobile phone in the H264 video stream format, wherein the audio is mixed into the calling channel and transferred to the called mobile phone.

[0069] It should also be noted that when playing a private video format file, it is necessary to push audio and video data frames to the called mobile phone at fixed time intervals. Whether the time of the audio frame is used as a benchmark or the time of the video is used as a benchmark will affect the user experience. On the whole, the human body's reaction time to sound is usually faster than the reaction time to the image. From the perspective of the complete process of sound or image information being transmitted from the sensory organs to the cerebral cortex for processing, the reaction time of sound (90ms-200ms) is usually shorter than that of image (100ms-400ms). When the eyes and ears receive information at the same time, the brain usually processes the sound information first, and then processes the image information. Therefore, the present invention plays files based on the interval of the audio.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. An asymmetric audio and video data collaboration method, characterized in that: include: S1: The calling enterprise user establishes an audio call with the called terminal, and the call platform establishes a video collaboration channel; S2, obtaining the service data of the enterprise user on the calling side, converting the service data into a private format video file by format preprocessing, parsing the private format video file, reading the audio frame data in the private format video file at a fixed frequency during playback, and caching it in an audio buffer; wherein the service data includes static files without sound and audio and video files with sound; S3, obtaining the enterprise RTP audio data decoded by the enterprise user on the calling side during the calling call; S4, resampling the enterprise RTP audio data and the first frame of audio data in the audio buffer and merging them to generate a new RTP audio packet to achieve a mixing function; S5, sending the mixed RTP audio packet to the called mobile phone, and the user of the called mobile phone can hear the sound of the calling phone and the sound in the video file.

2. The asymmetric audio and video data collaboration method according to claim 1, characterized in that: In S2, the static files without sound include: PPT files, WORD files, PDF files, JPEG files; the audio and video files with sound are MP4 video format files.

3. The asymmetric audio and video data collaboration method according to claim 2, characterized in that: In S2, a silent static file is converted into a private format video file, including: converting the static file into a 25FPS MP4 video format file, inserting two silent frames before each video frame in the converted video format file to achieve synchronization with the caller side voice, and performing format preprocessing on the converted MP4 video format file to obtain a private format video file.

4. The asymmetric audio and video data collaboration method according to claim 3, characterized in that: The length of the two silent frames is 20 milliseconds.

5. The asymmetric audio and video data collaboration method according to claim 2, characterized in that: In S2, the audio and video file with sound is converted into a private format video file, including: performing format preprocessing on the MP4 video format file with sound to obtain the private format video file.

6. The asymmetric audio and video data collaboration method according to claim 3 or 5, characterized in that: The MP4 video format file is format preprocessed, the audio data and the video data are separated, the separated audio data is converted into audio, the separated video data is converted into video stream, and then the converted data is saved to generate a private format file.

7. The asymmetric audio and video data collaboration method according to claim 1, characterized in that: In S2, during the file playback process, the audio data in the private format video file is read at a fixed rate and saved to the end of the audio buffer.

8. The asymmetric audio and video data collaboration method according to claim 1 or 7, characterized in that: The fixed rate is 20 milliseconds.

9. The asymmetric audio and video data collaboration method according to claim 1, characterized in that: In S4, the enterprise RTP audio data is communication, presentation, and advertising audio associated with the business data.

10. The asymmetric audio and video data collaboration method according to claim 1, characterized in that: In S5, the calling platform pushes the mixed RTP audio packet to the called mobile phone in H264 video stream format.

Images