Communication method, device and system

Abstract

The invention provides a communication method, device and system. The method provides a scheme capable of solving the problem of synchronization between an audio frame and a video frame. The method comprises the following steps: a receiving end determines that a video frame corresponding to a first video timestamp is a synchronous video frame of an audio mixing frame corresponding to the first audio timestamp according to the first video timestamp and the first audio timestamp which are simultaneously sent by network equipment, and the receiving end can determine the video timestamps of the video frames corresponding to the other audio mixing frames according to the audio mixing frame sampling rate, the video frame sampling rate, the first video timestamp, the first audio timestamp and the audio timestamps of the other audio mixing frames, so as to synchronously play the other audio mixing frames and the video frames corresponding to the other audio mixing frames. Therefore, synchronization between the audio frame and the video frame is realized.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and more specifically to a communication method, device and system. Background Art

[0002] The video conferencing media plane architecture includes two solutions: multipoint conferencing unit (MCU) solution and selective forwarding unit (SFU) solution. In the MCU solution, the server mixes the audio and video streams from all terminal devices in the same conference to obtain a mixed audio and video stream, and then sends the mixed audio and video stream to all terminal devices in the same conference. In the SFU solution, the server directly forwards the audio and video stream from a terminal device to other terminal devices.

[0003] In any of the above single solutions, there will be no obvious asynchrony problem between the audio frames and video frames sent by the server to the terminal device, because the timestamps of the audio / video frames are generated by the same entity. If the timestamps of the audio / video frames are generated by different entities, for example, the video frame adopts the SFU solution (that is, the video timestamp is generated by the terminal device) and the audio frame adopts the MCU solution (that is, the audio timestamp is generated by the server after mixing), it may cause obvious asynchrony between the audio frame and the video frame. Therefore, how to achieve synchronization between audio frames and video frames is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a communication method, device and system, which can achieve synchronization between audio frames and video frames.

[0005] In a first aspect, a communication method is provided, including: a network device sends a first audio frame and a first audio timestamp to a second terminal device, and sends a first video frame and a first video timestamp to the second terminal device, the first audio frame is a mixed frame obtained by the network device based on at least a second audio frame, the first video frame and the second audio frame are sent synchronously by the first terminal device, the first audio timestamp is a timestamp generated by the network device for the first audio frame, and the first video timestamp is a timestamp generated by the first terminal device for the first video frame; the network device sends a second video frame, a second video timestamp, and a second audio timestamp to the second terminal device, the second video timestamp is a timestamp generated by the first terminal device for the second video frame, the second audio timestamp is a timestamp generated by the network device for the third audio frame, the third audio frame is a mixed frame obtained by the network device based on at least a fourth audio frame, and the second video frame and the fourth audio frame are sent synchronously by the first terminal device. The second audio timestamp, the second video timestamp, the first audio timestamp, and the first video timestamp are used to determine that the first video frame and the first audio frame should be played synchronously.

[0006] Specifically, the network device simultaneously sends the video timestamp of the video frame and the audio timestamp of the mixed frame to the terminal device. The mixed frame and the video frame are synchronized with each other. In this way, the receiving end can determine that there is a corresponding relationship between the video timestamp and the audio timestamp, and can determine the video frame synchronized with other mixed frames based on the corresponding relationship, thereby achieving synchronization between the audio frame and the video frame.

[0007] In a possible implementation manner, the method also includes: the network device receives a third audio timestamp and a fourth audio timestamp sent by the first terminal device, the third audio timestamp is a timestamp generated by the first terminal device for the second audio frame, and the fourth audio timestamp is a timestamp generated by the first terminal device for the fourth audio frame; the network device determines the second audio timestamp based on the third audio timestamp, the fourth audio timestamp, the mixing frame sampling rate, the audio frame sampling rate and the first audio timestamp.

[0008] In this way, the network device can obtain the second audio timestamp corresponding to the third audio frame before obtaining the third audio frame.

[0009] A possible implementation method is that the network device determines the second audio timestamp based on the third audio timestamp, the fourth audio timestamp, the mixing frame sampling rate, the audio frame sampling rate and the first audio timestamp, including: the network device determines the difference between the fourth audio timestamp and the third audio timestamp; the network device determines the ratio of the difference to the audio frame sampling rate; the network device determines the second audio timestamp based on the product of the ratio and the mixing frame sampling rate and the sum of the first audio timestamp.

[0010] Through the above technical solution, the network device can determine the second audio timestamp.

[0011] A possible implementation method is that the network device sends a second video frame, a second video timestamp and a second audio timestamp to a second terminal device, including: the network device sends a first data packet to the second terminal device, and the first data packet includes the second video frame, the second video timestamp and the second audio timestamp.

[0012] In this way, the second terminal device can determine the correspondence between the second video frame corresponding to the second video timestamp and the second audio frame corresponding to the second audio timestamp.

[0013] In a possible implementation manner, the first data packet is a real-time transport protocol data packet.

[0014] A possible implementation method is that the network device sends a first audio frame and a first audio timestamp to a second terminal device, and sends a first video frame and a first video timestamp to the second terminal device, including: the network device sends a second data packet and a third data packet to the second terminal device, the second data packet includes the first audio frame and the first audio timestamp, and the second data packet includes the first video frame and the first video timestamp.

[0015] In a possible implementation manner, the second data packet and the third data packet are both real-time transport protocol data packets.

[0016] In a possible implementation manner, the first video frame and the second video frame are sent by the network device through an IP multimedia subsystem (IMS) data channel, and the first audio frame is sent by the network device through an IMS voice bearer channel.

[0017] In a second aspect, a communication method is provided, comprising: receiving a first audio frame and a first audio timestamp sent by a network device, and receiving a first video frame and a first video timestamp sent by the network device, the first video timestamp being a timestamp generated by the first terminal device for the first video frame, the first audio timestamp being a timestamp generated by the network device for the first audio frame, the first audio frame being a mixed frame obtained by the network device based on at least a second audio frame, and the first video frame and the second audio frame being sent synchronously by the first terminal device; receiving a second video frame, a second video timestamp and a second audio timestamp sent by the network device, the second video timestamp being a timestamp generated by the first terminal device for the second video frame, the second audio timestamp being a timestamp generated by the network device for the third audio frame, the third audio frame being a mixed frame obtained by the network device based on at least a fourth audio frame, and the second video frame and the fourth audio frame being sent synchronously by the first terminal device; and synchronously playing the first video frame and the first audio frame based on the second video timestamp, the second audio timestamp, the first audio timestamp and the first video timestamp.

[0018] Specifically, the execution subject of the solution described in the second aspect may be the second terminal device, or a chip or module in the second terminal device.

[0019] Through the above scheme, the second terminal device determines that the video frame corresponding to the second video timestamp is a synchronized video frame of the mixed frame corresponding to the second audio timestamp based on the second video timestamp and the second audio timestamp sent simultaneously by the network device, and the second terminal device can determine that the first video frame is a synchronized video frame of the first audio frame based on the mixed frame sampling rate, the video frame sampling rate, the second video timestamp, the second audio timestamp, the first audio timestamp and the first video timestamp, and synchronously play the first video frame and the first audio frame. In this way, synchronization between audio frames and video frames can be achieved.

[0020] A possible implementation method, which synchronously plays the first video frame and the first audio frame based on the second video timestamp, the second audio timestamp, the first audio timestamp and the first video timestamp, includes: determining a first difference between the second audio timestamp and the first audio timestamp; determining a ratio of the first difference to the mixed frame sampling rate; determining the product of the ratio and the video frame sampling rate; when the difference between the second video timestamp and the product is the first video timestamp, determining that the first video frame corresponds to the first audio frame; and synchronously playing the first video frame and the first audio frame.

[0021] It is understandable that the above parameters such as the audio mixing frame sampling rate and the video frame sampling rate can be configured by default configuration or by negotiation between the two parties.

[0022] In this way, the second terminal device can determine that the first video frame is a synchronized video frame of the first audio frame.

[0023] In a possible implementation, receiving a second video frame, a second video timestamp, and a second audio timestamp sent by a network device includes: receiving a first data packet sent by the network device, the first data packet including a second video frame, a second video timestamp, and a second audio timestamp.

[0024] In a possible implementation manner, the first data packet is a real-time transport protocol data packet.

[0025] A possible implementation method, which receives a first audio frame and a first audio timestamp sent by a network device, and receives a first video frame and a first video timestamp sent by the network device, includes: receiving a second data packet and a third data packet sent by the network device, the second data packet including a first audio frame and a first audio timestamp, and the second data packet including a first video frame and a first video timestamp.

[0026] In a possible implementation manner, the second data packet and the third data packet are both real-time transport protocol data packets.

[0027] In a possible implementation manner, the first video frame and the second video frame are sent by the network device through an IMS data channel, and the first audio frame is sent by the network device through an IMS voice bearer channel.

[0028] According to a third aspect, a communication method is provided, comprising: a network device sends a first audio frame and a first audio timestamp to a second terminal device, and sends a first video frame and a first video timestamp to the second terminal device, wherein the first audio frame is a mixed frame obtained by the network device based on at least the second audio frame, the first video frame and the second audio frame are sent synchronously by the first terminal device, the first audio timestamp is a timestamp generated by the network device for the first audio frame, and the first video timestamp is a timestamp generated by the first terminal device for the first video frame; the second terminal device receives the first audio frame, the first audio timestamp, the first video frame and the first video timestamp; the network device sends a first audio frame to the second terminal device, wherein the first audio frame is a mixed frame obtained by the network device based on at least the second audio frame, the first video frame and the second audio frame are sent synchronously by the first terminal device, The terminal device sends a second video frame, a second video timestamp and a second audio timestamp, the second video timestamp is a timestamp generated by the first terminal device for the second video frame, the second audio timestamp is a timestamp generated by the network device for the third audio frame, the third audio frame is a mixed frame obtained by the network device based on at least a fourth audio frame, and the second video frame and the fourth audio frame are sent synchronously by the first terminal device; the second terminal device receives the second video frame, the second video timestamp and the second audio timestamp; the second terminal device synchronously plays the first video frame and the first audio frame based on the second video timestamp, the second audio timestamp, the first audio timestamp and the first video timestamp.

[0029] A possible implementation method is that the second terminal device synchronously plays the first video frame and the first audio frame based on the second video timestamp, the second audio timestamp, the first audio timestamp and the first video timestamp, including: the second terminal device determines the first difference between the second audio timestamp and the first audio timestamp; the second terminal device determines the ratio of the first difference to the mixing frame sampling rate; the second terminal device determines the product of the ratio and the video frame sampling rate; when the difference between the second video timestamp and the product is the first video timestamp, the second terminal device determines that the first video frame corresponds to the first audio frame; the second terminal device synchronously plays the first video frame and the first audio frame.

[0030] A possible implementation method is that the network device sends a second video frame, a second video timestamp and a second audio timestamp to the second terminal device, including: the network device sends a first data packet to the second terminal device, and the first data packet includes the second video frame, the second video timestamp and the second audio timestamp.

[0031] In a possible implementation manner, the first data packet is a real-time transport protocol data packet.

[0032] A possible implementation method is that a network device sends a first audio frame and a first audio timestamp to a second terminal device, and sends a first video frame and a first video timestamp to the second terminal device, including: the network device sends a second data packet and a third data packet to the second terminal device, the second data packet includes the first audio frame and the first audio timestamp, and the second data packet includes the first video frame and the first video timestamp.

[0033] In a possible implementation manner, the second data packet and the third data packet are both real-time transport protocol data packets.

[0034] In a possible implementation manner, the first video frame and the second video frame are sent by the network device through an IMS data channel, and the first audio frame is sent by the network device through an IMS voice bearer channel.

[0035] In a fourth aspect, a communication method is provided, comprising: a network device sends a sequence of audio data packets and a sequence of video data packets in parallel to a first terminal device in a multi-party communication service. Each audio data packet in the sequence of audio data packets contains data and a mixing timestamp of a mixing frame, the mixing frame is obtained by the network device based on mixing of an audio frame from at least a second terminal device in the multi-party communication service, and the mixing timestamp is generated by the network device for the mixing frame; each video data packet in the sequence of video data packets contains data and a video timestamp of a video frame, the video frame comes from the second terminal device, and the video timestamp is generated by the second terminal device for the video frame; at least the first video data packet in the sequence of video data packets also contains a first mixing timestamp, the first mixing timestamp is the same as the mixing timestamp contained in the first audio data packet in the sequence of audio data packets, and the first mixing timestamp and the video timestamp in the first video data packet are used to determine that the video frame in the second video data packet in the sequence of video data packets and the mixing frame in the second audio data packet in the sequence of audio data packets should be played synchronously.

[0036] In a possible implementation manner, the video data packet sequence and the audio data packet sequence are transmitted through different channels.

[0037] In a possible implementation, the video data packet sequence is transmitted through an IMS data channel, and the audio data packet sequence is transmitted through an IMS voice bearer channel.

[0038] In a possible implementation, both the video data packet and the audio data packet are real-time transport protocol data packets.

[0039] In a fifth aspect, a communication method is provided, comprising: a first terminal device in a multi-party communication service receives a sequence of audio data packets and a sequence of video data packets in parallel from a network device. Each audio data packet in the sequence of audio data packets contains data and a mixing timestamp of a mixing frame, the mixing frame is obtained by the network device based on mixing of an audio frame from at least a second terminal device in the multi-party communication service, and the mixing timestamp is generated by the network device for the mixing frame; each video data packet in the sequence of video data packets contains data and a video timestamp of a video frame, the video frame comes from the second terminal device, and the video timestamp is generated by the second terminal device for the video frame; at least the first video data packet in the sequence of video data packets also contains a first mixing timestamp, the first mixing timestamp is the same as the mixing timestamp contained in the first audio data packet in the sequence of audio data packets; the first terminal device synchronously plays the video frame in the second video data packet in the sequence of video data packets and the mixing frame in the second audio data packet in the sequence of audio data packets based on the first mixing timestamp and the video timestamp in the first video data packet.

[0040] In a possible implementation manner, the video data packet sequence and the audio data packet sequence are transmitted through different channels.

[0041] In a possible implementation, the video data packet sequence is transmitted through an IMS data channel, and the audio data packet sequence is transmitted through an IMS voice bearer channel.

[0042] In a possible implementation, both the video data packet and the audio data packet are real-time transport protocol data packets.

[0043] In a sixth aspect, a communication device is provided, which may be a network device, or a device in a network device (for example, a chip, or a chip system, or a circuit), or a device that can be used in conjunction with a network device.

[0044] In one possible implementation, the communication device includes a module or unit corresponding to each of the methods / operations / steps / actions described in any one of the first aspects. The module or unit may be a hardware circuit, software, or a combination of a hardware circuit and software.

[0045] In a seventh aspect, a communication device is provided, which may be a terminal device, or a device in the terminal device (for example, a chip, or a chip system, or a circuit), or a device that can be used in conjunction with the terminal device.

[0046] In one possible implementation, the communication device includes a module or unit corresponding to each of the methods / operations / steps / actions described in any one of the second aspects. The module or unit may be a hardware circuit, software, or a combination of a hardware circuit and software.

[0047] In an eighth aspect, a communication device is provided, which may be a network device, or a device in a network device (for example, a chip, or a chip system, or a circuit), or a device that can be used in conjunction with a terminal device.

[0048] In one possible implementation, the communication device includes a module or unit corresponding to each of the methods / operations / steps / actions described in any one of the fourth aspects. The module or unit may be a hardware circuit, software, or a combination of a hardware circuit and software.

[0049] In a ninth aspect, a communication device is provided, which may be a terminal device, or a device in the terminal device (for example, a chip, or a chip system, or a circuit), or a device that can be used in conjunction with the terminal device.

[0050] In one possible implementation, the communication device includes a module or unit corresponding to each of the methods / operations / steps / actions described in any one of the fifth aspects. The module or unit may be a hardware circuit, software, or a combination of a hardware circuit and software.

[0051] In the tenth aspect, a communication device is provided, comprising a processor, wherein the processor is used to, by executing a computer program or instruction or through a logic circuit, enable the communication device to perform the method described in the first aspect and any possibility of the first aspect; or, enable the communication device to perform the method described in the second aspect and any possibility of the second aspect; or, enable the communication device to perform the method described in the third aspect and any possibility of the third aspect; or, enable the communication device to perform the method described in the fourth aspect and any possibility of the fourth aspect; or, enable the communication device to perform the method described in the fifth aspect and any possibility of the fifth aspect.

[0052] In a possible implementation, the communication device further includes a memory for storing the computer program or instruction.

[0053] In a possible implementation manner, the communication device further includes a communication interface, which is used to input and / or output signals.

[0054] In the eleventh aspect, a communication device is provided, comprising a logic circuit and an input / output interface, the input / output interface being used to input and / or output signals, the logic circuit being used to execute the method described in the first aspect and any possibility of the first aspect; or, the logic circuit being used to execute the method described in the second aspect and any possibility of the second aspect; or, the logic circuit being used to execute the method described in the third aspect and any possibility of the third aspect; or, the logic circuit being used to execute the method described in the fourth aspect and any possibility of the fourth aspect; or, the logic circuit being used to execute the method described in the fifth aspect and any possibility of the fifth aspect.

[0055] In the twelfth aspect, a computer-readable storage medium is provided, on which a computer program or instruction is stored. When the computer program or the instruction is run on a computer, the method described in the first aspect and any possibility of the first aspect is executed; or, the method described in the second aspect and any possibility of the second aspect is executed; or, the method described in the third aspect and any possibility of the third aspect is executed; or, the method described in the fourth aspect and any possibility of the fourth aspect is executed; or, the method described in the fifth aspect and any possibility of the fifth aspect is executed.

[0056] In the thirteenth aspect, a computer program product is provided, comprising instructions, which, when executed on a computer, cause the method described in the first aspect and any possibility of the first aspect to be executed; or, cause the method described in the second aspect and any possibility of the second aspect to be executed; or, cause the method described in the third aspect and any possibility of the third aspect to be executed; or, cause the method described in the fourth aspect and any possibility of the fourth aspect to be executed; or, cause the method described in the fifth aspect and any possibility of the fifth aspect to be executed.

[0057] In the fourteenth aspect, a communication system is provided, comprising: a terminal device and a network device, the terminal device being used to execute the method described in the second aspect and any one of the second aspects, and the network device being used to execute the method described in the first aspect and any one of the first aspects; or, the terminal device and the network device being used to execute the method described in the third aspect and any one of the possibilities of the third aspect; or, the terminal device being used to execute the method described in the fifth aspect and any one of the fifth aspects, and the network device being used to execute the method described in the fourth aspect and any one of the fourth aspects.

[0058] For the description of the beneficial effects of the fourth aspect to the fourteenth aspect, reference can be made to the description of the beneficial effects of the first aspect to the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a schematic diagram of a communication system 100 applicable to an embodiment of the present application.

[0060] Figure 2 This is a diagram of an IMS session network architecture supporting data channels according to an embodiment of the present application.

[0061] Figure 3 This is a schematic diagram of the MCU solution architecture.

[0062] Figure 4 This is a schematic diagram of the SFU solution architecture.

[0063] Figure 5 It is a schematic diagram of the interaction flow of the communication method 500 according to an embodiment of the present application.

[0064] Figure 6 It is a schematic diagram of the relationship between the audio timestamp of the audio frame and the audio timestamp of the mixing frame.

[0065] Figure 7 It is a schematic diagram of the relationship between the audio timestamp of the mixed frame and the video timestamp of the video frame.

[0066] Figure 8 It is a flow diagram of audio frames and video frames according to an embodiment of the present application.

[0067] Fig. 9 It is a schematic diagram of the interaction flow of the communication method 900 according to an embodiment of the present application.

[0068] Fig.10 It is a schematic diagram of the correspondence between the audio data packet sequence and the video data packet sequence of the embodiment of the present application.

[0069] Fig.11 It is a schematic block diagram of a communication device 1100 according to an embodiment of the present application.

[0070] Fig.12 It is a schematic block diagram of a communication device 1200 according to an embodiment of the present application.

[0071] Fig.13 It is a schematic block diagram of a communication device 1300 according to an embodiment of the present application.

[0072] Fig.14 It is a schematic block diagram of a communication device 1400 according to an embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to facilitate understanding of the embodiments of the present application, the following points are first explained.

[0074] 1. In this application, unless otherwise specified, "plurality" means two or more.

[0075] 2. In this application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other. The technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0076] 3. The various digital numbers involved in this application are only used for the convenience of description and are not used to limit the scope of protection of this application. The size of the serial numbers involved in this application does not mean the order of execution. The execution order of each process should be determined by its function and internal logic. For example, the terms "first", "second", "third", "fourth" and other various terminology labels (if any) in the specification and claims and drawings of this application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. Among them, the data used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than what is illustrated or described here.

[0077] At the same time, any embodiment or design described in the present application as "exemplary" or "for example" should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0078] 4. The terms "comprise", "include", "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product or apparatus.

[0079] 5. In this application, "pre-configuration" may include pre-definition, such as protocol definition. Among them, "pre-definition" can be implemented by pre-saving corresponding codes, tables or other methods that can be used to indicate relevant information in the device, and this application does not limit its specific implementation method.

[0080] 6. The term "storage" or "saving" as used in this application may refer to saving in one or more memories. The one or more memories may be provided separately or integrated in an encoder or decoder, a processor, or a communication device. The one or more memories may also be partially provided separately and partially integrated in a decoder, a processor, or a communication device. The type of memory may be any form of storage medium, which is not limited thereto.

[0081] VII. The “protocol” referred to in this application may refer to a standard protocol in the field of communications, such as the fourth generation (4G) network, the fifth generation (5G) network protocol, the new radio (NR) protocol, the 5.5G network protocol, the sixth generation (6 th The present application does not limit the network protocols used in future communication systems.

[0082] 8. The dotted arrows or boxes in the schematic diagrams of the drawings in the specification of this application represent optional steps or optional modules.

[0083] IX. In this application, unless otherwise specified, “ / ” indicates that the objects associated with each other are in an “or” relationship. For example, A / B can represent A or B. “And / or” in this application is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural.

[0084] First, the communication system to which the technical solution of the present application is applicable is described.

[0085] The technical solution of the present application can be applied to various communication systems, such as: 5G system, 4G system (or called long term evolution (LTE) system), LTE frequency division duplex (FDD) system, LTE time division duplex (TDD) system and future communication systems such as 6G system.

[0086] Combine the following Figure 1 The communication system 100 to which the embodiment of the present application is applicable is described. The communication system 100 is only an example and does not constitute any limitation to the present application. The communication system 100 includes: a first terminal device, a second terminal device, and an IMS network. Among them, the IMS network in the communication system 100 can also be a web real-time communication (WEBRTC) network. For ease of description, the embodiment of the present application is described by taking the IMS network as an example.

[0087] Specifically, the first terminal device and the second terminal device communicate via the IMS network. The first terminal device is a calling device (ie, a device initiating a call), and the second terminal device is a called device (ie, a device being called).

[0088] The IMS network can be a collection of multiple devices that process voice and multimedia services, or a collection of multiple servers that process audio, video and multimedia services. The IMS network includes one or more network devices, such as call session control function (CSCF) network element, IMS access media gateway (AGW), data channel signaling function (DCSF) network element, data channel media function (DCMF) network element, home subscriber server (HSS), etc.

[0089] When the first terminal device is used as a calling device, it can be any electronic device with a call function. Exemplarily, the first terminal device is a user equipment (UE), which can be any device that can access a network, and can also be called terminal equipment (terminal equipment), terminal device, access terminal, user unit, user station, mobile station, mobile station (MS), mobile terminal (MT), remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user device, etc. UE can be a device that provides voice / data connectivity to users, such as a handheld device with wireless connection function, a vehicle-mounted device, etc. At present, some examples of terminals may be: mobile phones, tablet computers, computers with wireless transceiver functions (such as laptops, PDAs, etc.), mobile Internet devices (MID), virtual reality (VR) devices, AR devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication functions, computing devices or other processing devices connected to wireless modems, vehicle-mounted devices, wearable devices, terminal devices in 4G / 5G networks, or public land mobile communication networks that will evolve in the future. network, PLMN) and other terminal devices.

[0090] In addition, UE can also be a UE in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection. IoT technology can achieve massive connections, deep coverage, and terminal power saving through narrow band (NB) technology, for example. UE can also include smart printers, train detectors, etc. Its main functions include collecting data (part of the terminal equipment), receiving control information and downlink data from network equipment, and sending electromagnetic waves to transmit uplink data to network equipment.

[0091] UE can be used to act as a base station. For example, the user equipment can act as a scheduling entity that provides sidelink signals between user equipment in vehicle to everything (V2X) or device to device (D2D). For example, a cell phone and a car communicate with each other using sidelink signals. Cell phones and smart home devices communicate with each other without relaying the communication signal through a base station.

[0092] When the second terminal device is used as the called device, it can be any electronic device with a call function. Exemplarily, the second terminal device is a user device, or the second terminal device is a public safety answering point (such as PSAP) in an IMS emergency call scenario (i.e., communication device 1 initiates an IMS emergency call (emergency call) to communication device 2), or any electronic device with a call function under a public safety answering point.

[0093] The above-mentioned network architecture applied to the embodiment of the present application is only an example, and the network architecture applicable to the embodiment of the present application is not limited to this. Any network architecture that can realize the functions of the above-mentioned network elements is applicable to the embodiment of the present application, that is, the network architecture and business scenarios described in the embodiment of the present application are for the purpose of more clearly illustrating the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of network architecture and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0094] The network elements or devices listed in the above network architecture are only exemplary. The network architecture applicable to the present application may also include other network elements or devices, which are not limited in the present application. The naming of the above network elements or devices is only defined for the convenience of distinguishing different functions and shall not constitute any limitation to the present application. The present application does not exclude the possibility of adopting other names in 4G networks, 5G networks and other networks in the future. For example, in a 6G network, some or all of the above network elements may use the terminology in 4G / 5G, or may use other names.

[0095] The communication device in the embodiment of the present application includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through a process (Process), for example, a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a windows operating system. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software. In addition, the embodiment of the present application does not specifically limit the specific structure of the execution subject of the method provided in the embodiment of the present application, as long as it can communicate according to the method provided in the embodiment of the present application by running a program that records the code of the method provided in the embodiment of the present application, for example, the execution subject of the method provided in the embodiment of the present application can be a terminal device or a network device, or a functional module in a terminal device or a network device that can call a program and execute a program.

[0096] In addition, various aspects or features of the embodiments of the present application can be implemented as methods, devices or products using standard programming and / or engineering techniques. The term "product" used in this application covers computer programs that can be accessed from any computer-readable device, carrier or medium. For example, computer-readable media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks or tapes, etc.), optical disks (e.g., compact discs (CDs), digital versatile discs (DVDs), etc.), smart cards and flash memory devices (e.g., erasable programmable read-only memory (EPROM), cards, sticks or key drives, etc.). In addition, the various storage media described herein may represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing and / or carrying instructions and / or data.

[0097] Figure 2 : is an IMS session network architecture diagram supporting data channels in an embodiment of the present application. Figure 2 A brief introduction is given to some network elements or devices involved in the network architecture shown.

[0098] 1. CSCF network element: CSCF is a functional entity within IMS and is the core of the entire IMS. It is mainly responsible for processing signaling control during multimedia call sessions. It manages IMS user authentication, IMS bearer plane quality of service (QoS), cooperates with other network elements to control SIP sessions, as well as service negotiation and resource allocation. For the sake of convenience, the CSCF network element in this application is referred to as "CSCF".

[0099] The CSCF may communicate with the terminal device or with the gateway device. For example, the CSCF selects the gateway device for communicating with the terminal device, and may also allocate routing information, such as an IP address or port, to the terminal device and the gateway device.

[0100] As an example but not limitation, CSCF is divided into proxy CSCF (P-CSCF), interrogating CSCF (I-CSCF), serving CSCF (S-CSCF), etc. according to function.

[0101] Among them, P-CSCF is the entry point for users to access the IMS network, and is mainly responsible for forwarding SIP signaling between IMS users and their home networks. I-CSCF is the unified entry point for IMS users' home networks, and is responsible for allocating or querying S-CSCFs that serve users. S-CSCF is the unified entry point for IMS users' home networks, and is responsible for allocating or querying S-CSCFs that serve users.

[0102] The above-mentioned P-CSCF, S-CSCF, and I-CSCF may be independently configured in different entities, or may be integrated in the same entity, without limitation.

[0103] 2. IMS-AGW: IMS-AGW can provide IMS network access gateway and media gateway functions.

[0104] The CSCF and the IMS AGW may be collectively referred to as the IMS core. The present application does not limit the signaling interaction method between network elements in the IMS core.

[0105] 3. DCSF network element: DCSF is a part of the logical functional entity in the data channel server (DCS). DCSF is mainly used to provide data channel signaling control functions.

[0106] 4. DCMF network element: DCMF is also a part of the logical functional entity in DCS. DCMF is mainly used to provide data channel media resource management functions.

[0107] 5. Multimedia Telephony application server (MMTelAS): MMTelAS is an application layer device located at the upper layer of the IMS system, providing basic and supplementary services, such as multimedia conferencing, converged communications, SMS gateways, standard switchboards, and other services. The IMS network is an open system based on IP bearer that provides users with various multimedia services. MMTelAS interacts with CSCF to trigger and execute various network services. In addition, MMTelAS supports DC interaction of IMS sessions, that is, MMTelAS can establish DC for terminal devices. MMTelAS can be called a session control function device, which is used to control the DC establishment process.

[0108] 5. Data channel application (DC App)

[0109] The DC App described in the present application is used to transmit additional communication information through a data channel between multiple terminal devices in the same IMS call service. As an example, the DC App is a web content containing hypertext markup language (HTML), JavaScript scripts, pictures, and cascading style sheets (CSS), which describes a graphical user interface (UI) and can implement interactive business logic.

[0110] DC App is usually downloaded from the network side by the call application on the terminal device and runs during the IMS call without installation or uninstallation. DC App can exist in the form of a web page, or in the form of a mini program / quick app / light app, etc., without limitation.

[0111] The DC App involved in this application may be a DC App in a 4G / 5G network. The DC App can transmit various data such as text, pictures, locations, files, etc. before and after an IMS call is established, and realize communication between the two parties in addition to voice calls or video calls, which can greatly enhance the experience of both parties.

[0112] 6. Data channel application repository (DCAR): DCAR is a repository for storing data channel applications (DC App).

[0113] After a developer completes the development of a DC App, he / she uploads it to the operator's DCS, which then saves it to DCAR. When needed, DCS downloads the DC App from DCAR to the local computer for subsequent processing.

[0114] 7. IMS application server (AS): IMS AS is the top-level application layer device in the IMS system, providing basic and supplementary services, such as multimedia conferencing, converged communications, SMS gateway, standard switchboard, etc. The IMS network is an open system based on IP bearer that provides users with various multimedia services. IMS AS and CSCF interact to trigger and execute various network services. The terminal device communicates with IMS AS, and IMS AS can establish a DC for the terminal device.

[0115] It can be understood that the above-mentioned network architecture applied to the present application is only an example, and the network architecture applicable to the present application is not limited to this. Any network architecture that can realize the functions of the above-mentioned network elements is applicable to the present application, that is, the network architecture and business scenarios described in the present application are for the purpose of more clearly illustrating the technical solution of the present application, and do not constitute a limitation on the technical solution provided in the present application. Ordinary technicians in this field can know that with the evolution of network architecture and the emergence of new business scenarios, the technical solution provided in the present application is also applicable to similar technical problems.

[0116] The network elements or devices listed in the above network architecture are only exemplary. The network architecture applicable to the present application may also include other network elements or devices, which are not limited in the present application. The naming of the above network elements or devices is only defined for the convenience of distinguishing different functions and shall not constitute any limitation to the present application. The present application does not exclude the possibility of adopting other names in 4G networks, 5G networks and other networks in the future. For example, in a 6G network, some or all of the above network elements may use the terminology in 4G / 5G, or may use other names.

[0117] The communication equipment of the present application (for example, the above-mentioned terminal equipment, CSCF equipment or AS equipment, etc.) includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU) and a memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through a process, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system or a windows operating system. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software. In addition, the present application does not specifically limit the specific structure of the execution subject of the method provided by the present application, as long as it can communicate according to the method provided by the present application by running a program that records the code of the method provided by the present application. For example, the execution subject of the method provided by the present application can be a terminal device or a network device, or a functional module in a terminal device or a network device that can call and execute a program.

[0118] In addition, various aspects or features of the present application can be implemented as methods, devices or products using standard programming and / or engineering techniques. The term "product" used in this application covers computer programs that can be accessed from any computer-readable device, carrier or medium. For example, computer-readable media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks or tapes, etc.), optical disks (e.g., compact discs (CDs), digital versatile discs (DVDs), etc.), smart cards and flash memory devices (e.g., erasable programmable read-only memory (EPROM), cards, sticks or key drives, etc.). The various storage media described herein may represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing and / or carrying instructions and / or data.

[0119] Next, the terms or concepts involved in the embodiments of the present application are described.

[0120] 1. Timestamp

[0121] Timestamp: It is used to reflect the sampling time of the first byte of data in the real-time transmission protocol (RTP) packet. At the beginning of a session, the value of the timestamp is randomly determined. For example, the initial value of the timestamp is 100. As time goes by, the timestamp value accumulates from the initial value 100 to 2000. The receiving end can determine which audio frame or video frame to restore at what time based on the current value of the timestamp.

[0122] Taking the audio frame as an example, the sampling duration of the audio frame is fixed, and the receiving end can determine the corresponding audio frame according to the timestamp value. In one example, the audio frame sampling rate is 8000 (Hz), 50 audio frames can be transmitted in 1 second, and the duration of each audio frame is 20 milliseconds. Assuming that the initial value of the timestamp is 0, each time an audio frame is transmitted, the timestamp value accumulates 160 (160 = 8000 * 20 / 1000). Within 1 second, when the second audio frame is transmitted, the timestamp value accumulates from 0 to 160, when the third audio frame is transmitted, the timestamp value is 320, when the fourth audio frame is transmitted, the timestamp value is 480, and so on. When the current timestamp value is 1600, the receiving end determines that the timestamp value 1600 corresponds to the 11th audio frame, and accordingly, the receiving end can restore the 11th audio frame. Among them, the audio frame sampling rate refers to the number of times the recording device samples the sound analog signal in a unit time. The higher the sampling frequency, the more real and natural the waveform of the mechanical wave is, and the unit is Hz.

[0123] 2. MCU Solution

[0124] Figure 3 This is a schematic diagram of the MCU solution architecture. Figure 3 As shown, the MCU solution includes an MCU and multiple terminal devices (such as terminal device 1 to terminal device 4). Multiple terminal devices in the same conference send their own audio and video streams that need to be shared to the server respectively. The MCU mixes the audio and video streams from the multiple terminal devices to obtain a mixed audio and video stream, and then sends it to all terminal devices (receiving ends) of the same conference. In the MCU solution, the mixed audio and video stream sent by the MCU to the receiving end is based on the time reference system of the MCU. The receiving end can determine the corresponding audio frames and video frames based on the same time reference system (that is, the time reference system of the network device).

[0125] 3. SFU Program

[0126] Figure 4 This is a schematic diagram of the SFU solution architecture. Figure 4 As shown, the SFU solution includes an SFU and multiple terminal devices (such as terminal device 1 to terminal device 4). Exemplarily, the SFU forwards the audio and video streams from terminal device 1 (as the sending end) to other terminal devices (such as terminal device 2 to terminal device 4) (as the receiving end) in the same conference. In the SFU solution, the audio and video streams forwarded by the SFU to the receiving end are based on the time reference system of the sending end, and the receiving end can determine the corresponding audio frames and video frames based on the same time reference system (i.e., the time reference system of the sending end).

[0127] When the audio frames and video frames received by the receiving end are based on the same time reference system, the receiving end can determine the synchronized audio frames and video frames based on the same time reference system. When the audio frames and video frames received by the receiving end are based on different time reference systems, the receiving end cannot determine the synchronized audio frames and video frames, which will cause the problem of asynchrony between the audio frames and the video frames. Therefore, in a hybrid solution involving the MCU solution and the SFU solution, that is, when the video adopts the SFU solution and the audio adopts the MCU solution, how to achieve synchronization between the audio frames and the video frames is a technical problem that needs to be solved urgently.

[0128] It should be uniformly explained that the "synchronization of audio frames and video frames" described in the embodiments of the present application does not mean that the audio frames and video frames are exactly the same in time, but that there may be a certain time deviation. For example, the sending time of the audio frame differs from the sending time of the video frame by 1 millisecond, or the receiving time of the audio frame differs from the receiving time of the video frame by 1 millisecond.

[0129] The communication method of the embodiment of the present application is described below.

[0130] For ease of description, this application specifies the expression of some terms, namely: a=audio=audio, v=video=video, f=frame=frame, t=timestamp=timestamp, p=packet=packet.

[0131] Figure 5 It is a schematic diagram of the interaction flow of the communication method 500 according to an embodiment of the present application. Figure 5 The method shown can be executed by the terminal device 120, the terminal device 130 and the network device 110, or by a module and / or device (for example, a chip or an integrated circuit, etc.) with corresponding functions installed in the terminal device 120, the terminal device 130 and the network device 110, without limitation. The following description is made by taking the terminal device 120, the terminal device 130 and the network device 110 as an example. The terminal device 130 and the terminal device 120 can be the second terminal device and the first terminal device in the communication system 100, respectively. Figure 5 As shown, the method 500 includes:

[0132] Optionally, S510a, the terminal device 120 sends the audio frame af2, the audio timestamp at3, the video frame vf1 and the video timestamp vt1 to the network device 110.

[0133] Accordingly, the network device 110 receives the audio frame af2, the audio timestamp at3, the video frame vf1, and the video timestamp vt1.

[0134] The audio timestamp at3 is generated by the terminal device 120 for the audio frame af2, and the video timestamp vt1 is generated by the terminal device 120 for the video frame vf1. The terminal device 120 sends the audio frame af2 and the video frame vf1 to the network device 110 synchronously, that is, the network device 110 determines that the audio frame af2 is a synchronous audio frame of the video frame vf1, and the video frame vf1 is a synchronous audio frame of the audio frame af2. In other words, the network device 110 can determine that the video frame vf1 and the audio frame af2 are sent synchronously by the terminal device 120 through the respective timestamps of the video frame vf1 and the audio frame af2.

[0135] Other terminal devices may also synchronously send audio frames to the network device 110, such as the terminal device 140 ( Figure 5 (not shown) sends the audio frame af3 to the network device 110, and the sending time of the audio frame af3 is the same as the sending time of the audio frame af2 or there is a certain time deviation.

[0136] Furthermore, the network device 110 performs mixing processing (including decoding, mixing, and re-encoding processes) on at least the audio frame af2 and the audio frame af3 to obtain an audio frame af1 (mixed frame), and generates an audio timestamp at1 for the audio frame af1 (mixed frame).

[0137] S510, the network device 110 sends an audio frame af1 (such as a first audio frame) and an audio timestamp at1 (such as a first audio timestamp) to the terminal device 130, and sends a video frame vf1 (such as a first video timestamp) and a video timestamp vt1 (such as a first video timestamp) to the terminal device 130.

[0138] Accordingly, the terminal device 130 receives the audio frame af1 (mixed audio frame) and the audio timestamp at1, and receives the video frame vf1 and the video timestamp vt1. Furthermore, the terminal device 130 buffers the audio frame af1 and the video frame vf1.

[0139] The network device 110 may first send the video frame vf1 and the video timestamp vt1 to the terminal device 130, and then send the audio frame af1 (mixed audio frame) and the audio timestamp at1. Therefore, the receiving time of the audio frame af1 (mixed audio frame) is different from the receiving time of the video frame vf1.

[0140] It can be understood that the terminal device 130 cannot determine the correspondence between the video frame vf1 and the audio frame af1 (mixed frame), that is, the terminal device 130 cannot determine whether the video frame vf1 is a synchronized video frame of the audio frame af1 (mixed frame), and whether the audio frame af1 (mixed frame) is a synchronized audio frame of the video frame vf1.

[0141] Optionally, S520a, the terminal device 120 sends the audio frame af5, the audio timestamp at4, the video frame vf2, and the video timestamp vt2 to the network device 110.

[0142] Accordingly, the network device 110 receives the audio frame af5, the audio timestamp at4, the video frame vf2, and the video timestamp vt2.

[0143] The audio timestamp at4 is generated by the terminal device 120 for the audio frame af5, and the video timestamp vt2 is generated by the terminal device 120 for the video frame vf2. The terminal device 120 sends the audio frame af5 and the video frame vf2 to the network device 110 synchronously, that is, the network device 110 can determine that the audio frame af5 is a synchronous audio frame of the video frame vf2, and the video frame vf2 is a synchronous audio frame of the audio frame af5. The network device 110 can determine that the video frame vf2 and the audio frame af5 are sent synchronously by the terminal device 120 through the respective timestamps of the video frame vf2 and the audio frame af5.

[0144] It is understood that other terminal devices may also synchronously send audio frames to the network device 110, such as the terminal device 140 ( Figure 5 (not shown) sends the audio frame af6 to the network device 110. The sending time of the audio frame af6 is the same as the sending time of the audio frame af5 or there is a certain time deviation.

[0145] Furthermore, the network device 110 performs mixing processing (including decoding, mixing, and re-encoding processes) on at least the audio frame af5 and the audio frame af6 to obtain an audio frame af4 (mixed frame), and generates an audio timestamp at2 for the audio frame af4 (mixed frame).

[0146] S520 , the network device 1120 sends a video frame vf2 (such as a second video frame), a video timestamp vt2 (such as a second video timestamp), and an audio timestamp at2 (such as a second audio timestamp) to the terminal device 130 .

[0147] Accordingly, the terminal device 130 receives the video frame vf2, the video timestamp vt2, and the audio timestamp at2.

[0148] It should be noted that, since it takes a certain amount of time for the network device 110 to mix the audio frames af5 and af6 to obtain the audio frame 4, the network device 110 can determine the audio timestamp at2 according to the audio timestamp 1, the mixing frame sampling rate, the audio frame sampling rate, the audio timestamp at3 (such as the third audio timestamp) and the audio timestamp at4 (such as the fourth audio timestamp). For details, see Figure 6 The mixing frame sampling rate refers to the number of times the recording device samples the sound analog signal per unit time, and the unit is Hz.

[0149] Figure 6 This is a schematic diagram of the relationship between the audio timestamp of the audio frame and the audio timestamp of the mixing frame. Figure 6 As shown, the audio timestamp at3 (corresponding to the audio frame vf2) and the audio timestamp at1 (corresponding to the audio frame af1, i.e., the mixing frame) correspond to the same absolute time, or the deviation between the respective corresponding absolute times is less than a preset threshold. When the audio frame sampling rate and the mixing frame sampling rate are known, the network device 110 determines the audio timestamp at2 according to the audio timestamp at1, the mixing frame sampling rate, the audio frame sampling rate, the audio timestamp at3, and the audio timestamp at4.

[0150] Further, the network device 110 determines the difference between the audio timestamp at4 and the audio timestamp at3, and determines the ratio of the difference to the audio frame sampling rate, and determines the audio timestamp at2 according to the product of the ratio and the mixed frame sampling rate and the sum of the audio timestamps at1. In this way, the network device 110 can obtain the audio timestamp at1 corresponding to the audio frame af4 before obtaining the audio frame af4 (mixed frame).

[0151] Exemplarily, the absolute time corresponding to the audio timestamp at3 and the audio timestamp at1 is: 10:30:40 (indicating 10:30:40), the audio frame sampling rate is 1000 (Hz), the mixing frame sampling rate (or the mixing stream sampling rate) is 2000 (Hz), the value of the audio timestamp at3 is 6000, the value of the audio timestamp at4 is 8000, the value of the audio timestamp at1 is 10000, and the value of the audio timestamp at2 is (8000-6000) / 1000*2000+10000=14000. Accordingly, the absolute time corresponding to the audio timestamp at2 is: 10:30:40+(8000-6000) / 1000=10:30:42. The absolute time corresponding to the audio timestamp at4 is: 10:30:40+(14000-10000) / 2000=10:30:42.

[0152] S530 , the terminal device 130 synchronously plays the video frame vf1 and the audio frame af1 (mixed frame) based on the video timestamp vt2 , the audio timestamp at2 , the audio timestamp at1 , and the video timestamp vt1 .

[0153] Specifically, the terminal device 130 can determine that the video frame vf1 is a synchronized video frame of the audio frame af1 (mixed audio frame) according to the above parameters. Figure 7 .

[0154] The video timestamp vt2 and the audio timestamp at2 can also be used to determine the correspondence between other audio frames and video frames sent by the network device 110, for example, to determine the correspondence between the audio frame af7 and the video frame vf3, the audio frame af7 is obtained by the network device 110 mixing the audio frame af8 from at least the terminal device 120, the video frame vf3 is the synchronized video frame of the audio frame af8, and the audio frame af8 is the synchronized audio frame of the video frame vf3. The sending time of the audio frame af8 and the video frame vf3 can be after the sending time of the audio frame af2 and the video frame vf1.

[0155] Figure 7 This is a schematic diagram of the relationship between the audio timestamp of the mixed audio frame and the video timestamp of the video frame. Figure 7As shown, the video timestamp vt2 and the audio timestamp at2 correspond to the same absolute time, or the deviation between the respective corresponding absolute times is less than a preset threshold. When the video frame sampling rate and the mixed audio frame sampling rate are known, the terminal device 130 can determine that the video frame vf1 is a synchronized video frame of the audio frame af1 based on the video timestamp vt2, the audio timestamp at2, the audio timestamp at1, the video frame sampling rate, the mixed audio frame sampling rate and the video timestamp vt1. The video frame sampling rate refers to the number of times the video recording device samples the video analog signal per unit time, and the unit is Hz.

[0156] Furthermore, the terminal device 130 determines the first difference between the audio timestamp at2 and the audio timestamp at1, and determines the ratio of the first difference to the mixing frame sampling rate, and can also determine the product of the ratio and the video frame sampling rate; when the difference between the video timestamp vt2 and the product is the video timestamp 1, the terminal device 130 can determine that the video frame vf1 corresponds to the audio frame af1.

[0157] Exemplarily, the absolute time corresponding to the video timestamp vt2 and the audio timestamp at2 is: 10:40:40, the video frame sampling rate is 1000 (Hz), the mixing frame sampling rate is 2000 (Hz), the value of the video timestamp vt2 is 6000, the value of the audio timestamp at1 is 8000, and the value of the audio timestamp at2 is 10000. Then the value of the video timestamp vt1 is: 6000-(10000-8000) / 2000*1000=5000. Correspondingly, the absolute time corresponding to the audio timestamp at1 is: 10:40:40-(10000-8000) / 2000=10:40:39. The absolute time corresponding to the video timestamp vt1 is: 10:40:40-(6000-5000) / 1000=10:40:39.

[0158] Through the above method, the terminal device 130 can determine whether the audio frame af1 corresponds to the video frame vf1 according to whether the value of the video timestamp vt1 obtained based on the above formula is consistent with the value of the video timestamp corresponding to the video frame vf1.

[0159] It should be noted that the above parameters such as the video frame sampling rate and the mixing frame sampling rate can be configured or configured through default configuration or negotiation between the two parties, and there is no limitation on this.

[0160] Furthermore, after playing the audio frame af1, the terminal device 130 may synchronously play the video frame vf1.

[0161] It should be noted that the above solution does not limit the order of S510 and S520, that is, the sending time of the audio frame af1 can be before the sending time of the audio frame af2, and the sending time of the audio frame af1 can also be after the sending time of the audio frame af2, and there is no limitation on this. The terminal device 130 can determine the corresponding relationship between the video frame vf1 and the audio frame af1 according to the video timestamp vt2 and the audio timestamp at2 in S520.

[0162] Through the above scheme, the terminal device 130 determines that the video frame corresponding to the video timestamp vt2 is a synchronized video frame of the mixed frame corresponding to the audio timestamp at2 according to the video timestamp vt2 and the audio timestamp at2 sent simultaneously by the network device 110, and the terminal device 130 can determine that the video frame vf1 is a synchronized video frame of the audio frame af1 according to the mixed frame sampling rate, the video frame sampling rate, the video timestamp vt1, the audio timestamp at1, the audio timestamp at2 and the video timestamp vt2, and synchronously play the video frame vf1 and the audio frame af1, so that the synchronization between the audio frame and the video frame can be achieved.

[0163] It should be noted that the method 500 can be applied not only to the DC conference architecture, but also to the WEBRTC conference architecture. When the method 500 is applied to the DC conference architecture, the audio channel ( Figure 5 Not shown) is a voice bearer channel (e.g., a 4G voice bearer channel (voice on LTE, VOLTE) or a NR voice bearer channel (voice onNR, VONR)), a video channel between a network device and a terminal device ( Figure 5 (not shown) is a data channel; when method 500 is applied to the WEBRTC architecture, the audio channel and video channel between the network device and the terminal device are both RTP channels.

[0164] It should also be noted that the voice bearer channel in the DC conference architecture and the RTP channel in the WEBRTC conference architecture can both be called IMS voice bearer channels, and the data channel in the DC conference architecture and the RTP channel in the WEBRTC conference architecture can both be called IMS data channels.

[0165] In addition, in the above content, the network device 110 processes audio frames based on the MCU solution and processes video frames based on the SFU solution. Therefore, the above solution can be used in a hybrid solution of the MCU solution and the SFU solution to solve the problem of synchronization between audio frames and video frames.

[0166] Combined with the following Figure 8 The flow of the above audio frames and video frames is further described.

[0167] Figure 8Schematic diagram of the flow of audio frames and video frames in the embodiment of the present application. Figure 8 As shown, the terminal device 120 sends an audio frame af2 to the network device 110 through the audio channel 1, and sends a video frame vf1 to the network device 110 through the video channel 1, and the terminal device 140 sends an audio frame af3 to the network device 110 through the audio channel 2. Accordingly, the network device 110 performs mixing processing on the audio frame af2 and the audio frame af3 (using the MCU solution for the audio frame), and sends the audio frame af1 to the terminal device 130 through the audio channel 3. The network device 110 directly forwards the video frame vf1 to the terminal device 130 through the video channel 2 (using the SFU solution for the video frame).

[0168] exist Figure 8 In the method 500, the terminal device 130 can determine that the synchronized video frame of the audio frame af1 is the video frame vf1, so that synchronization between the audio frame and the video frame can be achieved.

[0169] The following further describes the method 500 by taking the application of the method 500 to the DC conference architecture as an example.

[0170] Fig. 9 It is a schematic diagram of the interaction process of the communication method 800 according to an embodiment of the present application. Fig. 9 The method shown can be executed by terminal device 120, terminal device 130, terminal device 140 and IMS network 110 (the aforementioned network device 110), or by a module and / or device (for example, a chip or an integrated circuit, etc.) with corresponding functions installed in terminal device 120, terminal device 130, terminal device 140 and IMS network 110, without limitation. The following description is made by taking terminal device 120, terminal device 130, terminal device 140 and IMS network 110 as an example. Fig. 9 As shown, method 900 includes:

[0171] Optionally, S901a , a DC multi-party audio and video session is established between the terminal devices 120 , 130 , 140 , and the IMS network 110 .

[0172] Specifically, the terminal device 120 is speaking, and the terminal device 130 selects to watch the screen of the terminal device 120. In the process of establishing a DC multi-party audio and video session, the IMS network 110 and the terminal device 130 can negotiate the audio timestamp extension capability of RTP (the audio timestamp corresponds to the mixing frame) through the session description protocol (SDP). For example, the IMS network 110 and the terminal device 130 can negotiate the audio timestamp extension capability through the code: a=extmap:1urn:ietf:params:rtp-hdrext:sdes:mixer-ts.

[0173] S901, the terminal device 120 sends data packets p1 and p2 to the IMS network 110, where data packet p1 includes data of audio frame af2 and audio timestamp at3, and data packet p2 includes data of video frame vf1 and video timestamp vt1. The terminal device 140 sends data packet p3 to the IMS network 110, where data packet p3 includes data of audio frame af3 and audio timestamp at5.

[0174] Accordingly, the IMS network 110 receives the data packet p1 , the data packet p2 , and the data packet p3 .

[0175] Specifically, the audio timestamp at3 is a timestamp generated by the terminal device 120 for the audio frame af2, the audio timestamp at5 is a timestamp generated by the terminal device 140 for the audio frame af3, and the video timestamp vt1 is a timestamp generated by the terminal device 120 for the video frame vf1. The terminal device 120 synchronously sends the data of the audio frame af2 and the data of the video frame vf1 to the IMS network 110, so the IMS network 110 can determine that the audio frame af2 is a synchronized audio frame of the video frame vf1, and the video frame vf1 is a synchronized video frame of the audio frame af2.

[0176] It should be noted that the data of the audio frame af2 included in the data packet p1 may be all the data of the audio frame af2 or part of the data of the audio frame af2, which is not limited. The description of the data packet p1 is also applicable to other data packets described below, and will not be repeated hereafter.

[0177] The sending time of the data packet p1 and the sending time of the data packet p3 are the same.

[0178] S902 : The IMS network 110 sends a data packet p4 to the terminal device 130 . The data packet p4 includes data of the video frame vf1 and a video timestamp vt1 .

[0179] When the IMS network 110 does not change the format of the data packet p3, the data packet p4 is the data packet p3.

[0180] Accordingly, the terminal device 130 receives the data packet p4.

[0181] S903 . The terminal device 130 caches the video frame vf1 .

[0182] S904 . The IMS network 110 mixes the audio frame af2 and the audio frame af3 to obtain an audio frame af1 , and generates an audio timestamp at1 for the audio frame af1 .

[0183] For the description of S804, please refer to S510, which will not be repeated here.

[0184] It should be noted that the IMS network 110 may randomly generate an audio timestamp at1 for the audio frame af1.

[0185] S905 . The IMS network 110 sends a data packet p5 to the terminal device 130 . The data packet p5 includes data of the audio frame af1 and an audio timestamp at1 .

[0186] Accordingly, the terminal device 130 receives the data packet p5.

[0187] Furthermore, the terminal device 130 performs decoding and other processing on the audio frame af1.

[0188] It should be noted that the terminal device 130 cannot determine the video frame corresponding to the audio frame af1.

[0189] S906. The terminal device 120 sends data packets p6 and p7 to the IMS network 110. The data packet p6 includes the data of the audio frame af5 and the audio timestamp at4. The data packet p7 includes the data of the video frame vf2 and the video timestamp vt2. The terminal device 140 sends data packet p8 to the IMS network 110. The data packet p8 includes the data of the audio frame af6 and the audio timestamp at6.

[0190] Accordingly, the IMS network 110 receives data packets p6 to p8.

[0191] The audio timestamp at4 is a timestamp generated by the terminal device 120 for the audio frame af5, the audio timestamp at6 is a timestamp generated by the terminal device 140 for the audio frame af6, and the video timestamp vt2 is a timestamp generated by the terminal device 120 for the video frame vf2. The terminal device 120 synchronously sends the data of the audio frame af5 and the data of the video frame vf2 to the IMS network 110, so the IMS network 110 can determine that the audio frame af5 is a synchronized audio frame of the video frame vf2, and the video frame vf2 is a synchronized video frame of the audio frame af5.

[0192] S907 . The IMS network 110 determines the audio timestamp at2 according to the audio timestamp at1 , the audio timestamp at4 , the audio mixing frame sampling rate, the audio frame sampling rate, and the audio timestamp at3 .

[0193] For a detailed description of S907, please refer to Figure 6 , I will not go into details here.

[0194] S908 . The IMS network 110 sends a data packet p9 to the terminal device 130 . The data packet p9 includes data of the video frame vf2 , a video timestamp vt2 , and an audio timestamp at2 .

[0195] Accordingly, the terminal device 130 receives the data packet p9.

[0196] S909. The terminal device 130 caches the video frame vf2.

[0197] S910 . The terminal device 130 synchronously plays the video frame vf1 and the mixed audio frame af1 based on the video timestamp vt2 , the audio timestamp at2 , the audio timestamp at1 , and the video timestamp vt1 .

[0198] For a detailed description of S910, please refer to Figure 7 and S530, which will not be described here.

[0199] In the above solution, the embodiment of the present application is described by taking the audio frame af1 before the audio frame af4 as an example, but is not limited to the scenario where the audio frame af1 is after the audio frame af4.

[0200] Through the above scheme, the present application supports carrying the audio timestamp at of the mixed frame af synchronized with the video frame vf in the data packet carrying the video frame vf and the video timestamp vt corresponding to the video frame vf. The receiving end can determine the video frame vf corresponding to the mixed frame af based on the video timestamp vt and audio timestamp at carried in the data packet and the audio timestamp at of the mixed frame af that needs to be played synchronously. In this way, synchronization between the video frame and the audio frame can be achieved.

[0201] In method 900, the embodiment of the present application supports the IMS network 110 to periodically (for example, every 5 seconds) send a data packet carrying both the audio timestamp at and the video timestamp vt to the terminal device 130. The packaging duration of the audio frame af generally does not change, and the packaging duration of the video frame vf (corresponding to the frame rate) may be adjusted according to the network conditions. For more accuracy, the terminal device can obtain the latest packaging duration each time.

[0202] In method 900, after the terminal device 130 completes the synchronization of a video frame vf and a mixed audio frame af, the terminal device 130 can play the subsequent video frame vf at the inherent frame rate until the next data packet carrying the audio timestamp at and the video timestamp vt is received, and then the next synchronization between the video frame vf and the mixed audio frame af is performed.

[0203] It should be noted that the present application does not limit the execution order between steps S901 to S910. For example, S903 and S904 can be executed at the same time.

[0204] It should also be noted that the method 900 can also be applied to the WEBRTC conference architecture. When the method 800 is applied to the WEBRTC conference architecture, the IMS network 110 can be a web server (which is responsible for the business logic control of all multi-party calls, such as creating a multi-party conference, users actively joining a multi-party conference, users leaving a multi-party conference, and users inviting other people to join a multi-party conference), and the terminal devices 120 to 140 can be web clients, etc.

[0205] It should also be noted that the data packets mentioned above are all RTP data packets.

[0206] Combined with the following Fig.10 right Figure 5 and Figure 8 The method is further described.

[0207] It should be noted that Fig.10 The numbers of the audio frames, video frames, audio timestamps, video timestamps, and data packets in the method 500 and the audio frames, video frames, audio timestamps, and video timestamps in the method 800 are not related to each other. Fig.10 The audio frames, video frames, audio timestamps, video timestamps, and data packets in the are independent numbers and are described uniformly here.

[0208] Fig.10 Schematic diagram of the corresponding relationship between the audio data packet sequence and the video data packet sequence in the embodiment of the present application. Fig.10 As shown, the network device 110 can send an audio data packet sequence and a video data packet sequence to the terminal device 130 in parallel. Each audio data packet in the audio data packet sequence contains the data of a mixed audio frame and a mixed audio timestamp. For the description of the mixed audio frame and the audio timestamp, please refer to the above text. Each video data packet in the video data packet sequence contains the data of a video frame and a video timestamp. The data of the video frame in each video data packet comes from the terminal device 120.

[0209] Exemplarily, the audio data packet sequence includes: audio data packet ap1, audio data packet ap2, audio data packet ap3 and audio data packet ap4. Audio data packet ap1 includes data of audio frame af1 and audio timestamp at1, audio data packet ap2 includes data of audio frame af2 and audio timestamp at2, audio data packet ap3 includes data of audio frame af3 and audio timestamp at3, and audio data packet ap4 includes data of audio frame af4 and audio timestamp at4. The audio timestamp is generated by the network device 110.

[0210] Exemplarily, the video data packet sequence includes: video data packet vp1, video data packet vp2, video data packet vp3 and video data packet vp4. Video data packet vp1 includes data of video frame vf1 and video timestamp vt1, video data packet vp2 includes data of video frame vf2 and video timestamp vt2, video data packet vp3 includes data of video frame vf3 and video timestamp vt3, and video data packet vp4 includes data of video frame vf4 and video timestamp vt4. The video timestamp is generated by the terminal device 120.

[0211] In the embodiment of the present application, the video data packet vp3 also includes the audio timestamp at2 as an example. The terminal device 130 determines that the mixed frame af0 corresponds to the video frame vf0, the mixed frame af1 corresponds to the video frame vf1, and the mixed frame af3 corresponds to the video frame vf3, etc. according to the video timestamp vt2 and the audio timestamp at2, and in combination with the method 500.

[0212] In summary, the network device 110 sends a sequence of audio data packets and a sequence of video data packets in parallel to the terminal device 130 in the multi-party communication service. Each audio data packet in the sequence of audio data packets contains data of a mixed audio frame and an audio timestamp, the mixed audio frame is obtained by the network device 110 based on mixing of audio frames from at least the terminal device 120 in the multi-party communication service, and the audio timestamp is generated by the network device 110 for the mixed audio frame. Each video data packet in the video data packet sequence contains data of a video frame and a video timestamp, the video frame comes from the terminal device 120, and the video timestamp is generated by the terminal device 120 for the video frame; at least the first video data packet in the video data packet sequence (taking video data packet vp3 as an example) also contains a first audio timestamp (taking audio timestamp at2 as an example), the first audio timestamp (audio timestamp at2) is the same as the audio timestamp (audio timestamp at2) contained in the first audio data packet in the audio data packet sequence (taking audio data packet ap3 as an example), the first audio timestamp (audio timestamp at2) and the video timestamp (taking video timestamp vt2 as an example) in the first video data packet (video data packet vp3) are used to determine that the video frame (taking video frame vf2 as an example) in the second video data packet (video data packet vp2) in the video data packet sequence and the mixed audio frame (taking mixed audio frame af2 as an example) in the second audio data packet (audio data packet vp2) in the audio data packet sequence should be played synchronously.

[0213] It should be noted that the above-mentioned multi-party communication service may be a video conference, such as the aforementioned DC conference or WEBRTC conference, or a multi-party video call.

[0214] It should also be noted that the audio data packet sequence is sent by the network device 110 through the IMS voice bearer channel, and the video data packet sequence is sent by the network device 110 through the IMS data channel. Each audio data packet in the audio data packet sequence is an RTP data packet, and each video data packet in the video data packet sequence is an RTP data packet.

[0215] Finally, the device embodiment of the embodiment of the present application is introduced.

[0216] In order to implement the functions in the method provided in this application, the terminal device 130 and the network device 110 may include a hardware structure and / or a software module, and implement the above functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether one of the above functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.

[0217] Fig.1111 is a schematic block diagram of a communication device 1100 according to an embodiment of the present application. The communication device 1100 includes a processor 1110 and a communication interface 120, and the processor 1110 and the communication interface 1120 may be connected to each other via a bus 1130. The communication device 1100 may be a network device 110 or a terminal device 130.

[0218] Optionally, the communication device 1100 may further include a memory 1140. The memory 1140 includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a portable read-only memory (CD-ROM), and the memory 1140 is used for related instructions and data.

[0219] The processor 1110 may be one or more central processing units (CPUs). When the processor 1110 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0220] When the communication device 1100 is a terminal device 130, illustratively, the processor 1110 is used to receive an audio frame af1, an audio timestamp at1, a video frame vf1, and a video timestamp vt1 sent by the network device 110. The processor 1110 is also used to receive a video frame vf2, a video timestamp vt2, and an audio timestamp at2 sent by the network device 110. The processor 1110 is also used to synchronously play the video frame vf1 and the audio frame af1 according to the video timestamp vt2, the audio timestamp at2, the mixing frame sampling rate, the video frame sampling rate, the audio timestamp at1, and the video timestamp vt1.

[0221] When the communication device 1100 is the network device 110, exemplarily, the processor 1110 is used to send the audio frame af1 and the audio timestamp at1, as well as the video frame vf1 and the video timestamp vt1 to the terminal device 130; the processor 1110 is also used to send the video frame vf2, the video timestamp vt2 and the audio timestamp at2 to the terminal device 130.

[0222] The above contents are only exemplary descriptions. When the communication device 1100 is the network device 110 / terminal device 130, it will be responsible for executing the methods or steps related to the network device 110 / terminal device 130 in the above method embodiments.

[0223] The above description is only an exemplary description. For specific content, please refer to the content shown in the above method embodiment. Fig.11 The implementation of each operation in can also refer to Figures 5 to 10 The corresponding description of the method embodiment shown.

[0224] In addition, the communication device 1100 may also be a terminal device 120 or a terminal device 140, which is used to execute the methods or steps related to the terminal device 120 / terminal device 140 in the aforementioned method embodiments.

[0225] Fig.12 1 is a schematic block diagram of a communication device 1200 according to an embodiment of the present application. The communication device 1200 may be a network device 110 or a terminal device 130, or may be a chip or module in the network device 110 or the terminal device 130, for implementing the method involved in the above embodiment. The communication device 1200 includes a transceiver unit 1210. The transceiver unit 1210 is exemplarily introduced below.

[0226] The transceiver unit 1210 may include a transmitting unit and a receiving unit. The transmitting unit is used to perform a transmitting action of the communication device, and the receiving unit is used to perform a receiving action of the communication device. For ease of description, the embodiment of the present application combines the transmitting unit and the receiving unit into one transceiver unit. A unified description is given here, and no further description is given later.

[0227] When the communication device 1200 is a terminal device 130, illustratively, the transceiver unit 1210 is used to receive the audio frame af1, the audio timestamp at1, the video frame vf1, and the video timestamp vt1 sent by the network device 110. The transceiver unit 1210 is also used to receive the video frame vf2, the video timestamp vt2, and the audio timestamp at2 sent by the network device 110.

[0228] The communication device 1200 may further include a processing unit 1220 for synchronously playing the video frame vf1 and the audio frame af1 based on the video timestamp vt2, the audio timestamp at2, the audio timestamp at1 and the video timestamp vt1.

[0229] When the communication device 1200 is the network device 110, exemplarily, the transceiver unit 1210 is used to send audio frame af1 and audio timestamp at1, as well as video frame vf1 and video timestamp vt1 to the terminal device 130; and is also used to send video frame vf2, video timestamp vt2 and audio timestamp at2 to the terminal device 130.

[0230] The above contents are only exemplary descriptions. When the communication device 1200 is the network device 110 / terminal device 130, it will be responsible for executing the methods or steps related to the network device 110 / terminal device 130 in the above method embodiments.

[0231] Optionally, the communication device 1200 further includes a storage unit 1230, where the storage unit 1230 is used to store a program or code for executing the aforementioned method.

[0232] Fig.11 and Fig.12 The device embodiment shown is used to implement Figures 5 to 10 The content described. Fig.11 and Fig.12 The specific execution steps and methods of the device shown can refer to the contents described in the aforementioned method embodiment.

[0233] In addition, the communication device 1200 may also be a terminal device 120 or a terminal device 140, which is used to execute the methods or steps related to the terminal device 120 / terminal device 140 in the aforementioned method embodiments.

[0234] Fig.13 1 is a schematic block diagram of a communication device 1300 according to an embodiment of the present application. The communication device 1300 is used to implement the functions of the network device 110 / terminal device 130. The communication device 1300 may be a chip in the network device 110 / terminal device 130.

[0235] The communication device 1300 includes: an input / output interface 1320 and a processor 1310. The input / output interface 1320 may be an input / output circuit. The processor 1310 may be a signal processor, a chip, or other integrated circuit that can implement the method of the present application. The input / output interface 1120 is used for inputting or outputting signals or data.

[0236] For example, when the communication device 1300 is a terminal device 130, the input / output interface 1320 is used to receive the audio frame af1, the audio timestamp at1, the video frame vf1, and the video timestamp vt1 sent by the network device 110. The input / output interface 1320 is also used to receive the video frame vf2, the video timestamp vt2, and the audio timestamp at2 sent by the network device 110. The processor 1110 is used to synchronously play the video frame vf1 and the audio frame af1 according to the video timestamp vt2, the audio timestamp at2, the audio timestamp at1, and the video timestamp vt1.

[0237] For example, the communication device 1300 is the network device 110, and the input / output interface 1320 is used to send the audio frame af1, the audio timestamp at1, the video frame vf1, and the video timestamp vt1 to the terminal device 130. The input / output interface 1320 is also used to send the video frame vf2, the video timestamp vt2, and the audio timestamp at2 to the terminal device 130.

[0238] In one possible implementation, the processor 1310 implements the functions implemented by the network device 110 or the terminal device 130 by executing instructions stored in the memory.

[0239] Optionally, the communication device 1300 also includes a memory.

[0240] Optionally, the processor and memory are integrated together.

[0241] Optionally, the memory is outside the communication device 1300 .

[0242] In a possible implementation, the processor 1310 may be a logic circuit, and the processor 1310 inputs / outputs messages or signals through the input / output interface 1320. The logic circuit may be a signal processor, a chip, or other integrated circuit that can implement the method of the embodiment of the present application.

[0243] The above description of the communication device 1300 is only an exemplary description. The communication device 1300 can be used to execute the method described in the above embodiment. The specific content can be found in the description of the above method embodiment, which will not be repeated here.

[0244] In addition, the communication device 1300 may also be a terminal device 120 or a terminal device 140, which is used to execute the methods or steps related to the terminal device 120 / terminal device 140 in the aforementioned method embodiments.

[0245] Fig.14 1 is a schematic block diagram of a communication device 1400 of an embodiment of the present application. The communication device 1400 may be a terminal device (not limited to the terminal device 120, the terminal device 130 or the terminal device 140), a processor of the terminal device, or a chip. The communication device 1400 may be used to perform the operations performed by the terminal device or the communication device in the above method embodiment.

[0246] When the communication device 1400 is a terminal device, Fig.14 FIG. 1 shows a simplified schematic diagram of the structure of a terminal device. Fig.14 As shown, the terminal device includes a processor, a memory, and a transceiver. The memory can store computer program codes, and the transceiver includes a transmitter 1431, a receiver 1432, and a radio frequency circuit ( Fig.14 Not shown), antenna 1433 and input / output device ( Fig.14 not shown).

[0247] The processor is mainly used to process communication protocols and communication data, as well as to control terminal devices, execute software programs, process software program data, etc. The memory is mainly used to store software programs and data. The radio frequency circuit is mainly used for converting baseband signals and radio frequency signals and processing radio frequency signals. The antenna is mainly used to send and receive radio frequency signals in the form of electromagnetic waves. Input and output devices. For example, touch screens, display screens, keyboards, etc. are mainly used to receive data input by users and output data to users. It should be noted that some types of terminal devices may not have input and output devices.

[0248] When data needs to be sent, the processor performs baseband processing on the data to be sent and outputs the baseband signal to the RF circuit. The RF circuit performs RF processing on the baseband signal and then sends the RF signal outward in the form of electromagnetic waves through the antenna. When data is sent to the terminal device, the RF circuit receives the RF signal through the antenna, converts the RF signal into a baseband signal, and outputs the baseband signal to the processor. The processor converts the baseband signal into data and processes the data. For ease of explanation, Fig.14 Only one memory, processor and transceiver are shown. In an actual terminal device product, there may be one or more processors and one or more memories. The memory may also be referred to as a storage medium or a storage device. The memory may be set independently of the processor or integrated with the processor, and the embodiments of the present application do not limit this.

[0249] In the embodiment of the present application, the antenna and the radio frequency circuit with transceiver functions can be regarded as the transceiver module of the terminal device, and the processor with processing function can be regarded as the processing module of the terminal device.

[0250] like Fig.14 As shown, the terminal device includes a processor 1410, a memory 1420 and a transceiver 1430. The processor 1410 may also be referred to as a processing unit, a processing board, a processing module, a processing device, etc., and the transceiver 1430 may also be referred to as a transceiver unit, a transceiver, a transceiver device, etc.

[0251] Optionally, the device for implementing the receiving function in the transceiver 1430 may be regarded as a receiving module, and the device for implementing the sending function in the transceiver 1430 may be regarded as a sending module, that is, the transceiver 1430 includes a receiver and a transmitter. A transceiver may sometimes be referred to as a transceiver, a transceiver module, or a transceiver circuit, etc. A receiver may sometimes be referred to as a receiver, a receiving module, or a receiving circuit, etc. A transmitter may sometimes be referred to as a transmitter, a transmitting module, or a transmitting circuit, etc.

[0252] For example, in one implementation, the processor 1410 is used to execute Figures 5 to 10 In the embodiment shown, the processing action on the terminal device side is performed by the transceiver 1430. Figures 5 to 10 The sending and receiving actions on the terminal device side.

[0253] For example, in one implementation, the processor 1410 is used to execute Figures 5 to 10 In the embodiment shown, the processing action on the terminal device side is performed by the transceiver 1430. Figures 5 to 10 The sending and receiving actions on the terminal device side.

[0254] It should be understood that Fig.14 This is only an example and not a limitation. The terminal device including the transceiver module and the processing module may not rely on Figures 11 to 13 The structure shown.

[0255] When the communication device 1400 is a chip, the chip includes a processor, a memory and a transceiver. The transceiver may be an input / output circuit or a communication interface; the processor may be a processing module or a microprocessor or an integrated circuit integrated on the chip. The sending operation of the terminal device in the above method embodiment may be understood as the output of the chip, and the receiving operation of the terminal device in the above method embodiment may be understood as the input of the chip.

[0256] The present application also provides a chip, including a processor, for calling and executing instructions stored in a memory from the memory, so that a communication device equipped with the chip executes the methods in the above examples.

[0257] The present application also provides a chip, including: an input interface, an output interface, and a processor, wherein the input interface, the output interface, and the processor are connected via an internal connection path, and the processor is used to execute the code in the memory, and when the code is executed, the processor is used to execute the method in each of the above examples. Optionally, the chip also includes a memory, and the memory is used to store computer programs or codes.

[0258] The present application also provides a processor, which is coupled to a memory and is used to execute the methods and functions involving a network device or a terminal device in any of the above embodiments.

[0259] The present application provides a computer program product including instructions. When the computer program product is run on a computer, the method of the above embodiment is implemented.

[0260] The present application also provides a computer program. When the computer program is executed in a computer, the method of the above embodiment is implemented.

[0261] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a computer, the method described in the above embodiment is implemented.

[0262] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0263] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0264] In several embodiments provided in the present application, the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0265] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the technical solution of the embodiment of the present application.

[0266] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0267] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of each method embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0268] The above are only specific implementations of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be included in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application shall be based on the protection scope of the claims.

Claims

1. A communication method, characterized in that: include: The network device sends a first audio frame and a first audio timestamp to a second terminal device, and sends a first video frame and a first video timestamp to the second terminal device, wherein the first audio frame is a mixed frame obtained by the network device based on at least a second audio frame, the first video frame and the second audio frame are sent synchronously by the first terminal device, the first audio timestamp is a timestamp generated by the network device for the first audio frame, and the first video timestamp is a timestamp generated by the first terminal device for the first video frame; The network device sends a second video frame, a second video timestamp, and a second audio timestamp to the second terminal device, where the second video timestamp is a timestamp generated by the first terminal device for the second video frame, the second audio timestamp is a timestamp generated by the network device for a third audio frame, the third audio frame is a mixed audio frame obtained by the network device based on at least a fourth audio frame, and the second video frame and the fourth audio frame are sent synchronously by the first terminal device; The second audio timestamp, the second video timestamp, the first audio timestamp and the first video timestamp are used to determine that the first video frame and the first audio frame should be played synchronously.

2. The method according to claim 1, characterized in that The method further comprises: The network device receives a third audio timestamp and a fourth audio timestamp sent by the first terminal device, where the third audio timestamp is a timestamp generated by the first terminal device for the second audio frame, and the fourth audio timestamp is a timestamp generated by the first terminal device for the fourth audio frame; The network device determines the second audio timestamp according to the third audio timestamp, the fourth audio timestamp, a mixing frame sampling rate, an audio frame sampling rate, and the first audio timestamp.

3. The method according to claim 2, characterized in that The network device determines the second audio timestamp according to the third audio timestamp, the fourth audio timestamp, a mixing frame sampling rate, an audio frame sampling rate, and the first audio timestamp, including: The network device determines a difference between the fourth audio timestamp and the third audio timestamp; The network device determines a ratio of the difference to the audio frame sampling rate; The network device determines the second audio timestamp according to a sum of a product of the ratio and the mixing frame sampling rate and the first audio timestamp.

4. The method according to any one of claims 1 to 3, characterized in that The network device sends a second video frame, a second video timestamp, and a second audio timestamp to the second terminal device, including: The network device sends a first data packet to the second terminal device, where the first data packet includes the second video frame, the second video timestamp, and the second audio timestamp.

5. The method according to claim 4, characterized in that The first data packet is a real-time transport protocol data packet.

6. The method according to any one of claims 1 to 5, characterized in that The network device sends a first audio frame and a first audio timestamp to a second terminal device, and sends a first video frame and a first video timestamp to the second terminal device, including: The network device sends a second data packet and a third data packet to the second terminal device, the second data packet includes the first audio frame and the first audio timestamp, and the second data packet includes the first video frame and the first video timestamp.

7. The method according to claim 6, characterized in that The second data packet and the third data packet are both real-time transport protocol data packets.

8. The method according to any one of claims 1 to 7, characterized in that The first video frame and the second video frame are sent by the network device through an IP Multimedia Subsystem IMS data channel, and the first audio frame is sent by the network device through an IMS voice bearer channel.

9. A communication method, characterized in that: include: Receive a first audio frame and a first audio timestamp sent by a network device, and receive a first video frame and a first video timestamp sent by the network device, wherein the first video timestamp is a timestamp generated by a first terminal device for the first video frame, the first audio timestamp is a timestamp generated by the network device for the first audio frame, the first audio frame is a mixed audio frame obtained by the network device based on at least a second audio frame, and the first video frame and the second audio frame are synchronously sent by the first terminal device; receiving a second video frame, a second video timestamp, and a second audio timestamp sent by the network device, wherein the second video timestamp is a timestamp generated by the first terminal device for the second video frame, the second audio timestamp is a timestamp generated by the network device for a third audio frame, the third audio frame is a mixed audio frame obtained by the network device based on at least a fourth audio frame, and the second video frame and the fourth audio frame are synchronously sent by the first terminal device; The first video frame and the first audio frame are synchronously played based on the second video timestamp, the second audio timestamp, the first audio timestamp, and the first video timestamp.

10. The method according to claim 9, characterized in that The synchronously playing the first video frame and the first audio frame based on the second video timestamp, the second audio timestamp, the first audio timestamp and the first video timestamp includes: determining a first difference between the second audio timestamp and the first audio timestamp; Determining a ratio of the first difference to a sampling rate of a mixing frame; Determining a product of the ratio and a video frame sampling rate; When the difference between the second video timestamp and the product is the first video timestamp, determining that the first video frame corresponds to the first audio frame; The first video frame and the first audio frame are played synchronously.

11. The method according to claim 9 or 10, characterized in that: The receiving a second video frame, a second video timestamp, and a second audio timestamp sent by the network device includes: A first data packet sent by the network device is received, where the first data packet includes the second video frame, the second video timestamp, and the second audio timestamp.

12. The method according to claim 11, characterized in that The first data packet is a real-time transport protocol data packet.

13. The method according to any one of claims 9 to 12, characterized in that The receiving a first audio frame and a first audio timestamp sent by a network device, and receiving a first video frame and a first video timestamp sent by the network device, comprises: A second data packet and a third data packet sent by the network device are received, wherein the second data packet includes the first audio frame and the first audio timestamp, and the second data packet includes the first video frame and the first video timestamp.

14. The method according to claim 13, characterized in that The second data packet and the third data packet are both real-time transport protocol data packets.

15. The method according to any one of claims 9 to 14, characterized in that The first video frame and the second video frame are sent by the network device through an IMS data channel, and the first audio frame is sent by the network device through an IMS voice bearer channel.

16. A communication method, characterized in that: include: The network device sends a first audio frame and a first audio timestamp to a second terminal device, and sends a first video frame and a first video timestamp to the second terminal device, wherein the first audio frame is a mixed frame obtained by the network device based on at least a second audio frame, the first video frame and the second audio frame are sent synchronously by the first terminal device, the first audio timestamp is a timestamp generated by the network device for the first audio frame, and the first video timestamp is a timestamp generated by the first terminal device for the first video frame; The second terminal device receives the first audio frame, the first audio timestamp, the first video frame, and the first video timestamp; The network device sends a second video frame, a second video timestamp, and a second audio timestamp to the second terminal device, where the second video timestamp is a timestamp generated by the first terminal device for the second video frame, the second audio timestamp is a timestamp generated by the network device for a third audio frame, the third audio frame is a mixed audio frame obtained by the network device based on at least a fourth audio frame, and the second video frame and the fourth audio frame are sent synchronously by the first terminal device; The second terminal device receives the second video frame, the second video timestamp, and the second audio timestamp; The second terminal device synchronously plays the first video frame and the first audio frame based on the second video timestamp, the second audio timestamp, the first audio timestamp and the first video timestamp.

17. The method according to claim 16, characterized in that The second terminal device synchronously plays the first video frame and the first audio frame based on the second video timestamp, the second audio timestamp, the first audio timestamp and the first video timestamp, including: The second terminal device determines a first difference between the second audio timestamp and the first audio timestamp; The second terminal device determines a ratio of the first difference to a sampling rate of a mixing frame; The second terminal device determines the product of the ratio and the video frame sampling rate; When the difference between the second video timestamp and the product is the first video timestamp, the second terminal device determines that the first video frame corresponds to the first audio frame; The second terminal device synchronously plays the first video frame and the first audio frame.

18. The method according to claim 16 or 17, characterized in that The network device sends a second video frame, a second video timestamp, and a second audio timestamp to the second terminal device, including: The network device sends a first data packet to the second terminal device, where the first data packet includes the second video frame, the second video timestamp, and the second audio timestamp.

19. The method according to claim 18, characterized in that The first data packet is a real-time transport protocol data packet.

20. The method according to any one of claims 16 to 19, characterized in that The network device sends a first audio frame and a first audio timestamp to a second terminal device, and sends a first video frame and a first video timestamp to the second terminal device, including: The network device sends a second data packet and a third data packet to the second terminal device, the second data packet includes the first audio frame and the first audio timestamp, and the second data packet includes the first video frame and the first video timestamp.

21. The method according to claim 20, characterized in that The second data packet and the third data packet are both real-time transport protocol data packets.

22. The method according to any one of claims 16 to 21, characterized in that The first video frame and the second video frame are sent by the network device through an IMS data channel, and the first audio frame is sent by the network device through an IMS voice bearer channel.

23. A communication device, characterized in that: include: a processor for executing a computer program stored in the memory, so that the communication device performs the method according to any one of claims 1 to 8, or, so that the communication device performs the method according to any one of claims 9 to 15; or, So that the communication device performs the method as claimed in any one of claims 16 to 22.

24. A computer program product, characterized in that The computer program product comprises: Instructions for executing the method according to any one of claims 1 to 8, or, Instructions for executing the method according to any one of claims 9 to 15; or, Instructions for performing the method of any one of claims 16 to 22.

25. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, The computer is caused to execute the method according to any one of claims 1 to 8, or causing the computer to execute the method according to any one of claims 9 to 15; or, The computer is caused to execute the method according to any one of claims 16 to 22.

26. A communication system, characterized in that: include: A network device, configured to execute the method according to any one of claims 1 to 8; A terminal device, configured to execute the method according to any one of claims 9 to 15; or The network device and the terminal device are used to execute the method as described in any one of claims 16 to 22.

Images